Remote JMX connection

As I've been working for the past while on remote testing in JMeter, I'm also trying to monitor the memory usage as well as monitoring attributes of objects and classes.  I learned that the plugin from JVM called MBeans was able to monitor this type of information.  Aftering do more research, I learned that a remote JMX connection needs to be instantiated in order for MBeans to read the information.

With further research, I encountered a very useful blog here and  a follow entry here which gave me a better idea of how it all works.  Answers to questions that hadn't even arose yet were all answered a thorough walkthrough was provided.

To start things off, JMX is a technolgy that allows users and developers to manage their Java applications and services.  To start the JMX agent, arguments in the format of -Dpropertyname=value  are placed after the java command (java -Dpropertyname=value ./javaapp).

For my monitoring environment, the arguments I specifically need are:

• -Dcom.sun.management.jmxremote (since I am running remote monitoring)
• -Dcom.sun.management.jmxremote.port=portnum (where portnum is neither the port number of my jstatd connection nor jmeter-server)
• -Dcom.sun.management.jmxremote.ssl=false (allows me to connect without needing my ssl configurations)
• -Dcom.sun.management.jmxremote.authenticate=false (allows me to connect without a password file)

And voila!  The connection is made and I can successfully monitor my applications and the attributes :).  

Troubleshooting JMeter remote testing issues

This issue revolves around JMeter remote testing.  I currently have a Windows master machine running JMeter tests remotely on a Linux (redhat) slave machine.  All properties were changed and connections were successful (until a power outage happened which caused remote connections to fail).

There were two issues were:

1. Since the power outage, all machines were rebooted.  Unknowingly, Redhat Linux has a default firewall.  Run iptables on command prompt to check for firewall. The firewall blocks any remote connections attempted on the server.
2. After connecting successfully to the slave machine, an issue where the JMeter GUI not responding with test results occurred. I was not able to tell whether a test was being run or completed at all unless I looked from the cmd window. This issue ended up being a network issue. The slave machine was responding (marked in the command line) and the remote JVM memory monitor was also increasing, but there was still no response. As a matter of fact, the responding server got lost in the network connection. The master machine was able to connect to the slave, but the responding data got lost in the master's network. The master machine has multiple network connections, and the server was sending data back to a network that the JMeter client wasn't on. As a workaround, I had to disable networks that were not being used at that moment.

Since this is just a workaround, it would be nice to figure out how to specify a network to open JMeter instead of randomly assigning this variable. Any comments or suggestions are appreciated.

Cache GET Failure

When running performance testing on an cache using Ehcache, it would run smoothly on a cache with:

• 3,000 entries
• JVM shows that only 50 GB of data was occupied (60 GB is allocated for the application)
• takes approximately 20 minutes to complete entire test plan

There was 10 GB of memory left available and unused, so I wanted to maximize usage.  After doing some math, (3,000 entries/50 GB = 60 entries/GB), I increased the number of entries to 3,500 (just to give a little leeway).  Strangely, whenever I did this, even when increasing the number of entries to 3001, I would have to wait over an hour for the test plan to complete.  The results did not look good either.  I was getting a relatively high percentage of errors.  

Since I am using JMeter for this performance test, I used the SampleResult class to configure my data collection.  I set the jmeter.save.saveservice.samplerData=true in bin/jmeter.properties.  I used a "View Results Tree" listener to see the data collection.  As the test was running, once it reached the GET transactions, I noticed it would fail immediately on the first transaction.  The error was a validation error exception.  

I then looked at my cache configuration xml file.  I finally found the cause of this issue!  I had set the maxEntriesLocalHeap=30,000,000, which is fine for 3000 entries (of size 10,000).  For any added entry, the cache size isn't big enough, so I using the memoryStoreEvictionPolicy (mine was set to LRU), it would override old entries (I presume that later GET transactions would actually succeed since it doesn't override the later X amount of entries).  This was the cause of the issue!  I increased the size maxEntries, and it solved the problem! Ta-da!