Troubleshooting Slow Grid Services

When a service is taking longer than expected to do things, the very first step in trying to speed up the service is to determine what it is spending its time doing.

Java comes with a tool called Java VisualVM. This tool can be used to determine what parts of its code a service is spending its time in. It can also be used to determine how the service is using its memory and threads.

Java VisualVM documentation is part of the Java 1.6 SE Documentation. This documentation explains most of what you need to know to use VisualVM. However, there is one important issue for caGrid services that is very hard to find in the documentation unless you already know what to look for.

The documentation basically says that you just start VisualVM and select, in VisualVM's Applications Window, the application (Java VM process) that you want to analyze. The problem with this is that your Tomcat process may not show up in VisualVM's Applications Window. The following instructions will allow the caGrid service's Tomcat process to appear in VisualVM's Applications Window.

These instructions assume that you start tomcat with the startup or catalina scripts that are provided in tomcat's bin directory.

The first step is to set some environment variables that will make tomcat accessible to VisualVM. In a Linux or Mac environment, issue this command:

export JAVA_OPTS='-Xshare:off -Dcom.sun.management.jmxremote.port=8086  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false'

In a Windows environment, issue this command:

set JAVA_OPTS="-Xshare:off -Dcom.sun.management.jmxremote.port=8086  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Note that you can specify a different port number than the 8086 shown above. Just be sure to specify the same port number as on the command line in the following steps.

Start VisualVM and navigate to its Applications Window. Right click in any empty part of the applications window. When the menu pops up, select "Add JXM Connection...".

Fill in the Add JMX Connection dialog like this:

When editing the dialog, be sure that you use the same port number you specified on the command line.

After you click on the dialog's OK button, you grid service's tomcat should appear like this: