In the first two editions of this series, I looked at how you could do some simple network analysis and some simple monitoring of your Jazz servers. In the third edition I looked at nmon, for showing you what the server is doing from a system perspective. In this blog post, we get a even deeper into seeing what is happening on your Jazz server. One of the nice little tools that I have encountered in my work with Jazz deployments, is a little tool called WAIT. This tool works for ALL environments.
What is WAIT?
WAIT (Whole system Analysis of Idle Time) is a great little tool that analyzes javacores to tell you what is going on in your system. It uses these javacore files as input, and does an analysis of the information within them. Very handy to use, and it can be quite informative.
Setting up WAIT
First you need to go and find WAIT, so go to the WAIT homepage. Now click on the new user button on this page, and get yourself registered with the WAIT tool. So register yourself (it will be worth it).
Now that you are registered, you’re ready to move on. You will see an odd screen with text informing you that your registration was a success. Just hit the green Use Wait button, and then acknowledge the license agreement.
Now we get to the real user interface for the WAIT tool. You will see a series of tabs across the top of the screen, for Submit Data, Report Gallery, Example Input, and Data Collection Scripts. You might want to start out by checking out the demo, so you can see what WAIT will do for you. Go ahead, it will give you some more background on WAIT.
The first thing we should do now is to check out the Data Collection Scripts page. Click on the menu entry for Data Collection Scripts, and then take a look at the page. Download the appropriate data collection utility for the platform that you will be working on. These data collection scripts are really quite simple. They just force the generation of a series of javacores, and then package these up for the WAIT tool to process.
ONE WORD OF CAUTION: The generation of javacores will stop the JVM briefly while the javacore is generated. This can make your Jazz application “pause”, so be prepared. In addition, these javacores can take up disk space. I have had customers who have started generating javacores, and have forgotten to STOP generating them. The system eventually crashes when the space on the disk gets completely filled with javacores.
Now lets try launching that data collection script. I run on Ubuntu Linux (I know, it’s not officially supported for Jazz, but it works). On my machine, I just put the script in a utils directory under my user directory. Then I brought up the system monitor and found the java process that was running the RTC server that I have running on my system. I took note of the PID of this process.
So now that I know my Jazz PID, I can kick off a collection of data. I used this command line to get what I wanted:
./waitDataCollector.sh --iters 10 --javacoreDir /home/dtoczala/utils/WAIT_tools/cores --outputDir /home/dtoczala/utils/WAIT_tools/logs 19748
So let’s look at this command:
- The ./waitDataCollector.sh invokes my script
- The – – iters 10 option tells the tool to collect 10 javacores (at the 30 second default interval)
- The – – javacoreDir path is the path to the javacore files
- The – – outputDir path is the path to the resulting zip file of data collected.
Just execute the script without arguments to see a list of the arguments available to you. After 5 minutes, I can go out and see the resulting zip file in the /logs directory.
Now that I have something, what do I do with it?
So now that I have the zip file with my data, I can go back out to the WAIT website. At this point we can click on the Submit Data tab, and pull up that page. Now I select my file to be analyzed, enter in a description, and then I Submit for Analysis.
So now the WAIT tool will upload my zip file, and then process my javacores. This can take a little while, depending on the speed of your network, so now is a good time to go and grab a beverage.
Checking the Results
Once the tool is done you will see some colorful output, like the boring sample that I am showing below.
The top section has time running from left to right, and CPU utilization broken down over that time period. Using this graph you can easily determine if other processes on your Jazz server are running out of control, and just how hard your Jazz instance(s) are straining your CPU capacity.
The next section shows the runnable threads. This section, and the sections that follow, allow you to see what the threads of your Jazz server are doing. Click on one of these colored sections, and look at the Stack Viewer section in the lower right. You will see all of the various processes associated with that particular thread. The rest of the report is just like this, with you being able to drill down and get more information on particular threads and processes.
Using this and the Pop Out button in the Stack Viewer section, you can see exactly what is happing on your Jazz server at various points in time. You will want to look for threads and services that are taking up the most resources, and determine which operations are causing those to run. This will help you determine what is impacting the performance of your Jazz server. This is just another way to see what is going on “under the hood” of your Jazz application server. One more note, don’t neglect the memory usage section. Make sure to take a look at that too.
What Next?
This series of articles is continued. Here is a list of the topics covered:
- Jazz Performance Part 1 – Is my Network Causing me Pain?
- Jazz Performance Part 2 – Using the Admin Console
- Jazz Performance Part 3 – What does nmon have to do with my Jazz Server?
- Jazz Performance Part 5 – Keep an http watch on your Jazz Server
- Jazz Performance Part 6 – Obeying the Laws of Physics
- Jazz Performance Part 7 – What is JazzMon and Why Should I Care?
Dan Toczala's Blog 