In the first two editions of this series, I looked at how you could do some simple network analysis and some simple monitoring of your Jazz servers. In this blog post, we get a little bit deeper into seeing what is happening on your Jazz server. One of the nice little tools that I have encountered in my work with Jazz deployments, is a little tool called nmon. This tool works on AIX and Linux environments.Author’s Note: I run Jazz in an unsupported Ubuntu Linux environment, and nmon works fine on my system as well. All of the screenshots in this article are from my own Ubuntu system.
What does nmon do and how do I get it?
Nmon is a great tool that allows you to see how your system is performing. On Windows (or even Solaris) you have the perfmeter that does essentially the same thing. A lot of of my customers tend to run their Jazz servers on Linux or AIX, so nmon becomes my friend in these instances.
You can learn about nmon from a a variety of locations, and even check out the nmon Introductory Workshop. there are a variety of download areas for this as well, but I trust the download area on SourceForge. It is now open source, so you can even dig in and help improve the tool if you want.
You will also want to download the nmonanalyser spreadsheet. I know, it’s in Windows format, but I am not too proud to use Windows apps where it makes sense. I am sure that you can make it work for your particular situation. We’ll go over using this later in this article.
I’ve got it! Now what do I do?
Once you have nmon installed, it is usually helpful to run through some of the interactive screens. Start up nmon (just type nmon), and you will see a menu screen.
Take some time and check out the displays and data for some of the options that you see here. The things that we will be primarily interested in when looking at our Jazz server will be CPU, memory, disks, network, and top-processes. Bring up each of those displays and get familiar with the data that they are showing you. Now that you seem to have this all under control, you are probably wondering exactly how you use this to tell you ANYTHING. It’s showing me things in real time, and in order to debug performance issues you need data over a period of time. You want to see how the system resources perform before, during, and after any performance issues that you might be having.
Using nmon from the command line
The nice thing about nmon is that you can run it from the command line, have it run for a period of time and collect statistics about your system, and then have all of that data free for you to analyze. Just type “nmon –help” to see the help screen for the nmon utility, and you will see all of the different possible switches that you can use.
I like using the following command to launch nmon:
nmon -f -s 300 -c 288 -t
This will launch nmon to output to a spreadsheet (-f), making measurements every 5 minutes (-s 300), for a day (-c 288), while collecting information on the top processes (-t). You would need to relaunch this every day. This is good if you are doing this to find a current issue.
If you are just monitoring things, over the long term, then I would look at using the following command to launch nmon:
nmon -f -s 1800 -c 336 -t
Now keep in mind that the directory that you launch nmon from is where the results files will be written. Make sure that you launch nmon in a spot where you will be able to easily organize the results (which come out named “<hostname>_yymmdd_hhmm.nmon”).
Launch this daily/weekly with a cron job (if on Linux), or a scheduled batch file (on Windows). Use daily dumps for looking at current issues, and use weekly (or even monthly) collections for the ongoing monitoring of Jazz server performance.
What Do I Look For?
For a Jazz based server, I like to monitor the following areas, for the following reasons:
- CPU – I like to know what the general load on the CPU is, but I am also interested in spikes at different times. Seeing consistent spikes at the same time every day can indicate that something unique is occurring each day at that time. For example, it could be the data warehouse ETL jobs, it could be some automated reports, or it could be the beginning of a workday for a particular site.
- Memory – I like to make sure that we’re not running out of memory.
- Disk I/O – I like to see how much we are going out to the local disk. This is less important if you do not have a local Lucene index being maintained.
- Network I/O – I like to monitor this to see if we are reaching the limits of what our network pipe can handle. It also allows me to identify periods of high traffic.
One key thing to keep in mind – this information is not going to tell you where your problems are all by itself. You need to correlate this data with times where you see performance degradation, and with expected user activities during those times. Typical areas that I like to look into include:
- Spikes in network I/O – often indicates a large amount of data which can impact performance, and which will often point us towards poor/slow network connections.
- Spikes in disk I/O – may indicate issues with local storage.
- Spikes in memory consumption – this will help you determine if you need to look more closely at JVM heap sizes and other memory parameters.
- Saturation of CPU – if you see that the CPU is consistently running about 50%, then it might be time to think about setting up another Jazz server instance.
Using nmon with the nmonanalyser
So now that you have these nmon data files being generated, how do you take a good look at them? This is where you will want to check out the nmonanalyser spreadsheet. The spreadsheet will use macros (so you need to enable macros) to read in those nmon data files, and will dump the data into a large number of spreadsheets, each on a separate tab. Each spreadsheet will cover a different aspect of system performance, and most of them have some built in graph which will show you what your system is doing.
As you can see in the example above, the graphs provide an easy way to visualize the various facets of system performance over a period of time. To use the nmonanalyser spreadsheet, I grabbed my download and made a copy of it. I then renamed the copy to something meaningful, so i always have a backup copy. When you open the spreadsheet, you will see the initial startup screen.
First you need to make sure that you ENABLE macros from this spreadsheet. Then you will want to press the button to analyze your nmon data (which you collected earlier with that nmon -f -s 1800 -c 336 -t command). When you hit the button, a file selection dialog will pop up, and you will select your nmon output file. Once selected, you will see the macro begin to process your input file. Once it has completed, it will prompt you for an output file. Just select a good name, and location, for the resulting spreadsheet. Exit the analyzer then sit back and enjoy your results.
Flip through the tabs on the bottom of the spreadsheet, and you will see graphs for all sorts of different system parameters. There are 22 different tabs on the spreadsheet, most of which have graphs to provide a nice visual representation of the data that was collected.
When looking at Jazz servers with nmon, it is important to keep in mind that application server performance is not the only thing that will impact the performance of your Jazz solution. Jazz uses multiple applications (most notably the JTS, CCM, QM, and RM) all working in concert, to provide a software development experience with deep integration capabilities. It will also rely on a database server for storing the repository contents, and the network is also a factor in performance. What this will do is allow you to either eliminate the application server performance, or dig into it more deeply, in the search for better Jazz performance.
This series of articles is continued. Here is a list of the topics covered:
- Jazz Performance Part 1 – Is my Network Causing me Pain?
- Jazz Performance Part 2 – Using the Admin Console
- Jazz Performance Part 4 – Why WAIT, see what your Jazz Server is doing
- Jazz Performance Part 5 – Keep an http watch on your Jazz Server
- Jazz Performance Part 6 – Obeying the Laws of Physics
- Jazz Performance Part 7 – What is JazzMon and Why Should I Care?