Jazz Performance Part 7 – What is JazzMon and Why Should I Care?

Note: I have not lost the ability to count.  I realize that I am publishing Part 7 of this series before parts 5 and 6.  I just feel that this information on JazzMon can be extremely useful to people today, and want to get it out as soon as possible.  (I am NOT saying that parts 5 and 6 will be useless – grin)

Note #2:  John Camelon, who leads the RTC development team working on the SCM portion of the tool, pointed out some errors in what I had written earlier about scm.common.IScmService.refreshWorkspaces.  I have fixed that, and thank him for keeping me honest.

Recently I have been working with a new tool that we are using to assess the performance and health of deployed Jazz systems.  JazzMon is a great tool to use for this purpose.  It was developed by our Jazz performance team, and it does a nice job of allowing you to monitor the performance and health of your Jazz infrastructure.

The tool is referenced as a zip file on Jazz.net, and the jar file and properties files need to be installed.  You can reference the READ_ME file that is included in the downloaded zip file, and I suggest that you also take a quick look at the JazzMon FAQs.  The key things that it needs to know is the name of the servers to be monitored, the length of time to monitor, and some user/password information. The key here is to let the JazzMon application run continuously over a long period of time (say for a week), and take samples of data every hour.  With this level of granularity it will give you a much better idea of your trends in usage over time.

Just today I spent some time working with some of my IBM comrades looking at the JazzMon data associated with a particular customer issue that has become a high priority for us.  We looked at the JazzMon data by taking a dump of data from the customer site.  The data from JazzMon is captured as a CSV file (actually a text file, which has tab delimited data), which can then be imported into your favorite spreadsheet tool.  One of the table rows included in the file is a reference sample of what we have seen for values in our own internal Jazz implementation on Jazz.net.  Use this as a rough comparison for the data that you see from your own Jazz environment.

Some of the things that we noticed when analyzing the particular files that we were looking at were the settings associated with builds and workspace refreshing.  Some of the counters that we found most interesting were:

• scm.common.IScmService.refreshWorkspaces – this shows the number of workspace refreshes happening.  This has the potential to be an expensive operation.  This checks the status of the workspace, and if someone is doing a “clean” build, you will see this followed by a long list of IVersionedContentService.GET requests, as it pulls files over the network and establishes the workspace contents.  If the workspace is up-to-date, or close to being up-to-date, then this operation is not that expensive.
• scm.common.IScmService.compareWorkspaces – this shows the number of workspace comparisons being done, which is another example of an expensive operation, often associated with builds.
• scm.common.IVersionedContentService.GET – this is the actual fetch of a file from the SCM repository.

One of the tough things to understand is the relative meaning of all of the files output from JazzMon. Some of the ones that seem to be popular to look at are:

• licence_flVal.txt – this has license usage data for Jazz floating licenses (good for long term planning, and determining actual user loading trends)
• service_brTot.txt – this has the total number of bytes received by each service (good for determining network loading and network interface loading)
• service_bsTot.txt – this has the total number of bytes sent by each service (good for determining network loading and network interface loading)
• service_etAvg_sorted.txt – this is a sorted list of the average number of invocations of each service (key here is to look for BIG deviations from the baseline Jazz.net data shown)

One of my Jumpstart team members made the comment that the tough part here is looking at the data that seems out of the normal ranges, and then determining what the root cause of these measurements really are.  Some of this is obvious based on the service name, but some of these may be more subtle.  I encourage anyone using JazzMon to post those types of questions to the jazz.net forum, and use the tag “jazzmon” on your question.

These are just some of the things that I saw that were interesting.  Try JazzMon yourself, even if you are not seeing any performance issues.  It will give you a baseline of data that you can then use if/when you have performance issues in the future.

Other Articles in the Series

This series of articles is completed.  Here is a list of the topics covered:

• Jazz Performance Part 1 – Is my Network Causing me Pain?
• Jazz Performance Part 2 – Using the Admin Console
• Jazz Performance Part 3 – What does nmon have to do with my Jazz Server?
• Jazz performance Part 4 – Why WAIT, see what your Jazz server is doing
• Jazz Performance Part 5 – Keep an http watch on your Jazz Server
• Jazz Performance Part 6 – Obeying the Laws of Physics

Recent Interesting blog on RTC for a Mac

Just read this post on Installing Worklight 5 and RTC on a Mac, written by Aaron Allsbrook of ClearBlade. The guys at ClearBlade have done some very good work with RTC in the mainframe environment, and it’s really interesting to see them expanding their skills to cover software development in the mobile market, making use of the integration with Worklight.

Another Good Recent Blog Post

Boris Kuschel on my team sent me this link to a blog post that illustrates the differences in liberal and conservative software development cultures.  It is written by Steve Yegge of Google (whom I don’t know and have never met).  The blog post is pretty lengthy, but if you start reading it you will get sucked into it.  He makes some really good observations about how the relative importance we place on risk management influences the way we develop software, select programming languages, and do our daily jobs.  I love this post because he comes out and states what his preferences are, and then goes on to give a rather even-handed analysis of the situation.

Other Observations

I am seeing more demand in the market for basic Jazz administrative skills. I see job postings for Jazz Administrators, and people to architect Jazz solutions, almost daily. It all feels a little bit like the 90’s, when I could spend an entire month teaching Admin courses for ClearCase and ClearQuest.  There seems to be a growing demand for these skills, and some of these job postings seem to stay open for quite awhile. I am also seeing more independent consulting firms (like ClearBlade) involved with some of our Jazz customers. This is a good sign of a healthy Jazz community.