Jazz Performance Part 1 – Is my Network Causing me Pain?

Note: This is the first in a series of blog entries on Jazz Server performance.  Links to additional blog entries appear at the bottom of this blog.

I have spent the past couple of weeks doing some interesting investigation into some performance issues and some system tuning with one of our customers.  It has been a really interesting exercise for me, and like most of the times that I work on things like this, I have been learning a lot.  There are a lot of things that exist out there that can help you check the operational health of your Jazz infrastructure.  Some of these I knew about, some of these I didn’t know about, and some of these I knew about but had forgotten.  Now let’s look at some basic techniques that just about anyone can use to identify issues in their Jazz infrastructure, and improve performance.

No Software Needed – Using Metronome to Gauge Network Throughput and Latency

The Jazz architecture is based on a set of Jazz servers located in some central location, like a data center.  You communicate to these servers using either the RTC Eclipse Client, or through your web browser.  This communication occurs over your network, and the speed and throughput of your network has a direct impact on the performance of the Jazz tools.  Slow or low-bandwidth networks will make the use of ANY web based application slow and painful.

What it Does

There is a great little tool included with your RTC Eclipse client, called Metronome.  Now I know what you are going to say, “Our performance with the Eclipse client is fine, we just have some web client issues”.  Metronome can help with those as well!!  The tool does a repeated ping test to the Jazz server, which gives you a range of ping times and a more accurate indication of latency in your network.  Networks with high latency will demonstrate poor performance, with either the Web client or the Eclipse client.

The tool will also do an upload and subsequent download of a 2MB file to the RTC server.  This gives you an indication of the network bandwidth that you have available to you.  This is important, since you may have a fast network, or a network with large bandwidth, but if the network is fully loaded, the effective speed of your transactions will be slower.  So don’t be fooled by a gigabit backbone, or a muscular NIC card.  use Metronome to find out what your throughput REALLY is.

If your performance is poor in the Eclipse client, the tool provides a lot of information which you can use to help you determine where you might be experiencing issues.  Read more details about the Metronome tool, and some of the other information that it provides, in the Jazz.net article The Jazz metronome tool keeps us honest.

How to Get Your Data

Metronome comes packaged with your RTC Eclipse client in RTC 3.x.  Once you have the Eclipse client launched, connect to your RTC server.  Then from the menu bar select Window -> Preferences.  In the resulting dialog, navigate to Team -> Jazz Source Control, and check the box for the Metronome tool.

Now you will see a small metronome icon at the bottom of your Eclipse client.  

Double click on the icon, and select the connect that you want to test.  You will now see the Metronome results and guidance dialog.  Click on the Connection tab, and then click the Test Connection Speed button at the top of the dialog.  After the tests have run, you should then be able to see your results.  There is also some guidance on the types of results that you should expect to see.

What I See

I get pretty good performance with my laptop accessing the Jazz.net server.  When connected to my home network, with my cable modem, I get ping times in the range of 80ms – 120ms.  Anything below 200ms will give you acceptable travelling performance, and if you can stay below 100ms you should see acceptable performance.  Values greater than 200ms indicate that you will experience slower performance due to network latency issues.  Talk to you network administrator and try to get your client machine “closer” (from a network perspective) to the RTC server.

For the effective upload/download speed, I see for 10KB/s – 15KB/s for my upload speed, and a range of 80KB/s – 100KB/s for my download speed.  This seems consistent with what I see when I use either my Eclipse or Web client.  Downloading files is fairly responsive, but uploads of files (attachments, code, blog postings with lots of diagrams) can be a bit slow.  We like to see upload/download speeds in the 60KB/s range for what would be considered acceptable performance.  If you are not seeing this, you can either look to increase the effective network bandwidth, or move your Jazz infrastructure to a less “noisy” part of your network.  In my case, all I can do is curse my ISP, and broadband cable technology in general.

One thing to note is that your results will vary depending on when you run the tool.  If you see good performance in the early morning (when nobody is loading up the corporate network), but poor performance at lunchtime (as everyone in the office watches their favorite YouTube videos), run metronome at those times and note the differences in the values returned.  The key thing to remember is that while a single measurement will not prove or disprove adequate network performance, it can help you understand your actual network performance better.

Other Things to Try

You can also use the Netalyzer tool from Berkley labs.  This tool does an analysis of your network performance, and will give you an idea of what kind of performance you are seeing.  It is not just a simple dumb speed test (like Speedtest), or a simple ping measurement, it does a full assessment of the performance of your browser/network.  It is a web based tool that runs a Java applet within your browser.  It worked on Firefox for me, although my Chrome browser had some issues with it.  The output is great, and does a great job explaining everything.  Here is a look at what I saw when I ran it from my home office.

Interestingly enough, it pointed out that my ISP has a poorly performing DNS server, and over buffers packet traffic.  That refers to Buffer Bloat, a common issue today with network performance on the Internet.  I was impressed with the information provided by the tool – it took roughly 3 minutes to pull all of this together.

What Next?

This series of articles is continued.  Here is a list of the topics covered:

• Jazz Performance Part 2 – Using the Admin Console
• Jazz Performance Part 3 – What does nmon have to do with my Jazz Server?
• Jazz Performance Part 4 – Why WAIT, see what your Jazz Server is doing
• Jazz Performance Part 5 – Keep an http watch on your Jazz Server
• Jazz Performance Part 6 – Obeying the Laws of Physics
• Jazz Performance Part 7 – What is JazzMon and Why Should I Care?

Concerned partner, or intrusive tool vendor? Tell me.

This week I have been involved in a series of discussions about the upgrade of the Jazz products from their 2.x versions to the 3.x versions.  As we discussed some of the challenges that our customers have faced, and the impact that an improved upgrade process would have, we were at a loss for being able to assess the impact to ALL of our customers.  We know the customers that we have sold products to, but we don’t know how big their Jazz repositories are, we don’t know how many licenses have been deployed,  and we don’t know how the products have been deployed.  So we began to discuss how to best find out some basic information from our customers.  People began to discuss how we should best ask our customers for this information.

After a few minutes of discussion, it occurred to me that we were thinking about this all wrong.  Why should we constantly be calling our customers and asking them about their Jazz deployments?  The Jazz servers capture a number of statistics about themselves and the repositories that hold their artifacts.  You can see these in the administrative reports available on your Jazz servers.  If we could somehow harvest this information from our customers who have deployed the Jazz solution, then we would be able to be more proactive about the advice and guidance that we give our customers.

My idea is to have a small task that runs each time the data warehouse jobs run (typically nightly).  With each run, a small email is written which contains some basic high level information, which then gets sent back to the Jazz team.  The email is in plain text, so customers can easily monitor (and scan) the information that is being sent out.  Each server would have a switch where they could configure if the emails are sent out at all (so customers could elect not to participate), and a setting to configure where the email is sent (for those customers that want to review the contents of the email before forwarding to the Jazz team).  The basic information to collect would be the following:

• License usage – average, and peak for each license type
• Repository sizes
• Number of application servers, type, and locations (ie. a customer may have one JTS, one RTC, and two RQM applications, in different data centers, deployed)
• Type of application server used (Tomcat or WAS), and type of database used (DB2, Oracle, SQL Server)
• Basic REST response times (as a rough measure of performance)
• Potentially a series of simple performance metrics

I would then propose that we make this information readily available on Jazz.net, with customer names obscured, so our customers would be able to see what other Jazz deployments look like.  Are there other customers using the full set of CLM tools?  How many other customers are using two RTC applications sharing the same JTS?  Are our repositories considered large?  In addition, we would be able to monitor customer situations and offer predictive help.  We might also find that customers using the full set of CLM tools have RTC repositories that grow at an average rate of 20% per year, while stand alone RTC repositories only grow at a rate of 15% per year.

So now I have a question for the people who read this blog.  Does this seem like a good idea for you and your organization?  Does this seem like a vendor that is looking to work with it’s customers to monitor it’s deployed products to improve them, or an intrusive vendor seeking to grab and gather information for their own use?  Would you turn on this capability?  Would you find value in seeing these metrics on Jazz.net?  Are there things that I should include in my list of collected data?  Things I should take out?

I am looking for feedback on this idea.  If I get enough support for this, I will try to promote it so that it is included in a future version of the Jazz products.  So don’t be shy – let me know what you think.