Note: I have had a tremendous response to this article, and the attached PDF. I also received some great feedback which I am incorporating into the document.
2/13/2013 – updated the document based on comments and feedback.
2/14/2013 – updated the PDF document based on additional feedback. Added content on reporting and weekly administrative tasks.
A lot of my time has been spent working on Jazz performance over the past year. As I worked with different customers, I saw a lot of different approaches and a lot of different performance issues. As the year has rolled on, I have tried to capture things that I have learned, and share them in my blog series on Jazz performance (see Jazz Performance Part 1 – Is My Network Causing Me Pain?). I have also been capturing a lot of the best practices and explanations about Jazz performance. I have compiled all of this information, and checked with the experts, and put it all into one place.
Last year, some of my IBM friends also mentioned the concept of a performance guide to our customers. They called this guide a “Purple Book”. What you see attached here is the initial version of that “purple book”. Why am I releasing this right now, on my blog? Why isn’t this on the “official IBM site” or up on Jazz.net? This is an early copy of that “Purple Book”, but there is enough good information in here that it may be worthwhile for a lot of our customers. It still needs more information, more refinement and more review, before it can get released on Jazz.net. This information will actually become the foundation of a Deployment wiki site which should go live later this year. I expect the information to be updated and expanded upon in that deployment wiki. This document is a “point in time” view of Jazz performance. I know that it will improve and change over time, as new tools and techniques for monitoring and improving Jazz performance are uncovered. Please feel free to provide comments (and even content!) for this guide.
So what is this guide about? It has sections on what impacts the performance of a Jazz solution, with some basic explanation of why performance is impacted. It has a guide on monitoring your Jazz implementation, and then finally ends with a section of troubleshooting advice and tips. Here is a link to the Jazz Performance Guide, please give it a read and let me know what you think.
Finally I get to the final chapter of my blog postings on Jazz Performance. I ended up doing parts 7, 6 & 5 in reverse order, because those seemed to be the most relevant at the time. So I guess that this makes this post the least relevant of all! It isn’t, but the area that this post covers tends to be a very focused one, and one that may not apply to many of our customers. This post focuses on the performance of the web client in your Jazz environment, and how you can tell where time is being spent in the responses to user requests through the web browser clients.
Using Web Browser Monitors
The best way to determine how time is being spent, is to use one of the web browser debugger and monitor tools that are commonly used. I will cover a tool called Http Watch. I start with Http Watch because it works for both Firefox and Internet Explorer. It runs on Windows, and is an add-on/plugin for both Internet Explorer and Firefox. Once you have it installed, go to the Tools menu of either browser to activate it.
Once you launch Http Watch, you will see a window at the bottom of your browser. This is where you will be able to see exactly what is going on in the browser. Now navigate out to Jazz.net, and check out one of the plans. Go out to Jazz.net and log in, using your user ID and password (because I know that you already have one). Now before you navigate to the plan, make sure to hit the record button in the Http Watch window, which tells Http Watch to begin recording and timing your browser interactions. Then navigate to the RTC Product Backlog Plan. It might be easier to navigate to the plan first, start recording with Http Watch, and then do a page refresh. Be careful though, because the browser cache may influence the actual load time in this case. Either way, you should see a graph begin populating in the lower window indicating what has happened during your display of this plan. Wait until the plan is fully displayed and rendered, then press the Stop button in the Http Watch window. Now you can go and look at the data.
Checking out The Data
What Does It Mean?
One of the things that I have used this for is to highlight the differences between the various browsers and browser versions that my customers have available to them. In many instances, customers are limited in what they can install on their machines, and they may be on earlier versions of some browsers. This allows them to compare browser performance in their environment. Not only are you able to compare the overall time needed to display page contents, you can also see how much time is being spent in processing and transferring the information, and contrast the differences between your browsers. It is something that I strongly suggest that you do, and let your users know what you have seen. End users want to know the differences in browser performance, and they also like to see benchmarks of what is considered “normal” performance in your environment. If they see that a particular plan takes 10 seconds to render for you, then they know to expect similar performance when they use the tools.
Setting expectations and getting performance baselines is important. Recording them somewhere visible to your users is even more important. If you begin getting complaints about slower performance three months from today, you will have a set of tools and measurements so you can objectively determine if things have deteriorated over time, or if the end users just have rising expectations. This objective data is critical when trying to address performance issues in your environment. It allows you to weed out the users complaining about normal performance, and allows you to quickly zero in on real problems. it will then allow you to measure the impact of any changes that you make to your configuration.
Other Articles in the Series
This series of articles is completed. Here is a list of the topics covered:
There are a number of things that can impact the performance of your Jazz servers. Some of these things you can address with better architecture. Some things can be addressed with better tuning of the web servers, JVMs, databases and Jazz applications. Some things cannot be changed. Some things are just slaves to the laws of physics. Electrons can only move so fast (“186,000 miles per second – Not just a good idea, it’s the law“), and there are distinct physical limitations that we need to be aware of in our environments. What are some of these limitations that you need to be aware of?
The biggest physical limitation to the performance of any computing system is latency. The amount of time it takes for your data to make the physical journey from your Jazz server to the end user machine is just one place where latency comes into play. There is also the amount of time data needs to travel between the various servers that support your Jazz applications. What are some of the more common causes of latency?
Since the Jazz applications often depend on LDAP for user authentication, the latency between the Jazz JTS application and the LDAP server can have an impact on end user performance. That is why we always recommend deploying Jazz in a data center, since most corporate data centers have a local LDAP repository, thus minimizing the amount of latency between the JTS and any associated LDAP servers.
Likewise, the Jazz application servers, web servers, and backend database servers should be co-located if at all possible. Some Jazz customers do have them in different locations, but I always insist that these be located within the same physical location. A 100ms ping time may not seem like much, but when a query needs to return data from 100’s of work items, and the round trip between the Jazz server and the database server ends up being 200ms for every operation, the time adds up.
You cannot control where end users will access your Jazz solution from, but you can control expectations. Users that are overseas need to understand that their performance will not be as good as the performance for the people who are in a more local region. There are some ways that you can address this, using SCM caching proxies and distributed Jazz SCM capabilities, but you always have to be aware of the issue. Make sure that your end users have realistic expectations.
One of the less known causes of latency, and one that most people ignore, is the presence of switches and routers in a network. A physical database server may be only 30 feet away from the Jazz application servers, but if these are on different subnets, and require routing between multiple switches to communicate, then the net effect is the same as having the machines thousands of miles apart. Because of this is it always important to have a feel for what the typical round trip time is for your data. You can use network tools to determine this, or you can use the Jazz Performance Health Check widget. Do NOT use ping as a measure of latency. Ping sends small packets, and networks will handle ping packets differently from Jazz application packets. Trust the results that you see in the performance health check widget, since it does not use ping, but sends actual packets of data between the web client and the Jazz application servers.
Jazz applications are intended to perform in wide area networks, but anything that you can do to further minimize latency will result in improved end user performance.
The second big physical factor that will limit your Jazz performance is the concept of throughput. This is the amount of data that can be transitioned or transferred from one of the components of your solution architecture over a given period of time. The common places that most people will observe throughput as a bottleneck to performance is at the web server layer. A web server can only service a certain amount of requests in a given period of time. This is a limitation of the Jazz solution that is imposed by the web server technology that you choose to deploy with (either Tomcat or WebSphere).
A more common limitation is the network. Your network can only supply a certain volume of data in a given time period, regardless of the number of requests waiting to go onto the network. Network bandwidth is not typically an issue, but it can become an issue in a couple of different situations. In the case large builds, often teams will first create a workspace in which to execute the build. In cases where the code base is large (like a full Android build), all of the code needs to be transferred from the repository to the local workspace. Users need to understand that this is the equivalent of downloading the entire code base over the network. In the case of large code bases, this transfer of data would take tens of minutes (if not hours) if just done via FTP over the network. It will not go any quicker with Jazz, the files still need to be physically copied from the repository to the workspace area. Coordinating large builds, and doing incremental builds, can help spread out and reduce some of the pain. Having a build farm that is close to the Jazz application servers is another way to reduce the impact of builds on your network.
The second area where I have seen network bandwidth come into play is in virtual environments. In some virtual environments, the virtual machine appears to have a full 100BaseT or gigabit ethernet connection to the network. However, the virtual machine is actually sharing this connectivity with other virtual machines that exist on the same physical hardware. So if you have 4 Jazz application servers running in their own virtual machines, on the same physical hardware, then the maximum amount of throughput for ALL machines will be limited by the network bandwidth. in this situation, a large build being done out of the repository of one of your Jazz SCM instances, could potentially impact the performance of ALL of your Jazz applications, since the network bandwidth is being saturated by the files being transferred in support of the build.
Another area where throughput and bandwidth come into play is with the disk I/O used with a Jazz solution. The Jazz application servers are not heavy users of the disk storage on their systems, since data is stored in the repositories, which are located on the database servers. Since Jazz deployments do store a large number of artifacts, some of which have large sizes, the disk I/O on the database servers is critical. The disk I/O controllers will limit how fast data can be pulled from the database, so it can be returned to the Jazz application server, and ultimately to the user. Make sure that you are monitoring the speed and load put on the disk I/O controllers, so you can see when disk I/O throughput issues are limiting your performance.
Jazz performance has been a hot topic with many customers recently. Everyone wants to know how to optimize their Jazz deployments. What most people REALLY want is to balance the costs associated with the hardware needed to support a Jazz deployment, with the user expectations of acceptable performance. Jazz administrators can tune and adjust their deployments to make them more efficient, but they need to realize that basic architecture and the physical limitations of the hardware environment will also have an impact on the overall performance of their deployments. Being aware of how these physical limitations impact how you deploy your Jazz solution will allow you to make better architectural choices for your specific deployment.
Other Articles in the Series
This series of articles is completed. Here is a list of the topics covered: