I have spent the past couple of weeks doing some interesting investigation into some performance issues and some system tuning with one of our customers.  It has been a really interesting exercise for me, and like most of the times that I work on things like this, I have been learning a lot.  There are a lot of things that exist out there that can help you check the operational health of your Jazz infrastructure.  Some of these I knew about, some of these I didn’t know about, and some of these I knew about but had forgotten.  Now let’s look at some basic techniques that just about anyone can use to identify issues in their Jazz infrastructure, and improve performance.

No Software Needed – Using Metronome to Gauge Network Throughput and Latency

The Jazz architecture is based on a set of Jazz servers located in some central location, like a data center.  You communicate to these servers using either the RTC Eclipse Client, or through your web browser.  This communication occurs over your network, and the speed and throughput of your network has a direct impact on the performance of the Jazz tools.  Slow or low-bandwidth networks will make the use of ANY web based application slow and painful.

What it Does

There is a great little tool included with your RTC Eclipse client, called Metronome.  Now I know what you are going to say, “Our performance with the Eclipse client is fine, we just have some web client issues”.  Metronome can help with those as well!!  The tool does a repeated ping test to the Jazz server, which gives you a range of ping times and a more accurate indication of latency in your network.  Networks with high latency will demonstrate poor performance, with either the Web client or the Eclipse client.

The tool will also do an upload and subsequent download of a 2MB file to the RTC server.  This gives you an indication of the network bandwidth that you have available to you.  This is important, since you may have a fast network, or a network with large bandwidth, but if the network is fully loaded, the effective speed of your transactions will be slower.  So don’t be fooled by a gigabit backbone, or a muscular NIC card.  use Metronome to find out what your throughput REALLY is.

If your performance is poor in the Eclipse client, the tool provides a lot of information which you can use to help you determine where you might be experiencing issues.  Read more details about the Metronome tool, and some of the other information that it provides, in the Jazz.net article The Jazz metronome tool keeps us honest.

How to Get Your Data

Metronome comes packaged with your RTC Eclipse client in RTC 3.x.  Once you have the Eclipse client launched, connect to your RTC server.  Then from the menu bar select Window -> Preferences.  In the resulting dialog, navigate to Team -> Jazz Source Control, and check the box for the Metronome tool.

Now you will see a small metronome icon at the bottom of your Eclipse client.  

Double click on the icon, and select the connect that you want to test.  You will now see the Metronome results and guidance dialog.  Click on the Connection tab, and then click the Test Connection Speed button at the top of the dialog.  After the tests have run, you should then be able to see your results.  There is also some guidance on the types of results that you should expect to see.

What I See

I get pretty good performance with my laptop accessing the Jazz.net server.  When connected to my home network, with my cable modem, I get ping times in the range of 80ms – 120ms.  Anything below 200ms will give you acceptable travelling performance, and if you can stay below 100ms you should see acceptable performance.  Values greater than 200ms indicate that you will experience slower performance due to network latency issues.  Talk to you network administrator and try to get your client machine “closer” (from a network perspective) to the RTC server.

For the effective upload/download speed, I see for 10KB/s – 15KB/s for my upload speed, and a range of 80KB/s – 100KB/s for my download speed.  This seems consistent with what I see when I use either my Eclipse or Web client.  Downloading files is fairly responsive, but uploads of files (attachments, code, blog postings with lots of diagrams) can be a bit slow.  We like to see upload/download speeds in the 60KB/s range for what would be considered acceptable performance.  If you are not seeing this, you can either look to increase the effective network bandwidth, or move your Jazz infrastructure to a less “noisy” part of your network.  In my case, all I can do is curse my ISP, and broadband cable technology in general.

One thing to note is that your results will vary depending on when you run the tool.  If you see good performance in the early morning (when nobody is loading up the corporate network), but poor performance at lunchtime (as everyone in the office watches their favorite YouTube videos), run metronome at those times and note the differences in the values returned.  The key thing to remember is that while a single measurement will not prove or disprove adequate network performance, it can help you understand your actual network performance better.

Other Things to Try

You can also use the Netalyzer tool from Berkley labs.  This tool does an analysis of your network performance, and will give you an idea of what kind of performance you are seeing.  It is not just a simple dumb speed test (like Speedtest), or a simple ping measurement, it does a full assessment of the performance of your browser/network.  It is a web based tool that runs a Java applet within your browser.  It worked on Firefox for me, although my Chrome browser had some issues with it.  The output is great, and does a great job explaining everything.  Here is a look at what I saw when I ran it from my home office.

Interestingly enough, it pointed out that my ISP has a poorly performing DNS server, and over buffers packet traffic.  That refers to Buffer Bloat, a common issue today with network performance on the Internet.  I was impressed with the information provided by the tool – it took roughly 3 minutes to pull all of this together.

After a few minutes of discussion, it occurred to me that we were thinking about this all wrong.  Why should we constantly be calling our customers and asking them about their Jazz deployments?  The Jazz servers capture a number of statistics about themselves and the repositories that hold their artifacts.  You can see these in the administrative reports available on your Jazz servers.  If we could somehow harvest this information from our customers who have deployed the Jazz solution, then we would be able to be more proactive about the advice and guidance that we give our customers.

My idea is to have a small task that runs each time the data warehouse jobs run (typically nightly).  With each run, a small email is written which contains some basic high level information, which then gets sent back to the Jazz team.  The email is in plain text, so customers can easily monitor (and scan) the information that is being sent out.  Each server would have a switch where they could configure if the emails are sent out at all (so customers could elect not to participate), and a setting to configure where the email is sent (for those customers that want to review the contents of the email before forwarding to the Jazz team).  The basic information to collect would be the following:

• License usage – average, and peak for each license type
• Repository sizes
• Number of application servers, type, and locations (ie. a customer may have one JTS, one RTC, and two RQM applications, in different data centers, deployed)
• Type of application server used (Tomcat or WAS), and type of database used (DB2, Oracle, SQL Server)
• Basic REST response times (as a rough measure of performance)
• Potentially a series of simple performance metrics

I would then propose that we make this information readily available on Jazz.net, with customer names obscured, so our customers would be able to see what other Jazz deployments look like.  Are there other customers using the full set of CLM tools?  How many other customers are using two RTC applications sharing the same JTS?  Are our repositories considered large?  In addition, we would be able to monitor customer situations and offer predictive help.  We might also find that customers using the full set of CLM tools have RTC repositories that grow at an average rate of 20% per year, while stand alone RTC repositories only grow at a rate of 15% per year.

So now I have a question for the people who read this blog.  Does this seem like a good idea for you and your organization?  Does this seem like a vendor that is looking to work with it’s customers to monitor it’s deployed products to improve them, or an intrusive vendor seeking to grab and gather information for their own use?  Would you turn on this capability?  Would you find value in seeing these metrics on Jazz.net?  Are there things that I should include in my list of collected data?  Things I should take out?

I am looking for feedback on this idea.  If I get enough support for this, I will try to promote it so that it is included in a future version of the Jazz products.  So don’t be shy – let me know what you think.

During the class I made the point that the CLM solution is wonderful, and the integrations are powerful, but only if our customers are able to leverage these integrations to help improve the speed or quality of their software.  One way customers realize the benefits of this is through the real-time colaboartion enabled by being able to seamlessly jump from doing one thing, say code development, to another, maybe checking the requirements that you are developing code for.  the other way that customers realize this benefit is through improved cross-capability reporting.  Being able to report and correlate data from you change management, configuration management, requirements management and test management tools is a powerful thing.

So we discussed Rational Reporting for Development Intelligence (RRDI), and we discussed how this was something that any Jazz customer has the entitlement to use, and how critical it can be to the overall success of some of our Jazz customers.  Then we also spent some time discussing Rational Publishing Engine (RPE) and Rational Reporting for Document Generation (RRDG).  Since we don’t run into these so much, I had to admit that I didn’t know a lot about these, other then the fact that Rational Requirments Composer (RRC) uses the RRDG reporting engine to execute some of it’s reports.  Needless to say, I wasn’t providing too many answers to the questions my class was throwing out there.  All I could do is relate what I knew from having gone through the CLM 2011 Reporting Workshop (which is VERY good).

One of the other students in the class was able to rescue me, and provide some very good information on our reporting solutions.  He pointed out to me and the rest of the class that they could enable themselves very easily with RPE, using the online resources available to everyone.  He shared some links with us to some tutorials for using RPE.  I glanced at them briefly, found them useful, and figured that I should share them with the rest of the world:

• RPE and RQM Tutorial for version 3.x
• RPE and RRC Tutorial for version 3.x

So if you get a chance, run through these and let me know what you think.  I would be interested in hearing your feedback on how useful these tutorials are.

Comparing Open Source and Jazz CLM

Comparison of open source solutions and the Jazz CLM solution is extremely tough.  Where you end up seems to depend on your point of view, and in what you find to be important.  Open Source solutions appeal to smaller teams, because of the low cost of obtaining the tools in question (essentially $0 licensing), and the high degree of autonomy that teams have when deploying these tools.  In addition, when looking at this from just a software development team perspective, the tools do the job that they are intended for, are lightweight, and considered “good enough”.  The standard question is , “Why pay money for additional functionality which you will never use?”.  It is a valid point.

The Jazz CLM solution appeals to larger teams, and to people who deal with multiple software development teams.  When teams want to be able to share repository information, roll up software development metrics, and have visibility across multiple software projects/teams, then the Jazz CLM solution is a clear winner over open source solutions.  The integration of the data elements, and the visibility of information is much better with Jazz CLM.

You might have noticed that I am being very general at this point, and not going into specific functionality.  That is because the differences in developer desktop functionality are often quite minor.  People tend to like what they are comfortable with, and the relative “ease of use” and “look and feel” types of measures are extremely subjective and based largely on the experiences of the individual.

The broad general statement that you can use to differentiate Jazz CLM and just about every open source tool is that open source is cheaper to obtain, but the Jazz CLM offers more robust integrations and reporting capabilities.  This is a broad generalization, since in many cases the overall cost of ownership for Jazz CLM may be lower than open source, and also since a well crafted open source solution can offer adequate integrations and reporting capabilities.  The key here is determining the overall cost to your organization.  If you are building the infrastructure to integrate open source solutions, you will need to spend time/money both creating and maintaining these integrations.  If you are deploying to a large organization you should also factor in the administrative costs, in terms of overall tool administrators, as well as in terms of the amount of “self-service” administration done by each of the individual teams.  (see my blog posting Jazz Hands – Administration and Cost of Ownership)

One other cautionary statement.  Do your research, but consider your sources.  I am an employee of IBM, and I work on the Jazz development team.  You can Google blogs and articles on software development tools and get about 2000 different opinions.  Some love Jazz, some love open source, some love other commercial products.  Always read these things with an open mind, and consider the author.  Some of them you can trust to be fair and balanced (I hope that you have me in this category), while others will clearly have a highly biased view of things.

Other Considerations When Comparing Jazz and Open Source

Some of the other things that you need to consider when comparing Jazz and open source solutions can become quite important, and often they go unmentioned.  One of the key things that HAS to be considered are the interests of ALL of your stakeholders.  Software development environments impact many different levels of the organization, some directly, some indirectly.

Software developers and software testers are two core groups of stakeholders whose concerns need to be addressed.  Do the tools in question address their needs for automation, information sharing, and workflow management? How many of their opinions are based on objective measures, and how many are based on a level of comfort with a particular technology?

Software integrators and delivery teams also need to be considered. Can multiple development efforts be tracked and coordinated??  How can I tell the progress of the multiple teams whose software I depend on to deliver particular capabilities to the business?  Do I have visibility to the information that I need?

Your Operations team needs to be considered.  Can this solution be supported by a centralized service team?  Can the solution be easily deployed and scaled as needs change within the organization?  How easy is it to deploy and maintain the solution?  What SLA’s are expected?

Finally, your Management needs to be considered.  Sometimes they will be the driving force behind a decision, since they have to sign off on the purchase of licenses, hardware and network infrastructure to support a software development environment.  Will they be able to have the access to the information that they need?  Will they be able to satisfy any regulatory requirements?  Will they have transparent views and access to see how their development teams are doing?  Do they sufficiently understand the concerns of the other stakeholders?

Take all of your stakeholders into account before deciding on a course of action.  Weigh the costs and benefits of your approach to each group, and then clearly let them know what these costs and benefits are.  Regardless of your decision, and how it reflects the concerns of each group of stakeholders, ALL of your stakeholders will be more open and cooperative during the deployment of your solution if they understand the reasons for your choice, and they feel that their concerns were heard and understood.

Strategic Considerations of Software Development Tooling

The strategic considerations of what types of software development tools that you use, and how you deploy these tools to your organization are quite important.  So far in this article I have provided an overview of the things that you will want to consider, and some idea on some of the more common benefits of Jazz CLM and open source.

In my mind I like to look at the tools used to support a software development environment as an investment.  You are spending money on licenses, hardware, training, etc., in order to improve your software development capabilities.  Are you getting an adequate return on this investment?

The investing angle also helps me frame the entire conversation about if an organization should choose Jazz CLM or an open source stack of tools.  I would argue that you should use BOTH.  The Jazz CLM solution allows open source tools to be easily integrated into the Jazz environment.  This allows teams to choose the tools that best fit their needs, while maintaining the ability to provide reporting and visibility to the efforts of the software development teams.  You use the mix of tools that best fit your software development culture.

Like any investment portfolio, you need to be diversified.  It helps you avoid the risks associated with depending on a single company, technology, or approach.  You need to be free to try new technologies and approaches, without having to scrap your software development infrastructure every 5 years.  I recently read an article that talked about how new technologies have disrupted the industry in the past.  On Route 128 outside of Boston you can still see the shells of the old DEC and Wang headquarters.  During the 1980′s we all assumed that these vendors would always be around to support their products.  They are both long gone now.  Not relying on a single source for your solution will allow you flexibility in the future, and reduces the risks associated with the loss of support for any portion of your software development infrastructure.

Utilizing the Jazz CLM technologies mixed with open source, you reduce the risks of going with an IBM-only or open source-only approach.  If you choose tools that support the OSLC specification, you can also include tools and capabilities from other vendors as well.

I can hear your objections now – “We wanted to consolidate tools, to reduce costs and improve reporting and metrics!”.  With a Jazz CLM infrastructure you will get consistent metrics reporting, roll up of data, and transparency to the efforts of the software development teams. Now your administrative costs will increase with more tools (more things to train administrators and end users on), so I would not suggest adopting any open source tool with Jazz CLM.  I would select a strategic set of Jazz/open source/other vendor capabilities that will provide some real value to my teams.

Conclusions

So this article has discussed some of the tradeoffs and considerations that you need to be aware of when trying to decide between open source tools and Jazz CLM.  My argument is that these decisions need to be made with an understanding of the strategic implications of your decision, and an eye on reducing the long term risks of your decision.  The choice between open source and Jazz CLM is not an either/or proposition.  Having a mix of open source, the Jazz CLM technology and other vendor solutions (as long as they all are able to integrate via OSLC) allows teams to choose the appropriate tools for their particular task.  A healthy portfolio of software development tools and technologies should be diversified to reduce the risks associated with technological obsolescence and support for your solutions.

“Where do I start when I want to upgrade to CLM 2011?”.

Your first stop should be your local IBM Sales team.  They should be able to get you in touch with the right people inside of IBM to help you learn about, plan, and then execute your upgrade.  Keep in mind that this upgrade can be complex, and that you will need to make some choices that will impact your future options for scaling and the deployment of your solution.  So here is a list of what you really should do:

• Contact your IBM sales team – they will hand you this same list, but they may also be able to provide you with up-to-date information and some assistance in the planning of your upgrade.
• Remember to READ upgrade directions CAREFULLY – Many people get into trouble when they skip steps, or go looking for specific answers and read things out of context!
• Read about URI stability, and the Upgrade FAQ -  Do this before you plan your upgrade, it will help you understand some of the reasons for how the upgrade is done, and what limitations you have during upgrade.  Plan your URI’s, and read Tim Berners-Lee’s article called Cool URI’s Don’t Change.
• Fill out the upgrade assessment tool – This will help you understand your current Jazz environment, and help us understand your situation better.
• Run through the upgrade workshop BEFORE you plan your upgrade -  This is CRITICAL – you will understand more about your options after doing this workshop.
• PLAN your upgrade – Work with IBM, we may be able to help you with this planning.
• Read the Infocenter carefully – Make use of the interactive upgrade guide.  (again, skipping steps will cause you to have problems!)
• TEST your upgrade – Do this to test out your execution of the upgrade, and so you can time the procedures needed.

I know that this list may look large, but it is critical that your upgrade be done properly.  The question that I always seem to get when I present people with this list is, “OK, that is for your usual customers, but we are smart/in a hurry/adventurous.  What is the short version of things that we have to do?”.   THIS IS THE SHORT VERSION!!  The Jumpstart team spent months of testing effort helping to get the upgrade process as smooth as possible.  We also spent a lot of time developing the upgrade workshop, and everyone who has used it thus far has remarked on how much better they understood and were able to perform their CLM upgrade.  It may take a while to go through it, but it is an essential part of planning for your upgrade.

So spend the time doing your homework and planning BEFORE you upgrade, and your upgrade experience will be smooth and uneventful.

I have to thank Jim Ruehlin (read his blog, Jazz Practices) for providing the inspiration for this post.  As we were talking about this topic this afternoon, it became something that I urgently had to get down on paper.  Jim is part of my Jumpstart team and (along with the rest of the Jumpstart team)  is one of the foremost experts around Jazz upgrades.

Added on 9/21/2011 – Additional info:  Check out Jim Ruehlin’s blog on URI’s, “You Can’t Change the Public URI. Really. You Can’t.“, and David Chadwick’s blog, “Public URL’s are Cast in Stone – How Do I Plan for the Future?“

I had been talking to a customer about the relative merits of the jazz platform, and how some of the core architectural decisions that we made have had some very profound impacts on our Jazz based products.  We spent some time discussing how the use of linked data and REST based objects in our Jazz architecture has made it possible to have a robust and flexible framework for integrating software development tools.  They had an interesting issue that they wanted to explore.

They have a set of homegrown tools for their mainframe development environment, and they are looking at the Jazz CLM solution for their distributed (what they called their mid-range) development teams.  They had a lot of questions about OSLC, and were very interested in how easy or hard it would be to do an OSLC interface to their mainframe SCM systems.  How would they do it?  (Check out the Lyo project for Eclipse)  The reason they were interested is that they wanted to be able to reference change sets from both the mid-range development teams, as well as the mainframe development teams, when they had work items that had work that was required in both domains.

I shared with them some of the concepts behind OSLC, as well offering them some use cases for this kind of work (like using a parent work item with child work items on either side of the mid-range/mainframe divide). Then we talked about flexibility of the Jazz environment, and the balance that you need to strike between having the power to modify and customize systems without limit, and the need to have systems that are easy to maintain and operate.  If that flexibility was without cost in terms of development and maintenance, then everyone would create their own custom software development environments.  We discussed how your development tools need to be good enough that they help speed your development, improve your quality, and enhance you collaboration, without having a high overhead expense associated with keeping them maintained and administered.

They kept coming back to the relative difficulty of producing an OSLC interface for their current mainframe SCM.  Without time to look at what they had in place, it was impossible for me to really tell them anything intelligent.  They seemed to be searching for an answer that I just couldn’t give them.  It was a good conversation, but it felt a bit unfinished.

It wasn’t until I was in the cab on the way back to the hotel that I realized what I had failed to mention.  It didn’t really matter how much effort was required to create an OSLC interface for their mainframe system.  What matters is the relative cost of that interface.  If they wanted to do this on their own, without OSLC, then they would need to create something similar to OSLC.  Which means they would need to do that, AND the effort that they had been asking me about.  What other alternative is there if you want to link your data?  The value of OSLC is that it is already there, already proven, and can be used as a framework enabling communication between different software development tools, without worrying about any vendor specific or proprietary interfaces.

I guess the lesson here is that the REAL value of OSLC isn’t in the bits and bytes, but it is in the concept.  Tools should be able to discover each other other, determine services, and create links and relationships between their artifacts regardless of vendor or technology.  OSLC is the common language that software development tools can use to seamlessly communicate with each other, without brittle and hard to maintain point-to-point integrations.  You can either use what is already there, and has been proven, or take the time and expense of trying to create it yourself.  When you frame it in those terms, the problem seems much less daunting, and the answer seems a bit more obvious.

Now all I have to do is find those guys so I can tell them…….

So what are the differences between Tomcat and WebSphere?  At the most simple level, Tomcat is open source, and WebSphere is an IBM product.  You can Google “WebSphere vs Tomcat”, and see articles that claim that Tomcat is much better, and others that claim that WebSphere is superior.  In what I have seen in practice, and what my customers have told me, WebSphere seems to scale better, and has better support for enterprise types of functionality (like clustering, identity and authentication, and so on).  Some like Tomcat for it’s simplicity, which is pretty nice.  I think that it probably boils down to what your experiences have been, and where your organization has expertise.

From a higher level, IBM is able to sell WebSphere because it has advanced capabilities that the open source alternatives do not have.  WebSphere is easier to administer, and allows you to deploy things in different profiles, which allows you to take down one Jazz application without impacting the other running applications.  Some other Jazz related things that you can do with WebSphere that are difficult (or impossible) with Tomcat:

• Single Sign On solutions are easier to support with WebSphere, and Tomcat does not support a distributed single sign on capability (if you have different web servers for the various Jazz products).
• WebSphere has an Administration UI that greatly simplifies the following operations:
• Installation of applications
• Stopping and starting those applications (on an individual basis)
• The configuration of the JVM properties used by the applications
• The setting up of a reverse proxy
• More flexibility with LDAP security and support, with an ability to map LDAP groups and roles to Jazz application groups
• Ability to more easily use monitoring capabilities to observe web server performance

In addition, some of the newer planned capabilities for the Jazz applications will be initially implemented on WebSphere.  Now I can’t tell you what those capabilities might be, but a quick look at our plans out on Jazz.net should give you some idea of what those might be.

Keep in mind that you will not have all of the functionality that you would have if you were using Jazz SCM, the integration just provides for the linking of Perforce changelists to Jazz work items.  This is not an integration that is done at the OSLC level.  For those customers that have an absolute need to keep Perforce as their SCM solution, this allows them to enjoy the advantages of the Jazz architecture without having to change SCM systems.

For those of you who don’t care about Perforce, the article is still informative.  It provides a nice example and reference implementation for the linking of other SCM solutions with the Jazz architecture.  Ken’s approach is something that you can use to link other tools and repositories into the jazz environment.

The Jazz Cafe is going to be much more interactive this year.  We have had customers give us very positive feedback on the informal help and conversations that they have had with our experts in the Jazz Cafe.  This year we will also be running our own miniature conference, with subjects covering a wide array of topics.  Even if you have questions on a topic that is not on the schedule, I strongly encourage you to drop by.  We will have members of the Jazz Jumpstart team available to answer questions and address your issues (even if they aren’t on the schedule, or if you missed one of our sessions).  We’ll also have some people just grabbing coffee and relaxing between sessions in the Jazz Cafe.

So be sure to drop by the Jazz Cafe to make connections with our experts, meet other Jazz users, exchange ideas and approaches to address software development issues, and just relax in an informal and creative atmosphere.  The Cafe will be open on Monday, Tuesday and Wednesday, from 9:45am to 11:00am, and again from 2:00pm to 4:00pm.

Here is a link to the schedule of events in the Jazz Cafe:

Jazz_Cafe_2011

Here is a quick look at it – but it might be small in your browser….

What if you are new to the jazz world, and don’t have a collection of RSS feeds?  Don’t worry, a friend of mine has pulled together a collection of “essential” Jazz feeds for your RSS reader.  Darrel Rader works in the Rational brand with me, and he has collected a bunch of very good Jazz feeds into one bunch, in an OPML file that you can download and import into your own RSS feed reader.  Just click on this link for Darrel’s Jazz RSS Feeds.  It’s a great way for new users of the Jazz platform to  stay on top of current events, and noteworthy news.

I would provide some more help and guidance on different RSS feed readers to use (I use Liferea, Darrel likes RSSOwl, and there is an OPML addon for Firefox), and how to import the OPML file, but like I said, I am insanely busy.  Just download Darrel’s feeds, look at some of the FAQ’s and help with your favorite RSS reader, and begin to feel more connected.  It shouldn’t take you longer than 10 or 15 minutes to set up.