Jazz Hands – Administration and Cost of Ownership

I get inspiration from a lot of sources.  My latest came from my customers, and my daughter.  I was relaxing last weekend, sitting around a campfire with my wife and my daughter.  It was one of those times when you talk and exchange ideas with your kid.  I was thinking about how nice things were, but thoughts of work crept into my head, and I was thinking about the many customer requests for “typical” administrative headcounts that I have seen in the past few weeks.  Right about this time, my daughter pipes in with the final piece of her story about her skating program (she figure skates), “..and we decided that we needed to be more visual, you know, have more Jazz hands”.  It seemed like the perfect title for a blog entry on administration of Jazz based technology, and specifically Rational Team Concert (RTC).

Jazz is a centrally based repository, which relies on a simple architecture to provide software development services to it’s users.  The details of this architecture are not particularly important for the purposes of this discussion, what is important is how Jazz manages the way it is configured, and how it holds data.  At the core of Jazz is the concept of a process template.  This is an XML file which describes in detail how a process works within the Jazz framework, what roles exist, the permissions that the roles established will have, the types of objects (work items) that exist, their associated workflows, and approval processes.  Modification of this XML file can be done via text editor, or by using the GUI in Rational Team Concert.  It is much more than a state engine, it helps define a process, and it allows that process to be applied to individual projects in a uniform manner.  The key point here is that the process is just another piece of data that lives within a Jazz based repository.

Why is this important?  Most tools allow you to configure how they will work (where files are stored, how records are kept, etc.), and they also let you enforce workflows and process.  The key difference is that in order to modify the process, you need to have some understanding of how the tool works, and then you need to interact with the tool to implement these process features.  ClearCase used triggers, which could be implemented through the writing of scripts that were attached to the repository environment in a very tool specific manner.  ClearQuest used hook code (specific implementation of Visual Basic or Perl which interacted with the API to effect specific business rules), or it’s own interface for defining records, workflows, fields and so on.  Other tools all follow a similar paradigm.

With a Jazz based implementation, the process can be manipulated with a GUI, and can be done without specialized knowledge of the tool API, or the tool inner workings.  I know what you are thinking, what in the world does this have to do with tool administration?  The subtle difference manifests itself in some significant ways in the real world.

  1. Tool Administrators no longer need to know some programming or scripting language.  Everything is GUI driven, and it all happens on a centrally located server, not some distributed set of resources that need to be managed and coordinated.  Tool administrators can support users from around the world, without leaving the data center.
  2. Database Administrators no longer have to perform tool specific steps to do backup and recovery operations on the repository databases.  They do not need to have ANY tool knowledge.  Tool administrators do not need to have any specialized database knowledge.  Each can live safely within their own area of expertise.
  3. Process engineers, Subject Matter Experts (SMEs), and project leaders can now modify individual project processes on their own (assuming that their roles permit them to) without having to get tool administrators involved.  They can modify the processes used for a project, without impacting the processes used by the organization as a whole.
  4. Jazz based solution architectures are more simple and straightforward, enabling easier understanding of the architecture, easier configuration, and reduced maintenance.

While this may not seem significant, the impacts on cost of ownership, and the number of “Jazz Hands” that an organization needs to support their software development infrastructure are significant.  In the past, the ratio of users to administrators usually had these types of typical values:

  • ClearCase (complex deployment) – 100 : 1
  • ClearCase (simple deployment) – 350 : 1
  • ClearQuest (complex deployment) – 200 : 1
  • ClearQuest (simple deployment) – 500 : 1
  • Subversion (complex deployment) – 200 : 1
  • Subversion (simple deployment) – 400 : 1

Now with Jazz, IBM is hosting an internal deployment of almost 20000 users, with a staff of 5 administrators.  This is a ratio of roughly 4000 : 1.  Granted, the administrators that we have internally are a pretty sharp bunch, and they do have a responsive support channel to our product development teams (grin).  Even if I assumed that a “regular” Jazz administrator was only half as effective, that still leaves us with a gaudy ratio of 2000 : 1.  This represents an enormous cost savings for our customers deploying Jazz technologies.  In addition, these administrative resources no longer need to know and understand tools, software development processes, and scripting languages.  They just need to know a little bit about Jazz, which is pretty easy to pick up, and easier to become enabled with than any of the other ALM tools out in the marketplace.  The cost savings are big, and the reduced complexity is an additional benefit.

So let’s bring it back to administrative costs.  If I have an organization of 200 people in software development, the only gain I will realize is that my overall environment is simpler and easier to understand.  It means less complexity.  If my organization is larger, say 2000 people, then I begin to realize some impressive savings.  For 2000 developers, a ClearCase/ClearQuest environment may have used 3 ClearQuest administrators, and 5 ClearCase administrators worldwide.  If I were to assume a relatively cheap fully burdened engineer rate of $60000 per person, this would cost me roughly $480,000 a year.  With some other less functional tools, I might need 4 or 5 overall administrators, for a cost of $240,000 per year.  With a Jazz based infrastructure, an RTC implementation would need a single administrator, for a cost of $50000 per year.  This represents a significant cost savings, and when coupled with the reduced hardware footprint needed for Jazz based offerings, it can quickly add up.

12 thoughts on “Jazz Hands – Administration and Cost of Ownership

    1. dtoczala

      I have a lot of data and models for value of the tools, as well as the Cost of Ownership. The best model that I have looks at things from a 30,000 foot level, and is only good for a rough order of magnitude (ROM) type of estimate. Trying to do anything finer is usually a waste of time, because then the assumptions will get inaccurate, and you end up getting an answer that isn’t too close to reality. It might make people feel better, because someone worked on it, but the accuracy is misleading. I would rather stay on the factual end of things, and not make any wild claims about accuracy. If you would like to walk through something like this, drop me an email (dtoczala@us.ibm.com), and we can set up a half hour or so to walk through the model.
      In general, I find that most larger deployments of Jazz technology will realize a breakeven point after deploying to about 75% of their development organization. Since these deployments aren’t done overnight, by the time we begin deploying to the “late adopters” within an organization, the benefits to the early adopters have already paid for the tools and the tool deployment activities.

    1. dtoczala

      IBM has internal deployments of Jazz that have over 20000 users, both developers and contributors, and the implementation seems to be growing every week.
      Jazz.net is a different beast. There you probably have more than 20000 “logins”, but all of those users are in a contributor role, and thus much more lightweight and easier to manage. These casual users visit weekly, or less frequently, as opposed to a developer who may be using the SCM functionality and living in the tool for 50-75% of their workday.

  1. Daniel Cox

    I’m not sure it is quite reasonable to compare administration costs for an active enterprise development community with those on jazz.net. While jazz.net has a large number of users, most of them are infrequent – at best – users of the actual Jazz/RTC implementation. First, no external users use source code. Second, even those users that have some interaction with work items are only reporting occasional defects or viewing existing entries.

    In an enterprise deployment you have a much more active user base which in turn leads to more administrative issues. Also the number of separate projects that are maintained are probably much higher than jazz.net’s two projects (though the RTC project represents the very high end of complexity for a single project).

    Having said that – I absolutely agree that Jazz/RTC are a VAST improvement over ClearCase and ClearQuest in terms of administration as well as many other areas.

    1. dtoczala

      A very good point. Jazz.net actually uses fewer administrators, and as you say the user base there is more passive. The implementation that I reference in this post is a US based implementation that is hosting a wide array of software development efforts. Some of these are simple, and some are complex. Again, the project profiles may not match what a “typical” IT shop might have, but even if we increase the ratio of administrators to users by a factor of two, the cost savings are significant.

  2. James

    I am putting together an internal presentation in an attempt to get momentum for us to take a serious look at this technology. We are currently a complex ClearCase/ClearQuest shop. Is there a reference point that you used to get the administrative ratios?

    ClearCase (complex deployment) – 100 : 1
    ClearQuest (complex deployment) – 200 : 1
    RTC – 2000 : 1

    I would like to include this info in a slide. While the data is good, it would be really nice if there was an official IBM report or a Forrester report that I could reference. As you can imagine, the credibility of a blog isn’t as strong (even if it is a blog from somebody that works for IBM).

    1. dtoczala

      Those numbers come from the customers that I have seen around the world in the past 15 years. I have seen the CC/CQ ratios get even worse in some situations, especially with embedded systems development shops (think airplanes, lasers, missiles, pacemakers, etc.). I have seen better ratio’s in other shops (IT heavy shops with consistent processes), and they may get up to a 400:1 or 500:1 ratio. I have even seen customers at our user conference with ratios as high as 800:1.

      As far as RTC/Jazz are concerned, our data is more limited, but early adopting customers are seeing ratios of 200:1, with a steady increase as they migrate their development teams into the RTC environment. Most have somewhere from 200-800 users deployed, with a couple of administrators (not necessarily full time). Now internal within IBM, we have what I stated in the blog. That number is somewhat inflated (read the comments on the blog, one of my readers had some good observations). So I wouldn’t suggest that you would see this kind of ratio, at least not with a straight face. I can honestly state that your administrative headcount for supporting Jazz should be somewhere in the range of 10%-40% of what it was with CC/CQ (depending on the complexity of your implementation, number of users, etc.). The architecture of Jazz just makes it much easier to administer.

      Just be careful in what you claim!! While the end state needs less administration, you will not realize those gains on day 1. You need to migrate users, and while you are doing this, you will need to support your legacy CC/CQ environment as well as your Jazz environment. Most of my customers see a small decrease in cost of ownership in the first year or so (during this transition phase), and my earliest customers are now starting to see significant reduction in administrative costs in year 2 and beyond.

  3. Pingback: Supporting Jazz – Everyone Needs a Good Beat « Dan Toczala’s Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.