Real World Jazz Results – Our Customers are not “Kind of Blue”

Real World Jazz Results – Our customers are notKind of Blue
I have been wandering the world (electronically) and North America (physically) for the past year and half helping IBM customers understand the capabilities of Jazz, and the implications of the deployment of those capabilities for their organizations.  I have deployed countless Proof of Concept environments, run many Proof of Technology shows, and spent many a day discussing the relative merits of a variety of software development tools, methodologies, and just plain old war stories from the age of “stone knives and bearskins“.  I have also helped some of our larger customers begin to deploy the Jazz technology, and leverage the benefits associated with adopting the Jazz platform as a foundation for software development activities.  The most important thing that I have done, is not deploying the Jazz technology, it is measuring the results and benefits that the Jazz technology has given our customers.
IBM has been deploying Jazz internally for a while now, and we do have some measures of the benefits that we are realizing.  This is great… for IBM.  The issue is that IBM has a unique access to resources and circumstances that will not be replicated in the situation of ANY of our customers.  Just because the guy who lives next door to me realizes 20% lower medical costs by self insuring than I do, doesn’t mean that it is the right approach for me to take.  The guy is a doctor, so the that might skew the results a bit.  What the examples I am going to share in the article show is that it is possible to realize improvements in productivity and software development performance with Jazz, even if your company isn’t called IBM.
The first example is a typical example for many small organizations that adopt the Jazz technology.  This particular group is within a line of business with one of our larger customers, but they operate like a small software group.  They had a small team doing Agile development, and using a mix of spreadsheets and open source tools to help them.  We decided to use this team as a pilot project, after we assured ourselves that we could easily migrate them off of the Jazz platform after a handful of sprints.  That was 10 months ago, and the team still refuses to get off of the Jazz platform.  They bought it so they wouldn’t have to lose it.  They like it that much.
It been 10 months, and we have had a good amount of time to collect data from this team.  I will stay away from using actual dollar amounts and talk in rough percentages, since some of this information is considered sensitive by this customer.  The benefits that this team saw over time began with some dramatic improvement, once they got comfortable with the tools, and broke down along these lines (in terms of time):
  • Developers – saved roughly 18% of their time by using/collaborating with Jazz (90 minutes a day)
  • Business Analysts – saved roughly 11% of their time because stories were easier to manipulate, status and prioritize with Jazz (55 minutes a day)
  • Systems Analysts – saved roughly 40% of their time, since following stories and approved changes was so much easier (200 minutes a day)
  • QA Analysts – saved roughly 10% of their time, with easier traceability between stories, test cases, and builds (50 minutes a day)
  • Program Management – saved roughly 15% of their time with the improved transparency of Jazz (75 minutes a day)
My second example is a much larger organization, with over 400 software development staff.  This group has a large number of Agile teams (hence a much larger sample than my first example), that range in size from 8 to 18 people per team.  This particular group has decided to quantify their benefit in terms of dollars, by taking their average rate for a fully burdened resource, and multiplying it by the time savings.  After roughly 6 months, this group was able to summarize their annual benefits from a Jazz based implementation:
  • Planning Automation – $389,000
  • Daily scrum meetings (time reduced) – $810,000
  • Quality Improvement –  $194,000
  • Reduced effort to manage work items – $1,200,000
  • Project reporting – $130,000
  • Reduced time determining project status – $990,000
  • Reduced time to establish new teams – $432,000
  • Reduction of duplicate efforts – $648K

This comes out to a grand total of $4,793,000.  That is almost five million dollars.  These numbers look a bit unbelievable, and represent a HUGE cost savings and capability improvement.  Even if we assume that people are in “happy” accounting mode, and cut these numbers in half, it still represents almost two and a half million dollars in savings and increased capability.  This is significant.

Closing Thoughts
It has been a great first year of posting my insights, thoughts, and observations of Jazz technology to this blog.  I anticipate that I will continue to do so in the coming year.  This final post of the year is a significant one for me.  It is significant because it represents real capability increases and cost savings that have been realized by my Jazz customers, not just some projected benefits derived from a small pilot effort.  Our Jazz baby is beginning to grow up and make it’s own way in the world.  While these cost benefits are impressive, what holds more promise for me are the strategic benefits that Jazz provides to an organization:
  • It allows an organization to have a balanced portfolio of software tools that reduce it’s risk, and allow it to leverage successful technologies more quickly.
  • It provides an organization a high degree of transparency into IT development efforts, and thus improves the accountability with teams, groups, and the entire IT organization.
  • It allows our customers to actually do active monitoring and management offshore and outsourced IT efforts.
  • It helps software development people focus on what they are good at – developing software.
I hope that 2010 provides me with more inspiration and stories for this blog, and I encourage you to post comments and questions here.  Like the rest of the Jazz based universe, this small outpost is only as informative and entertaining as my readers make it.  Send me your stories, observations and questions, and I will get to them as best I can.

What is MCIF, and why does Jazz care?

Recently a lot of us at IBM/Rational have been talking about something called MCIF.  What is MCIF?  It stands for Measured Capability Improvement Framework.  You can read Per Kroll’s whitepaper on the overall concept, or just check out the IBM overview of MCIF.  I often have to explain it to our customers (or even to people outside of work who have nothing better to do than listen to some software guy attempt to explain what he does), and I use this analogy:

When you drive a car, you use a variety of feedback mechanisms.  Let’s take a trip to the grocery store to buy more coffee (so you can stay awake during this analogy).  You plan out your route.  You prepare.  You grab your car keys, and you hop in the car.  You rely on a core set of metrics to make sure that you reach your destination safely.  You watch the road, and you adjust your steering to guide the car around obstacles.  You watch the signs and signals, and stop for red lights and merging traffic.  The key point here is that while you have an overall plan on how to get to where you are going (the route that you will take), you don’t just blindly steer, accelerate and brake the same way every time you make the trip.  External events have an impact on the amount of time, the route, and the stopping and starting that you do on the way to your destination.  This is an agile type of philosophy.  Don’t plan your trip in intricate detail and blindly follow that plan.  Have a vision of where you want to go, and then rely on continuous feedback to get to your destination in the most efficient manner.

So that works for a single trip.  As a business, you don’t want single successful trips, you want a culture of excellence.  You want governance over the trips being taken.  You accept that fact that some of your projects will be failures, and some will be fantastic successes.  The key is to be able to differentiate the winners from the losers as early as possible.  You can only do this if you are measuring your projects, your people, and your organization.  In the software world, we always refer to these as metrics.  Let’s go back to our analogy.

So I drive my car a few times, but after about 8000 miles, the engine begins to run rough.  My car requires more fuel to get me to the same places that it used to.  By having a service record, and by measuring things like gas mileage, I know that the car needs an oil change.  Maybe it needs new tires, or a new battery.  Is the temperature gauge going up more frequently?  Maybe I need to get my radiator flushed.  The key here is that while I may take pride in being able to get to the pizza place and back in 10 minutes, over the long run I look at different measures to determine the relative health and capability of my car.

So this brings us back to MCIF.  MCIF is a framework of industry proven measures that can be utilized at a variety of levels.  Some measure the long term health and improvement in organizational performance.  Some are used to measure short term project performance.  Some are used to gauge organizational health and the impact of changes to software development methods, teams, and tools.  The key is to MEASURE things.  Not all of the metrics called out in MCIF may be useful to you in your industry.  That’s OK.  No metric is perfect.  Know that, and accept it.  The measures just give you a way to quantify results, and the ability to ask more probing questions to get to the root of organizational issues.

This is all pretty straightforward and simple.  The old maxim that, “You cannot manage what you cannot measure“, has been around for a long time.  Is any of this new?  Not really.  A large portion of this is just the application of long held business concepts, to the wild, wild west of software development.  During the 1990’s, when there was a web startup around every corner, nobody asked about returns on investment, returns on capital, or some of the more basic productivity measures.  Now that the industry has grown up, it gets to be treated just like any other facet of the business.

So what does all of this have to do with Jazz?  Jazz technology is a means to an end.  Teams perform better when they know they are being measured.  That measurement needs to be done intelligently.  The performance of teams and the organization as a whole needs to be available to the teams, their management, and the business executives.  Teams that are measured, but never know how they are being measured, will never be able to improve.  Having an environment where teams can see their performance, and where they can see the impact of their individual actions on that performance, begins to set up a vicious cycle of excellence.  The transparency and visibility that the Jazz technology gives to all members of an organization, helps drive accountability.  The capability that Jazz has to be able to bring together data from different disciplines, form different stakeholders, with different concerns, allows an organization to begin to make correlations between measures, and determine leading and lagging indicators of project success.

This is all great stuff.  But there must be a dark side, a sinister evil secret hiding in this scenario.  There is.  This visibility and transparency can lead to negative behavior.  Using the enhanced transparency to punish will lead to negative behavior in the organization.  Punishing teams for high defect rates?  Guess what, they will stop reporting their defects, stop tracking potential issues, and your quality will suffer. You will lose customers, market share or capability.  Executives need to focus on demonstrating leadership.  Metrics that do not measure up need to be addressed, but the teams need to be empowered to solve the issues.  High defect rates?  The first reaction shouldn’t be, “Cut them all”, instead it should be to ask the question, “Why are there high defect rates?”.  Does it correlate to a decrease in quality?  Does it impact our time to market?  Is it a statistical anomaly?  The metrics do not answer questions, they indicate areas where management focus is needed.

The most visible benefit of the Jazz technology, and having Executive management with a healthy attitude about measuring their teams, was provided to me by one of the early Jazz adopters.  The manager told me half jokingly that the biggest example of the success of Jazz was a horribly failed project that had just been killed.  I was taken aback, but I asked him why this was.  “In the past it would have taken us 6 to 9 months to determine that the project was doomed to be a failure.  Using Agile development methods and Jazz, we were able to kill it in 2 months.  We saved 4 months of development costs, and we were able to allocate those people to a project that provided a positive ROI for the company”.