Basics for Upgrading ANY Software

Recently I became involved with a customer upgrading UrbanCode Deploy.  I want to share my experiences, some lessons learned, and some IT basics.

Our customer was looking to upgrade their UrbanCode Deploy to the latest version.  Doing this meant that they had to upgrade in steps, as outlined in the IBM Technote on Upgrading UrbanCode Deploy.  The customer understood this, and they began to work with the IBM support team on getting their upgrade done.  They had a simple plan which outlined the high level steps involved, and they had received a couple of patches to support issues specific to their environment.  I became involved when they were one week away from doing the upgrade on their production systems.

As I became more involved, I became increasingly alarmed at what I was seeing.  The migration plan was too simple – it had no step-by-step guidance on which values to select for configuration options, installation paths, or even server and environment names.  This left fer too much room for error, and made the upgrade process prone to errors when executed in the production environment.  That leads us to our first general IT lesson, which is:

General Lesson #1 – Any upgrade plan must be able to be executed by someone OTHER than the tool administrator or the author of the plan

One other factor that concerned me was that I saw no detailed “rollback plan”.  Part of the upgrade plan HAS to include what staff should do if the production upgrade goes bad.  It could be due to power outages, lack of resources, or some other unforeseen circumstance.  You need to have detailed instructions (see General Lesson #1 above) on how to restore the existing environment if the upgrade fails.  Nobody likes to do this, but if the upgrade does fail for some reason, people will be under pressure and tired.  They need to have easy to understand and easy to execute instructions on how to restore the production environment.  This is our addition to the first lesson,

General Lesson #1a – Any upgrade plan without a “rollback” section instructing how to restore production to it’s pre-upgrade configuration, is not complete

One of the other areas where I had concerns was due to the fact the customer was planning on moving ahead, even though they were receiving delivery of post-migration scripts the day before the planned upgrade of their production environment.  They kept insisting that they had tested everything in their staging environment, but I knew that they would not be able to adequately test the post-migration scripts (more on those later) prior to upgrading production.  They had tested things extensively in their staging environment, but they had only tested individual parts of the upgrade process.  This leads us to our second general IT lesson, namely:

General Lesson #2 – NEVER upgrade software on production systems unless you have done a FULL upgrade scenario in your staging environment

I can hear the grumbling already.  “We can’t push this out, it’s our only upgrade window for the next month”.  I understand that this can be frustrating, and can seem like be over prepared at times, but this is a lesson drilled into people in our business based on traumatic experiences of the past.  A decision to proceed with an upgrade of a production environment should only be done when you have prepared enough so the risk of ANYTHING going wrong is negligible.  Our systems are critical to the organizations that employ us, treating them with a cavalier attitude only puts our profession on trial.  The decision to upgrade, without having done a full upgrade in a staging environment, has been called “IT Malpractice” by some of my friends in the industry.  It’s a great term, and one I plan to use in the future.  The basic question is this: “Is the potential pain in a botched upgrade worse than delaying the upgrade?”.  If you haven’t covered the lessons spelled out above, assume that your upgrade will NOT be successful.

The customer was also upgrading from version 4.x to the latest version of UrbanCode Deploy.  This meant some changes to the architecture of the product which has a direct impact on the end users.  The first of these is a change to the way that security is handled by UrbanCode.  You really need to be aware of your security settings and the changes that may occur as part of the upgrade.  If you are unfamiliar with the UrbanCode Security Model, then review it and make sure that you have a clear understanding of how roles, permissions, and teams impact the ability of end users to deploy software in your environment.

UrbanCode Lesson #1 – Understand the UrbanCode Security Model, and know how it is impacted by the upgrade

Another thing that happens when upgrading to UrbanCode Deploy 6.x is that your resources move from being in a flat list, to a tree structure.  This allows you to organize your resources into common groups, and find them much more easily in the UI (ie. no 2000 item pull down menus).  During the upgrade to 6.x the urbanCode Deploy resources will be reorganized into a “flat” tree structure.  This has an adverse effect on performance for tool administrators, as page loads become slow if you have a large number of resources.

In order to address this, and as a way to better organize your resources, you should refactor your resources into the tree structure provided.  There is a simple script that IBM can share with you that will refactor resources based on the name of the resource.  You can read Boris Kuschel’s blog on how he deployed this on Bluemix.  Essentially you just have a script that breaks up the resources alphabetically.  You’ll probably want to alter the script to refactor your resources based on some other criteria, but the code is all there.

UrbanCode Lesson #2 – Understand the changes to the UrbanCode Resource Model, and know how it impacts your upgrade

Also keep in mind that some of this refactoring could potentially impact your procedures, depending on how you reference those resources.

Summary

Upgrading any software in your production environments is a risk.  We often think of upgrades as being “simple”, but a tool upgrade is often dictated by a foundational change in a product.  These foundational changes will often impact how the product is supported and how it operates, and this may have an impact on your environment.  ALWAYS follow standard IT best practices and TEST upgrades in legitimate testing environments.  Make sure that you have a script (especially for manual steps) for the upgrade that has been fully run through without issues in your testing environment prior to attempting to upgrade your production environments.  I hate seeing my customers going through painful situations that could have been easily avoided with some risk management and planning.

Advertisements

How Do I Deploy Jazz?

In my recent travels and discussions with Jazz customers and potential Jazz customers, I am hearing a common theme.  This theme has been building for the past couple of months, and I keep hearing different variations on the theme.  it has made me think, and has had me talking with some of the really smart people on the Jazz team (as opposed to myself).  The big question out there seems to boil down to this:

“I bought into your vision of Jazz making software development more streamlined, more transparent, and making teams accountable.  Now that I own this thing, how do I roll it out quickly to my organization without turning my teams upside down?”

It’s a good question.  Many of my new Jazz customers have a significant investment (in terms of time, effort, and money) in their current tools and processes.  Everyone understands their current tools and processes, even in the worse case scenario where the environment is a patchwork of different tools glued together by a bunch of scripts.  Now they have a whole new paradigm and environment to work with (called Jazz), and it represents risk to every project that considers implementing it.

Consider the Risks

Nobody likes a pessimist, but it is important to understand the risks that an organization faces when deploying ANY new solution to their entire software development environment.  Everybody believes in standard tools, as long as it’s THEIR standard tools.  Once you decide to roll Jazz based tools out to your organization, part of your development community will love the idea, and part of the community will hate it.  Neither group is really right or wrong, they all just have different concerns, different fears, and different levels of uncertainty that they are comfortable with.  It’s human nature, and you need to be aware of it, and acknowledge it.  So if I have to sum up risk #1, it would be, “Not everyone is going to embrace the changes that you are going to make”.

The other risk to consider for the teams that will be implementing a Jazz solution is the risk to their productivity.  Anytime you change the processes or tools in use by a team, their productivity will decrease, regardless of the tool being implemented.  This is because people need some time to become effective with the tool.  After they become comfortable and effective with the tool, they will become more productive then they previously were.  The goal is to minimize the loss of productivity.

Loss and Gain of Productivity With New Tools
Loss and Gain of Productivity With New Tools

Teams with aggressive delivery schedules may be resistant to change, because they do not think that they will realize enough of the increased productivity (the area in green) to make up for the time lost during tool implementation (the area in red).  We can sum up this risk by stating, “Teams are afraid that they will be unable to realize the benefits quickly enough”.

Consider Your Goals

Always keep in mind what it is that you are attempting to accomplish.  You want to have a software development environment that serves many different communities.

For the software development teams, it has to help them adhere to some process, without being a constant roadblock and source of wasted time.  It should help them collaborate on their tasks, and should help them operate as a team, and not as a bunch of individuals.   It should help them become more productive, and allow them to do their jobs with a minimal amount of  “static” introduced by misinterpreted requirements, lost work items, manual processes, and overall lack of good communications.

For software development managers and project managers, the environment has to keep the developers focused on the work (and not busy complaining about a lack of support or knowledge), and it has to be able to provide accurate and objective project metrics.  The environment should provide visibility at many different levels, and allow the capability to quickly react to changes in market conditions, the business environment, and overall priorities.

Finally, for the software executive, the software development environment needs to provide two things.  It needs to provide visibility.  Software executives should have visibility into lines of business, divisions, groups and projects, so they can easily identify what is working within their corporate culture, and what is not working.  Along with this visibility comes an implied accountability.  Teams that know they are being monitored are accountable, they alone determine their success or failure, and that result is out there for the rest of the corporate culture to see.

Within these three different communities are many sub-communities.  There is the quality community, the security community, and others.  These communities all have their own specific needs that need to be met.

With all of these competing stakeholders, it might seem that a solution that will satisfy them all would be impossible.   With Jazz, you are able to balance these needs, and implement a solution that will satisfy the core needs of each of these constituencies.  The key is to involve each of these communities at the beginning of any implementation, and to make sure that their CORE needs are articulated.  Make sure that the needs are articulated, and not specific technologies.  Needing easy and immediate visibility to key project metrics is a core need, seeing something in a vendor specific tool (IBM or otherwise) is not.

I guess we can sum this up by stating this as risk #3, “Make sure that the core needs of ALL key stakeholders are articulated and understood.”

Consider The Reality

There are a lot of different approaches that an organization can take to deploy Jazz technologies.  So far we have been in what I call a “rainbows and unicorns” world, where we have focused on the potential for a significantly improved software development environment in terms of collaboration, productivity, and interoperability.  We haven’t really addressed the reality of YOUR situation.   Hopefully you are still reading this and have not dismissed me as a starry eyed idealist.  This is where reality rears it’s head, and where the real benefits of the Jazz approach become apparent.

When organizations change the tools and processes that they use to develop software, they incur costs associated with the purchase, deployment and support of those new tools and processes.  As I have discussed in previous blog entries (see How Does Rational Jazz Compare with Open Source Solutions), the most expensive part of any change are the costs associated with migration.  This includes the migration of development artifacts, tools, and most importantly, the minds and attitudes of the software development population.

So since this is expensive, and any new tool requires this migration, what makes Jazz so special?  With the introduction of connectors and bridges to other common tools, a Jazz infrastructure allows an organization to plug current repositories and tools into the Jazz infrastructure.  This allows organizations to realize some of the benefits of a Jazz infrastructure, without costly and time consuming migration activities.

This is a vital consideration for many organizations.  For a project in a maintenance phase, where the costs of migration might exceed the benefits, it allows an organization to leave these assets “in place”.  The organization can then selectively migrate projects based on the return that the new infrastructure will give them, while still realizing some of the incremental benefits associated with Jazz.

In this current economic environment, having this flexibility allows an organization to effectively control the costs associated with migration, since projects can be migrated selectively over time.  By controlling the pace of these migrations, and organization can control the costs (and measure the benefits) and limit the risks associated with migration activities.  Ask anyone who has done one of those “over the weekend”, full cutover activities, about the risk and time spent in planning for this activity.  The response will probably include rolling eyes, claims of “never again”, and the comment that the person involved “isn’t paid enough for this kind of thing”.  The key point here is that one size does not fit all, but there are some common approaches and migration patterns that can be utilized.

If you had to sum this up, you could state this risk as, “The reality of your situation is unique, and migration has significant costs associated with it”

Iterative Rollouts and  Adoptions

Migrations to a Jazz infrastructure can be done on a project by project basis, and can often be done while leaving the current tool infrastructure in place.  This is the key building block to an effective strategy for migration.  On the surface, you are migrating your organization to a new set of tools, technology, and processes.  In reality, you are implementing change across the entire organization.

So if the capability to do this is a building block, what is the overall approach to migration that is going to save me money, sleepless nights, and reduce the risks associated with implementing change?  The key here is to implement change in small, easy to control and easy to monitor pieces.  You need to implement change in an iterative process, focusing on a small subset of teams and/or functionality in each iteration.

Implementing change in an iterative process allows you to control the amount of data, people, and infrastructure being migrated.  This results in less complexity, less planning, and a workload that can be handled more effectively by a more limited support staff.  Doing this alone improves your chances for successful migration, since the team implementing change is more focused on a handful of projects, and not just scrambling trying to keep everyone happy.

There are two more important aspects that need to be followed through on, in order to realize all of the benefits of deploying change in an iterative manner.  The first of these is the feedback at the end of each iteration.  This feedback should come from the teams that are being migrated.  Find out which approaches worked, what techniques were most effective, and find out where the implementation of desired changes to the organization can be improved.  The key here is to be open and honest about your migration activities.  You need to acknowledge your mistakes and shortcomings, and then work to improve on those areas.  Leave your ego at the door, and acknowledge that constructive criticism makes you stronger and more effective in the long run.

The final key is the one that is most often overlooked in the implementation of any change across an organization.  Too often we lose sight of what we REALLY want to accomplish.  The goal of any Jazz migration is not to get Jazz deployed to the organization.  The true goal is to IMPROVE THE ORGANIZATION!  Go back to those core needs of your stakeholders.  Are those needs being met?  You need to measure the impacts on cost, performance, reliability, collaboration, productivity, and support that this organizational change has had.  Tailor your measures to the key needs of the organization.  Is speed to market important?  Then measure release cycle times.  Is quality important?  Then measure defect density.  Identify key measures that are important to your organization, and measure these before, during, and after each iteration of change.  By measuring and monitoring each iteration, you can determine the impact that your changes are having on the organization, and you can easily determine if the costs of migration are exceeding the benefits of the organizational change.  If the benefits do not outweigh the costs, then why bother changing?

Conclusion

We have to acknowledge the risks inherent in the deployment of any new capability to an organization.  These can be summed up in the following bullet points we discussed earlier:

  • Not everyone is going to embrace the changes that you are going to make.
  • Teams are afraid that they will not be able to realize the benefits quickly enough
  • Make sure that the core needs of ALL key stakeholders are articulated and understood
  • The reality of your situation is unique, and migration has significant costs associated with it

How do you address these very real risks?  By deploying change across your organization in a controlled and iterative manner.  You measure the benefits and costs after each phase, and make determinations on the continued value of migration, as well as areas where you can improve the effectiveness of migration activities.  Having this constant feedback allows you to determine  if the results are worth the effort.  In these days of intense budget pressure, being able to measure the costs and benefits of new technologies and capabilities is critical to the continued success of any software development organization.

Having objectives measures of effectiveness and capability also allow you to address the risks associated with migration activities.  Showing project managers and team leads real data on the benefits, costs, and time associated with change helps remove the fear of the unknown.  It helps them identify the risk to their projects, and allows them to make decisions based on real data, not on emotion, conjecture, and fear.