Recently I became involved with a customer upgrading UrbanCode Deploy. I want to share my experiences, some lessons learned, and some IT basics.
Our customer was looking to upgrade their UrbanCode Deploy to the latest version. Doing this meant that they had to upgrade in steps, as outlined in the IBM Technote on Upgrading UrbanCode Deploy. The customer understood this, and they began to work with the IBM support team on getting their upgrade done. They had a simple plan which outlined the high level steps involved, and they had received a couple of patches to support issues specific to their environment. I became involved when they were one week away from doing the upgrade on their production systems.
As I became more involved, I became increasingly alarmed at what I was seeing. The migration plan was too simple – it had no step-by-step guidance on which values to select for configuration options, installation paths, or even server and environment names. This left fer too much room for error, and made the upgrade process prone to errors when executed in the production environment. That leads us to our first general IT lesson, which is:
General Lesson #1 – Any upgrade plan must be able to be executed by someone OTHER than the tool administrator or the author of the plan
One other factor that concerned me was that I saw no detailed “rollback plan”. Part of the upgrade plan HAS to include what staff should do if the production upgrade goes bad. It could be due to power outages, lack of resources, or some other unforeseen circumstance. You need to have detailed instructions (see General Lesson #1 above) on how to restore the existing environment if the upgrade fails. Nobody likes to do this, but if the upgrade does fail for some reason, people will be under pressure and tired. They need to have easy to understand and easy to execute instructions on how to restore the production environment. This is our addition to the first lesson,
General Lesson #1a – Any upgrade plan without a “rollback” section instructing how to restore production to it’s pre-upgrade configuration, is not complete
One of the other areas where I had concerns was due to the fact the customer was planning on moving ahead, even though they were receiving delivery of post-migration scripts the day before the planned upgrade of their production environment. They kept insisting that they had tested everything in their staging environment, but I knew that they would not be able to adequately test the post-migration scripts (more on those later) prior to upgrading production. They had tested things extensively in their staging environment, but they had only tested individual parts of the upgrade process. This leads us to our second general IT lesson, namely:
General Lesson #2 – NEVER upgrade software on production systems unless you have done a FULL upgrade scenario in your staging environment
I can hear the grumbling already. “We can’t push this out, it’s our only upgrade window for the next month”. I understand that this can be frustrating, and can seem like be over prepared at times, but this is a lesson drilled into people in our business based on traumatic experiences of the past. A decision to proceed with an upgrade of a production environment should only be done when you have prepared enough so the risk of ANYTHING going wrong is negligible. Our systems are critical to the organizations that employ us, treating them with a cavalier attitude only puts our profession on trial. The decision to upgrade, without having done a full upgrade in a staging environment, has been called “IT Malpractice” by some of my friends in the industry. It’s a great term, and one I plan to use in the future. The basic question is this: “Is the potential pain in a botched upgrade worse than delaying the upgrade?”. If you haven’t covered the lessons spelled out above, assume that your upgrade will NOT be successful.
The customer was also upgrading from version 4.x to the latest version of UrbanCode Deploy. This meant some changes to the architecture of the product which has a direct impact on the end users. The first of these is a change to the way that security is handled by UrbanCode. You really need to be aware of your security settings and the changes that may occur as part of the upgrade. If you are unfamiliar with the UrbanCode Security Model, then review it and make sure that you have a clear understanding of how roles, permissions, and teams impact the ability of end users to deploy software in your environment.
UrbanCode Lesson #1 – Understand the UrbanCode Security Model, and know how it is impacted by the upgrade
Another thing that happens when upgrading to UrbanCode Deploy 6.x is that your resources move from being in a flat list, to a tree structure. This allows you to organize your resources into common groups, and find them much more easily in the UI (ie. no 2000 item pull down menus). During the upgrade to 6.x the urbanCode Deploy resources will be reorganized into a “flat” tree structure. This has an adverse effect on performance for tool administrators, as page loads become slow if you have a large number of resources.
In order to address this, and as a way to better organize your resources, you should refactor your resources into the tree structure provided. There is a simple script that IBM can share with you that will refactor resources based on the name of the resource. You can read Boris Kuschel’s blog on how he deployed this on Bluemix. Essentially you just have a script that breaks up the resources alphabetically. You’ll probably want to alter the script to refactor your resources based on some other criteria, but the code is all there.
UrbanCode Lesson #2 – Understand the changes to the UrbanCode Resource Model, and know how it impacts your upgrade
Also keep in mind that some of this refactoring could potentially impact your procedures, depending on how you reference those resources.
Summary
Upgrading any software in your production environments is a risk. We often think of upgrades as being “simple”, but a tool upgrade is often dictated by a foundational change in a product. These foundational changes will often impact how the product is supported and how it operates, and this may have an impact on your environment. ALWAYS follow standard IT best practices and TEST upgrades in legitimate testing environments. Make sure that you have a script (especially for manual steps) for the upgrade that has been fully run through without issues in your testing environment prior to attempting to upgrade your production environments. I hate seeing my customers going through painful situations that could have been easily avoided with some risk management and planning.
Dan Toczala's Blog 