Basics for Upgrading ANY Software

Recently I became involved with a customer upgrading UrbanCode Deploy.  I want to share my experiences, some lessons learned, and some IT basics.

Our customer was looking to upgrade their UrbanCode Deploy to the latest version.  Doing this meant that they had to upgrade in steps, as outlined in the IBM Technote on Upgrading UrbanCode Deploy.  The customer understood this, and they began to work with the IBM support team on getting their upgrade done.  They had a simple plan which outlined the high level steps involved, and they had received a couple of patches to support issues specific to their environment.  I became involved when they were one week away from doing the upgrade on their production systems.

As I became more involved, I became increasingly alarmed at what I was seeing.  The migration plan was too simple – it had no step-by-step guidance on which values to select for configuration options, installation paths, or even server and environment names.  This left fer too much room for error, and made the upgrade process prone to errors when executed in the production environment.  That leads us to our first general IT lesson, which is:

General Lesson #1 – Any upgrade plan must be able to be executed by someone OTHER than the tool administrator or the author of the plan

One other factor that concerned me was that I saw no detailed “rollback plan”.  Part of the upgrade plan HAS to include what staff should do if the production upgrade goes bad.  It could be due to power outages, lack of resources, or some other unforeseen circumstance.  You need to have detailed instructions (see General Lesson #1 above) on how to restore the existing environment if the upgrade fails.  Nobody likes to do this, but if the upgrade does fail for some reason, people will be under pressure and tired.  They need to have easy to understand and easy to execute instructions on how to restore the production environment.  This is our addition to the first lesson,

General Lesson #1a – Any upgrade plan without a “rollback” section instructing how to restore production to it’s pre-upgrade configuration, is not complete

One of the other areas where I had concerns was due to the fact the customer was planning on moving ahead, even though they were receiving delivery of post-migration scripts the day before the planned upgrade of their production environment.  They kept insisting that they had tested everything in their staging environment, but I knew that they would not be able to adequately test the post-migration scripts (more on those later) prior to upgrading production.  They had tested things extensively in their staging environment, but they had only tested individual parts of the upgrade process.  This leads us to our second general IT lesson, namely:

General Lesson #2 – NEVER upgrade software on production systems unless you have done a FULL upgrade scenario in your staging environment

I can hear the grumbling already.  “We can’t push this out, it’s our only upgrade window for the next month”.  I understand that this can be frustrating, and can seem like be over prepared at times, but this is a lesson drilled into people in our business based on traumatic experiences of the past.  A decision to proceed with an upgrade of a production environment should only be done when you have prepared enough so the risk of ANYTHING going wrong is negligible.  Our systems are critical to the organizations that employ us, treating them with a cavalier attitude only puts our profession on trial.  The decision to upgrade, without having done a full upgrade in a staging environment, has been called “IT Malpractice” by some of my friends in the industry.  It’s a great term, and one I plan to use in the future.  The basic question is this: “Is the potential pain in a botched upgrade worse than delaying the upgrade?”.  If you haven’t covered the lessons spelled out above, assume that your upgrade will NOT be successful.

The customer was also upgrading from version 4.x to the latest version of UrbanCode Deploy.  This meant some changes to the architecture of the product which has a direct impact on the end users.  The first of these is a change to the way that security is handled by UrbanCode.  You really need to be aware of your security settings and the changes that may occur as part of the upgrade.  If you are unfamiliar with the UrbanCode Security Model, then review it and make sure that you have a clear understanding of how roles, permissions, and teams impact the ability of end users to deploy software in your environment.

UrbanCode Lesson #1 – Understand the UrbanCode Security Model, and know how it is impacted by the upgrade

Another thing that happens when upgrading to UrbanCode Deploy 6.x is that your resources move from being in a flat list, to a tree structure.  This allows you to organize your resources into common groups, and find them much more easily in the UI (ie. no 2000 item pull down menus).  During the upgrade to 6.x the urbanCode Deploy resources will be reorganized into a “flat” tree structure.  This has an adverse effect on performance for tool administrators, as page loads become slow if you have a large number of resources.

In order to address this, and as a way to better organize your resources, you should refactor your resources into the tree structure provided.  There is a simple script that IBM can share with you that will refactor resources based on the name of the resource.  You can read Boris Kuschel’s blog on how he deployed this on Bluemix.  Essentially you just have a script that breaks up the resources alphabetically.  You’ll probably want to alter the script to refactor your resources based on some other criteria, but the code is all there.

UrbanCode Lesson #2 – Understand the changes to the UrbanCode Resource Model, and know how it impacts your upgrade

Also keep in mind that some of this refactoring could potentially impact your procedures, depending on how you reference those resources.

Summary

Upgrading any software in your production environments is a risk.  We often think of upgrades as being “simple”, but a tool upgrade is often dictated by a foundational change in a product.  These foundational changes will often impact how the product is supported and how it operates, and this may have an impact on your environment.  ALWAYS follow standard IT best practices and TEST upgrades in legitimate testing environments.  Make sure that you have a script (especially for manual steps) for the upgrade that has been fully run through without issues in your testing environment prior to attempting to upgrade your production environments.  I hate seeing my customers going through painful situations that could have been easily avoided with some risk management and planning.

Advertisements

An Easier Cloud Calendar

Timing is……………… everything.  About 4 hours after I did my last post on How About a Generic Calendar in the Cloud?, I saw a post from one of my team members.  It was a post from Sean Wilbur called, Streamlining Your Bluemix Project for One Button Sharing.  It was a great post, and once I followed the directions that Sean outlined, I was able to add a simple little “Deploy to Bluemix” button on my project.

So now if you would like to get a copy of my Generic Calendar project to play with for yourself, it is really easy.  Just make sure that you have a Bluemix account, and that you have a linked DevOps Services account.  Then just navigate to my project in DevOps Services (it’s the dtoczala|ULLCloudCalendar project).  Once there, you can look at the README.md file displayed there, and look for the “Deploy to Bluemix” button.  It looks like this:

deploy-to-bluemix
The Deploy to Bluemix button

Just press that button and you will get a project created in the DevOps services that is a fork of my original project.  The automatic creation of the project will throw an error during the deployment, but you will be able to easily fix this.  The error is due to a problem in the manifest.yml file, we are currently unable to create and bind services for our application through the manifest (see Sean’s question on this in the forum).  You can easily fix this by doing three things:

  1. In your DevOps services console, configure your Deploy stage – In your newly created project, press the build and deploy button, and then configure the deploy stage.  You will add a job to the deploy configuration, a deploy job, that will do a “cf push” of your application.  Then try executing it.  It will still fail (because our MongoDB service is not present), but it will create a new Bluemix application for you.  This is your version of the ULL Cloud Calendar app.
  2. In your Bluemix console, add and bind the MongoDB service – This is straightforward.  Just add the MongoDB service and make sure to bind it to your new application.  When you add the service, Bluemix will ask if you would like to restage your application.  Answer yes, and wait for the application to be deployed again.
  3. In your Bluemix console, click on the link for your new app – just click on the link for the route to your new application. 

Now once that little issue with the manifest.yml is cleared up, you will be able to share your Bluemix applications with the press of a button.  Bringing up applications and capabilities in the cloud is getting easier and easier!

How about a Generic Calendar in the Cloud?

Working with my team is often fun and rewarding.  I learn a lot from the people I work with, and I get the chance to try and learn new technologies all of the time, in an effort to solve real business problems.  One of our most recent challenges is having the ability to have a “team calendar” that we can all update.  We wanted to have some light weight way to coordinate our activities, and to keep on top of vacations and travel plans.

We wanted something that would work within the External Content widget of RTC, because we wanted to expose this calendar on our RTC dashboard.  The dashboard is where we track our work, watch our progress, and document our progress on our key measures, so it seemed to be a logical place for the calendar to live.  RTC doesn’t have any kind of native calendar ability, and it is something we miss.  I considered just using a Google Calendar, but we dismissed that because often our calendar entries will contain sensitive information.  So we wanted something that could be done within the IBM firewall.  That led me down the path of creating a simple calendar application using Bluemix.  IBM has a small internal implementation of Bluemix behind our firewall, so our simple privacy and security needs could be met by this.

I decided on a simple implementation using Node,js (which recently announced it’s own Node.js foundation).  I thought about using Cloudant for the underlying datastore, but in the end I decided on using Mongo DB, because I didn’t want this to be an “IBM solution”.

Keep in mind that this is a lightweight solution, it uses the Sandbox plan for the MongoDB service, and the code is not expected to be robust enough for 50 or 100 people to use.  It’s meant as a nice sample project, and one that could be useful to a small team.  It’s not going to replace your enterprise calendar solution.  It also uses the dhtmlxScheduler component, which has it’s own licensing concerns.  dhtmlxScheduler Standard Edition is available under GNU GPLv2, so be aware of the implications of this.

An example of the calendar application
An example of the calendar application

Would you like to see how to deploy it for yourself?  Then read on……

Deploying the Generic Calendar on Bluemix

You’ll need a Bluemix account with all of the usual capabilities.  You’ll first want to grab the code for this from my Github project called ULLCloudCalendar, and save it on your laptop/workstation somewhere.  Just hit the button to download the zip of the contents of the project.  I developed this on a Linux box, so hopefully the character sets and encoding don’t screw you up too much.  Once you have a copy of the code, you’ll want to login to Bluemix.  Once there, you will create a new application using the SDK for Node.js runtime.  Give the app a good name (like “AcmeCalendar“), and wait for Bluemix to create your skeleton app for you.

Once Bluemix is done, you should see you new application on the Bluemix console.  Now you will want to go and click on the box to “Add a Service or API”.  Scroll through the list of services until you come to the “MongoLab” service.  Click on the icon for this service, and on the next screen, create a new instance of the MongoLab service (which is a cloud hosted Mongo DB).  Make sure to give it a name that corresponds to the name of your application (like “MongoLab-AcmeCalendar“).  Also make sure that it is being set up in the correct space, and that it will be bound to the correct application (in my case, the “AcmeCalendar” application).  When you have checked everything, press the “Create” button to create your service.

At this point, you will be ready for that code that you copied earlier.  Have the code in a directory by itself.  Make sure that you have downloaded the Cloud Foundry CF Command Line interface, and have installed it on your computer.  Open up a command line interface, and navigate to the new directory where your code lives, at the top level directory.  This directory contains the manifest.yml, app.js, and package.json files.  Now we’ll log into our Cloud Foundry instance, set into the right environment, and push all of this code up to the cloud.

  • Login to the Cloud Foundry instance.
cf api https://api.ng.bluemix.net
  • Login to your account space on the Cloud Foundry/Bluemix instance.  Use your Bluemix ID for the owner (-o) and user (-u) parameters, and the space name of your space on Bluemix for the space (-s) parameter.  After you do this, you will be prompted for your password.
cf login -u dtoczala@us.ibm.com -o dtoczala@us.ibm.com -s 'dev'
  • Now you will want to modify the manifest.yml file to make sure that you can find your new project.  Edit the manifest.yml file and change all of the “ULLCloudCalendar” entries to the name of your application (Acme Calendar in this example).
host: AcmeCalendar
name: AcmeCalendar
  • Now modify the package.json file to reflect the new name of your application as well.  Edit this file and change the line with the name to your new application name.
"name": "AcmeCalendar",
  • Finally, you can now push all of your code up to the Bluemix infrastructure.  Use the name of your application (which is Acme Calendar in my example)
cf push AcmeCalendar

Keep in mind that there are other ways to do this (using an Eclipse plugin is one of them), so do a little research and find out the method that works best for you.  Once you do the cf push of your code, you will see Bluemix/Cloud Foundry do it’s work.  At some point it will tell you that your application has been deployed in the cloud, and that it is running.

Accessing Your Calendar

On the Bluemix console for your application, you will see a link to your new calendar application.  It will be something like https://AcmeCalendar.mybluemix.net (depending on your application name and the route that you have chosen).  You can change the route, but that is a technique for a future blog post (it’s not that hard, I just don’t want to get into it here).  Clicking on that link will launch you to the website where you can access your new calendar.  Play around with it.  Double clicking on a day will open a dialog box for adding a new event.  Double clicking on an event will allow you to change or delete the event.  It’s pretty simple.

Adding this to your RTC dashboard is pretty simple too.  Just create a new tab on your RTC dashboard.  Click on the caret next to the tab name and be sure to set the tab up to display widgets in a single column (otherwise the calendar becomes too small to be useful).  Now click on the “Add Widget” button on the upper right of the dashboard.  Select the “External Content” widget.  Once the widget is displayed on the dashboard, click on the small pencil in the widget menu bar.  You will now enter in

  • The External URL (which is the web address of your new app, maybe something like “https://AcmeCalendar.mybluemix.net“)
  • The height of the widget (try 550 pixels for starters, adjust it as you need to)
  • The refresh rate (go with a simple 5 minutes, or 300 seconds)

Once you do this, you should now see you calendar application right on your RTC dashboard.  You can even navigate through the calendar and add/change/delete events.

How Does it Work?

If you’ve read this far, you have enough knowledge to be able to deploy a simple cloud based calendar for your team.  If you want to get into the code, and possibly change and enhance this calendar app, then read this section.

The calendar has three big pieces that control everything.  The first piece is the dhtmlxscheduler piece.  The code for this component (which controls the calendar look and feel, and drives functionality) is in the public folder.  I didn’t touch this stuff, but if you want to try messing around with the CSS files to change the look and feel of things, be my guest.

The next big piece is the code that controls the rendering of the HTML page.  This is in the index.html file in the public folder.  There are two important pieces of code in this file.  There is the script.  The script first will go and make some configuration settings to the calendar, it then sets up the basic colors used, and sets up the look of the dialog box to add/modify/delete events.  The script then initializes the calendar component with these settings.  At the end of the script, the default date/time format is specified, your existing data is loaded, and a data processor is set up to handle the interactive user requests.  Then there is the body of the HTML page, which sets up the display of the calendar itself, and initializes things.  I didn’t fool around with this section.

The final piece is the app itself, in the top level directory in the file app.js.  This file handles the storage and retrieval of data from the MongoDB, and does some data checking and data manipulation to help format things appropriately.

The script starts out by setting a bunch of global variables and reading the various VCAP settings provided by the Bluemix environment.  This allows the application to connect to the MongoDB that is bound to this application with the correct credentials, and it also provides some other important runtime information.  You will notice that there are references to SSO and certs that are in the code, but have not been tested.

Once this initial code is complete, you can see where we connect to the MongoDB.  Following this is a section of code that is NOT tested (and not used) that deals with the SSO and passport functionality.  This all ends up with the section on customer authentication middleware.

Finally we get to the Application routes.

  • The code for the /init route is simple, it just adds a single event on New Year’s Day to get you started.
  • The get code for the /data route supplies the calendar object with all of it’s events from the MongoDB.  It retrieves ALL of the events in the datastore, builds a JSON object with these events, and provides them as a stream of JSON data to the calendar object in a response.  Be careful with the formatting of your dates in the JSON response, an invalid date can cause problems.
  • The post code for the /data route processes the creation/modification/deletion of events by the calendar object.  A user who changes something in the calendar will post the change to the /data route.  This section handles the request, and processes it accordingly.

Finally at the end of the app.js file, we start the app.

What’s Next?

There are things you can do to change how this calendar works, and expand or change it’s functionality.  I’ll cover a couple of the simple things here.

I Hate The Colors

You hate the colors that I picked?  So change them yourself.  There are two sections that deal with the colors in the calendar interface.  In the index.html file, there is a variable called colorpicker.  You can change the names of the colors by changing the label property of the array entries, or the color itself by changing the key property of the array entries.  This key property defines the RGB mix of colors.  I used the HTML color picker to get these values.  You can even add even more colors by adding more entries to the array.

These key values (the “rgb(x,y,z)” entries) are stored with the events in the database.  If you look at the code in the app.js file, look at the get /data section of code.   In here you see a section of code where we check the color property of the returned event data.  This represents the color of the event box.  Based on this, the if statement will either assign a grey box and black text (if no color information is provided), or the proper color and black text, unless the box is indigo in color, in which case white text is selected because it shows up better.  Kind of hard to explain – easier to see in the code.

            if (!data[i].color)
                {
                color = 'rgb(204,204,204)';  // grey block
                textcolor = 'rgb(0,0,0)';    // black text
                }
            else
                {
                color = data[i].color;
                textcolor = 'rgb(0,0,0)';    // black text
                if (color == 'rgb(77,77,184)')  // if block is indigo
                    textcolor = 'rgb(255,255,255)';    // go with white text
                }

What About Repeating Calendar Entries?

The dhtmlxscheduler will support repeating entries.  To implement this, check out this entry in their documentation on implementing repeating entries.  In fact, take a look at their documentation overall.  I found the sections on custom event colors, Lightbox manipulations (Lightbox is the user entry popup), and the code associated with the coloring events example to be quite helpful.

Weeks start on Mondays, not Sundays

This one is an easy change in the index.html file.  Just find this line of code:

scheduler.config.start_on_monday = false;

and change it to:

scheduler.config.start_on_monday = true;

Summary

I wanted a calendar that was stand-alone, that could be displayed in an RTC widget, and that I could easily deploy in a Bluemix environment.  I hope this guide has shown you how easy this is to do, and allows you to add this calendar ability to your RTC environment.  If you have comments or issues, please comment and I will do my best to answer your questions.b

Bluemix gets your deployments rolling, rolling, rolling…..

I recently spent some time with some of our most knowledgeable Bluemix people.  We were helping one of our customers explore the possibilities of Bluemix for their organization.  Dan Berg and Scott Rich are two of the sharper guys I know, and they were helping us.  Dan Berg came up with a great technique that enables “Blue/Green” (or rolling) deployments on Bluemix.

You can read his developerWorks article, Automate Rolling Deployments with IDS for Bluemix , and learn how to do this.  The code samples and public Bluemix project that he provides allow anyone to be able to expose this kind of capability for their Bluemix users.  I was going to post my version of the directions for doing rolling deployments in this blog article, but Dan’s is very clear and easy to follow.  He even has links to the Bluemix project with sample code, and links to a post on Blue/Green deployment.

The “meat” of this is the deploy.sh script, but there are changes to the build script (in Ant) as well.  The build script changes that he discusses will copy your updated deploy.sh script to the archiving directory.  This is important to remember (see below).  Dan discusses the deploy shell script, and you should pay particular attention to where you would add the binding of services to your new application.  The nice thing is that you can test this out in a QA pipeline deploy, before you even attempt to do something like this for real.

Just some words of caution as you go through this.  The first is that you need to remember that your build and deployment scripts are under Git control.  This is great, because it allows you to see changes in your scripts, and gives you the ability to rollback to a previous version if necessary.  It also means that as you change your scripts, you will need to COMMIT and PUSH your script changes.  I have made the mistake of editing my scripts in the web browser editor, and then wondering why the changes I make don’t seem to do anything.  It’s because my script changes have not been pushed into the cloud yet.  Another issue is that even after you have pushed your changes into the Git repository, you still need to build the application to get them to execute.  Why?  Because the copy of the deploy.sh script to the archive directory occurs during the build, and the auto-deploy runs from the archive directory.  This isn’t an issue if you have auto-deploy turned on, since the Git commit will automatically kick off a build, which then kicks off an automatic deployment.

The scripts work great.  Ina session with one of our customers last week, the students set up a test that exercised the application being deployed.  It was a simple application, and we just changed some of the titles on a web page.  We then implemented a rolling deployment, and did a push of our changes in Bluemix.  We could then watch as the changes were built, and then auto-deployed in Bluemix.  Their tests noted no interruption of service when we deployed our changes.  It was pretty cool.