Basics for Upgrading ANY Software

Recently I became involved with a customer upgrading UrbanCode Deploy.  I want to share my experiences, some lessons learned, and some IT basics.

Our customer was looking to upgrade their UrbanCode Deploy to the latest version.  Doing this meant that they had to upgrade in steps, as outlined in the IBM Technote on Upgrading UrbanCode Deploy.  The customer understood this, and they began to work with the IBM support team on getting their upgrade done.  They had a simple plan which outlined the high level steps involved, and they had received a couple of patches to support issues specific to their environment.  I became involved when they were one week away from doing the upgrade on their production systems.

As I became more involved, I became increasingly alarmed at what I was seeing.  The migration plan was too simple – it had no step-by-step guidance on which values to select for configuration options, installation paths, or even server and environment names.  This left fer too much room for error, and made the upgrade process prone to errors when executed in the production environment.  That leads us to our first general IT lesson, which is:

General Lesson #1 – Any upgrade plan must be able to be executed by someone OTHER than the tool administrator or the author of the plan

One other factor that concerned me was that I saw no detailed “rollback plan”.  Part of the upgrade plan HAS to include what staff should do if the production upgrade goes bad.  It could be due to power outages, lack of resources, or some other unforeseen circumstance.  You need to have detailed instructions (see General Lesson #1 above) on how to restore the existing environment if the upgrade fails.  Nobody likes to do this, but if the upgrade does fail for some reason, people will be under pressure and tired.  They need to have easy to understand and easy to execute instructions on how to restore the production environment.  This is our addition to the first lesson,

General Lesson #1a – Any upgrade plan without a “rollback” section instructing how to restore production to it’s pre-upgrade configuration, is not complete

One of the other areas where I had concerns was due to the fact the customer was planning on moving ahead, even though they were receiving delivery of post-migration scripts the day before the planned upgrade of their production environment.  They kept insisting that they had tested everything in their staging environment, but I knew that they would not be able to adequately test the post-migration scripts (more on those later) prior to upgrading production.  They had tested things extensively in their staging environment, but they had only tested individual parts of the upgrade process.  This leads us to our second general IT lesson, namely:

General Lesson #2 – NEVER upgrade software on production systems unless you have done a FULL upgrade scenario in your staging environment

I can hear the grumbling already.  “We can’t push this out, it’s our only upgrade window for the next month”.  I understand that this can be frustrating, and can seem like be over prepared at times, but this is a lesson drilled into people in our business based on traumatic experiences of the past.  A decision to proceed with an upgrade of a production environment should only be done when you have prepared enough so the risk of ANYTHING going wrong is negligible.  Our systems are critical to the organizations that employ us, treating them with a cavalier attitude only puts our profession on trial.  The decision to upgrade, without having done a full upgrade in a staging environment, has been called “IT Malpractice” by some of my friends in the industry.  It’s a great term, and one I plan to use in the future.  The basic question is this: “Is the potential pain in a botched upgrade worse than delaying the upgrade?”.  If you haven’t covered the lessons spelled out above, assume that your upgrade will NOT be successful.

The customer was also upgrading from version 4.x to the latest version of UrbanCode Deploy.  This meant some changes to the architecture of the product which has a direct impact on the end users.  The first of these is a change to the way that security is handled by UrbanCode.  You really need to be aware of your security settings and the changes that may occur as part of the upgrade.  If you are unfamiliar with the UrbanCode Security Model, then review it and make sure that you have a clear understanding of how roles, permissions, and teams impact the ability of end users to deploy software in your environment.

UrbanCode Lesson #1 – Understand the UrbanCode Security Model, and know how it is impacted by the upgrade

Another thing that happens when upgrading to UrbanCode Deploy 6.x is that your resources move from being in a flat list, to a tree structure.  This allows you to organize your resources into common groups, and find them much more easily in the UI (ie. no 2000 item pull down menus).  During the upgrade to 6.x the urbanCode Deploy resources will be reorganized into a “flat” tree structure.  This has an adverse effect on performance for tool administrators, as page loads become slow if you have a large number of resources.

In order to address this, and as a way to better organize your resources, you should refactor your resources into the tree structure provided.  There is a simple script that IBM can share with you that will refactor resources based on the name of the resource.  You can read Boris Kuschel’s blog on how he deployed this on Bluemix.  Essentially you just have a script that breaks up the resources alphabetically.  You’ll probably want to alter the script to refactor your resources based on some other criteria, but the code is all there.

UrbanCode Lesson #2 – Understand the changes to the UrbanCode Resource Model, and know how it impacts your upgrade

Also keep in mind that some of this refactoring could potentially impact your procedures, depending on how you reference those resources.

Summary

Upgrading any software in your production environments is a risk.  We often think of upgrades as being “simple”, but a tool upgrade is often dictated by a foundational change in a product.  These foundational changes will often impact how the product is supported and how it operates, and this may have an impact on your environment.  ALWAYS follow standard IT best practices and TEST upgrades in legitimate testing environments.  Make sure that you have a script (especially for manual steps) for the upgrade that has been fully run through without issues in your testing environment prior to attempting to upgrade your production environments.  I hate seeing my customers going through painful situations that could have been easily avoided with some risk management and planning.

An Easier Cloud Calendar

Timing is……………… everything.  About 4 hours after I did my last post on How About a Generic Calendar in the Cloud?, I saw a post from one of my team members.  It was a post from Sean Wilbur called, Streamlining Your Bluemix Project for One Button Sharing.  It was a great post, and once I followed the directions that Sean outlined, I was able to add a simple little “Deploy to Bluemix” button on my project.

So now if you would like to get a copy of my Generic Calendar project to play with for yourself, it is really easy.  Just make sure that you have a Bluemix account, and that you have a linked DevOps Services account.  Then just navigate to my project in DevOps Services (it’s the dtoczala|ULLCloudCalendar project).  Once there, you can look at the README.md file displayed there, and look for the “Deploy to Bluemix” button.  It looks like this:

deploy-to-bluemix

The Deploy to Bluemix button

Just press that button and you will get a project created in the DevOps services that is a fork of my original project.  The automatic creation of the project will throw an error during the deployment, but you will be able to easily fix this.  The error is due to a problem in the manifest.yml file, we are currently unable to create and bind services for our application through the manifest (see Sean’s question on this in the forum).  You can easily fix this by doing three things:

  1. In your DevOps services console, configure your Deploy stage – In your newly created project, press the build and deploy button, and then configure the deploy stage.  You will add a job to the deploy configuration, a deploy job, that will do a “cf push” of your application.  Then try executing it.  It will still fail (because our MongoDB service is not present), but it will create a new Bluemix application for you.  This is your version of the ULL Cloud Calendar app.
  2. In your Bluemix console, add and bind the MongoDB service – This is straightforward.  Just add the MongoDB service and make sure to bind it to your new application.  When you add the service, Bluemix will ask if you would like to restage your application.  Answer yes, and wait for the application to be deployed again.
  3. In your Bluemix console, click on the link for your new app – just click on the link for the route to your new application. 

Now once that little issue with the manifest.yml is cleared up, you will be able to share your Bluemix applications with the press of a button.  Bringing up applications and capabilities in the cloud is getting easier and easier!

How about a Generic Calendar in the Cloud?

Working with my team is often fun and rewarding.  I learn a lot from the people I work with, and I get the chance to try and learn new technologies all of the time, in an effort to solve real business problems.  One of our most recent challenges is having the ability to have a “team calendar” that we can all update.  We wanted to have some light weight way to coordinate our activities, and to keep on top of vacations and travel plans.

We wanted something that would work within the External Content widget of RTC, because we wanted to expose this calendar on our RTC dashboard.  The dashboard is where we track our work, watch our progress, and document our progress on our key measures, so it seemed to be a logical place for the calendar to live.  RTC doesn’t have any kind of native calendar ability, and it is something we miss.  I considered just using a Google Calendar, but we dismissed that because often our calendar entries will contain sensitive information.  So we wanted something that could be done within the IBM firewall.  That led me down the path of creating a simple calendar application using Bluemix.  IBM has a small internal implementation of Bluemix behind our firewall, so our simple privacy and security needs could be met by this.

I decided on a simple implementation using Node,js (which recently announced it’s own Node.js foundation).  I thought about using Cloudant for the underlying datastore, but in the end I decided on using Mongo DB, because I didn’t want this to be an “IBM solution”.

Keep in mind that this is a lightweight solution, it uses the Sandbox plan for the MongoDB service, and the code is not expected to be robust enough for 50 or 100 people to use.  It’s meant as a nice sample project, and one that could be useful to a small team.  It’s not going to replace your enterprise calendar solution.  It also uses the dhtmlxScheduler component, which has it’s own licensing concerns.  dhtmlxScheduler Standard Edition is available under GNU GPLv2, so be aware of the implications of this.

An example of the calendar application

An example of the calendar application

Would you like to see how to deploy it for yourself?  Then read on……

Deploying the Generic Calendar on Bluemix

You’ll need a Bluemix account with all of the usual capabilities.  You’ll first want to grab the code for this from my Github project called ULLCloudCalendar, and save it on your laptop/workstation somewhere.  Just hit the button to download the zip of the contents of the project.  I developed this on a Linux box, so hopefully the character sets and encoding don’t screw you up too much.  Once you have a copy of the code, you’ll want to login to Bluemix.  Once there, you will create a new application using the SDK for Node.js runtime.  Give the app a good name (like “AcmeCalendar“), and wait for Bluemix to create your skeleton app for you.

Once Bluemix is done, you should see you new application on the Bluemix console.  Now you will want to go and click on the box to “Add a Service or API”.  Scroll through the list of services until you come to the “MongoLab” service.  Click on the icon for this service, and on the next screen, create a new instance of the MongoLab service (which is a cloud hosted Mongo DB).  Make sure to give it a name that corresponds to the name of your application (like “MongoLab-AcmeCalendar“).  Also make sure that it is being set up in the correct space, and that it will be bound to the correct application (in my case, the “AcmeCalendar” application).  When you have checked everything, press the “Create” button to create your service.

At this point, you will be ready for that code that you copied earlier.  Have the code in a directory by itself.  Make sure that you have downloaded the Cloud Foundry CF Command Line interface, and have installed it on your computer.  Open up a command line interface, and navigate to the new directory where your code lives, at the top level directory.  This directory contains the manifest.yml, app.js, and package.json files.  Now we’ll log into our Cloud Foundry instance, set into the right environment, and push all of this code up to the cloud.

  • Login to the Cloud Foundry instance.
cf api https://api.ng.bluemix.net
  • Login to your account space on the Cloud Foundry/Bluemix instance.  Use your Bluemix ID for the owner (-o) and user (-u) parameters, and the space name of your space on Bluemix for the space (-s) parameter.  After you do this, you will be prompted for your password.
cf login -u dtoczala@us.ibm.com -o dtoczala@us.ibm.com -s 'dev'
  • Now you will want to modify the manifest.yml file to make sure that you can find your new project.  Edit the manifest.yml file and change all of the “ULLCloudCalendar” entries to the name of your application (Acme Calendar in this example).
host: AcmeCalendar
name: AcmeCalendar
  • Now modify the package.json file to reflect the new name of your application as well.  Edit this file and change the line with the name to your new application name.
"name": "AcmeCalendar",
  • Finally, you can now push all of your code up to the Bluemix infrastructure.  Use the name of your application (which is Acme Calendar in my example)
cf push AcmeCalendar

Keep in mind that there are other ways to do this (using an Eclipse plugin is one of them), so do a little research and find out the method that works best for you.  Once you do the cf push of your code, you will see Bluemix/Cloud Foundry do it’s work.  At some point it will tell you that your application has been deployed in the cloud, and that it is running.

Accessing Your Calendar

On the Bluemix console for your application, you will see a link to your new calendar application.  It will be something like https://AcmeCalendar.mybluemix.net (depending on your application name and the route that you have chosen).  You can change the route, but that is a technique for a future blog post (it’s not that hard, I just don’t want to get into it here).  Clicking on that link will launch you to the website where you can access your new calendar.  Play around with it.  Double clicking on a day will open a dialog box for adding a new event.  Double clicking on an event will allow you to change or delete the event.  It’s pretty simple.

Adding this to your RTC dashboard is pretty simple too.  Just create a new tab on your RTC dashboard.  Click on the caret next to the tab name and be sure to set the tab up to display widgets in a single column (otherwise the calendar becomes too small to be useful).  Now click on the “Add Widget” button on the upper right of the dashboard.  Select the “External Content” widget.  Once the widget is displayed on the dashboard, click on the small pencil in the widget menu bar.  You will now enter in

  • The External URL (which is the web address of your new app, maybe something like “https://AcmeCalendar.mybluemix.net“)
  • The height of the widget (try 550 pixels for starters, adjust it as you need to)
  • The refresh rate (go with a simple 5 minutes, or 300 seconds)

Once you do this, you should now see you calendar application right on your RTC dashboard.  You can even navigate through the calendar and add/change/delete events.

How Does it Work?

If you’ve read this far, you have enough knowledge to be able to deploy a simple cloud based calendar for your team.  If you want to get into the code, and possibly change and enhance this calendar app, then read this section.

The calendar has three big pieces that control everything.  The first piece is the dhtmlxscheduler piece.  The code for this component (which controls the calendar look and feel, and drives functionality) is in the public folder.  I didn’t touch this stuff, but if you want to try messing around with the CSS files to change the look and feel of things, be my guest.

The next big piece is the code that controls the rendering of the HTML page.  This is in the index.html file in the public folder.  There are two important pieces of code in this file.  There is the script.  The script first will go and make some configuration settings to the calendar, it then sets up the basic colors used, and sets up the look of the dialog box to add/modify/delete events.  The script then initializes the calendar component with these settings.  At the end of the script, the default date/time format is specified, your existing data is loaded, and a data processor is set up to handle the interactive user requests.  Then there is the body of the HTML page, which sets up the display of the calendar itself, and initializes things.  I didn’t fool around with this section.

The final piece is the app itself, in the top level directory in the file app.js.  This file handles the storage and retrieval of data from the MongoDB, and does some data checking and data manipulation to help format things appropriately.

The script starts out by setting a bunch of global variables and reading the various VCAP settings provided by the Bluemix environment.  This allows the application to connect to the MongoDB that is bound to this application with the correct credentials, and it also provides some other important runtime information.  You will notice that there are references to SSO and certs that are in the code, but have not been tested.

Once this initial code is complete, you can see where we connect to the MongoDB.  Following this is a section of code that is NOT tested (and not used) that deals with the SSO and passport functionality.  This all ends up with the section on customer authentication middleware.

Finally we get to the Application routes.

  • The code for the /init route is simple, it just adds a single event on New Year’s Day to get you started.
  • The get code for the /data route supplies the calendar object with all of it’s events from the MongoDB.  It retrieves ALL of the events in the datastore, builds a JSON object with these events, and provides them as a stream of JSON data to the calendar object in a response.  Be careful with the formatting of your dates in the JSON response, an invalid date can cause problems.
  • The post code for the /data route processes the creation/modification/deletion of events by the calendar object.  A user who changes something in the calendar will post the change to the /data route.  This section handles the request, and processes it accordingly.

Finally at the end of the app.js file, we start the app.

What’s Next?

There are things you can do to change how this calendar works, and expand or change it’s functionality.  I’ll cover a couple of the simple things here.

I Hate The Colors

You hate the colors that I picked?  So change them yourself.  There are two sections that deal with the colors in the calendar interface.  In the index.html file, there is a variable called colorpicker.  You can change the names of the colors by changing the label property of the array entries, or the color itself by changing the key property of the array entries.  This key property defines the RGB mix of colors.  I used the HTML color picker to get these values.  You can even add even more colors by adding more entries to the array.

These key values (the “rgb(x,y,z)” entries) are stored with the events in the database.  If you look at the code in the app.js file, look at the get /data section of code.   In here you see a section of code where we check the color property of the returned event data.  This represents the color of the event box.  Based on this, the if statement will either assign a grey box and black text (if no color information is provided), or the proper color and black text, unless the box is indigo in color, in which case white text is selected because it shows up better.  Kind of hard to explain – easier to see in the code.

            if (!data[i].color)
                {
                color = 'rgb(204,204,204)';  // grey block
                textcolor = 'rgb(0,0,0)';    // black text
                }
            else
                {
                color = data[i].color;
                textcolor = 'rgb(0,0,0)';    // black text
                if (color == 'rgb(77,77,184)')  // if block is indigo
                    textcolor = 'rgb(255,255,255)';    // go with white text
                }

What About Repeating Calendar Entries?

The dhtmlxscheduler will support repeating entries.  To implement this, check out this entry in their documentation on implementing repeating entries.  In fact, take a look at their documentation overall.  I found the sections on custom event colors, Lightbox manipulations (Lightbox is the user entry popup), and the code associated with the coloring events example to be quite helpful.

Weeks start on Mondays, not Sundays

This one is an easy change in the index.html file.  Just find this line of code:

scheduler.config.start_on_monday = false;

and change it to:

scheduler.config.start_on_monday = true;

Summary

I wanted a calendar that was stand-alone, that could be displayed in an RTC widget, and that I could easily deploy in a Bluemix environment.  I hope this guide has shown you how easy this is to do, and allows you to add this calendar ability to your RTC environment.  If you have comments or issues, please comment and I will do my best to answer your questions.b

Dashboards, Transparency and Strategy

I recently started writing an article on Using Scrum Methods with IBM DevOps Services (IDS), I mentioned that I would follow up with some other topics of interest to users of IDS, like dashboards.  As I began to write that article, I could feel a series of more general observations coming out.  There are some things that I have noticed about the general use of dashboards that I have seen, and how this fits with the strategies and transparency of the organizations that I work with.  This article will talk a little bit about the observations that I have made, and some of the important general concepts to keep in mind.

Now I have seen the things that I mention here done with Jazz dashboards, or dashboards in IDS.  I have also seen it with similar dashboarding capabilities in other products.  I have even seen wiki’s used with some amount of success.  The techniques and observations that I mention here work regardless of technology – so focus on the concepts.

Why Does Everyone Want a Dashboard?

Dashboards are a nice way to see how things are going for your team/project/organization at a glance.  Executives and leaders love them, even when they don’t fully understand them.  I find that dashboards can have a profound impact on teams and organizations when they are used correctly, and can have some negative effects when not used wisely.  Here are some basic and important things that you should remember when you use dashboards:

  • Dashboards should be used for transparency – everyone in your organization should have a common view of how things are going, based on common data.  Individual teams may focus on things within their scope, but they should be able to put their mission and their success (or struggles) into the broader context of the entire organization.  Software people can get blinders on, it’s important to remind them that there are higher business goals that they are contributing to.
  • Transparency is good – Transparency means sharing information, both good and bad.  This helps teams see themselves in a more realistic light, and it pushes teams to perform more efficiently and with higher quality.  Transparency will also help reduce miscommunication and wasted effort within your organization.
  • Strategy is key – Transparency for the sake of transparency is just a lot of noise.  “I had a burger for lunch”, “I’m wearing a green shirt today”, and “I just finished testing some code”, are all logically equivalent statements, when not tied to a strategy.  All of this transparency needs to be focused and tied to a solid strategy.

How Do I Make Transparency Effective?

In order to make transparency effective, you need to consider a few things.  The most important is strategy.  Your transparent communications should be aligned with the strategy of your organization.  In my experience, it is best to have a small list of key organizational objectives.  These can be things like, “Capture 10% of additional market share”, “Increase Revenues by 7%”, or “Improve quality by 10%”.  They key here is that they need to be measurable.  Some people like the so called “Smart Metrics“, which are Specific, Measurable, Attainable, Realistic and Timely (although there are other slightly different Smart Metric definitions).  They are usually a good benchmark for what you want for goals.

This is where some discipline comes in handy.  Defining these types of goals can be difficult, so don’t just spend 5 minutes whipping up some strategic goals that don’t really say anything.  Spend some time, think about it, and come up with some specific goals for your teams.  Your strategy is going to be implemented by these goals, so make sure that they clearly articulate your business strategy.

Once you have these high level goals, you can begin to think about how to reflect them on a dashboard.  We often take 1 or 2 team level metrics for each strategic goal, and measure them over time.  I would say that you should not have more than 3 metrics for each goal reflected in a dashboard, otherwise you tend to dilute the focus of your team.  Do this for each of your strategic goals, and you are all set.

Now let your team know about the dashboard, so they can work to make themselves look good.  Also let the rest of the organization see your dashboard, so they can see how well (or poorly) your team is doing on meeting their goals.  Don’t be afraid of “bad news”, or measures that come up short.  It just highlights areas where you team needs to focus more.  In order to have a high performing team, you need to build a sense of pride and accomplishment, and have meaningful goals – putting team results out for the rest of the organization to see is one way of doing that.  (Note: Read The Wisdom of Teams by Jon R. Katzenbach & Douglas K. Smith – or check this Wisdom of Teams cheat sheet.  This is a must read for anyone leading Agile teams.)

The key here is to focus on the team – and not on individuals within the team.  Your team succeeds or fails as a team, make sure that they all understand this.  Once this attitude is part of your team culture, collaboration within the team will increase, and this will lead to better collaboration with other teams.  A dashboard focused on team metrics and accomplishments that drive towards meaningful goals can help drive that culture.

I Can’t Possibly Manage My Team This Way

At this point, I can hear a lot of people saying, “My team does complex work, and we cannot just run off of a single dashboard showing 1or 2 metrics for each strategic goal”.  Guess what?  I agree.  Most teams do a lot of work supporting activities that do not get reflected in your strategic goals.  Having a dashboard will help you focus on what really is important.  If these things are necessary, you and your team will know it, and continue to do it.  If not – then they will die off (which is a good thing).  Use your dashboard to help your team eliminate work with no value.

You also have other things that you need to track, or other things that you want to track.  These things can go on an internal team dashboard – one that is visible to your team, and one that they pay attention to.  This “internal dashboard” should have all of the same metrics and goals on your team dashboard that you share with the rest of the organization.  However, it will have a few additional things.

TOXTOXTOX

I Can’t Expose Our Strategy, That Would Be Suicide!

Mention Sun Tzu and the Art of War.  Mention US military doctrine in the Gulf War.

Things Change

Mention that dashboards change as the business environment changes.

Using Scrum Methods with IBM DevOps Services

Note: I co-authored this with Patchanee Petprayoon, so you can also find this on her WordPress blog, Ppatchanee.

This article is in response to a number of frequently asked questions about sprint planning and scrum methods, as well as some links to other resources we have created or found to answer those requests.  Before we dive deeper into the topic, let’s take a quick look at what type of scrum work items are provided in IBM DevOps Services. If you have never used any scrum methods before, this can be overwhelming at first.  That’s OK, just read through this and try to digest as much as you can.  If you are familiar with Scrum and Agile methodologies, just skim through all of the explanations to get to the meat on how to do things with IDS.

We call out some sections of this whole process in this indented green text.  These represent best practices, or areas where you can dive more deeply into how IDS works. 

You might also note that a lot of the links for terminology in this article lead to Wikipedia references.  It’s not because we love wikipedia, it’s because too many of the good articles out there have a lot of product references (like this one does – with IDS).  I went looking for some content that was tool agnostic – and Wikipedia is often the best bet for that type of material.

Before We Begin

If you haven’t already registered for IBM DevOps Services (IDS) and Bluemix, you should register now.  It will help you follow along with our examples and will allow you to try a few things for yourself.  You can use IDS alone, use Bluemix without IDS, or use them together.  You may have used the Scrum Process with Rational Team Concert in the past.  If you did, then a lot of this will look quite familiar.

Getting Started with IBM DevOps Services (IDS)

Once you have registered with IDS, you will want to create a project.  To do this, go out to your IDS dashboard.  Click on Create a Project.  You will see a new project dialog come up.  Give your project a name (like “Sample Scrum Project”), something that will identify it.  Then select where your source code repository will be.  For the purposes of this article, it doesn’t really matter, since we’ll just be looking at some common Scrum methods, and will not be dealing with code.  Then make sure that you have a checkmark in the box for “Add features for Scrum development”.  If you are an adventurer, and want to play around with some code, and then deploy it to Bluemix, you might also want to check the box for “Make this a Bluemix project”.  Finally, hit the “Create” button in the lower right, which will create your new scrum project.

So now we have a Scrum project created, and you can see it in the browser.  Now we want to set it up correctly.

In the future, we’ll want to add other members of our team to this project.  To do this, just click on the link for “Members” on the left hand nav bar, and then click on the “Invite Members” icon to add members of your team to the project.

Right now, we just want to learn how to use IDS in conjunction with some common Scrum practices.  One of the first things you’ll need to do is to decide on the length of time of each of you iterations.  In Scrum, these are called Sprints.  Each Sprint represents a period of time where the development team will develop and test code, and commits to have an executable application at the conclusion of the sprint.  That means that you have a running application at the end of each sprint.  It may not expose all of the functionality that has been developed, and some functionality may not be present, but it will execute.

Most scrum teams run with 2 week sprints.  To set up your sprints, start on the overview screen, and press the button labelled “Track & Plan”.  Once on this screen, select “Sprint Planning” from the nav bar on the left.  Since you have just started your project, you don’t have any sprints right now.  So let’s create some.  Click on the “Add Sprints” button in the Sprint Planning window, and add 4 new sprints, and make them 2 weeks in length.

If you click on “Edit Sprints” at this point, you can see how you can now go and add new sprints, rename your sprints, and change the dates on your sprints.  When working with IDS, we suggest naming your Sprints the same as the end date of the sprint.  That way your stakeholders who are unfamiliar with Agile can look and see when their user stories will be implemented at a glance.  They really don’t care about sprints or Agile development, they want to know when they can expect things.

Defining Your Initial Work

Now you have the basic project setup done.  It’s now time to start thinking about doing actual work.  The first thing to understand is the different types of work items available to you to capture your work.

Work Item Type What it is used for
 Story (sometimes called User Story) A user story is a description of some functionality needed, told from the perspective of the end user of the application.  Some people find this equivalent to a use-case.
 Task Tasks are single user tasks used to accomplish some well defined and well scoped effort.  Stories are typically broken down into the multiple tasks, representing the work needed to accomplish the implementation of a Story.
 Epic Epics are stories that are too large to be contained within a single sprint.  Typically an Epic can be broken down into two or more Stories.
 Defect Defects are bugs, problems with the application.  Many Scrum teams will treat them similar to how they treat stories, by decomposing them into the tasks needed to be done to fix a particular defect.
 Impediment These are the risks, issues, and environmental factors that stop work from being accomplished.  Typical impediments may include things like waiting for properly signed SSL certificates, waiting on hardware availability, lack of stakeholder availability, and other similar situations.
 Adoption Item Adoption items indicate when changes made by one team need to be accepted (and integrated) with another team.  An upgrade to a new database version is an example of this.
 Retrospective Retrospectives are held at the end of each sprint, and they allow the team to reflect on what is working well, and what isn’t working well.  It is an attempt to keep the team focused on continuously improving their effectiveness.  Retrospective work items capture the notes and observations made by the team during a retrospective.
 Track Build Item Used to track builds, and relate them to them functionality that they will deliver.  Not a lot of teams use these.

So at this point you should begin creating user stories, epics, impediments, and adoption items for all of the work that you are aware of.  Just click on the “+” in the backlog box to begin creating your first work items.  Once you click on it, just type in a brief headline or general description of the work.  Don’t worry, you will be able to fill in more detail later.  Click on the small icons below the text, to change the work item type, add a more detailed description, and so on.  As you add work items, they will begin to display in your team backlog.

First work item

The edit sprints button, some text for a first work item, and the icons to change the work item attributes

Backlogs are a critical part of scrum development.  Your backlog contains all of the work that your team thinks that it may need to do, but that the team has not yet committed to doing.  If you haven’t scheduled it, and assigned the work to a particular sprint, then it should be in the backlog.  Mature products may have backlogs with 100 or more items in them.  We’ll discuss how to manage your backlog in the next section.

After you have created your initial work items, take a look at your backlog.  It should show all of your work items.  Click on any individual work item, and you will see the details about that work item.  Go back to your sprint planning view, and you will see the work items displayed along with some summary information.

When looking at the details of a work item, it is important to note the fields in the work item.  All of them are self-explanatory, but they do impact how your team does it’s work.  Often lists that are displayed (like your backlog) will have higher priority items at the top.  The discussion field is used to discuss work items, and to provide a running status on the work items. 

When you want to ask someone something in a discussion, click on the link icon with a person, and you will get a dialog where you can choose the specific person that you want to identify.  Now once you save your changes, that person will get an email notifying them that you have asked them a question, along with a link to the workitem.

If you have people always asking about the status of some work, just go to the links tab on a detailed work item, and add that person as a subscriber.  You can add yourself or other people as subscribers to a work item.  whenever a work item is modified, all subscribers get an email that notifies that of the modifications made to the work item, as well as a link to the work item.  We end up using much less email, and having far fewer status meetings, because by using subscribers we are able to keep everyone informed on the progress of critical work items.

Backlog Management

Let’s face it; even though we try our best to estimate things, we’re often very wrong.  We’re not wrong because we are dumb, we’re wrong because we don’t have enough information.  So when using Scrum, we look at each story on our backlog, and we estimate the size/complexity of the story using story points.  Teams will use various methods for estimating story points (some use Planning Poker).  The way that we size the story is very easy and straightforward. If you think that the user story can take a short amount of time to complete, then you give it a low number. If you think it’s going to take a lot of effort to get the user story done, then you give it a high number.  As time goes on, you will get a better feel for estimating work in this way.  You can do all of this work while in that same sprint planning view, capturing your decisions and discussions in the work items as you go along.

There is heated debate about assigning story points to user stories.  Some people think that it is a measure of risk and complexity, some think it is a measure of difficulty, some think of it as a measure of effort.  Do what works for you.  I like Michael Tang’s quick view of story points, but you can find other viewpoints from Mike Cohn and others.  Just spend 30 minutes Googling this and educate yourself.

Once you have estimated the relative sizes of your stories, you are now ready to do some backlog grooming.  In the left hand nav bar, you will notice a link called “Backlog”.  Click on that link, and you should see a nice list of all of your stories.

BacklogGrooming

Backlog Grooming Example

Now we can begin grooming this backlog.  Drag the most important story, the most critical one, to the top of the list.  Now drag the second most important story, and drop it below the first story.  Keep doing this until you have a ranked list of your user stories.

Clicking on a user story will bring up details about it, which you can use when discussing them.  Be careful when dragging and dropping user stories.  If you drop a user story on top of another user story, it will make that user story a child of the user story it was dropped on.  Sometimes you want to do this.  If you didn’t want to do this, then click on the small plus sign “+” on the left of the user story, which will display it’s children.  Then click the small “x” on the left of the child user story, to remove it from being a child.  If you drop it in-between two user stories, it will get ranked between them.

This is your ranked backlog.  This is a critical step for your team, because this indicates the relative priorities of the work that you have been asked to do.  When we get into sprint planning in the next section, you will see how we look at the top of the backlog when planning our next sprint.

Backlog grooming is an important and ongoing activity.  Some scrum teams will do backlog grooming once per sprint, others will do it monthly, and still others will try to do it on an ongoing basis.  The thing to remember is that stories on the backlog need to be reviewed periodically, to ensure that the estimates are accurate, and that the business and development environment have not changed and impacted their priority in relation to other stories on the backlog.

Sprint Planning

Now we get to the key piece of software delivery planning, Sprint Planning.  In your IDS browser, in the left hand nav bar, click on the link called “Sprint Planning”.  When you do this, on the left hand side you will see the ranked list of your backlog items, with the highest raked items on top (I told you that backlog grooming was important!).  On the right hand side you will see an empty list, which represents your first sprint.  Now at this point, you would begin to drag user stories and work items from the backlog (on the left), to your sprint (on the right).  Go ahead and try it.  Move the top ranked user story from the backlog to your sprint.

When you do this, you will notice some changes.  Now in the right hand list, click on the link for “Team Progress”.  What should you see?  Well you should see that the story is now scheduled to be done during the first sprint.  You’ll also notice that the story points assigned to this story show up in the team progress (you now show as having completed 0 out of x story points).  Observant users will also see that the ranked list has updated, and that the next item on the backlog now has a ranking of 1.

Continue to assign stories to your first sprint for what your team can GUARANTEE that they will finish in the timeframe of the first sprint.  That means COMPLETELY FUNCTIONAL software that satisfies the user stories.  Do not over-promise – this is your team’s commitment to complete the work in the timeframe of the first sprint.  Keep in mind that you’ll also have to do testing, and that some bugs may pop up and need to be done as part of the work of this sprint.

The amount of story points that a team can normally complete within a single sprint is called the Team Velocity.  You need to have a gut feel for how much your team can do in your first few sprints, until you get enough history of doing Agile development to understand the velocity of your teams.  Once you have this history, it is easy to assign work to sprints up to three of four sprints in the future, because you have an understanding of how your teams perform.

There is a pile of articles and blogs on Sprint Planning.  What we have seen work the best is when teams commit to doing about 60% of their capacity in a sprint.  This leaves them time for unanticipated surprises, bugs, and other things. It also allows them to build confidence and meet their commitments.

If a team notices that they are half way through their sprint, and that they are almost done, they can then go back to the backlog (or even a future sprint), and pick one or more appropriately sized stories to complete early, in the remaining time that they have in this sprint.  Nobody has ever complained about getting functionality early.

Sprint planning is only half done at this point.  You now have a set of user stories that you are committing to implement in your sprint.  But HOW are you going to do it?  You’re going to have the entire team take those user stories and break them down into the discrete tasks needed to implement those stories.  Look at your sprint backlog now, and on the top story click on the icon for “Open child task breakdown”.  You’ll see that label when you hover over the icon.

SprintPlanning

Sprint planning – creating the tasks for your user story

Once you click on the icon, a new screen appears.  This is your area to enter all of the child tasks for the story that you are breaking down.  You should just click on the plus sign to create a new task work item, that will be a child of the user story that you selected.

TaskEntry_1

Starting your work item entry – use icon to designate this as a task

As you enter in some text to define your new task, you’ll also notice some icons below the text area.  Use the icon on the far left to set the work item type.  In our case, we will want to make this a task.  Notice that this adds some additional “shorthand” to your task description.  Try the other icons, and become familiar with the attributes that you can change in your work item.

At this point you should be getting used to how the work items work within a sprint.  You could easily decompose a large story into smaller stories, making them children of the larger story.  Don’t go crazy with parent-child relationships, use them where it makes sense.  It can be effective to have everyone looking at the same screen during these planning meetings, so the whole team is aware of the tradeoffs discussed, and the whole team has input on the estimates and work breakdown.

Use the icons to set attributes for your work item

Use the icons to set attributes for your work item

Work with your team to define all of the tasks needed for the completion of the user story.  When you are done, you should have a list of tasks that are in the Open column.  Now click on the small down arrows at the bottom of the first task in the list, and you will see some detail information for your task.  You and your team should now look at estimating how much effort each task will take.  You can also further clarify things if you need to.

Adding estimates to your tasks

Adding estimates to your tasks

Go through all of the tasks for this story, and then do the same exercise with your team for the remainder of the stories in your sprint.

At this point you are almost done with your Sprint planning.  You now need to go back and review what you have done, and check your estimated work against the calendar.  Navigate back to your original sprint planning page, and then click on the link for Team Progress.

Checking your sprint estimates

Checking your sprint estimates

You will see that you have an total of the estimates of all of your tasks in hours displayed, a number of work items assigned to the sprint, and a total of the story points.  These are all represented as the second number (0/n) in those progress indicators.  Make sure that you have not overcommitted to the sprint in terms of hours (a two week sprint, means 80 hours of work for each person on the team), or in terms of story points.  Also make sure that you have not undercommitted, and if you have, look for additional stories off of the backlog that you can assign to this sprint.  You’ll spend the remainder of your time in sprint planning making adjustments.

In a lot of the literature on Sprint Planning, they discuss a Sprint Backlog (which is what you are building here), and a Product Backlog (which we just call a Backlog here).  The terminology can be confusing, but in the end the important thing is that you commit to right amount of work for your sprint, and that you correctly define that work for your team.

When do we start doing work?

All of that effort, and we have not written a single line of code.  Ugh!  It can be tough sometimes for people eager to dive in and start prototyping and coding.  Planning is important – it helps us focus on the things that will bring value to our stakeholders and to the business.  There is nothing worse than working long and hard on something that never gets used, or has no value.

The daily scrum

So now that everything is planned, we can kick off our first sprint.  Most Agile teams will have a daily standup meeting.  The daily standup (or scrum) is a chance for each member of the team to briefly talk about what they are doing, what they plan to do, and any impediments or obstacles to their achieving these goals.  These are sometimes referred to as The Three Questions.  It isn’t a status meeting.  It’s a chance to share technical news, and other team members are encouraged to share information that can help other members of the team.  The daily standup can be run in a number of different ways, and there are a lot of mistakes that you can make in how you conduct these meetings.

Most people suggest running these daily standup meetings in front of the scrum taskboard.  You can do this with IDS by using the Team work view.  Just go to the original sprint planning view, and look at the nav bar on the left.  In the nav bar, there is an entry for “Team’s Work”, click on it and you will see the team view of the work in your sprint.

Scrum task board - with viewing options highlighted

Scrum task board – with viewing options highlighted

By choosing the viewing option icons at the top right of the view, you can look at your work as a list, as a table, or in lanes.  Most mature scrum teams will choose to view things in lanes.  If you look at your view, you will notice a lane for open work items (which have not started), and In Progress work items (which are currently being worked on).  If you look closely, there is a horizontal slider at the bottom of this view, and you can scroll to the right and see a lane for completed items as well.  Make minor updates to the taskboard as you go, as long as it doesn’t impact the meeting.  You can drag and drop work items between the lanes, and you can update the attributes if needed.

During the standup meeting, members should be notifying others of their progress, what they plan to do next, and highlight any risks or impediments to their progress.    This not only makes every team member accountable to the team (who is going to stand up and say, “Today I am going to finalize my fantasy football lineup, and watch YouTube videos of kittens?“), it also helps team members to collaborate.  They can help each other with suggestions, short cuts, and help avoid known technical issues.  You want to foster a collaborative team environment, not a structured status reporting environment (filled with Scrum zombies).

Ending the sprint

As you get closer to the end of the sprint, most of your work items should be in the resolved column in your scrum board.  You should be seeing steady progress throughout the sprint.  Your sprint should end with a demonstration of running software that demonstrates the new functionality that has been implemented.  Once the stakeholders sign off on the implementation, your sprint is almost done.

At this point you should bring your team together and have a sprint retrospective.  Go out and create one more work item for your sprint (you should do this at the beginning of the sprint, so people don’t forget it at the end).  They type for the new work item should be “Retrospective”.  During the retrospective, the team should reflect on what they did well, what they struggled with, and what they can improve.  All of this information should be captured in the notes section of the Retrospective work item.  Once the retrospective is over, the Scrummaster may want to create some new work items to reflect things that the team can do in future sprints, that will improve overall team performance.

Often these retrospective observations will end up creating stories that do not serve an end user need, but instead that serve a development team need.  Some people have referred to this as technical debt, but there is some argument about how to best characterize this work.  Examples of stories coming from retrospectives may include the refactoring of portions of the code, development of automated testing scripts, implementation of build and delivery automation capabilities, and other related issues.  Often these “tech debt” stories are the ones that are pulled from the backlog when a team finishes their planned sprint work early, and is able to take on additional work within a sprint.

Wrapping Up

At this point we’ve done a quick overview of how scrum teams can use IDS to help organize and track their sprint planning and execution, how they can groom their backlogs, and do planning in a transparent manner so all stakeholders and team members can have a shared understanding of the goals and the progress of the scrum team during a sprint.

In our next article, we’ll touch on some more intermediate topics.  We’ll talk about how to effectively use dashboards with IDS, effective ways to work with code in IDS, and some small tips and tricks that can help your teams use IDS to deliver end user results that will make your customers smile.

UrbanCode Plugins – Where Do I Begin?

As a manager of a highly technical team, it can be tough to stay on top of everything that happens within my team.  Recently I have been seeing an increasing number of requests for UrbanCode Plugins, those nifty little add-ons that make UrbanCode extensible and much easier to use.  Now you can go and grab an already existing plugin from the UrbanCode Plugins site, because that site has about 336 of them (I just looked at the list today – by the time you read this the number will have changed).  The point is, there are a lot of existing plugins out there already.  Note that some of these are “community” plugins, meaning that they are community supported.

But what should you do if you need a plugin that has not already been developed?  Well…. you should go and create a plugin for yourself.  Where should you start?  One of my team members (Darrell Schrag – check out his blog, DevOps for Everyone) created an UrbanCode Deploy and UrbanCode Release Plugin workshop recently.  I knew about it, but never realized how much it could help our UrbanCode customers.  My mistake – this stuff is good!

It is part of the IBM UrbanCode Learning Circle, which is a great way to get yourself up to speed with UrbanCode.  It provides a roadmap for how you can become a self-taught DevOps guru (or at least proficient with the UrbanCode Deploy and UrbanCode Release tools).  The lab workbook is 150 pages of hands-on training, exposing you to all of the different facets of building your own UrbanCode plugin.  I know that 150 pages sounds like a lot, but you should be able to invest an afternoon and learn what you need to know about creating your own UrbanCode plugin.

If you still have questions, you can always go out to the UrbanCode Forum and ask the community for answers.  The forum is monitored by the IBM development and support teams, as well as by other UrbanCode users.  Just use the tags “urbancode” and “plugin” on your post, to make sure that the right people see your questions.  The key thing to remember is that you are not alone – there are plenty of other people who are trying to do similar things to what you want to do.  If you’re lucky, maybe there is someone who has already developed what you want.

 

 

Resume inflation – Welcome to the New World

In North America, there has been a lot of focus on the recent stories involving television newscasters/journalists who have “inflated” their resumes.  Brian Wiliams was the first big one, and is now facing a 6 month suspension for exaggerating claims for battlefield experiences during the Iraq conflict.  Now Bill O’Reilly is facing similar questioning of his claims of journalistic heroism during the Falkland’s war in 1982.  In both cases you see a massive outpouring of words and opinions on how these two men have lied, or embellished their experiences, in an effort to establish themselves in their profession.

The whole thing puts a small sour grin on my face.  In the technical world, we have been dealing with this for at least the last two decades.  For almost as long as I can remember, I have been talking to software vendors, software contractors, and people that I am looking to hire, and silently asking myself the same question, “Is this person telling me what they can do, or are they telling me what they think they can do?”.

In the software products field, it has become something of a game, where purchasers put the various solution vendors through a series of trials to demonstrate capability.  I have always jokingly referred to this as the “Vendor Olympics”.  The scary part is that these competitions are often decided on criteria that have little to do with technical capability, and more to do with trust.  “Can I trust this vendor to do what they tell me they are going to do?”, is often the most important question.  That type of thinking has led to organizations implementing flawed solutions, because, “nobody ever got fired for selecting IBM/Microsoft/Cisco“.  This kind of thinking leads organizations and individuals to indulge in some strange behaviors.

It leads organizations into “pushing the envelope” when looking to recruit talent.  “We’re an Agile development shop”, is a quote that you hear from almost any place looking to hire top talent.  Top talent wants to work with Agile, which emphasizes a degree of reasonable expectations on a developer.  If you go and look at the development methodologies in use at these places, you will find a mix of agile, waterfall, scrummerfall, eXtreme Programming (XP), and mass confusion.  They are telling you what they think they can be, and not really what they are.

Job seekers are just as bad.  I cannot begin to tell you how many candidates I have interviewed who claim to possess skills that they don’t really have.  I have had candidates tell me that they know Agile, and then struggle to tell me about how story points work.  These are the good candidates, because I can get to the truth with a few simple questions.  The tough ones are the people who claim to have deep technical skills with REST, HEAT, Cloud, Enterprise software, and other things that I cannot ascertain with a few simple questions.

Why do they do it?  Have you LOOKED at the job posting boards out there?  Companies are looking for “perfect” fits, advertising jobs with a laundry list of critical skills, and HR organizations are looking for keywords in applicant resumes.  Do you have experience in complex technical sales involving Cloud solutions?  I do, but not many other people do.  Many are just as qualified as I am, they just need a minimal amount of time to learn some new concepts.  So in an effort to get through the HR screening process, many job seekers “inflate” their resumes, so they can get that initial conversation with a hiring manager.  It becomes a big game.

So why bother writing about this?  Because I am growing tired of having to play this game.  Don’t inflate your experiences, just tell me what you know and how deep you know it.  Don’t tell me, “oh yeah, I know Node”.  I would much rather hear about how you are familiar with Node, haven’t done any real projects with it, but are extremely interested in learning it.  I am tired of HR finding me “perfect” fits.  I want imperfect fits.  I want people who are going to be challenged, and who like learning and stretching themselves and their skills.  I value the following qualities over all else:

  • Honesty
  • Willingness to learn
  • Ability to learn

So I am imploring my HR staff, now and in the future, to quit finding me the perfect candidate, and to stop sifting through resumes looking for keywords.  I want to build flexible teams, with high trust cultures, that can rapidly become high performing teams.  If I can’t trust you to be honest on your resume, how can I trust you to do a quality job for me?  I do NOT want a bunch of specialists, who are as busy looking for their next position as they are in working on my project.  We need to stop the inefficient hiring games, and we need to get down to the business of actually creating something of value.  We need to reestablish trust in the workplace, and it all begins in the hiring process.

So be warned – if I catch a hint of resume inflation, I will not hire you.  Even if you would have been a good fit otherwise.  My three values are listed above – if you want to impress me, then demonstrate those values to me.


No Nonsense Tech Talk Member

These are my PERSONAL views and observations

The postings on this site are my own and don't necessarily represent IBM's position, strategies or opinions. Anyone is free to use, copy, distribute, modify or sell the source code and other materials directly linked from dtoczala.wordpress.com and is provided "as is" without warranties. I am not responsible for any harm or damage caused to your computer, software or anything else caused by the material.

Recent Twitter Traffic


Follow

Get every new post delivered to your Inbox.

Join 273 other followers

%d bloggers like this: