What Does Watson Really Do?

Moving to a new role is always exciting, but always tough.  There is a learning curve, new technology, new terminology, new business processes, and new people to meet and work with.  Part of my starting as the Manager of the Watson architects was to address the learning curve around the technology and the terminology.  Since I know that any customer or partner looking to use Watson will need to go through a similar exercise, I thought that it might make sense to capture some of the better sources of information that helped me.

Getting Started

Once I started my new job, the first question that I had to answer is, “What exactly is Watson?  What does it do?”  I had to have an answer to this question, because my wife, kids, parents, and friends were asking me, and I felt dumb when I couldn’t answer this simple question.  one of the better answers that I have found is this 8 minute long video called, IBM Watson: How it Works.  I’m not usually big on videos, and I really don’t like marketing fluff (I’m a Tech weenie, I have always been a Tech weenie), but this one is short and to the point.  There are shorter snippets of this information on the SmarterPlanet – What is Watson page that you can look at as well.

The video introduces some key terminology that is important to understand.  It spends the first couple of minutes introducing cognitive computing, and how this is fundamentally different from more traditional computing.  If you’re not a technical person, this is pretty easy to understand.  This piece is important for more traditional programmers to understand, because it dispels the notion that this is just a turbo-charged expert system.  It is actually more than that.

It also talks about the differences between cognitive computing, and simple search capabilities.  The section on natural language processing is really interesting.  It should also help you begin to understand some of the concepts that are important to cognitive computing ( like data curation, corpus, machine learning, training, and question/answer pairs).

I also found an interesting article on No UI is the New UI, which covers some interesting ground on what the author calls artificial intelligence (which I call cognitive computing).  The processing of natural language, the conversion of speech-to-text, and conversions of text-to-speech are all Watson services that you can access on the Bluemix dashboard.  Bluemix is the IBM Cloud platform that allows you to select particular services (or APIs), and combine them to create customer Cloud applications.  The Watson services and capabilities are all cloud enabled and cloud ready for you, and you can see a list of them in the Bluemix catalog.  Not all of them are intuitively obvious at first (“What is the Alchemy API?”), but if you click on the tiles for each service, you will get a synopsis of what that service offers, and you can then drill down into more information on the service.

Cognitive computing is rapidly maturing and is beginning to be seen in the products and services that you use today.  Understanding the practical capabilities and limitations of cognitive computing will be critical in the future.  Because of this, I plan on following up on some of the services themselves, and the implications of using these services, in future blog posts.

Evolution, Change, and Success

I am now transitioning to a new role, and leaving my old job behind. People always seem to take this job change thing with either:

  • Malicious glee – basically laughing and telling everyone that they have left behind “so long suckers!”, or
  • False modesty – with a carefully worded message telling everyone (and if I didn’t mention you I am so VERY sorry) what an honor it was to work with them, and that we should all get together for lunch sometime.

I will try to do neither (although I am sure my close friends will be able to provide examples of both of the above behaviors), and instead I will attempt to focus on what is in front of me.  For me, it is an exciting time, a time when I am eager to look ahead to new challenges and new experiences.  I think the thing to do when changing jobs/roles is to look back and see how you’ve done, and think about what makes people successful.  I was reminded of this by a recent LinkedIn post by Patrick Leddin, which outlined the Five Invaluable Behaviors of Top Performers.  Now I have read similar posts from other writers on performance, but I read his post and it made me reflect on the role that I had played as leader of the Jazz Jumpstart, Emerging Technologies, and DevOps/Cloud teams.  His five behaviors read like this:

  • Deliver Results; Don’t Just Pleasantly Accomplish Activities
  • Solve Problems; Don’t Just Point Them Out
  • Learn New Stuff; Don’t Just Be Comfortable
  • Experience the Customer’s World; Don’t Just Observe It
  • Provide Value That Is Not Easily Replaced; Don’t Just Do the Job

These all seem very simple things, and they are.  The hard part is maintaining a focus on these areas as you deal with the day-to-day challenges that you and your team face.  The real hard part is putting this into practice as you go about doing the non-glamorous parts of your job.

So why write about this in the first place?  I do it as a way to remind myself to focus on these important areas as I move into my new role, and as a way to help the teams that I leave behind to realize their value, and to remind them that they need to focus on these things even when I am not around to remind them about it.  I hope one of my new team members references this blog post in the future, while trying to convince me to change some decision I have made which doesn’t hold to these principles.

This will be the final blog in this series, and I have started rebranding this blog.  That means new pictures (like my dog Jack) and new areas of focus.  I will still be here at the same old address, and you can still come here to find my past articles on the economics of software development, Agile development, the Jazz tools, DevOps, and Cloud technologies.  But from now on you will read blog posts about new technologies, like IBM Watson, and my observations on how to lead a successful team in helping to launch new technologies.

Using Bluemix DevOps Services to Support Multiple Deployments

In my earlier blog post on using Bluemix to deploy a simple Cloud Calendar on Bluemix, I added something the next day or so that discussed using the Deploy to Bluemix button for an even Easier Cloud Calendar.  Well my post has gotten some responses from within IBM, with various different teams wanting to use this simple cloud based calendar to provide a simple widget that they can use to provide a team calendar capability on their RTC dashboards.  So now I have a few different teams that would like me to deploy this simple solution for them.

Well I started out just showing people how they could do this themselves on Bluemix, because it really is quite easy.  However, some of the people asking for this are not technical types, they’re business type or (worse) management types.  They are afraid or unable to do this for themselves.  I’ll address the fear of cloud in a different blog post in the future.  The main thing here is that I ended up using the same code (or similar code) to deploy about 4 different types of calendars for 4 different groups within IBM.

How do I manage to do this without driving myself crazy?  I use the DevOps services available to me in Bluemix, and I configured a delivery pipeline that allows me to build, test, and then deploy four different variants of the same basic application.  The project was already out there in the Bluemix hosted Git repository, what I needed to do is to make some small changes to how I deploy, and some small changes for each flavor (or variant) of my application.


The original application was a Node.js app, and was pretty simple.  For details on it, view my original blog post called How About a Generic Calendar in the Cloud?.  Now to deploy this project, I went into my Test Calendar project and clicked on the Build & Deploy button.  This brought me to the Build and Deploy pipeline area.  I then looked and saw a simple build already set up for my Node.js project.  This simple build executes every time I push new changes to the master branch of my Git repository.  That’s perfect, I now have a continuous build (not to be confused with Continuous Integration) environment set up for my project.

Simple Deploy

Next I need to add a new stage to my project.  So I add a new stage to deploy my test configuration.  This stage takes the outputs of the build stage as inputs, and deploys these out to my test calendar environment.  This “test environment” is a simple application/MongoDB pair that I have out in the Bluemix staging environment.  Now since I have multiple instances that I want to deploy, under different names, in different Bluemix environments/spaces, I will need different manifest.yml files.  The manifest.yml file indicates where the application gets deployed, and the services that it will use.

So I decide to create multiple manifest.yml files, with a naming convention that indicates which deployment they belong to.  So I have a manifest_TestCalendar.yml file with the settings for my TestCalendar (which is my test configuration), and a manifest_ToxCalendar.yml file with the settings for my team calendar (which is my production configuration).  I actually have five of these, but I’ll keep it simple and just highlight the two for the purposes of explaining things here.  So my manifest_TestCalendar.yml file looks like this:

- disk_quota: 512M
- services:
  - mongolab-TestCalendar
  host: TestCalendar
  name: TestCalendar
  random-route: true
  path: .
  domain: stage1.mybluemix.net
  instances: 1
  memory: 512M
  label: mongodb
  plan: 100

and my manifest_ToxCalendar.yml file looks the same, except for the host line (which specifies “ToxCalendar”), the name line (which specifies “ToxCalendar”), and the declared services (which name a different MongoDB instance).  Note that the name of the Mongo DB service instance MUST match the name of the service as shown in Bluemix.  You’ll need to go and create that service first, before you try using this to spin up new instances of your application.  Also note that the route here is pointing at an internal IBM instance of Bluemix, if you do this in the public Bluemix, you’ll use mybluemix.net as the domain.

Configuring the Test Stage

Back to our deployment pipeline.  When I look at the pipeline, I decide to leave the first stage of deployment set to automatically deploy by leaving the stage trigger set to “Run jobs when the previous stage is completed”.  This means that this stage will run when the build stage successfully completes.  Now since I want to have a continuous integration environment, the target of this deploy should be my test instance.  What I will have isn’t really a true continuous integration environment, as I don’t have any automated tests being run as part of the deployment, but you can see how you can easily modify the stage to support this.

DeployToTestSo we’ve decided to do an automated deploy of my test instance on any build.  Go click on the gear icon in the stage, and chose Configure Stage.  You will now see a dialog box where you can configure your DevOps pipeline stage.  In this dialog, check the Input tab to make sure that the stage is set to be run automatically, by making sure that the Run jobs when previous stage is completed option is selected.  Then click on the Jobs tab and make sure that you have a Deploy job selected (if you don’t, then go and create one).  The Deployer Type should be set to Cloud Foundry, and the Target (where this should be deployed) should be your Bluemix environment.  Once you select the target, your organization and space drop-down selections should get populated with organizations that you are a part of, and the spaces available in those organizations (For more on organizations and spaces in Bluemix/CloudFoundry, read Managing your account).  Also enter in your Application Name in the field provided, and make sure that this application name matches the name of the application in your Bluemix console, as well as the application name that you used in the manifest.yml file.  In this case, it is the name of the application that we used in the manifest_TestCalendar.yml file, which was “TestCalendar”.

Finally you will see a grey box with the deploy script.  Now in order to make sure that we use the manifest_TestCalendar.yml file to deploy this, we have to copy this over the existing manifest.yml file.  It is a simple Unix/Linux copy command, and your resulting deploy script should now look like this:

# Move proper manifest file into place
cp manifest_testCalendar.yml manifest.yml
# push code
cf push "${CF_APP}"
# view logs
#cf logs "${CF_APP}" --recent

The result of this is that we copy the version of the manifest file that we want into place, and then CloudFoundry just does the rest.  Go ahead and make a simple code change (just update a comment, or the README file), commit and push it (if you’re using Git), and watch as the application gets automatically built, and then the test calendar gets deployed.

Configuring the Production Stage

Now if we want to deploy using the same mechanism for our production instance, the process is simple.  We just click on the gear icon in the stage, and chose Clone Stage.  This creates a new stage, just like our stage to deploy our test instance.  Click on the gear icon for this new stage.  You will now see a dialog box where you can configure your cloned DevOps pipeline stage.  In this dialog, check the Input tab to make sure that the stage is NOT set to be run automatically, by making sure that the Run jobs only when this stage is run manually option is selected.  Then click on the Jobs tab and make sure that you have a Deploy job selected.  The Deployer Type and the Target (where this should be deployed) should remain the same.  Once you select the target, your organization and space drop-down selections should get populated with organizations that you are a part of, and the spaces available in those organizations.  If you want to deploy your production instance to a different organization and space, you will change the settings for these fields.  Enter in your Application Name (for the Production application) in the field provided, and make sure that this application name matches the name of the application in your Bluemix console, as well as the application name that you used in the manifest.yml file.  In the case of my production instance, it is the name of the application that we used in the manifest_ToxCalendar.yml file, which was “ToxCalendar”.

Finally you will see a grey box with the deploy script.  Now in order to make sure that we use the manifest_ToxCalendar.yml file to deploy this, we have to copy this over the existing manifest.yml file.  We’ll need to modify the cloned deploy script to reflect this change:

# Move proper manifest file into place
cp manifest_ToxCalendar.yml manifest.yml
# push code
cf push "${CF_APP}"
# view logs
#cf logs "${CF_APP}" --recent

The result of this is that for the production stage, we copy the version of the manifest file that we want (manifest_ToxCalendar.yml) into place for use by CloudFoundry.  Since this has been configured as a manual stage, you now will only deploy a new version of your production calendar when you manually drag a new build to the stage.

What does it mean?

PipelineNow we have a deployment pipeline built that allows us to do automated builds and deployments of changes to our calendar application to the test environment.  It should look something like the picture on the right.

Once we are satisfied with our changes, we can then just drag the appropriate build from the App Build stage, and drop it on the stage where we deploy to production (Deploy to Bluemix Staging – Production in the picture).  This will start the deployment process for the production instance of our calendar application.

What about different variants of our calendar application?

Now we can use this same technique of having multiple file variants to support different deployed “flavors” or variants of our calendar application.  I do the same thing with the Node code in the index.html file of the calendar application.  This file is in the public subdirectory.  I can create two variants of this file, and save them in different files (say index_ToxCalendar.html and index_TestCalendar.html).  The change to the index_TestCalendar.html file is the addition of a single line of code, which will display a heading on the calendar.  The code snippet looks like this:

<body onload="init();">
    <div id="scheduler_here" class="dhx_cal_container" style='width:100%; height:100%;'>
    <center><b>TEST CALENDAR</b></center>
        <div class="dhx_cal_navline">

The single added line just puts a title line (“TEST CALENDAR”) in the application.  For other instances, I can make similar changes to similar files.  To get the deployed application to use the correct version of the index.html field that I want, I need to make one more modification to the deploy script.  It should now look like this:

# Move proper manifest file into place
cp manifest_testCalendar.yml manifest.yml
# Move proper index.html file into place
cp public/index_TestCalendar.html public/index.html
# push code
cf push "${CF_APP}"
# view logs
#cf logs "${CF_APP}" --recent

Now we have a specific variant for our test calendar, deployed to the test environment.  You would then do similar changes and file variants for all of the other deployed variants of this that you want to deploy and support.

Basics for Upgrading ANY Software

Recently I became involved with a customer upgrading UrbanCode Deploy.  I want to share my experiences, some lessons learned, and some IT basics.

Our customer was looking to upgrade their UrbanCode Deploy to the latest version.  Doing this meant that they had to upgrade in steps, as outlined in the IBM Technote on Upgrading UrbanCode Deploy.  The customer understood this, and they began to work with the IBM support team on getting their upgrade done.  They had a simple plan which outlined the high level steps involved, and they had received a couple of patches to support issues specific to their environment.  I became involved when they were one week away from doing the upgrade on their production systems.

As I became more involved, I became increasingly alarmed at what I was seeing.  The migration plan was too simple – it had no step-by-step guidance on which values to select for configuration options, installation paths, or even server and environment names.  This left fer too much room for error, and made the upgrade process prone to errors when executed in the production environment.  That leads us to our first general IT lesson, which is:

General Lesson #1 – Any upgrade plan must be able to be executed by someone OTHER than the tool administrator or the author of the plan

One other factor that concerned me was that I saw no detailed “rollback plan”.  Part of the upgrade plan HAS to include what staff should do if the production upgrade goes bad.  It could be due to power outages, lack of resources, or some other unforeseen circumstance.  You need to have detailed instructions (see General Lesson #1 above) on how to restore the existing environment if the upgrade fails.  Nobody likes to do this, but if the upgrade does fail for some reason, people will be under pressure and tired.  They need to have easy to understand and easy to execute instructions on how to restore the production environment.  This is our addition to the first lesson,

General Lesson #1a – Any upgrade plan without a “rollback” section instructing how to restore production to it’s pre-upgrade configuration, is not complete

One of the other areas where I had concerns was due to the fact the customer was planning on moving ahead, even though they were receiving delivery of post-migration scripts the day before the planned upgrade of their production environment.  They kept insisting that they had tested everything in their staging environment, but I knew that they would not be able to adequately test the post-migration scripts (more on those later) prior to upgrading production.  They had tested things extensively in their staging environment, but they had only tested individual parts of the upgrade process.  This leads us to our second general IT lesson, namely:

General Lesson #2 – NEVER upgrade software on production systems unless you have done a FULL upgrade scenario in your staging environment

I can hear the grumbling already.  “We can’t push this out, it’s our only upgrade window for the next month”.  I understand that this can be frustrating, and can seem like be over prepared at times, but this is a lesson drilled into people in our business based on traumatic experiences of the past.  A decision to proceed with an upgrade of a production environment should only be done when you have prepared enough so the risk of ANYTHING going wrong is negligible.  Our systems are critical to the organizations that employ us, treating them with a cavalier attitude only puts our profession on trial.  The decision to upgrade, without having done a full upgrade in a staging environment, has been called “IT Malpractice” by some of my friends in the industry.  It’s a great term, and one I plan to use in the future.  The basic question is this: “Is the potential pain in a botched upgrade worse than delaying the upgrade?”.  If you haven’t covered the lessons spelled out above, assume that your upgrade will NOT be successful.

The customer was also upgrading from version 4.x to the latest version of UrbanCode Deploy.  This meant some changes to the architecture of the product which has a direct impact on the end users.  The first of these is a change to the way that security is handled by UrbanCode.  You really need to be aware of your security settings and the changes that may occur as part of the upgrade.  If you are unfamiliar with the UrbanCode Security Model, then review it and make sure that you have a clear understanding of how roles, permissions, and teams impact the ability of end users to deploy software in your environment.

UrbanCode Lesson #1 – Understand the UrbanCode Security Model, and know how it is impacted by the upgrade

Another thing that happens when upgrading to UrbanCode Deploy 6.x is that your resources move from being in a flat list, to a tree structure.  This allows you to organize your resources into common groups, and find them much more easily in the UI (ie. no 2000 item pull down menus).  During the upgrade to 6.x the urbanCode Deploy resources will be reorganized into a “flat” tree structure.  This has an adverse effect on performance for tool administrators, as page loads become slow if you have a large number of resources.

In order to address this, and as a way to better organize your resources, you should refactor your resources into the tree structure provided.  There is a simple script that IBM can share with you that will refactor resources based on the name of the resource.  You can read Boris Kuschel’s blog on how he deployed this on Bluemix.  Essentially you just have a script that breaks up the resources alphabetically.  You’ll probably want to alter the script to refactor your resources based on some other criteria, but the code is all there.

UrbanCode Lesson #2 – Understand the changes to the UrbanCode Resource Model, and know how it impacts your upgrade

Also keep in mind that some of this refactoring could potentially impact your procedures, depending on how you reference those resources.


Upgrading any software in your production environments is a risk.  We often think of upgrades as being “simple”, but a tool upgrade is often dictated by a foundational change in a product.  These foundational changes will often impact how the product is supported and how it operates, and this may have an impact on your environment.  ALWAYS follow standard IT best practices and TEST upgrades in legitimate testing environments.  Make sure that you have a script (especially for manual steps) for the upgrade that has been fully run through without issues in your testing environment prior to attempting to upgrade your production environments.  I hate seeing my customers going through painful situations that could have been easily avoided with some risk management and planning.

An Easier Cloud Calendar

Timing is……………… everything.  About 4 hours after I did my last post on How About a Generic Calendar in the Cloud?, I saw a post from one of my team members.  It was a post from Sean Wilbur called, Streamlining Your Bluemix Project for One Button Sharing.  It was a great post, and once I followed the directions that Sean outlined, I was able to add a simple little “Deploy to Bluemix” button on my project.

So now if you would like to get a copy of my Generic Calendar project to play with for yourself, it is really easy.  Just make sure that you have a Bluemix account, and that you have a linked DevOps Services account.  Then just navigate to my project in DevOps Services (it’s the dtoczala|ULLCloudCalendar project).  Once there, you can look at the README.md file displayed there, and look for the “Deploy to Bluemix” button.  It looks like this:


The Deploy to Bluemix button

Just press that button and you will get a project created in the DevOps services that is a fork of my original project.  The automatic creation of the project will throw an error during the deployment, but you will be able to easily fix this.  The error is due to a problem in the manifest.yml file, we are currently unable to create and bind services for our application through the manifest (see Sean’s question on this in the forum).  You can easily fix this by doing three things:

  1. In your DevOps services console, configure your Deploy stage – In your newly created project, press the build and deploy button, and then configure the deploy stage.  You will add a job to the deploy configuration, a deploy job, that will do a “cf push” of your application.  Then try executing it.  It will still fail (because our MongoDB service is not present), but it will create a new Bluemix application for you.  This is your version of the ULL Cloud Calendar app.
  2. In your Bluemix console, add and bind the MongoDB service – This is straightforward.  Just add the MongoDB service and make sure to bind it to your new application.  When you add the service, Bluemix will ask if you would like to restage your application.  Answer yes, and wait for the application to be deployed again.
  3. In your Bluemix console, click on the link for your new app – just click on the link for the route to your new application. 

Now once that little issue with the manifest.yml is cleared up, you will be able to share your Bluemix applications with the press of a button.  Bringing up applications and capabilities in the cloud is getting easier and easier!

How about a Generic Calendar in the Cloud?

Working with my team is often fun and rewarding.  I learn a lot from the people I work with, and I get the chance to try and learn new technologies all of the time, in an effort to solve real business problems.  One of our most recent challenges is having the ability to have a “team calendar” that we can all update.  We wanted to have some light weight way to coordinate our activities, and to keep on top of vacations and travel plans.

We wanted something that would work within the External Content widget of RTC, because we wanted to expose this calendar on our RTC dashboard.  The dashboard is where we track our work, watch our progress, and document our progress on our key measures, so it seemed to be a logical place for the calendar to live.  RTC doesn’t have any kind of native calendar ability, and it is something we miss.  I considered just using a Google Calendar, but we dismissed that because often our calendar entries will contain sensitive information.  So we wanted something that could be done within the IBM firewall.  That led me down the path of creating a simple calendar application using Bluemix.  IBM has a small internal implementation of Bluemix behind our firewall, so our simple privacy and security needs could be met by this.

I decided on a simple implementation using Node,js (which recently announced it’s own Node.js foundation).  I thought about using Cloudant for the underlying datastore, but in the end I decided on using Mongo DB, because I didn’t want this to be an “IBM solution”.

Keep in mind that this is a lightweight solution, it uses the Sandbox plan for the MongoDB service, and the code is not expected to be robust enough for 50 or 100 people to use.  It’s meant as a nice sample project, and one that could be useful to a small team.  It’s not going to replace your enterprise calendar solution.  It also uses the dhtmlxScheduler component, which has it’s own licensing concerns.  dhtmlxScheduler Standard Edition is available under GNU GPLv2, so be aware of the implications of this.

An example of the calendar application

An example of the calendar application

Would you like to see how to deploy it for yourself?  Then read on……

Deploying the Generic Calendar on Bluemix

You’ll need a Bluemix account with all of the usual capabilities.  You’ll first want to grab the code for this from my Github project called ULLCloudCalendar, and save it on your laptop/workstation somewhere.  Just hit the button to download the zip of the contents of the project.  I developed this on a Linux box, so hopefully the character sets and encoding don’t screw you up too much.  Once you have a copy of the code, you’ll want to login to Bluemix.  Once there, you will create a new application using the SDK for Node.js runtime.  Give the app a good name (like “AcmeCalendar“), and wait for Bluemix to create your skeleton app for you.

Once Bluemix is done, you should see you new application on the Bluemix console.  Now you will want to go and click on the box to “Add a Service or API”.  Scroll through the list of services until you come to the “MongoLab” service.  Click on the icon for this service, and on the next screen, create a new instance of the MongoLab service (which is a cloud hosted Mongo DB).  Make sure to give it a name that corresponds to the name of your application (like “MongoLab-AcmeCalendar“).  Also make sure that it is being set up in the correct space, and that it will be bound to the correct application (in my case, the “AcmeCalendar” application).  When you have checked everything, press the “Create” button to create your service.

At this point, you will be ready for that code that you copied earlier.  Have the code in a directory by itself.  Make sure that you have downloaded the Cloud Foundry CF Command Line interface, and have installed it on your computer.  Open up a command line interface, and navigate to the new directory where your code lives, at the top level directory.  This directory contains the manifest.yml, app.js, and package.json files.  Now we’ll log into our Cloud Foundry instance, set into the right environment, and push all of this code up to the cloud.

  • Login to the Cloud Foundry instance.
cf api https://api.ng.bluemix.net
  • Login to your account space on the Cloud Foundry/Bluemix instance.  Use your Bluemix ID for the owner (-o) and user (-u) parameters, and the space name of your space on Bluemix for the space (-s) parameter.  After you do this, you will be prompted for your password.
cf login -u dtoczala@us.ibm.com -o dtoczala@us.ibm.com -s 'dev'
  • Now you will want to modify the manifest.yml file to make sure that you can find your new project.  Edit the manifest.yml file and change all of the “ULLCloudCalendar” entries to the name of your application (Acme Calendar in this example).
host: AcmeCalendar
name: AcmeCalendar
  • Now modify the package.json file to reflect the new name of your application as well.  Edit this file and change the line with the name to your new application name.
"name": "AcmeCalendar",
  • Finally, you can now push all of your code up to the Bluemix infrastructure.  Use the name of your application (which is Acme Calendar in my example)
cf push AcmeCalendar

Keep in mind that there are other ways to do this (using an Eclipse plugin is one of them), so do a little research and find out the method that works best for you.  Once you do the cf push of your code, you will see Bluemix/Cloud Foundry do it’s work.  At some point it will tell you that your application has been deployed in the cloud, and that it is running.

Accessing Your Calendar

On the Bluemix console for your application, you will see a link to your new calendar application.  It will be something like https://AcmeCalendar.mybluemix.net (depending on your application name and the route that you have chosen).  You can change the route, but that is a technique for a future blog post (it’s not that hard, I just don’t want to get into it here).  Clicking on that link will launch you to the website where you can access your new calendar.  Play around with it.  Double clicking on a day will open a dialog box for adding a new event.  Double clicking on an event will allow you to change or delete the event.  It’s pretty simple.

Adding this to your RTC dashboard is pretty simple too.  Just create a new tab on your RTC dashboard.  Click on the caret next to the tab name and be sure to set the tab up to display widgets in a single column (otherwise the calendar becomes too small to be useful).  Now click on the “Add Widget” button on the upper right of the dashboard.  Select the “External Content” widget.  Once the widget is displayed on the dashboard, click on the small pencil in the widget menu bar.  You will now enter in

  • The External URL (which is the web address of your new app, maybe something like “https://AcmeCalendar.mybluemix.net“)
  • The height of the widget (try 550 pixels for starters, adjust it as you need to)
  • The refresh rate (go with a simple 5 minutes, or 300 seconds)

Once you do this, you should now see you calendar application right on your RTC dashboard.  You can even navigate through the calendar and add/change/delete events.

How Does it Work?

If you’ve read this far, you have enough knowledge to be able to deploy a simple cloud based calendar for your team.  If you want to get into the code, and possibly change and enhance this calendar app, then read this section.

The calendar has three big pieces that control everything.  The first piece is the dhtmlxscheduler piece.  The code for this component (which controls the calendar look and feel, and drives functionality) is in the public folder.  I didn’t touch this stuff, but if you want to try messing around with the CSS files to change the look and feel of things, be my guest.

The next big piece is the code that controls the rendering of the HTML page.  This is in the index.html file in the public folder.  There are two important pieces of code in this file.  There is the script.  The script first will go and make some configuration settings to the calendar, it then sets up the basic colors used, and sets up the look of the dialog box to add/modify/delete events.  The script then initializes the calendar component with these settings.  At the end of the script, the default date/time format is specified, your existing data is loaded, and a data processor is set up to handle the interactive user requests.  Then there is the body of the HTML page, which sets up the display of the calendar itself, and initializes things.  I didn’t fool around with this section.

The final piece is the app itself, in the top level directory in the file app.js.  This file handles the storage and retrieval of data from the MongoDB, and does some data checking and data manipulation to help format things appropriately.

The script starts out by setting a bunch of global variables and reading the various VCAP settings provided by the Bluemix environment.  This allows the application to connect to the MongoDB that is bound to this application with the correct credentials, and it also provides some other important runtime information.  You will notice that there are references to SSO and certs that are in the code, but have not been tested.

Once this initial code is complete, you can see where we connect to the MongoDB.  Following this is a section of code that is NOT tested (and not used) that deals with the SSO and passport functionality.  This all ends up with the section on customer authentication middleware.

Finally we get to the Application routes.

  • The code for the /init route is simple, it just adds a single event on New Year’s Day to get you started.
  • The get code for the /data route supplies the calendar object with all of it’s events from the MongoDB.  It retrieves ALL of the events in the datastore, builds a JSON object with these events, and provides them as a stream of JSON data to the calendar object in a response.  Be careful with the formatting of your dates in the JSON response, an invalid date can cause problems.
  • The post code for the /data route processes the creation/modification/deletion of events by the calendar object.  A user who changes something in the calendar will post the change to the /data route.  This section handles the request, and processes it accordingly.

Finally at the end of the app.js file, we start the app.

What’s Next?

There are things you can do to change how this calendar works, and expand or change it’s functionality.  I’ll cover a couple of the simple things here.

I Hate The Colors

You hate the colors that I picked?  So change them yourself.  There are two sections that deal with the colors in the calendar interface.  In the index.html file, there is a variable called colorpicker.  You can change the names of the colors by changing the label property of the array entries, or the color itself by changing the key property of the array entries.  This key property defines the RGB mix of colors.  I used the HTML color picker to get these values.  You can even add even more colors by adding more entries to the array.

These key values (the “rgb(x,y,z)” entries) are stored with the events in the database.  If you look at the code in the app.js file, look at the get /data section of code.   In here you see a section of code where we check the color property of the returned event data.  This represents the color of the event box.  Based on this, the if statement will either assign a grey box and black text (if no color information is provided), or the proper color and black text, unless the box is indigo in color, in which case white text is selected because it shows up better.  Kind of hard to explain – easier to see in the code.

            if (!data[i].color)
                color = 'rgb(204,204,204)';  // grey block
                textcolor = 'rgb(0,0,0)';    // black text
                color = data[i].color;
                textcolor = 'rgb(0,0,0)';    // black text
                if (color == 'rgb(77,77,184)')  // if block is indigo
                    textcolor = 'rgb(255,255,255)';    // go with white text

What About Repeating Calendar Entries?

The dhtmlxscheduler will support repeating entries.  To implement this, check out this entry in their documentation on implementing repeating entries.  In fact, take a look at their documentation overall.  I found the sections on custom event colors, Lightbox manipulations (Lightbox is the user entry popup), and the code associated with the coloring events example to be quite helpful.

Weeks start on Mondays, not Sundays

This one is an easy change in the index.html file.  Just find this line of code:

scheduler.config.start_on_monday = false;

and change it to:

scheduler.config.start_on_monday = true;


I wanted a calendar that was stand-alone, that could be displayed in an RTC widget, and that I could easily deploy in a Bluemix environment.  I hope this guide has shown you how easy this is to do, and allows you to add this calendar ability to your RTC environment.  If you have comments or issues, please comment and I will do my best to answer your questions.b

Using Scrum Methods with IBM DevOps Services

Note: I co-authored this with Patchanee Petprayoon, so you can also find this on her WordPress blog, Ppatchanee.

This article is in response to a number of frequently asked questions about sprint planning and scrum methods, as well as some links to other resources we have created or found to answer those requests.  Before we dive deeper into the topic, let’s take a quick look at what type of scrum work items are provided in IBM DevOps Services. If you have never used any scrum methods before, this can be overwhelming at first.  That’s OK, just read through this and try to digest as much as you can.  If you are familiar with Scrum and Agile methodologies, just skim through all of the explanations to get to the meat on how to do things with IDS.

We call out some sections of this whole process in this indented green text.  These represent best practices, or areas where you can dive more deeply into how IDS works. 

You might also note that a lot of the links for terminology in this article lead to Wikipedia references.  It’s not because we love wikipedia, it’s because too many of the good articles out there have a lot of product references (like this one does – with IDS).  I went looking for some content that was tool agnostic – and Wikipedia is often the best bet for that type of material.

Before We Begin

If you haven’t already registered for IBM DevOps Services (IDS) and Bluemix, you should register now.  It will help you follow along with our examples and will allow you to try a few things for yourself.  You can use IDS alone, use Bluemix without IDS, or use them together.  You may have used the Scrum Process with Rational Team Concert in the past.  If you did, then a lot of this will look quite familiar.

Getting Started with IBM DevOps Services (IDS)

Once you have registered with IDS, you will want to create a project.  To do this, go out to your IDS dashboard.  Click on Create a Project.  You will see a new project dialog come up.  Give your project a name (like “Sample Scrum Project”), something that will identify it.  Then select where your source code repository will be.  For the purposes of this article, it doesn’t really matter, since we’ll just be looking at some common Scrum methods, and will not be dealing with code.  Then make sure that you have a checkmark in the box for “Add features for Scrum development”.  If you are an adventurer, and want to play around with some code, and then deploy it to Bluemix, you might also want to check the box for “Make this a Bluemix project”.  Finally, hit the “Create” button in the lower right, which will create your new scrum project.

So now we have a Scrum project created, and you can see it in the browser.  Now we want to set it up correctly.

In the future, we’ll want to add other members of our team to this project.  To do this, just click on the link for “Members” on the left hand nav bar, and then click on the “Invite Members” icon to add members of your team to the project.

Right now, we just want to learn how to use IDS in conjunction with some common Scrum practices.  One of the first things you’ll need to do is to decide on the length of time of each of you iterations.  In Scrum, these are called Sprints.  Each Sprint represents a period of time where the development team will develop and test code, and commits to have an executable application at the conclusion of the sprint.  That means that you have a running application at the end of each sprint.  It may not expose all of the functionality that has been developed, and some functionality may not be present, but it will execute.

Most scrum teams run with 2 week sprints.  To set up your sprints, start on the overview screen, and press the button labelled “Track & Plan”.  Once on this screen, select “Sprint Planning” from the nav bar on the left.  Since you have just started your project, you don’t have any sprints right now.  So let’s create some.  Click on the “Add Sprints” button in the Sprint Planning window, and add 4 new sprints, and make them 2 weeks in length.

If you click on “Edit Sprints” at this point, you can see how you can now go and add new sprints, rename your sprints, and change the dates on your sprints.  When working with IDS, we suggest naming your Sprints the same as the end date of the sprint.  That way your stakeholders who are unfamiliar with Agile can look and see when their user stories will be implemented at a glance.  They really don’t care about sprints or Agile development, they want to know when they can expect things.

Defining Your Initial Work

Now you have the basic project setup done.  It’s now time to start thinking about doing actual work.  The first thing to understand is the different types of work items available to you to capture your work.

Work Item Type What it is used for
 Story (sometimes called User Story) A user story is a description of some functionality needed, told from the perspective of the end user of the application.  Some people find this equivalent to a use-case.
 Task Tasks are single user tasks used to accomplish some well defined and well scoped effort.  Stories are typically broken down into the multiple tasks, representing the work needed to accomplish the implementation of a Story.
 Epic Epics are stories that are too large to be contained within a single sprint.  Typically an Epic can be broken down into two or more Stories.
 Defect Defects are bugs, problems with the application.  Many Scrum teams will treat them similar to how they treat stories, by decomposing them into the tasks needed to be done to fix a particular defect.
 Impediment These are the risks, issues, and environmental factors that stop work from being accomplished.  Typical impediments may include things like waiting for properly signed SSL certificates, waiting on hardware availability, lack of stakeholder availability, and other similar situations.
 Adoption Item Adoption items indicate when changes made by one team need to be accepted (and integrated) with another team.  An upgrade to a new database version is an example of this.
 Retrospective Retrospectives are held at the end of each sprint, and they allow the team to reflect on what is working well, and what isn’t working well.  It is an attempt to keep the team focused on continuously improving their effectiveness.  Retrospective work items capture the notes and observations made by the team during a retrospective.
 Track Build Item Used to track builds, and relate them to them functionality that they will deliver.  Not a lot of teams use these.

So at this point you should begin creating user stories, epics, impediments, and adoption items for all of the work that you are aware of.  Just click on the “+” in the backlog box to begin creating your first work items.  Once you click on it, just type in a brief headline or general description of the work.  Don’t worry, you will be able to fill in more detail later.  Click on the small icons below the text, to change the work item type, add a more detailed description, and so on.  As you add work items, they will begin to display in your team backlog.

First work item

The edit sprints button, some text for a first work item, and the icons to change the work item attributes

Backlogs are a critical part of scrum development.  Your backlog contains all of the work that your team thinks that it may need to do, but that the team has not yet committed to doing.  If you haven’t scheduled it, and assigned the work to a particular sprint, then it should be in the backlog.  Mature products may have backlogs with 100 or more items in them.  We’ll discuss how to manage your backlog in the next section.

After you have created your initial work items, take a look at your backlog.  It should show all of your work items.  Click on any individual work item, and you will see the details about that work item.  Go back to your sprint planning view, and you will see the work items displayed along with some summary information.

When looking at the details of a work item, it is important to note the fields in the work item.  All of them are self-explanatory, but they do impact how your team does it’s work.  Often lists that are displayed (like your backlog) will have higher priority items at the top.  The discussion field is used to discuss work items, and to provide a running status on the work items. 

When you want to ask someone something in a discussion, click on the link icon with a person, and you will get a dialog where you can choose the specific person that you want to identify.  Now once you save your changes, that person will get an email notifying them that you have asked them a question, along with a link to the workitem.

If you have people always asking about the status of some work, just go to the links tab on a detailed work item, and add that person as a subscriber.  You can add yourself or other people as subscribers to a work item.  whenever a work item is modified, all subscribers get an email that notifies that of the modifications made to the work item, as well as a link to the work item.  We end up using much less email, and having far fewer status meetings, because by using subscribers we are able to keep everyone informed on the progress of critical work items.

Backlog Management

Let’s face it; even though we try our best to estimate things, we’re often very wrong.  We’re not wrong because we are dumb, we’re wrong because we don’t have enough information.  So when using Scrum, we look at each story on our backlog, and we estimate the size/complexity of the story using story points.  Teams will use various methods for estimating story points (some use Planning Poker).  The way that we size the story is very easy and straightforward. If you think that the user story can take a short amount of time to complete, then you give it a low number. If you think it’s going to take a lot of effort to get the user story done, then you give it a high number.  As time goes on, you will get a better feel for estimating work in this way.  You can do all of this work while in that same sprint planning view, capturing your decisions and discussions in the work items as you go along.

There is heated debate about assigning story points to user stories.  Some people think that it is a measure of risk and complexity, some think it is a measure of difficulty, some think of it as a measure of effort.  Do what works for you.  I like Michael Tang’s quick view of story points, but you can find other viewpoints from Mike Cohn and others.  Just spend 30 minutes Googling this and educate yourself.

Once you have estimated the relative sizes of your stories, you are now ready to do some backlog grooming.  In the left hand nav bar, you will notice a link called “Backlog”.  Click on that link, and you should see a nice list of all of your stories.


Backlog Grooming Example

Now we can begin grooming this backlog.  Drag the most important story, the most critical one, to the top of the list.  Now drag the second most important story, and drop it below the first story.  Keep doing this until you have a ranked list of your user stories.

Clicking on a user story will bring up details about it, which you can use when discussing them.  Be careful when dragging and dropping user stories.  If you drop a user story on top of another user story, it will make that user story a child of the user story it was dropped on.  Sometimes you want to do this.  If you didn’t want to do this, then click on the small plus sign “+” on the left of the user story, which will display it’s children.  Then click the small “x” on the left of the child user story, to remove it from being a child.  If you drop it in-between two user stories, it will get ranked between them.

This is your ranked backlog.  This is a critical step for your team, because this indicates the relative priorities of the work that you have been asked to do.  When we get into sprint planning in the next section, you will see how we look at the top of the backlog when planning our next sprint.

Backlog grooming is an important and ongoing activity.  Some scrum teams will do backlog grooming once per sprint, others will do it monthly, and still others will try to do it on an ongoing basis.  The thing to remember is that stories on the backlog need to be reviewed periodically, to ensure that the estimates are accurate, and that the business and development environment have not changed and impacted their priority in relation to other stories on the backlog.

Sprint Planning

Now we get to the key piece of software delivery planning, Sprint Planning.  In your IDS browser, in the left hand nav bar, click on the link called “Sprint Planning”.  When you do this, on the left hand side you will see the ranked list of your backlog items, with the highest raked items on top (I told you that backlog grooming was important!).  On the right hand side you will see an empty list, which represents your first sprint.  Now at this point, you would begin to drag user stories and work items from the backlog (on the left), to your sprint (on the right).  Go ahead and try it.  Move the top ranked user story from the backlog to your sprint.

When you do this, you will notice some changes.  Now in the right hand list, click on the link for “Team Progress”.  What should you see?  Well you should see that the story is now scheduled to be done during the first sprint.  You’ll also notice that the story points assigned to this story show up in the team progress (you now show as having completed 0 out of x story points).  Observant users will also see that the ranked list has updated, and that the next item on the backlog now has a ranking of 1.

Continue to assign stories to your first sprint for what your team can GUARANTEE that they will finish in the timeframe of the first sprint.  That means COMPLETELY FUNCTIONAL software that satisfies the user stories.  Do not over-promise – this is your team’s commitment to complete the work in the timeframe of the first sprint.  Keep in mind that you’ll also have to do testing, and that some bugs may pop up and need to be done as part of the work of this sprint.

The amount of story points that a team can normally complete within a single sprint is called the Team Velocity.  You need to have a gut feel for how much your team can do in your first few sprints, until you get enough history of doing Agile development to understand the velocity of your teams.  Once you have this history, it is easy to assign work to sprints up to three of four sprints in the future, because you have an understanding of how your teams perform.

There is a pile of articles and blogs on Sprint Planning.  What we have seen work the best is when teams commit to doing about 60% of their capacity in a sprint.  This leaves them time for unanticipated surprises, bugs, and other things. It also allows them to build confidence and meet their commitments.

If a team notices that they are half way through their sprint, and that they are almost done, they can then go back to the backlog (or even a future sprint), and pick one or more appropriately sized stories to complete early, in the remaining time that they have in this sprint.  Nobody has ever complained about getting functionality early.

Sprint planning is only half done at this point.  You now have a set of user stories that you are committing to implement in your sprint.  But HOW are you going to do it?  You’re going to have the entire team take those user stories and break them down into the discrete tasks needed to implement those stories.  Look at your sprint backlog now, and on the top story click on the icon for “Open child task breakdown”.  You’ll see that label when you hover over the icon.


Sprint planning – creating the tasks for your user story

Once you click on the icon, a new screen appears.  This is your area to enter all of the child tasks for the story that you are breaking down.  You should just click on the plus sign to create a new task work item, that will be a child of the user story that you selected.


Starting your work item entry – use icon to designate this as a task

As you enter in some text to define your new task, you’ll also notice some icons below the text area.  Use the icon on the far left to set the work item type.  In our case, we will want to make this a task.  Notice that this adds some additional “shorthand” to your task description.  Try the other icons, and become familiar with the attributes that you can change in your work item.

At this point you should be getting used to how the work items work within a sprint.  You could easily decompose a large story into smaller stories, making them children of the larger story.  Don’t go crazy with parent-child relationships, use them where it makes sense.  It can be effective to have everyone looking at the same screen during these planning meetings, so the whole team is aware of the tradeoffs discussed, and the whole team has input on the estimates and work breakdown.

Use the icons to set attributes for your work item

Use the icons to set attributes for your work item

Work with your team to define all of the tasks needed for the completion of the user story.  When you are done, you should have a list of tasks that are in the Open column.  Now click on the small down arrows at the bottom of the first task in the list, and you will see some detail information for your task.  You and your team should now look at estimating how much effort each task will take.  You can also further clarify things if you need to.

Adding estimates to your tasks

Adding estimates to your tasks

Go through all of the tasks for this story, and then do the same exercise with your team for the remainder of the stories in your sprint.

At this point you are almost done with your Sprint planning.  You now need to go back and review what you have done, and check your estimated work against the calendar.  Navigate back to your original sprint planning page, and then click on the link for Team Progress.

Checking your sprint estimates

Checking your sprint estimates

You will see that you have an total of the estimates of all of your tasks in hours displayed, a number of work items assigned to the sprint, and a total of the story points.  These are all represented as the second number (0/n) in those progress indicators.  Make sure that you have not overcommitted to the sprint in terms of hours (a two week sprint, means 80 hours of work for each person on the team), or in terms of story points.  Also make sure that you have not undercommitted, and if you have, look for additional stories off of the backlog that you can assign to this sprint.  You’ll spend the remainder of your time in sprint planning making adjustments.

In a lot of the literature on Sprint Planning, they discuss a Sprint Backlog (which is what you are building here), and a Product Backlog (which we just call a Backlog here).  The terminology can be confusing, but in the end the important thing is that you commit to right amount of work for your sprint, and that you correctly define that work for your team.

When do we start doing work?

All of that effort, and we have not written a single line of code.  Ugh!  It can be tough sometimes for people eager to dive in and start prototyping and coding.  Planning is important – it helps us focus on the things that will bring value to our stakeholders and to the business.  There is nothing worse than working long and hard on something that never gets used, or has no value.

The daily scrum

So now that everything is planned, we can kick off our first sprint.  Most Agile teams will have a daily standup meeting.  The daily standup (or scrum) is a chance for each member of the team to briefly talk about what they are doing, what they plan to do, and any impediments or obstacles to their achieving these goals.  These are sometimes referred to as The Three Questions.  It isn’t a status meeting.  It’s a chance to share technical news, and other team members are encouraged to share information that can help other members of the team.  The daily standup can be run in a number of different ways, and there are a lot of mistakes that you can make in how you conduct these meetings.

Most people suggest running these daily standup meetings in front of the scrum taskboard.  You can do this with IDS by using the Team work view.  Just go to the original sprint planning view, and look at the nav bar on the left.  In the nav bar, there is an entry for “Team’s Work”, click on it and you will see the team view of the work in your sprint.

Scrum task board - with viewing options highlighted

Scrum task board – with viewing options highlighted

By choosing the viewing option icons at the top right of the view, you can look at your work as a list, as a table, or in lanes.  Most mature scrum teams will choose to view things in lanes.  If you look at your view, you will notice a lane for open work items (which have not started), and In Progress work items (which are currently being worked on).  If you look closely, there is a horizontal slider at the bottom of this view, and you can scroll to the right and see a lane for completed items as well.  Make minor updates to the taskboard as you go, as long as it doesn’t impact the meeting.  You can drag and drop work items between the lanes, and you can update the attributes if needed.

During the standup meeting, members should be notifying others of their progress, what they plan to do next, and highlight any risks or impediments to their progress.    This not only makes every team member accountable to the team (who is going to stand up and say, “Today I am going to finalize my fantasy football lineup, and watch YouTube videos of kittens?“), it also helps team members to collaborate.  They can help each other with suggestions, short cuts, and help avoid known technical issues.  You want to foster a collaborative team environment, not a structured status reporting environment (filled with Scrum zombies).

Ending the sprint

As you get closer to the end of the sprint, most of your work items should be in the resolved column in your scrum board.  You should be seeing steady progress throughout the sprint.  Your sprint should end with a demonstration of running software that demonstrates the new functionality that has been implemented.  Once the stakeholders sign off on the implementation, your sprint is almost done.

At this point you should bring your team together and have a sprint retrospective.  Go out and create one more work item for your sprint (you should do this at the beginning of the sprint, so people don’t forget it at the end).  They type for the new work item should be “Retrospective”.  During the retrospective, the team should reflect on what they did well, what they struggled with, and what they can improve.  All of this information should be captured in the notes section of the Retrospective work item.  Once the retrospective is over, the Scrummaster may want to create some new work items to reflect things that the team can do in future sprints, that will improve overall team performance.

Often these retrospective observations will end up creating stories that do not serve an end user need, but instead that serve a development team need.  Some people have referred to this as technical debt, but there is some argument about how to best characterize this work.  Examples of stories coming from retrospectives may include the refactoring of portions of the code, development of automated testing scripts, implementation of build and delivery automation capabilities, and other related issues.  Often these “tech debt” stories are the ones that are pulled from the backlog when a team finishes their planned sprint work early, and is able to take on additional work within a sprint.

Wrapping Up

At this point we’ve done a quick overview of how scrum teams can use IDS to help organize and track their sprint planning and execution, how they can groom their backlogs, and do planning in a transparent manner so all stakeholders and team members can have a shared understanding of the goals and the progress of the scrum team during a sprint.

In our next article, we’ll touch on some more intermediate topics.  We’ll talk about how to effectively use dashboards with IDS, effective ways to work with code in IDS, and some small tips and tricks that can help your teams use IDS to deliver end user results that will make your customers smile.

No Nonsense Tech Talk Member

These are my PERSONAL views and observations

The postings on this site are my own and don't necessarily represent IBM's position, strategies or opinions. Anyone is free to use, copy, distribute, modify or sell the source code and other materials directly linked from dtoczala.wordpress.com and is provided "as is" without warranties. I am not responsible for any harm or damage caused to your computer, software or anything else caused by the material.

Recent Twitter Traffic


Get every new post delivered to your Inbox.

Join 307 other followers

%d bloggers like this: