I Love My Watson Chatbot - How Do I Make Changes To It?

Photo by Chris Lawton on Unsplash

This has also been published at Medium.com.

Note: This article was updated on April 29, to add links for a Leigh Williamson blog post that I found informative.

In an earlier blog post, IBM Watson Fueled Chatbots — For Our Health, I talk about the costs and benefits of a chatbot, and show you how a chatbot is within the reach of most organizations. I then branch out and discuss how to make updates to your chatbot, in my post called, I Love My Watson Chatbot — How Do I Update It? Now we look at doing some continuous testing, and how to do some basic version control and deploy new chatbot updates.

How Are We Doing?

In order to see how well your chatbot is performing, you need to put some automated testing of your chatbot in place. This will help you get some objective measures of chatbot accuracy and performance, and will also give you some insight into areas where your chatbot could stand to improve. Keep in mind that testing cognitive applications is fundamentally different from testing more traditional applications. Traditional applications relied on path testing, and code coverage. Due to the non-deterministic nature of cognitive applications, we end up having to use statistical models for our testing.

I STRONGLY suggest using the Python notebooks listed below. These can be run on your own machine, or from within Watson Studio. They are a great starting point for your automated analysis of your chatbot. Over time you can improve these notebooks and expand on some of the testing techniques in them.

  • Dialog Skill Analysis Notebook — a notebook that will do an analysis of your Dialog skill.
  • Log Analysis Workbooks — these two notebooks focus on slightly different areas, and do an analysis of your chatbot based on your logs.
  • My K-Fold Notebook — not as full-featured as the two notebooks above, but I like mine because it’s simple. For a new Python user, it also has a straightforward approach to k-fold testing that is easier to understand.

Getting the feeling that you need to know a little bit of Python? So am I. You don’t have to be a Python expert to do any of this, but you should at least understand the basics of Python, and how to use and manipulate Python notebooks. Don’t get intimidated — you don’t need to know EVERYTHING before you start — just jump in and learn things as you go.

Making Changes Without Impacting Production

Some change management scenarios are enabled by using Watson Assistant’s new versioning capability. This versioning is a bit different than some versioning that you might be used to — it’s not like a typical software development check out, check in paradigm. Instead, it is more like a baseline establishment paradigm, where a version (or baseline) is created as a snapshot of the model at some point in time. The current state of the model (which you see in the UI) is always the tip, or development version.

So if you go into your Watson Assistant Plus instance, and go into your skill, you can just click on “Versions” in the left-hand nav bar, and you will see the versions screen.

Versions Screen in Watson Assistant

In order to drop a new version (or baseline), just click on the “Save a new version” link in the upper right corner. You should then fill in something useful (think about naming conventions!) for the description, before pressing “Save”. We will talk more about naming conventions below, when we start to talk about some typical change management workflows.

Typical Change Management Workflows

A typical change management workflow will allow you to easily scope, contain, and then deploy changes to your chatbot. What I am proposing here is something that might not satisfy a “DevOps Purist”, since we have production and dev/test resources residing in the same service. 

Keep in mind that this approach CAN easily be modified so that instead of moving the versions around within a Watson Assistant instance, we could export the model from one Watson Assistant environment (say the test environment), and then import it into another environment (say the production environment). These environments could be located in different resource groups, with individualized access and permissions tailored for each environment/resource group.

So without further explanation, here is a high-level view of a typical lightweight change management process for your Watson Covid-19 chatbot:

  • Get suggested changes for the next cycle. Save these in some agreed upon format/tool with changes for your Covid-19 bot.
  • Agree on changes and scope of changes for the next cycle. For those of you who are familiar with Agile development principles, think of this as sprint planning….
  • Apply these changes in the tip (development) version of your Covid-19 bot
  • Test development Covid-19 bot version — validate your changes — make final revisions
  • Run automated quality tests against development bot version (yes, I wasn’t kidding about automated tests earlier, do this now — and you will thank me later)
  • Create a new version in the Covid-19 bot — call it “<YYYY-MM-DD> Production”
  • Get formal sign off for promotion of new version to production.
  • Move the Assistant pointer for the production Covid-19 bot to the new version (“<YY-MM-DD> Production”)
  • Backup the Covid-19 bot, with an export of the dialog skill to JSON — store the JSON file with name “<YYYY-MM-DD>_Production.json”
  • Move the Assistant pointer for the development Covid-19 bot to the tip version
  • Go back to step 1 and do it again……

Note that I have been non-specific about tickets/issues/tools to be using. I just trust that you are using some sort of change control — no matter how simple. You want to get your stakeholders in the habit of making change requests and then APPROVING those requests — just so we avoid any “Why did you do that?” situations. You can implement this on the IBM Cloud (a free version of GitHub hosted on the IBM Cloud), where you can do real lightweight change management, or you can use your existing change management tooling and infrastructure. It’s completely up to you.

Change Management for the Advanced User

The simple change management process outlined above will work for simple deployments, and is probably robust enough for a chatbot that is not expected to be deployed on a long term basis. What about change management for more established chatbots? Or for chatbots that are part of a larger system that is currently under some more formal type of change management? What can we do in those cases?

In those cases, and for users wishing for a more robust and formal change management or DevOps infrastructure,  you will want to read A comparison of DevOps choices for IBM Watson Assistant by Leigh Williamson. Leigh is a peer of mine who has spent some time thinking about this, and more importantly, he has spent some time actually implementing these types of change management systems.

What’s Next?

So if you have been following this series of blog posts, you have deployed a longtail chatbot that answers Covid-19 questions. You have learned how to update that chatbot, and how to test and version control your changes. What’s left to do? At this point you know enough to be dangerous — it might be time to fill in some of those learning gaps. My blog posts tell you just what you need to know, but there are so many other Watson techniques and capabilities that you can use to make your end-user experience even better. Maybe you should check out some of the Watson Covid-19 specific learning resources that are out there.

Another useless IBM Cloud Billing Script …. or not

Recently I have have had a customer ask me some questions about billing on the IBM Cloud. Specifically, they wanted to be able to download their billing information. Now I know that you can do this interactively from the IBM Cloud. You just go to your usage page and hit the little Export CSV button in the upper right-hand corner. This will dump out a CSV file of your current months usage for you.

“That’s nice…..but we want to be able to break down usage and get it into a spreadsheet”, was the response from my customer. “But CSV format IS a spreadsheet”, I thought to myself. So I decided to ask some more questions. It turns out that my customer is interested in AUTOMATING the administration of their IBM Cloud. This means that they want to dump not just one month’s worth of data, but a full 12 months of data into statements that will outline internal chargebacks (based on resource groups), as well as showing visual graphs that summarize usage and spending over the past 12 months.

Now I understood why the “button in the upper right” wasn’t going to work. They wanted some sort of script that would run automatically, to generate all of this content for their IBM Cloud user community. I had wanted to do some work with the Billing and Usage API and Resource Manager API on IBM Cloud, so I decided to script something up for them.

This project also led me to using the Resource Controller API – which is slightly different from the Resource Manager API. The Resource Controller API focuses on the various different service instances (resources) that have been created within an account. The Resource Manager API deals more with the resource groups defined within the account.

Note: If you’re reading this post and serious about doing this sort of automation for your IBM Cloud, PLEASE check out the API links above. It is possible to use different API calls to do more targeted operations – instead of the very broad implementation that I have done.

As usual, I decided to work within Watson Studio, and I decided to write my script in Python. Watson Studio includes some environments that have pre-defined collections of packages – including ones you will depend on like pandas, numpy, and IBM_Watson. It also allows me to run my script up on the IBM Cloud (not on my local machine), version my script, and dump my results to IBM Cloud’s Cloud Object Storage. My goal was to show this customer how to do a dump of just the services with usage (and not EVERY service, since there are many services out there which are on Lite plans). I also wanted to highlight how to get to some of the information available via the Resource Controller API.

I quickly learned that in order to get ALL of the information that I needed, I would need to cross-reference some of the data coming back from these various IBM Cloud API’s. That meant using all of the IBM Cloud API’s mentioned above.

So now I have this Python code all sitting in my new GitHub repository called IBMCloudBillingScript. This script only really does a lot of the same things that the “Export CSV” button does – so this makes this script kind of useless. The reason I built it, and have shared it on GitHub, is because we often want to AUTOMATE things based on this information. This script shows you how to get (and manipulate) this kind of information about your IBM Cloud account.

Getting Data from your IBM Cloud GitHub Project

Note: This post is also published on developerWorks, as Getting Data from your IBM Cloud GitHub Project.  Please refer to that article to catch any updates.

I blog when I have to answer questions that I want to more widely share the answers to.  It’s also a good way to remember things before a turkey induced amnesia sets in (it’s a week before U.S. Thanksgiving).

Recently I have seen some questions on being able to get access to the data in an IBM Cloud GitHub project.  I had just completed doing a quick internal activity for pulling information out of a GitHub Enterprise repository, so I figured that this would be simple.  It was…. and it wasn’t.  The IBM Cloud GitHub instance isn’t a GitHub Enterprise deployment, it is a GitLab deployment.  The GitLab API is a little different from the GitHub Enterprise API.  I managed to find a suitable Python package for using the GitLab API, and if you look at the resulting code, it is pretty simple.

So I have created a simple GitHub project (called IBMCloud_GitLab_CSV) that does a quick CSV export of issues from an IBM Cloud GitHub project.  It’s a simple example, written in Python, that you should be able to use and tailor, to fit your specific needs.  I use small Python programs like this to pull the open issues from a variety of projects, and then I can share the resulting CSV files with project managers and PowerPoint producers who want to report on these sorts of things.

If you need this type of capability, make sure to read the README file for this project, which instructs you on how to modify the code to point at YOUR IBM Cloud GitHub project, and tells you how to get an access token for yourself (which the program needs, in order to be able to log into your GitHub project).

If you want to improve on this example, or even create some type of generic tool for doing this type of thing, please join the IBMCloud_GitLab_CSV GitHub project and begin contributing to it.

An Easier Cloud Calendar

Timing is……………… everything.  About 4 hours after I did my last post on How About a Generic Calendar in the Cloud?, I saw a post from one of my team members.  It was a post from Sean Wilbur called, Streamlining Your Bluemix Project for One Button Sharing.  It was a great post, and once I followed the directions that Sean outlined, I was able to add a simple little “Deploy to Bluemix” button on my project.

So now if you would like to get a copy of my Generic Calendar project to play with for yourself, it is really easy.  Just make sure that you have a Bluemix account, and that you have a linked DevOps Services account.  Then just navigate to my project in DevOps Services (it’s the dtoczala|ULLCloudCalendar project).  Once there, you can look at the README.md file displayed there, and look for the “Deploy to Bluemix” button.  It looks like this:

deploy-to-bluemix
The Deploy to Bluemix button

Just press that button and you will get a project created in the DevOps services that is a fork of my original project.  The automatic creation of the project will throw an error during the deployment, but you will be able to easily fix this.  The error is due to a problem in the manifest.yml file, we are currently unable to create and bind services for our application through the manifest (see Sean’s question on this in the forum).  You can easily fix this by doing three things:

  1. In your DevOps services console, configure your Deploy stage – In your newly created project, press the build and deploy button, and then configure the deploy stage.  You will add a job to the deploy configuration, a deploy job, that will do a “cf push” of your application.  Then try executing it.  It will still fail (because our MongoDB service is not present), but it will create a new Bluemix application for you.  This is your version of the ULL Cloud Calendar app.
  2. In your Bluemix console, add and bind the MongoDB service – This is straightforward.  Just add the MongoDB service and make sure to bind it to your new application.  When you add the service, Bluemix will ask if you would like to restage your application.  Answer yes, and wait for the application to be deployed again.
  3. In your Bluemix console, click on the link for your new app – just click on the link for the route to your new application. 

Now once that little issue with the manifest.yml is cleared up, you will be able to share your Bluemix applications with the press of a button.  Bringing up applications and capabilities in the cloud is getting easier and easier!