Watson Text to Speech – The Costs of Personalization

I tend to write these blog posts to share interesting things that I have learned when working with our customers.  Just this past week I have had 2 or 3 blog worthy events happen, so I hope to be publishing these posts at a brisker pace in the coming months.

This week I had a customer that is using the Watson Text to Speech service.  They are using it to do short utterances, things like street names, addresses, and city names.  The utterances are relatively short.  They told me that they had no idea how they were being charged for the service.

This particular customer has a focus on producing a positive customer experience.  No tinny, mechanical voice for this customer!!  They are tweaking the speaking voice and customizing it, using SSML (Speech Synthesis Markup Language) to modify and “humanize” the synthesized speech from the Watson Text-to-Speech (TTS) service.  You have the ability to modify things like the emotion used in the speech generated (called expressive SSML), to more basic things like the pitch and glottal tension (and yes, I had to look up the definition of glottal tension).  The typical curl call that they use looked similar to this:

curl -X POST -u apikey:*****************************--header "Content-Type: application/json" --header "Accept: audio/wav" --data "{\"text\":\"<speak><voice-transformation type=\\\"Custom\\\" breathiness=\\\"35%\\\" pitch=\\\"-80%\\\" pitch_range=\\\"60%\\\" glottal_tension=\\\"-40%\\\" >$text</voice-transformation></speak>\"}" --output $finalFile "
https://stream.watsonplatform.net/text-to-speech/api/v1/synthesize?voice=en-US_MichaelVoice"

So this curl command will ask for some text (referenced by the $text parameter) that will have breathiness set to 35%, pitch at -80%, the pitch range set to 60%, with a glottal tension of -40%.  I’m sure that someone played with these values, before settling on this combination.  It’s a great way to customize the sound and the tone of your automated speaking responses. 

How Does This Impact Cost?

 The cost of doing something like this will vary, and this is where I learned how some small changes can have a HUGE impact on the costs associated with your Watson solution.  The basic price for using the Watson TTS service is $0.02 per thousand characters.  There are some interesting things to keep in mind here.  Whitespace is NOT counted, so only count the non-whitespace characters.  Also, remember that the voice customizations and everything between the “<speak>” and the “</speak> ” are included in this count.

Now let’s assume that the text being converted was a home address, something like, “9 Marine Drive, Round Rock, Texas 78681”.  Let’s also assume that the user is being referred to by name.  There will also be some other text (a meter reading, a service interruption, etc.) as well, informing the end user about something about to happen near their home address.  We want to figure out the monthly costs for something like this if we estimate that we’ll build and issue 100,000 of these notices in a month.  A sample utterance might sound/look like this:

“This message is for Dan Toczala.  We are informing you of a service interruption tomorrow morning at 9 Marine Drive, Round Rock, Texas 78681.  Please call us at 1-800-123-4567 if you have questions.”

Breaking It Down

Your application can look up the customer name and address, and build this entire text string for each individual event, and then submit each one to the Watson Text To Speech service.  Your typical call would look like this:

curl -X POST -u apikey:*****************************--header "Content-Type: application/json" --header "Accept: audio/wav" --data "{\"text\":\"<speak><voice-transformation type=\\\"Custom\\\" breathiness=\\\"35%\\\" pitch=\\\"-80%\\\" pitch_range=\\\"60%\\\" glottal_tension=\\\"-40%\\\" >This message is for Dan Toczala.  We are informing you of a service interruption tomorrow morning at 9 Marine Drive, Round Rock, Texas 78681.  Please call us at 1-800-123-4567 if you have questions.</voice-transformation></speak>\"}" --output $finalFile "
https://stream.watsonplatform.net/text-to-speech/api/v1/synthesize?voice=en-US_MichaelVoice"

For the purposes of this discussion, we’re going to just focus on the “payload”, or the part in the data section of the curl command.  The part that impacts what your costs are.  So this chunk:

<speak><voice-transformation type=\\\"Custom\\\" breathiness=\\\"35%\\\" pitch=\\\"-80%\\\" pitch_range=\\\"60%\\\" glottal_tension=\\\"-40%\\\" >This message is for Dan Toczala.  We are informing you of a service interruption tomorrow morning at 9 Marine Drive, Round Rock, Texas 78681.  Please call us at 1-800-123-4567 if you have questions.</voice-transformation></speak>\

Now in this example, we count ALL non-whitespace characters inside of the quotes.  We have 336 non-whitespace characters.  Multiply that by 100,000 notices in a month, and I get a rate of 33,600,000 characters a month.  Apply the TTS cost of $0.02 per thousand characters, and you get a final monthly cost of $672.

Now let’s see what happens if we change the way that we think about this.  What if we quit customizing so much of the voice?  Then we would end up with something looking like this:

<speak><voice-transformation>This message is for Dan Toczala.  We are informing you of a service interruption tomorrow morning at 9 Marine Drive, Round Rock, Texas 78681.  Please call us at 1-800-123-4567 if you have questions.</voice-transformation></speak>\

So for the non-customized example, we have 225 non-whitespace characters.  Multiply that by 100,000 notices in a month, and I get a rate of 22,500,000 characters a month.  Apply the TTS cost of $0.02 per thousand characters, and you get a final monthly cost of $450.  Customizing my voice could be looked at as a cheap way to have an impact on customer satisfaction (it’s only $222 a month), or a really expensive way to do this (it’s about 49% more expensive than the base translation).  Remember, it all depends on how you want to look at things.  I suggest focusing on your problem and the overall costs of your solution.

Now let’s look at a final example.  In this example, we’ll keep our customized voice, but we’ll try to stop converting the same text over and over again.  What if our message was built in a way that minimized what needed to be converted each time?  What if we converted a basic message once, and the rest of the customized part for each customer?  So we could do this for each customer:

<speak><voice-transformation type=\\\"Custom\\\" breathiness=\\\"35%\\\" pitch=\\\"-80%\\\" pitch_range=\\\"60%\\\" glottal_tension=\\\"-40%\\\" >This message is for Dan Toczala, who resides at 9 Marine Drive, Round Rock, Texas 78681</voice-transformation></speak>\

And then follow that with this “standard” section which we would only need to convert once (for a one time cost of fractions of a cent):

<speak><voice-transformation type=\\\"Custom\\\" breathiness=\\\"35%\\\" pitch=\\\"-80%\\\" pitch_range=\\\"60%\\\" glottal_tension=\\\"-40%\\\" >We are informing you of a service interruption tomorrow morning. Please call us at 1-800-123-4567 if you have any questions.</voice-transformation></speak>\

So for the modified script example, we have 244 non-whitespace characters.  Multiply that by 100,000 notices in a month, and I get a rate of 24,400,000 characters a month.  Apply the TTS cost and you get a final monthly cost of $488.

Final Conclusions

So let’s look at all of these options together:

ApproachCharacters /
Msg.
Characters /
Month
Monthly
Cost
% Change
Basic
22522,500,000$4500%
Full
Customization
33633,600,000$67249.3%
Modified
Customization
24424,400,000$4888.4%

Looking at things in this way helped us make a rational decision on what things really cost, and helped us look at ways we could maximize our impact and minimize our costs.

P.S.  For those of you who were patient enough to read through this entire article, you can save yourself even more by removing the <speak> and </speak> tags.  These are assumed by the Watson Text To Speech service, so you can omit using them and save yourself 15 characters per message.  For the purposes of this example, that would reduce the monthly cost of each of the above approaches by $30 a month.

Advertisements

Enabling Your IBM Cloud Users To Submit Support Tickets

Every once in a while I have a customer ask me a simple question, or bring me a simple problem, and I don’t know the answer for it.  So I make sure that I understand the question, and then I go off and find out the answer for them.  Sometimes these answers are simple, and sometimes these answers SHOULD be simple, but they are not well known.  Today I ran into one of those situations, and thought it would be good to share it.

The Problem – Users Cannot Open Support Tickets on IBM Cloud

My customer is relatively new to the IBM Cloud, let’s call them the Acme Corporation.  They have a subscription with a corporate account (ibmcloudadmin@acme.com).  One of their users (Sally@acme.com) goes into the account and is doing some work with an instance of Watson Assistant.   She runs into some odd behaviour and wants to open up a support ticket.   If you look in the upper right of the browser UI, you will see that she is in the proper context (ibmcloudadmin@acme.com).  When she tries to open a ticket, it tells her that she has Basic Support, and when she attempts to submit the ticket she gets an error dialog that says, “You do not have ticket add and view permissions”.  So what do we do to fix this?

The problem is that our user (Sally@acme.com) does not have permissions to open tickets for the account in question (ibmcloudadmin@acme.com).  

Solution #1 – Add Sally to the Account With Infrastructure Access

The first solution to this is to add Sally’s account (Sally@acme.com) to the list of users with access to the ibmcloudadmin@acme.com infrastructure resources.  This needs to be done by the account administrator, who will need to be logged in as an account administrator/owner.

You can now do this by inviting Sally as a new user, using the Invite New User page (part of the normal user management on the IBM Cloud).  When you do this, you will want to be sure to give Sally a “Basic User” role for her Infrastructure access (see the picture below).

Make sure to give your user “Basic User” permissions for Infrastructure Access

This will allow Sally’s account to create support tickets while working in the context of the corporate IBM Cloud account.

Solution #2 – Change Sally’s Infrastructure Access

The other thing that can be done is to modify the Infrastructure permissions assigned for Sally’s account (Sally@acme.com).  This also needs to be done by the account administrator, who will need to be logged in as an account administrator/owner.

You can do this by following the steps outlined here.  First, start by looking at the list of tickets for the account.   

You can do this by following the steps outlined here.  First, start by looking at the list of tickets for the account.  From the Menu bar, select Support -> View Tickets.

Then you will want to get a list of the users who have access to the infrastructure available in the IBM Cloud, so select Account -> Users -> User List.

Now in that list of IBM Cloud infrastructure users, you will want to find the users that you want to give the ability to submit support tickets, and you will select Edit Portal Access.  This will take you to the IBM Cloud Infrastructure portal access dialog for that particular user.

Once the portal access dialog comes up in the browser, make sure that you have selected the correct user, and then give that user permission to add, edit, view, and search thru support tickets.  Your screen should look similar to what you see below.

Conclusion: You need to give your users the ability to submit support tickets

The IBM Cloud has a finer-grained access, role, and permission model with the move towards resource groups and IAM.  Part of this finer-grained access is the ability to allow your users to submit, view, search, and edit support tickets.  Make sure that you get your users set up correctly the first time – so you can avoid these issues in the future.

Getting Started Right on the IBM Cloud

Note: This post was initially published here as Bluemix and Watson – Getting Started Right, on this site.  That article is also on developerWorks, as Bluemix and Watson – Getting Started Right. This blog post repeats a lot of that content, with some updates for new capabilities and functionality.  It was updated on September 21, as I added links to some great content that readers pointed out to me, that I felt compelled to share.  Please forgive the duplicated content….. In my role as a Customer Success Manager, I often talk to customers who are new to the IBM Cloud platform and the Watson services.  Often I will spend a good hour or more talking to customers and answering questions about the best way to get started with the organization of the IBM Cloud environment, and some of the more operational concerns around cloud and cognitive development.  Since I find myself answering these questions over and over, for over two years, I figured it would be good to update this blog post so I can easily share this information with even MORE customers.  It also helps me remember all of the important things when I do talk on the subject.

Getting Started on IBM Cloud

Signing up for IBM Cloud is easy, all you need to do is supply a valid email ID which will be associated with the IBM Cloud account.  This is fine for YOU, but what about your ORGANIZATION?  When setting up an IBM Cloud account for your company or organization, it is best to use a functional ID (some teams call them service accounts) associated with your company.  Just keep in mind that the IBM Cloud will be sending out automated emails to this account (warnings about service usage, services being deprecated, new services available, and other things).  Make sure that you choose an email address that is monitored by someone, so you don’t miss any important notifications. I KNOW it takes a little longer to set up a functional account within your company, and that it is a pain to go through the paperwork to justify it, but not having your IBM Cloud account tied to an individual will save you A LOT of time, effort and frustration in the future.

Learning the Cloud Concepts

This is a good spot to pause for a second, and take time to understand some of the basic cloud concepts that will impact how you organize yourself on the IBM Cloud.  Looking at this picture may help:
Orgs, Spaces and Resource Groups
The box in the upper left is my IBM Cloud Account, and this should be associated with a functional ID.  This is the functional ID and email address where you will receive communications from the IBM Cloud team.  It is also the Account which has access to EVERYTHING underneath it. Next, there are two blue boxes for the Organizations, which represent different functional organizations (or projects) within the larger IBM Cloud Account.  Even though the different Organizations reside in the same parent Account, the resources within those Organizations can have their visibility limited to only resources within that Organization. At this same conceptual level are the two Resource Groups.  These Resource Groups serve as the “home” for a variety of different infrastructure and IBM Cloud services.  Access to these Resource Groups can be given on an individual basis, or can be assigned through the use of Access Groups.  If you want to learn more about Resource Groups, Access Groups, and the IBM Access Management (IAM) system, read this great post on Getting Started with IBM Cloud IAM. The ovals below each Organization represent Spaces.  A Space can be thought of as an individual development environment, or development area, for the development of a cognitive application.  Access to these environments is controlled by assigning people one of three roles at the Organization level, and one of three roles at the Space level. Finally, at the base level, are the various IBM Cloud Services, which are deployed into either spaces or resource groups.  These can be Watson based services (like the Watson Assistant service), infrastructure services (like the Virtual Server service), or other IBM Cloud capabilities (like the Weather or Blockchain services).  Services essentially “live” within a particular Space or Resource Group, and the charges associated with a service are calculated and billed against the IBM Cloud Account.  These charges are broken down by Organization/Space or Resource Group, for your own internal billing and tracking purposes. The IBM Cloud documentation on Account Hierarchies does a great job describing all of the above and has a great diagram describing the relationships between all of these things.

Getting Organized on the IBM Cloud Platform

We’ll assume that you took my advice, and have a generic corporate ID (like IBMcloud@acme.com) that is your IBM Cloud account.  Now let’s get organized to support the development efforts of your organization on the IBM Cloud platform.  I’ll give you a “default” setup that you should use as the basis for your strategy in setting up the IBM Cloud Platform, and I’ll indicate options you have and why you might use those options. I do this from the command line, using the IBM Cloud command line interface (CLI).  I heavily use the IBM Cloud CLI in my tool referenced by the Administering Your IBM Cloud Account article.  For the purposes of this article, we want to create some new Organizations within our account, and we also want to create some new Resource Groups.  Some of our projects may be hybrid projects, and may use services supported by both models – so we need to have associated areas.  In this case, we’ll create working areas (which include organizations/spaces and resource groups) for the Tau project and the Epsilon project. Note: We introduce some naming conventions here that are important to note.  Here are our naming conventions:
  • Organizations -> <project_name>_ORG
    • For our example, the three organizations our three projects would be: Tau_ORG and Epsilon_ORG
  • Spaces -> <project_name>_<environment>_SPC
    • For our example, the Tau_ORG might have spaces called Tau_Dev_SPC and Tau_Test_SPC.
  • Resource Groups -> <project_name>_<environment>_RG
    • For our example, the Tau_ORG would have the following resource groups associated with the spaces above: Tau_Dev_RG and Tau_Test_RG.
  • Access Groups -> <project_name>_<environment>_<usertype>_AG
    • For our example, we might create the following access groups for our Tau_Dev_RG: Tau_Dev_Admins_AG, Tau_Dev_Managers_AG, Tau_Dev_Develop_AG
Here is what our command line looks like:
# Set the IBM Cloud API endpoint
ibmcloud api api.ng.bluemix.net
#
# Login to IBM Cloud, with -u username -p password
ibmcloud login -u IBMCloudAcct@acme.com -p "mypassword"
#
# Create new Organizations for Tau, Epsilon and Phi
ibmcloud iam org-create Tau_ORG
ibmcloud iam org-create Epsilon_ORG
#
# Print a list of all of the Organizations for this Account
ibmcloud iam orgs
The first command sets the API endpoint.  The second command gets you logged in.  The next two commands create your new organizations, and the last command lists all the IBM Cloud Organizations for your account.  Check out the Getting Started with the IBM Cloud CLI documentation for more details on the various CLI tools and commands available to you.  You will now see a listing of all of your new Organizations, along with your default organization. Why set up these separate organizations on the IBM Cloud Platform?  For a few reasons:
  1. It helps keep the work organized and segregated.  With these in place, and once we get spaces set up, we can limit user access to only certain areas.
  2. It allows us to see usage and billing details by organization.  I can see exactly what each organization (or project) is using, and how much of my total bill each project is responsible for.
Also keep in mind that these IBM Cloud Organizations are just a way to organize your IBM Cloud infrastructure.  After reading the guide, you may decide to go with IBM Cloud Organizations that break up areas based on the roles of your users, the environments being supported, or specific lines of business within your organization.

Getting a Subscription

To this point in our setup of the IBM Cloud for the fictional Acme Corporation, we have assumed that you have used a “free”, or “trial”, tier of services.  These give you enough resources to figure out what you would like to do on the IBM Cloud and allow you to build small demo applications based on the IBM Watson services. Now since we are talking about doing “real” application development work, we’re going to need more capability than these trial and free versions of the services provide.  That means that you will need to spend some amount of money to host these various services and capabilities.  Check out this nice overview of the various IBM Cloud Account Types to get an idea of your options.  The capabilities and limitations of these options differ, so it is important that you understand them. For our fictional Acme Corporation, we have decided to go with the popular subscription option.  To begin the process of getting a subscription, you will want to follow the directions on how to obtain a subscription on the account types page.  Make sure that your subscription is associated with the right IBM Cloud account (the one with the functional ID, remember that?).  Once you get your subscription, you will get a “code” that needs to be entered – make sure that you enter it in while logged into your Functional ID (we’re using something like IBMcloud@acme.com). Now that we have our organizations all set up, and our subscriptions associated with the correct projects within the fictional Acme Corporation, it’s time to set things up for our development teams.  I understand that some readers are working for smaller, more nimble organizations.  You may be fine with doing things in an ad-hoc manner, and that may actually be the best working model for you.  What follows is a broad outline of what larger organizations, seeking a better separation of duties, control of environments, and implementation of a more standard software development lifecycle and DevOps culture will want. We begin with our IBM Cloud Account and our IBM Cloud Organizations.  For our purposes here, we will focus on just one of our fictional Acme projects, the Tau project.  You begin by creating Spaces for each project.  To begin, you need to find out what environments each project will need.  Most will need a development environment, a test environment, and a production environment (at a minimum).  Some projects will want support for additional environments. For each project, go out and create the appropriate Spaces for the project.  Spaces need to have unique names.  For the Acme Tau project, we create support for two initial environments  – called Tau_Dev_SPC and Tau_Test_SPC.  We realize that we will need to build a production environment, but we’re not quite ready for that yet, so we can do that later. Now lets use the IBM Cloud CLI to go and create the spaces that we need (Note: This can also be done interactively from the IBM Cloud UI – but I know that you are interested in automating this, so I will focus on showing a script using the CLI): This results in an environment within the account which looks like the diagram below.
# Set the IBM Cloud API endpoint
ibmcloud api api.ng.bluemix.net
#
# Login to IBM Cloud, with -u username -p password -o organization
ibmcloud login -u IBMCloudAcct@acme.com -p "mypassword" -o Tau_ORG
#
# Create new Spaces within the Tau_ORG organization, 
# and within the Epsilon_ORG organization
ibmcloud iam space-create Tau_Dev_SPC -o Tau_ORG
ibmcloud iam space-create Tau_Test_SPC -o Tau_ORG
ibmcloud iam space-create Epsilon_Dev_SPC -o Epsilon_ORG
#
# Create related Resource Groups for Tau_ORG and Epsilon_ORG
ibmcloud resource group-create Tau_Dev_RG "Trial Quota"
ibmcloud resource group-create Tau_Test_RG "Trial Quota"
ibmcloud resource group-create Epsilon_Dev_RG "Trial Quota"
You can double-check your results either from the IBM Cloud CLI  (using the “ibmcloud iam spaces” and “ibmcloud resource groups” commands), or from the IBM Cloud dashboard.  This merely sets up the structure for your teams, the individual teams will need to populate these spaces and resource groups with services to make them useful.  As an IBM Cloud administrator, you should not have to do this.  The creation of any needed services should be left to the individual project teams.

Getting Everyone Else Onboard

So now that you have your projects and environments all set up, you are ready to go.  All that you have to do now is get your developers and other stakeholders into the environment.  The first step is to get your staff to register for the IBM Cloud and to get their own accounts.  I STRONGLY suggest that your force them to register with their company email addresses.  It will make things easier for you to administer things as time goes on. Note: Another reason that I STRONGLY suggest that your force users to register with their company email addresses is that most of my customers decide to federate their identities – meaning that they then will log in and authenticate with their corporate credentials (john.doe@acme.com, with the Acme password) for the IBM Cloud.  It allows you to do all sorts of great things – for more details see my GitHub page on IBM Cloud Onboarding. After your team has registered, you can invite them to the correct IBM Cloud Spaces and IBM Cloud Organizations.  So once again we will use  the IBM Cloud CLIand send the invitations that we need (Note: This can also be done interactively from the IBM Cloud UI – but I know that you are interested in automating this, so I will focus on showing a script using the CLI): This gets your users added to the proper organizations and spaces.  Some important things to keep in mind when assigning organization roles and space roles to your users.
  • Give everyone OrgAuditor for their ORG_ROLE.  This allows them to see what is going on at the organization level, but doesn’t allow them to change anything.  The only exceptions to this are the people who do administration of the organization (your IBM Cloud Administrator), and handle the billing associated with the organization (often the IBM Cloud Administrator as well).
  • Space roles are a bit different.
    • SpaceManager: This role can invite and manage users, and enable features for a given space.  You may want to give this to the development lead for the Development spaces, the test lead for the Test spaces, and your operations leader for the Production spaces.
    • SpaceDeveloper: This role can create and manage apps and services, and see logs and reports.  Give this role to developers in the Development spaces, but you probably don’t want anyone with this role in the Testing and Production spaces.  This allows you to maintain the stability of these environments (no code changes).
    • SpaceAuditor: This role can view logs, reports, and settings for the space.  This is used for people who might be interested in the development efforts in a given space – but not requiring access to change anything within the space.
# Set the IBM Cloud API endpoint
ibmcloud api api.ng.bluemix.net
#
# Login to IBM Cloud, with -u username -p password -o organization
ibmcloud login -u IBMCloudAcct@acme.com -p "mypassword" -o Tau_ORG
#
# Send invitations to the various team members within the Tau project
# arguments are: USER_NAME ORG_NAME ORG_ROLE SPACE_NAME SPACE_ROLE
ibmcloud iam account-user-invite john_smith@acme.com Tau_ORG OrgAuditor Tau_Dev_SPC SpaceDeveloper
ibmcloud iam account-user-invite jane_doe@acme.com Tau_ORG OrgAuditor Tau_Dev_SPC SpaceDeveloper
ibmcloud iam account-user-invite tom_tester@acme.com Tau_ORG OrgAuditor Tau_Test_SPC SpaceDeveloper
#
# Create access groups, and then assign team members
# to various access groups
ibmcloud iam access-group-create Tau_Dev_Admins_AG -d "Tau Dev environment administrators"
ibmcloud iam access group-user-add Tau_Dev_Admins_AG john_smith@acme.com
#
ibmcloud iam access-group-create Tau_Dev_Devs_AG -d "Tau Dev environment developers"
ibmcloud iam access group-user-add Tau_Dev_Devs_AG john_smith@acme.com
ibmcloud iam access group-user-add Tau_Dev_Devs_AG jane_doe@acme.com
#
# do the rest of your projects and users in a similar manner
...
You will probably want to automate this, or provide a self-service capability, so people can easily request access to the environments that they need.  If you’re not sure where to start with this, then go and read my post on Administering Your IBM Cloud Account – A script to help, it has a link to a GitHub repo with some Python scripting that will do some this for you.

Herding the Cats

At this point we have our IBM Cloud environments set up, and we have our users added into the environment where they need to be, with the access that they require.  Our fictional Acme environment looks a little something like this:
You’ll note that we now are showing the infrastructure resources (like virtual machines, file storage, block storage, object storage, etc.) being deployed from the IaaS account which is linked to your IBM Cloud account.  We haven’t spend a lot of time focused on this, as these resources are typically created on an as-needed basis – and they have a different set of concerns and networking and access concerns. Each of our projects now has the ability to control access to their IBM Cloud areas, and can effectively isolate their various environments.  Each is free to use whatever promotion process and DevOps tooling they prefer, or they can utilize the IBM Cloud Toolchains and the IBM Cloud Continuous Delivery service.  I strongly suggest that you use a DevOps solution which addresses ALL parts of your development process, regardless of where they reside. There are some important things to consider when thinking about the topic of DevOps in conjunction with the IBM Cloud and the IBM Watson services.
  • Watson services where you do not provide any training data (Document Conversion, Language Translator, Personality Insights, Tone Analyzer, Text-to-Speech (TTS) and Speech-to-Text (STT) ), can be deployed to additional Spaces (or Resource Groups) and do not need to have their data migrated from one environment to the next.  Keep in mind that on occasion the versions of the API and underlying services may increment, so you will want to control which API endpoints are used for these services.  (Note: If you have customized either Text-to-Speech or Speech-to-Text, then these services will need to be migrated from environment to environment.)
  • For Watson services with training data (Conversation, Discovery, Natural Language Classifier (NLC), Retrieve & Rank (R&R), Natural Language Understanding (NLU), and Visual Recognition), you will need to deploy these to environments and migrate their data.  There is currently no mechanism in place to copy service instances from one Space (or Resource Group) to another.  This has some very real implications beyond the operational aspects.
    • For services like Conversation which support an export/import functionality, it means that you can choose to do your training through the REST API, or interactively through the tooling provided on the IBM Cloud.  When you are ready to move an instance from one environment to another, you export from one environment, and then create a new instance in the target environment, and import your data.
    • For services like Discovery, which do not have an export/import capability, you should resist the temptation to train “by hand”, and instead have a script (or series of scripts) that can be used to train your system.  In this way, you can more easily create/recreate an instance of the service.

Conclusion

This should give you enough of an overview, and offer you links to enough information, so that you can begin using the IBM Cloud platform to develop cognitive applications with confidence.  Technology changes rapidly, so if you see problems with this basic approach, or have some best practices to share, please reach out to me.

Good Reference Materials

  • IBM Cloud command line interface (CLI) – The reference page for the IBM Cloud specific CLI.  Please use the IBM Cloud commands and not the Cloud Foundry equivalents, to ensure smooth operation of your Cloud environments.
  • IBM Cloud Onboarding Landing Page – a community maintained landing page with links to proven articles and technical content that is useful for teams just getting into the IBM Cloud.
  •  Administering Your IBM Cloud Account – A script to help – has a link to a GitHub repo with some Python scripting that will do some basic administrative operations for you.  Also logs the commands so you can HOW to do things from the IBM Cloud CLI.

IBM Cloud Content, Charges, and Delivery for Developer Teams

Note: I updated this a day or two after the original posting, as people sent me some good links to other resources that I wanted to share.

I have been getting this question constantly for the past month, and I have to do a presentation on it for one of my customers, so I figured that it is probably a good topic to share with a wider audience.  I am going to talk about how IBM Cloud customers can organize, manage, and use the IBM Cloud to develop applications and services, which they then can deliver to their customers.

First the Basics

First we need to cover the basics.  I have discussed the basic organization of an IBM Cloud account in my earlier post, “Bluemix and Watson – Getting Started Right” (Note that the IBM Cloud used to be called “Bluemix”).  In that article, I show you the basic organization of an IBM Cloud Account, it’s Organizations, and the Spaces underneath those organizations.  Most of our customers will organize their Accounts/Organizations/Spaces along the lines shown in Figure 1.

Figure 1 – A Typical Enterprise Account Structure

Note that right now there is no support for the concept of an Enterprise Account (or a parent of multiple IBM Cloud accounts), but when that capability DOES become available, I would see it being used as shown in the diagram above.  Now let’s look at what happens when you begin a project.

Launching a Project

When launching a project, you need to determine a few different things.  The most important piece is to figure out what KIND of a project you have.  I will divide projects into 4 major categories for the purposes of this conversation, and they are:

  • Internal Projects – projects that are done by your software development teams, and provide systems/applications for your organization.  This includes internal POCs, and other “exploratory” and “innovation” work.
  • Product Projects – projects that are done by your software development teams, that provide systems/applications that you market and sell as a product.  These products/services are then exposed or delivered to your customers.
  • Hosted Projects – projects that are done by your software development teams, that provide systems/applications that you host and maintain for a single customer.  This may also include products where you host unique copies (instances) for different customers.  Think of your favorite SaaS product.
  • Turnkey Projects – projects that are done by your software development teams, that provide systems/applications that you finish development on, and then deliver to your customer.

These project types are all going to require slightly different deployment and work environments.  The setup for each of these is based on the type of project, and the way that you need to react to and handle a couple of basic limitations that you need to be aware of.

The first limitation we will call the Billing Endpoint limitation.  It’s pretty simple, the bill for your Cloud services goes to the account owner – and nobody else.  So you need to be aware of the charges to any given account, how you will handle those charges (what one entity will pay for them), and how you will pass those charges along to your internal and external customers.

The second limitation is the Resource Portability limitation.  This one is pretty simple too.  You cannot “move” or “relocate” a service from one organization/space to a different organization/space in the IBM Cloud.  In order to move something from one environment to another, you need to recreate that service in the new environment in the same way that you did in the first environment.  This forces us to be disciplined in our software development – and brings us to our next section.

Importance of DevOps Tooling

The resource portability limitation means that we need to be able to recreate any cloud resource instance in some type of automated manner, in any environment we choose.  This demands a solid change management strategy, and solid DevOps tooling that can create the services and applications in your various IBM Cloud environments.

One way to do this is to use the DevOps Toolchains that are available on the IBM Cloud.  These toolchains are easy to use.  You can customize them to use tools that are “Cloud native” on the IBM Cloud, or you can use your own tools and processes.

A healthy Cloud development and deployment environment is strongly dependent on the DevOps environment that is supporting it.  Tools, standards, and automation can help development teams follow better engineering practices, and can help you deliver projects and products more quickly.  If you’re unfamiliar with DevOps, I suggest you Google it and read some of the stuff from Gene Kim, Sanjeev Sharma, Mik Kersten or Eric Minnick.

So keep in mind that setting up a DevOps framework and some administrative automation for your Cloud should be one of the first things that you do.  Investments supporting the automation of projects will pay huge dividends, and allow your teams to easily launch, execute, and retire projects in your Cloud environment.

Handling Projects

So now that I have convinced you that you need to invest some time and effort building up your DevOps capabilities on the Cloud, let’s get back to the main question of this blog post.  “How do I organize projects and content, and handle the financial aspects for these projects?”.

Handling Internal Projects

Internal projects are organized in the same way that I discuss in my earlier post, “Bluemix and Watson – Getting Started Right“.  The account level is a subscription owned by the Enterprise, and projects are run as Organizations in the Account, and the Spaces under those organizations represent the various different environments supported by a project (like development, test, QA, staging, production, etc.).

Figure 2 – Internal Project Organization

This project is going to be developed internally, and it will be deployed internally.  So our need to separate the “billing” is only from an internal perspective.  Since we can see billing at the organization and space levels (see Administering Your IBM Cloud Account – A Script to Help), it should be relatively simple to determine any chargebacks that you might want to do with your internal costs.

You’ll use the DevOps capabilities we discussed earlier to quickly establish the automation of continuous integration/continuous delivery (CI/CD) capabilities.  Teams can do development and can set up pipelines to deliver their project to staging environments, where operations teams can test the deployment and deliver to production environments.  This environment is straightforward and simple because we don’t need to worry about billing issues, and we don’t need to worry about visibility or ownership issues.  Things get more interesting with our other project types.

Handling Product Projects and Hosted Projects

Product and Hosted projects are organized in the same way, even though they are slightly different types of situations.  In these cases I recommend the use of a second IBM Cloud account.  The established Enterprise account that is already established (as described in the section on Internal Projects), should continue to be used for the development of this project.  This project is going to be developed internally, and we will track costs internally.  So our need to separate the “billing” from an internal perspective.  Since we can see billing at the organization and space levels, it should be relatively simple to determine any chargebacks that you need to do for this project.

Figure 3 – Hosted Project and Product Project Organization

You will still have development and test spaces for your project, but you will NOT have production, pre-production or staging areas.  You will use your DevOps capabilities to deliver your project to a different account/organization/space.

In the case of a product that you are hosting for general usage, you will deploy to a specific IBM Cloud account that has been created for the express purpose of hosting production versions of your product (or for the deployment of Kubernetes clusters that are running your product).  Your operations team will have access to this account, and these production environments, ensuring a separation of duties for your deployed products.

In the case of a product that you are hosting for usage by a specific customer, you will deploy to a specific IBM Cloud account that has been created for the express purpose of hosting production applications for that customer. Your operations team will have access to this account, ensuring a well defined set of duties for your hosted products.  This approach also allows you to easily collect any billing information for your customer.

Handling Turnkey Projects

Turnkey projects are organized almost exactly the same way as a hosted project, with two simple differences.  Just like a hosted project, you will need to create a new IBM Cloud account for the work being delivered.

Figure 4 – Turnkey Project Organization

The first big difference is that you are going to either have your customer own the new IBM Cloud account from it’s creation, or transfer ownership of the account to your customer.  Make sure that you are clear about who owns (and pays for) the various environments, and the timing of any account reassignment.

The second difference is that the new account may have more than just production spaces – since your customer will need development and test spaces to be able to do maintenance and further development of the application or system being delivered.

Things To Remember

Now that we’ve covered how to organize content and environments for project delivery, it’s time to remind you about some of key details that you will need to remember to make sure that your IBM Cloud development efforts go as smoothly as possible.

  • Make sure that you have a solid DevOps strategy.  This is key to being able to deliver project assets to specific environments.
  • Make sure that you have solid naming conventions and Administrative procedures.  These should be automated as much as possible (just like your DevOps support for development).  For some guidance on setting up roles and DevOps pipelines, check out some of these best practices for organizing users, teams and applications.
  • Know how you will set up your project – since this will have an impact on the contracts and costing that you have for your IBM Cloud hosted project.

 

Administering Your IBM Cloud Account – A script to help

Note: This post has been edited and updated multiple times, and the most recent and accurate copy of this post can be found on the IBM developerWorks website, in a blog post titled, Administering Your IBM Cloud Account – A script to help.

As many of you know, if I have to do something more than two or three times, I tend to put in some effort to script it.  I know a lot of what I can do on the command line with the IBM Cloud, but I don’t always remember the exact syntax for all of those bx command line commands.  I also like to have something that I can call from the command line, so I can just script up common administrative scenarios.

Other Options

There are some options which already exist out there.  I wasn’t aware of some of them, and none of them allow for scripting access.  One of the best that I have seen is the interactive application discussed in the blog post on Real-Time Billing Insights From Your IBM Cloud Account, written by Maria Borbones Garcia.  Her Billing Insights app is already deployed out on Bluemix.  It’s nice –  suggest you go and try it out.  She also points you to her mybilling project on GitHub, which means that you can download and deploy this app for yourself (and even contribute to the project).  Another project that I have seen is the My Console project, which will show a different view of your IBM Cloud account.

Why Create a Script?

This all came home to me this past week as I began to administer a series of accounts associated with a Beta effort at IBM (which I’ll probably expand upon once the closed beta is complete).  I have 20 different IBM Cloud accounts, and I need to manage the billing, users, and policies for each of these accounts.  I can do it all from the console, but that can take time, and I can make mistakes.  The other thing that I thought of was that I often get questions from our customers about, “How do I track what my users are using, and what our current bill is?”.  So that led me to begin writing up a Python script that would allow you to quickly and easily do these types of things.

So I began to develop the IBM_Cloud_Admin tool, which you can see the code for in its GitHub repository.  Go ahead and download a copy of it from GitHub.  This is a simple Python script, and it just executes a bunch of IBM Cloud CLI commands for you.  If you go through a session and then look at your logfile, you can see all the specific command line commands issued, and see the resulting output from those commands.  This allows you to do things in this tool, and then quickly look in the log file and strip out the commands that YOU need for your own scripts.

How To Use The Script

To run the script, you can just type in:

python IBM_Cloud_Admin.py -t apiKey.json

The script has a few different modes it can run in.

  • If you use the -t flag, it will use an API Key file, which you can get from your IBM Cloud account, to log into the IBM Cloud.  This is the way that I like to use it.
  • If you don’t use the -t flag, you’ll need to supply a username and password for your IBM Cloud account using the -u and -p flags.
  • If you use the -b flag (for billing information), then you will run in batch mode.  This will get billing information for the account being logged into, and then quit.  You can use this mode in a script, since it does not require any user input.
  • If you don’t use the -b flag (for billing information), then you will run in interactive mode.  This will display menus on the command line that you can choose from.

The Output Files

There are a number of output files from this tool.  There is the IBM_Cloud_Admin.output.log file, which contains a log of your session and will show you the IBM Cloud command line commands issued by the tool, and the responses returned.  This is a good way to get familiar with the IBM Cloud command line commands, so you can use them in custom scripts for your own use. 

You may also see files with names like, MyProj_billing _summary.csv and MyProj_billing _by_org.json.  These are billing reports that you generated from the tool.  Here is a list of the reports, and what they contain.

  • MyProj_billing _summary.csv – this CSV file contains billing summary data for your account for the current month.
  • MyProj_billing _summary.json – this JSON file contains billing summary data for your account for the current month.  It shows the raw JSON output from the IBM Cloud CLI.
  • MyProj_billing _by_org.csv – this CSV file contains billing details data for your account, split out by org and space, for the current month.
  • MyProj_billing _by_org.json – this JSON file contains billing details data for your account, split out by org and space, for the current month.  It shows the raw JSON output from the IBM Cloud CLI.
  • MyProj_annual_billing _summary.csv – this CSV file contains billing summary data for your account for the past year.
  • MyProj_annual_billing _summary.json – this JSON file contains billing summary data for your account for the past year.  It shows the raw JSON output from the IBM Cloud CLI.
  • MyProj_annual_billing _by_org.csv – this CSV file contains billing details data for your account, split out by org and space, for the past year.
  • MyProj_annual_billing _by_org.json – this JSON file contains billing details data for your account, split out by org and space, for the past year.  It shows the raw JSON output from the IBM Cloud CLI.

Use the JSON output files as inputs to further processing that you might want to do of your IBM Cloud usage data.  The CSV files can be used as inputs to spreadsheets and pivot tables that you can build that will show you details on usage from an account perspective, as well as from an organization and space perspective.

Getting Your Api Key File

I’ve mentioned the API key file a couple of times here.  If you are not familiar with what an API Key file is, then you’ll want to read this section.  An API Key is a small text file which contains some JSON based information, which when used properly with the IBM Cloud command line tool, will allow anyone to log into the IBM Cloud environment as a particular user, without having to supply a password.  The API Key file is your combined username/password.  Because of this, do NOT share API keyfiles with others, and you should rotate your API Key files periodically, just in case your keyfile has become compromised.

Getting an API Key on IBM Cloud is really easy.

  • Log into the IBM Cloud, and navigate to your account settings in the upper right hand corner of the IBM Cloud in your web browser. Select Manage > Security > Platform API Keys.
  • Click on the blue Create button.
  • In the resulting dialog, select a name for your API Key (something that will tell you which IBM Cloud account the key is associated with), give a short description, and hit the blue Create button.
  • You should now see a page indicating that your API Key has been successfully created. If not, then start over again from the beginning. If you have successfully created an API Key, download it to your machine, and store it somewhere secure.

Note: A quick note on API Keys. For security reasons, I suggest that you periodically destroy API Keys and re-create them (commonly called rotating your API keys or access tokens). Then if someone had access to your data by having one of your API keys, they will lose this access.

Other Tasks

Do you have other administrative tasks that you would like to see the tool handle?  Find a bug?  Want to help improve the tool by building a nice interface for it?  Just contact me through the GitHub repository, join the project, and add issues for problems, bugs, and enhancement requests.

A Final Thought

This script is a quick hacked together Python script – nothing more and nothing less.  The code isn’t pretty, and there are better ways to do some of the things that I have done here – but I was focused on getting something working quickly, and not on efficiency or Python coding best practices.  I would not expect anyone to base their entire IBM Cloud administration on this tool – but it’s handy to use if you need something quick, and cannot remember those IBM Cloud command line commands.

Deploying Production Cloud Applications – A Readiness Checklist

I just had a conversation today with my VP (Rob Sauerwalt – check him out on Twitter – time to do some shameless kissing up to my management team) about a recent internal communication that we both saw.  It was someone looking for a “readiness checklist” for the deployment of an application on the IBM Cloud.  Rob and I both agreed that this seems pretty simple, and we came up with a quick checklist of things to consider.

Now this list is not specific to the IBM Cloud, it’s pretty generic.  It’s just a quick checklist of things that you will want to make sure that you have considered, BEFORE you deploy that cloud based application into a production environment.  I am an Agile believer, so I would suggest that you address these checklist items in the SPIRIT of what they are trying to do, and that you should do what makes sense.  This means that each one of these areas does not need to represent some 59 page piece of documentation.  What you want to do is provide enough information so the poor guy who takes your job after you get promoted, is able to be effective and understand and maintain the application or system.

If you have suggestions about other things that should be on this list, please drop me a line and let me know.  I would love to add them to the list, and make this generic deployment readiness checklist even better.

Production Readiness Checklist

The Basics

⊗ Name and General Description of the Application – this includes the purpose of the application and the number of users that are anticipated to use the application.  Also have an idea of the types of users.  Is it for the general public?  Only for certain roles within our organization?  Is it only for your customers?  Do this in two to three paragraphs – anything more is adding complexity.

⊗ Description of Needed Software/Hardware/Cloud Resources – a list of the needed software packages, and the clou resources needed to run the application.  Do you use third party utilities or libraries?  Do you run on Cloud Foundry buildpacks?  Virtual machines?  Do you use Cloud services for database resources?  Often a high level architectural diagram is useful to help other people understand the system at a high level.  This should be done AS you build – so you can simplify things.  Are your developers using different libraries to accomplish the same thing?  Get them to standardize.  Reduce your dependencies, reduce your complexity, and you improve your software quality.

DevOps Considerations

⊗ Operating Systems and Patching Requirements – do you have specific OS requirements?  Do you require a particular framework to run properly (like .NET, Eclipse, or a particular Cloud Foundry buildpack)?  What OS versions have you tested and validated this application with – and do all of your components need to be on the same OS version?  This becomes important when fixes get deployed to containers, virtual machines get upgraded, and maintenance activities are done.

⊗ Installation and Configuration Guidelines – you should be deploying your application in some automated manner.  So your deployment and promotion scripts should be the only guide that you need…… except when they aren’t.  Take the time and DOCUMENT those scripts – explain WHAT you are doing and WHY, so your application can easily be reconfigured or deployed in different ways in the future.

⊗ Back-up, Data Retention and Data Archiving Policies – let your operations people know what data needs to be archived and retained.  How often do systems need to be backed up?  How will services be restored in the event of a crash?  Explain WHERE and HOW data needs to be retained.  Explain what your DEVELOPMENT teams need to review on a periodic basis.  This can be the biggest headache for development teams, because these are often scenarios that they have not considered.  Backup plans are not sufficient, they need to be executed at least once before you go into production – so you are sure that they are valid and that they work.

⊗ Monitoring and Systems Management – This includes runbooks – what do we need to do while the application is running?  Do we need to take the logs off of the system every day and archive them?  Or do we just let logs and error reports build up until the system crashes?  Should I monitor memory and heap usage on a daily basis?  Should I be monitoring CPU load?  Who do I notify if I see a problem, and what is a “problem”?  (CPU at 50%? CPU running at 20% with spikes to 100%?)  How will this application normally be supported?  You may not have complete information and definition of “problems” when you begin, bu define what you can and acknowledge that things will change as time goes on.

⊗ Incident Management – This details how you react to application incidents.  These could be bugs, outages, or both.  In the case of an outage, who needs to be called, and what actions should they take to collect needed data, and to get the application back up and running.  What logs are needed, what kind of data will aid in debugging issues?  Who is responsible for application uptime TODAY (get things back on track and running), and who is responsible for application uptime TOMORROW (who needs to find root cause, fix bugs, make design changes if needed, etc.).

⊗ Service Level Documentation -This is the “contract” between you and your customers.  How often will your application be down for maintenance?  If your application is down, how long before it comes back up?  Are there any billing or legal ramifications from a loss of service?  Do your customers get refunds – or cash back – when your Cloud application is unavailable?

⊗ Extra Credit – DevOps pipeline – you need to have an automated pipeline for the deployment of code changes into well defined development, test, and production environments.  You need to have a solid set of policies and procedures for the initiation and automation of these deployments.  Who has authority to deliver to test environments?  Production environments?

Software Architecture Considerations

⊗ Key Support & Maintenance Items – the team that built this thing knows where the weak spots are – share that knowledge!  Where does the team know that “tech debt” exists – and how is that impacting your application?  This information will help the teams maintaining and upgrading your application.  They will be able to do this with knowledge about how the application works, and why certain architectural choices were made.

⊗ Security Plan – Everyone is worried about the security of their applications and data on the cloud.  You need to be sensitve to this when deploying cloud based applications.  Your stakeholders and users will want to know that you have considered security, and that you are protecting their data from being exposed, stolen, or used without their knowledge/consent.

⊗ Application Design – This should include some high level description of your use case, a simple flowchart and dependencies.  Give enough detail so someone can easily get started in maintaining your application code, but not so much detail that you waste time and ultimately end up with documentation that does not match the code.

Is That Everything?

That’s not everything, but it is a good minimal list of things that you should have considered and/or documented.  Most applications need some sort of a support plan – who handles incoming problem tickets from customers?  Do you have a support process for your end users?  In your own environments and business context, you may have other things that need to be added to this list.  Do you need to check for compliance with some standard or regulation?  What are your policies for using Open Source software?

So this list is not meant to be exhaustive – but it is designed to make you think, and to help you ensure higher quality when deploying your Cloud applications.

Happy Holidays for 2017

With the end of the year quickly approaching, it is a great time to look back on the past year, and to look forward in anticipation for what is coming in 2018.

2017 was an interesting year.  I saw an explosion in the development of chatbots of various different types.  Some were very simple, others used both Watson Conversation and the Watson Discovery service to provide a deeper user experience – with an ability to answer both short tail and long tail questions.  I saw a huge uptick in interest in basic Cloud approaches, and a lot of interest in containers and Kubernetes.  I expect that both of these trends will continue into 2018.

In 2018 I expect to see the IBM Cloud mature and expand in it’s ability to quickly provide development, test and production computing environments for our customers.  I also expect that more people will become interested in hybrid cloud approaches, and will want to understand some best practices for managing these complex environments.  I am also excited about some of the excellent cognitive projects that I have seen which could soon be in production for our customers.  I also expect that some of our more advanced customers will be looking at how cognitive technologies can be incorporated into their current DevOps processes, and how these processes can be expanded into the cloud.

I hope that your 2017 was a good one, and I hope that you have a happy and safe holiday season.