In my role as a Watson and Cloud Adoption Manager, I often talk to customers who are new to the IBM Cloud platform (Bluemix) and the Watson services. Often I will spend a good hour or more talking to customers and answering questions about the best way to get started with the organization of the Bluemix environment, and some of the more operational concerns around cloud and cognitive development. Since I find myself answering these questions over and over, I figured it would be good to get this down in a blog post so I could easily share this information with MORE customers. It also helps me remember all the important things when I do talk on the subject.
Getting Started on Bluemix
Signing up for Bluemix is easy, all you need to do is supply a valid email ID which will be associated with the Bluemix account. This is fine for YOU, but what about your ORGANIZATION? When setting up a Bluemix account for your company or organization, it is best to use a functional ID associated with your company. Just keep in mind that the IBM Cloud will be sending out automated emails to this account (warnings about service usage, services being deprecated, new services available, and other things). So you will want to make sure that you choose an email address that is monitored by someone, so you don’t miss any important notifications.
I KNOW it takes a little longer to set up a functional account within your company, and that it is a pain to go through the paperwork to justify it, but not having your Bluemix account tied to an individual will save you A LOT of time, effort and frustration in the future.
Learning the Cloud Concepts
This is a good spot to pause for a second, and take time to understand some of the basic cloud concepts that will impact how you organize yourself on the IBM Cloud. Looking at this picture may help:
The outermost grey box is my Bluemix Account, and this should be associated with a functional ID. This is the functional ID and email address where you will receive communications from the IBM Cloud team. It is also the account which has access to EVERYTHING underneath it.
Inside of that grey box are three blue boxes. These are the Bluemix Organizations, which represent different functional organizations (or projects) within the larger Bluemix Account. Even though the different organizations reside in the same parent Bluemix Account, the resources within those organizations can be limited to visibility to only resources within that Bluemix Organization.
The green boxes inside of each Bluemix Organization represent Bluemix Spaces. A space can be thought of as an individual development environment, or development area, for the development of a cognitive application.
Finally, at the lowest level, are the various Bluemix Services, indicated with the orange circles. These can be Watson based services (like the Watson Conversation service), infrastructure services (like the OpenWhisk service), or other Bluemix capabilities (like the Weather or Blockchain services). Services essentially “live” within a particular space, and the charges associated with a service are calculated and billed against the Bluemix Account. These charges are broken down by Bluemix Organization, for your own internal billing and tracking purposes.
Getting Organized on the IBM Cloud Platform
We’ll assume that you took my advice, and have a generic corporate ID (like IBMcloud@acme.com) that is your Bluemix account. Now let’s get organized to support the development efforts of your organization on the IBM Cloud platform. I’m also going to assume that you have read the great guide to getting organized on the IBM Cloud Platform. It’s good, and it has more detail than what I cover here. It also doesn’t tell you how YOU should set up YOUR IBM Cloud Platform. I’ll give you a “default” setup that you should use as the basis for your strategy in setting up the IBM Cloud Platform, and I’ll indicate options you have and why you might use those options.
We’ll start with our Bluemix account for our fictional Acme Corporation (for this example, we’re using IBMcloud@acme.com). The first thing that we need to do is to create some Bluemix Organizations for ourselves. Now my fictional company is like a lot of other companies out there, it has some divisions, and these divisions don’t often share responsibilities or development assets. So I will start by giving each of the divisions that are looking at doing cognitive development their own Bluemix Organization.
I do this from the command line, using the Bluemix command line interface (CLI). I go over some of the uses for the Bluemix CLI in Getting Bluemix Information for Support and Automation. We want to create some new Bluemix Organizations within our account, specifically we want to create them for Delivery, Sales and Marketing.
Here is what our command line looks like:
# Set the Bluemix API endpoint bluemix api https://api.ng.bluemix.net # # Login to Bluemix, with -u username -p password bluemix login -u IBMCloudAcct@acme.com -p "mypassword" # # Create new Bluemix Organizations for Delivery, Sales, and Marketing bluemix iam org-create Acme_Delivery bluemix iam org-create Acme_Sales bluemix iam org-create Acme_Marketing # # Print a list of all of the Bluemix Organizations for this Bluemix Account bluemix iam orgs
The first command sets the API endpoint. The second command gets you logged in. The next three commands create your new organizations, and the last command lists all the Bluemix Organizations for your account. Check out the documentation for Bluemix commands for more details. You will now see a listing of all of your new Bluemix Organizations, along with your default organization. Be sure to provide a unique name for your organizations, and PLEASE establish some kind of naming convention. I used <company_name>_<company_division>, you might want to do this too.
Why set up these separate organizations on the IBM Cloud Platform? For a few reasons:
- It helps keep the work organized and segregated. With these in place, and once we get spaces set up, we can limit user access to only certain areas.
- It allows us to see usage and billing details by organization. I can see exactly what each organization is using, and how much of my total bill each organization is responsible for.
Also keep in mind that these Bluemix Organizations are just a way to organize your IBM Cloud infrastructure. After reading the guide, you may decide to go with Bluemix Organizations that break up areas based on the roles of your users, or specific projects.
Getting a Subscription
To this point in our setup of the IBM Cloud for the fictional Acme Corporation, we have assumed that you have used a “free”, or “trial”, tier of services. These give you enough resources to figure out what you would like to do on the IBM Cloud, and allowed you to build small demo applications based on the IBM Watson services.
Now since we are talking about doing “real” application development work, we’re going to need more capability than these trial and free versions of the services provide. That means that you will need to spend some amount of money to host these various services and capabilities. Check out this nice overview of the various IBM Cloud Account Types to get an idea of your options. The capabilities and limitations of these options differ, so it is important that you understand them.
For our fictional Acme Corporation, we have decided to go with the popular subscription option. To begin the process of getting a subscription, you will want to follow the directions on how to obtain a subscription on the account types page. Make sure that your subscription is associated with the right Bluemix account (the one with the functional ID, remember that?).
Getting Your Development Environments Right
Now that we have our organizations all set up, and our subscriptions associated with the correct divisions within the fictional Acme Corporation, it’s time to set things up for our development teams. Now I understand that some readers are working for smaller, more nimble organizations. You may be fine with doing things in an ad-hoc manner, and that may actually be the best working model for you. What follows is a broad outline of what larger organizations, seeking a better separation of duties, control of environments, and implementation of a more standard software development lifecycle and DevOps culture will want.
We begin with our Bluemix Account and our Bluemix Organizations. For our purposes here, we will focus on just one of our fictional Acme divisions, the Delivery division. You begin by identifying the projects within the division, and you then begin to create Bluemix Spaces for each project. You should also find out what environments each project will need. Most will need a development environment, a test environment, and a production environment (at a minimum). Some projects will want support for additional environments.
For each project, go out and create the appropriate Bluemix Spaces for the project. Spaces need to have unique names. For the sake of keeping things easily identifiable, unique, and organized, I have followed a naming convention of <Project>_<Environment> (you should probably do the same). So for the Acme Delivery division, we have three projects – called Phoenix, WH, and Dizzi. Each has different services that they use (some are common), and the Dizzi project has no need for a pre-production environment.
So now we use the Bluemix CLI to go and create the spaces that we need (Note: This can also be done interactively from the Bluemix UI – but I know that you are interested in automating this, so I will focus on using the CLI):
# Set the Bluemix API endpoint bluemix api https://api.ng.bluemix.net # # Login to Bluemix, with -u username -p password -o organization bluemix login -u IBMCloudAcct@acme.com -p "mypassword" -o Acme_Delivery # # Create new Bluemix Spaces within the Acme_Delivery organization bluemix iam space-create Phoenix_Dev -o Acme_Delivery bluemix iam space-create Phoenix_Test -o Acme_Delivery bluemix iam space-create Phoenix_PreProd -o Acme_Delivery bluemix iam space-create Phoenix_Prod -o Acme_Delivery bluemix iam space-create WH_Dev -o Acme_Delivery bluemix iam space-create WH_Test -o Acme_Delivery # # do the remaining spaces in a similar manner .....
This results in a Bluemix environment within the Acme_Delivery organization which looks like the diagram below.
You can double-check your results either from the Bluemix CLI (using the “bluemix iam spaces” command), or from the IBM Cloud dashboard. You will notice that there are some small yellow circles within each newly created space. These represent the IBM Cloud services that each project will create within these spaces, in order to implement their project. As a Bluemix administrator, you should not have to do this. The creation of needed services should be left to the individual project teams.
Getting Everyone Else Onboard
So now that you have your Bluemix Organizations, projects and environments (Bluemix Spaces) all set up, you are ready to go. All that you have to do now is get your developers and other stakeholders into the environment. The first step is to get your staff to register for Bluemix and to get their own accounts. I STRONGLY suggest that your force them to register with their company email addresses. It will make things easier for you to administer things as time goes on.
After your team has registered, you can invite them to the correct Bluemix Spaces and Bluemix Organizations. So once again we will use the Bluemix CLI, and send the invitations that we need (Note: This can also be done interactively from the Bluemix UI – but I know that you are interested in automating this, so I will focus on using the CLI):
# Set the Bluemix API endpoint bluemix api https://api.ng.bluemix.net # # Login to Bluemix, with -u username -p password -o organization bluemix login -u IBMCloudAcct@acme.com -p "mypassword" -o Acme_Delivery # # Send invitations to the various team members within the Delivery division # arguments are: USER_NAME ORG_NAME ORG_ROLE SPACE_NAME SPACE_ROLE bluemix iam account-user-invite firstname.lastname@example.org Acme_Delivery OrgAuditor Phoenix_Dev SpaceDeveloper bluemix iam account-user-invite email@example.com Acme_Delivery OrgAuditor Phoenix_Dev SpaceDeveloper bluemix iam account-user-invite firstname.lastname@example.org Acme_Delivery OrgAuditor Phoenix_Test SpaceDeveloper # # do the rest of your users in a similar manner ...
This gets your users added to the proper organizations and spaces. Some important things to keep in mind when assigning organization roles and space roles to your users.
- Give everyone OrgAuditor for their ORG_ROLE. This allows them to see what is going on at the organization level, but doesn’t allow them to change anything. The only exceptions to this are the people who do administration of the organization (your Bluemix Administrator), and handle the billing associated with the organization (often the Bluemix Administrator as well).
- Space roles are a bit different.
- SpaceManager: This role can invite and manage users, and enable features for a given space. You may want to give this to the development lead for the Development spaces, the test lead for the Test spaces, and your operations leader for the Production spaces.
- SpaceDeveloper: This role can create and manage apps and services, and see logs and reports. Give this role to developers in the Development spaces, but you probably don’t want anyone with this role in the Testing and Production spaces. This allows you to maintain the stability of these environments (no code changes).
- SpaceAuditor: This role can view logs, reports, and settings for the space. This is used for people who might be interested in the development efforts in a given space – but not requiring access to change anything within the space.
You will probably want to automate this, or provide a self-service capability, so people can easily request access to the environments that they need.
Herding the Cats
At this point we have our Bluemix environments set up, and we have our users added into the Bluemix Organizations and Bluemix Spaces where they need to be, with the access that they require. Our fictional Acme environment looks a little something like this:
Each of our divisions now has the ability to control access to their Bluemix areas, and can effectively isolate their various environments. Each is free to use whatever promotion process and DevOps tooling they prefer, or they can utilize the IBM Cloud Toolchains and the IBM Cloud Continuous Delivery service. Use a DevOps solution which addresses ALL parts of your development process, regardless of where they reside.
There are some important things to consider when thinking about the topic of DevOps in conjunction with the IBM Cloud and the IBM Watson services.
- Watson services where you do not provide any training data (Document Conversion, Language Translator, Personality Insights, Tone Analyzer, Text-to-Speech (TTS) and Speech-to-Text (STT) ), can be deployed to additional Bluemix Spaces and do not need to have their data migrated from one environment to the next. Keep in mind that on occasion the versions of the API and underlying services may increment, so you will want to control which API endpoints are used for these services. (Note: If you have customized either Text-to-Speech or Speech-to-Text, then these services will need to be migrated from environment to environment.)
- For Watson services with training data (Conversation, Discovery, Natural Language Classifier (NLC), Retrieve & Rank (R&R), Natural Language Understanding (NLU), and Visual Recognition), you will need to deploy these to environments and migrate their data. There is currently no mechanism in place to copy service instances from one space to another. This has some very real implications beyond the operational aspects.
- For services like Conversation which support an export/import functionality, it means that you can choose to do your training through the REST API, or interactively through the tooling provided on Bluemix. When you are ready to move an instance from one environment to another, you export from one environment, and then create a new instance in the target environment, and import your data.
- For services like Discovery, which do not have an export/import capability, you should resist the temptation to train “by hand”, and instead have a script (or series of scripts) that can be used to train your system. In this way, you can more easily create/recreate an instance of the service.
This should give you enough of an overview, and offer you links to enough information, so that you can begin using the IBM Cloud platform to develop cognitive applications with confidence. Technology changes rapidly, so if you see problems with this basic approach, or have some best practices to share, please reach out to me.
Good Reference Materials
- Setting up your Bluemix environment – Not a lot of guidance here, but this documentation page on Bluemix has a lot of the basic information that you will need to understand when setting up a DevOps supported development environment for your cloud and cognitive development.
- Bluemix CLI – The reference page for the Bluemix specific CLI. Please use the Bluemix commands and not the Cloud Foundry equivalents, to ensure smooth operation of your Cloud environments.