As many of you know, if I have to do something more than two or three times, I tend to put in some effort to script it. I know a lot of what I can do on the command line with the IBM Cloud, but I don’t always remember the exact syntax for all of those bx command line commands. I also like to have something that I can call from the command line, so I can just script up common administrative scenarios.
There are some options which already exist out there. I wasn’t aware of some of them, and none of them allow for scripting access. One of the best that I have seen is the interactive application discussed in the blog post on Real-Time Billing Insights From Your IBM Cloud Account, written by Maria Borbones Garcia. Her Billing Insights app is already deployed out on Bluemix. It’s nice – suggest you go and try it out. She also points you to her mybilling project on GitHub, which means that you can download and deploy this app for yourself (and even contribute to the project). Another project that I have seen is the My Console project, which will show a different view of your IBM Cloud account.
Why Create a Script?
This all came home to me this past week as I began to administer a series of accounts associated with a Beta effort at IBM (which I’ll probably expand upon once the closed beta is complete). I have 20 different IBM Cloud accounts, and I need to manage the billing, users, and policies for each of these accounts. I can do it all from the console, but that can take time, and I can make mistakes. The other thing that I thought of was that I often get questions from our customers about, “How do I track what my users are using, and what our current bill is?”. So that led me to begin writing up a Python script that would allow you to quickly and easily do these types of things.
So I began to develop the IBM_Cloud_Admin tool, which you can see the code for in its GitHub repository. Go ahead and download a copy of it from GitHub. This is a simple Python script, and it just executes a bunch of IBM Cloud CLI commands for you. If you go through a session and then look at your logfile, you can see all the specific command line commands issued, and see the resulting output from those commands. This allows you to do things in this tool, and then quickly look in the log file and strip out the commands that YOU need for your own scripts.
How To Use The Script
To run the script, you can just type in:
python IBM_Cloud_Admin.py -t apiKey.json
The script has a few different modes it can run in.
If you use the -t flag, it will use an API Key file, which you can get from your IBM Cloud account, to log into the IBM Cloud. This is the way that I like to use it.
If you don’t use the -t flag, you’ll need to supply a username and password for your IBM Cloud account using the -u and -p flags.
If you use the -b flag (for billing information), then you will run in batch mode. This will get billing information for the account being logged into, and then quit. You can use this mode in a script, since it does not require any user input.
If you don’t use the -b flag (for billing information), then you will run in interactive mode. This will display menus on the command line that you can choose from.
The Output Files
There are a number of output files from this tool. There is the IBM_Cloud_Admin.output.log file, which contains a log of your session and will show you the IBM Cloud command line commands issued by the tool, and the responses returned. This is a good way to get familiar with the IBM Cloud command line commands, so you can use them in custom scripts for your own use.
You may also see files with names like, MyProj_billing _summary.csv and MyProj_billing _by_org.json. These are billing reports that you generated from the tool. Here is a list of the reports, and what they contain.
MyProj_billing _summary.csv – this CSV file contains billing summary data for your account for the current month.
MyProj_billing _summary.json – this JSON file contains billing summary data for your account for the current month. It shows the raw JSON output from the IBM Cloud CLI.
MyProj_billing _by_org.csv – this CSV file contains billing details data for your account, split out by org and space, for the current month.
MyProj_billing _by_org.json – this JSON file contains billing details data for your account, split out by org and space, for the current month. It shows the raw JSON output from the IBM Cloud CLI.
MyProj_annual_billing _summary.csv – this CSV file contains billing summary data for your account for the past year.
MyProj_annual_billing _summary.json – this JSON file contains billing summary data for your account for the past year. It shows the raw JSON output from the IBM Cloud CLI.
MyProj_annual_billing _by_org.csv – this CSV file contains billing details data for your account, split out by org and space, for the past year.
MyProj_annual_billing _by_org.json – this JSON file contains billing details data for your account, split out by org and space, for the past year. It shows the raw JSON output from the IBM Cloud CLI.
Use the JSON output files as inputs to further processing that you might want to do of your IBM Cloud usage data. The CSV files can be used as inputs to spreadsheets and pivot tables that you can build that will show you details on usage from an account perspective, as well as from an organization and space perspective.
Getting Your Api Key File
I’ve mentioned the API key file a couple of times here. If you are not familiar with what an API Key file is, then you’ll want to read this section. An API Key is a small text file which contains some JSON based information, which when used properly with the IBM Cloud command line tool, will allow anyone to log into the IBM Cloud environment as a particular user, without having to supply a password. The API Key file is your combined username/password. Because of this, do NOT share API keyfiles with others, and you should rotate your API Key files periodically, just in case your keyfile has become compromised.
Getting an API Key on IBM Cloud is really easy.
Log into the IBM Cloud, and navigate to your account settings in the upper right hand corner of the IBM Cloud in your web browser. Select Manage > Security > Platform API Keys.
Click on the blue Create button.
In the resulting dialog, select a name for your API Key (something that will tell you which IBM Cloud account the key is associated with), give a short description, and hit the blue Create button.
You should now see a page indicating that your API Key has been successfully created. If not, then start over again from the beginning. If you have successfully created an API Key, download it to your machine, and store it somewhere secure.
Note: A quick note on API Keys. For security reasons, I suggest that you periodically destroy API Keys and re-create them (commonly called rotating your API keys or access tokens). Then if someone had access to your data by having one of your API keys, they will lose this access.
Do you have other administrative tasks that you would like to see the tool handle? Find a bug? Want to help improve the tool by building a nice interface for it? Just contact me through the GitHub repository, join the project, and add issues for problems, bugs, and enhancement requests.
A Final Thought
This script is a quick hacked together Python script – nothing more and nothing less. The code isn’t pretty, and there are better ways to do some of the things that I have done here – but I was focused on getting something working quickly, and not on efficiency or Python coding best practices. I would not expect anyone to base their entire IBM Cloud administration on this tool – but it’s handy to use if you need something quick, and cannot remember those IBM Cloud command line commands.
I just had a conversation today with my VP (Rob Sauerwalt – check him out on Twitter– time to do some shameless kissing up to my management team) about a recent internal communication that we both saw. It was someone looking for a “readiness checklist” for the deployment of an application on the IBM Cloud. Rob and I both agreed that this seems pretty simple, and we came up with a quick checklist of things to consider.
Now this list is not specific to the IBM Cloud, it’s pretty generic. It’s just a quick checklist of things that you will want to make sure that you have considered, BEFORE you deploy that cloud based application into a production environment. I am an Agile believer, so I would suggest that you address these checklist items in the SPIRIT of what they are trying to do, and that you should do what makes sense. This means that each one of these areas does not need to represent some 59 page piece of documentation. What you want to do is provide enough information so the poor guy who takes your job after you get promoted, is able to be effective and understand and maintain the application or system.
If you have suggestions about other things that should be on this list, please drop me a line and let me know. I would love to add them to the list, and make this generic deployment readiness checklist even better.
Production Readiness Checklist
⊗ Name and General Description of the Application – this includes the purpose of the application and the number of users that are anticipated to use the application. Also have an idea of the types of users. Is it for the general public? Only for certain roles within our organization? Is it only for your customers? Do this in two to three paragraphs – anything more is adding complexity.
⊗ Description of Needed Software/Hardware/Cloud Resources – a list of the needed software packages, and the clou resources needed to run the application. Do you use third party utilities or libraries? Do you run on Cloud Foundry buildpacks? Virtual machines? Do you use Cloud services for database resources? Often a high level architectural diagram is useful to help other people understand the system at a high level. This should be done AS you build – so you can simplify things. Are your developers using different libraries to accomplish the same thing? Get them to standardize. Reduce your dependencies, reduce your complexity, and you improve your software quality.
⊗ Operating Systems and Patching Requirements – do you have specific OS requirements? Do you require a particular framework to run properly (like .NET, Eclipse, or a particular Cloud Foundry buildpack)? What OS versions have you tested and validated this application with – and do all of your components need to be on the same OS version? This becomes important when fixes get deployed to containers, virtual machines get upgraded, and maintenance activities are done.
⊗ Installation and Configuration Guidelines – you should be deploying your application in some automated manner. So your deployment and promotion scripts should be the only guide that you need…… except when they aren’t. Take the time and DOCUMENT those scripts – explain WHAT you are doing and WHY, so your application can easily be reconfigured or deployed in different ways in the future.
⊗ Back-up, Data Retention and Data Archiving Policies – let your operations people know what data needs to be archived and retained. How often do systems need to be backed up? How will services be restored in the event of a crash? Explain WHERE and HOW data needs to be retained. Explain what your DEVELOPMENT teams need to review on a periodic basis. This can be the biggest headache for development teams, because these are often scenarios that they have not considered. Backup plans are not sufficient, they need to be executed at least once before you go into production – so you are sure that they are valid and that they work.
⊗ Monitoring and Systems Management – This includes runbooks – what do we need to do while the application is running? Do we need to take the logs off of the system every day and archive them? Or do we just let logs and error reports build up until the system crashes? Should I monitor memory and heap usage on a daily basis? Should I be monitoring CPU load? Who do I notify if I see a problem, and what is a “problem”? (CPU at 50%? CPU running at 20% with spikes to 100%?) How will this application normally be supported? You may not have complete information and definition of “problems” when you begin, bu define what you can and acknowledge that things will change as time goes on.
⊗ Incident Management – This details how you react to application incidents. These could be bugs, outages, or both. In the case of an outage, who needs to be called, and what actions should they take to collect needed data, and to get the application back up and running. What logs are needed, what kind of data will aid in debugging issues? Who is responsible for application uptime TODAY (get things back on track and running), and who is responsible for application uptime TOMORROW (who needs to find root cause, fix bugs, make design changes if needed, etc.).
⊗ Service Level Documentation -This is the “contract” between you and your customers. How often will your application be down for maintenance? If your application is down, how long before it comes back up? Are there any billing or legal ramifications from a loss of service? Do your customers get refunds – or cash back – when your Cloud application is unavailable?
⊗ Extra Credit – DevOps pipeline – you need to have an automated pipeline for the deployment of code changes into well defined development, test, and production environments. You need to have a solid set of policies and procedures for the initiation and automation of these deployments. Who has authority to deliver to test environments? Production environments?
Software Architecture Considerations
⊗ Key Support & Maintenance Items – the team that built this thing knows where the weak spots are – share that knowledge! Where does the team know that “tech debt” exists – and how is that impacting your application? This information will help the teams maintaining and upgrading your application. They will be able to do this with knowledge about how the application works, and why certain architectural choices were made.
⊗ Security Plan – Everyone is worried about the security of their applications and data on the cloud. You need to be sensitve to this when deploying cloud based applications. Your stakeholders and users will want to know that you have considered security, and that you are protecting their data from being exposed, stolen, or used without their knowledge/consent.
⊗ Application Design – This should include some high level description of your use case, a simple flowchart and dependencies. Give enough detail so someone can easily get started in maintaining your application code, but not so much detail that you waste time and ultimately end up with documentation that does not match the code.
Is That Everything?
That’s not everything, but it is a good minimal list of things that you should have considered and/or documented. Most applications need some sort of a support plan – who handles incoming problem tickets from customers? Do you have a support process for your end users? In your own environments and business context, you may have other things that need to be added to this list. Do you need to check for compliance with some standard or regulation? What are your policies for using Open Source software?
So this list is not meant to be exhaustive – but it is designed to make you think, and to help you ensure higher quality when deploying your Cloud applications.
With the end of the year quickly approaching, it is a great time to look back on the past year, and to look forward in anticipation for what is coming in 2018.
2017 was an interesting year. I saw an explosion in the development of chatbots of various different types. Some were very simple, others used both Watson Conversation and the Watson Discovery service to provide a deeper user experience – with an ability to answer both short tail and long tail questions. I saw a huge uptick in interest in basic Cloud approaches, and a lot of interest in containers and Kubernetes. I expect that both of these trends will continue into 2018.
In 2018 I expect to see the IBM Cloud mature and expand in it’s ability to quickly provide development, test and production computing environments for our customers. I also expect that more people will become interested in hybrid cloud approaches, and will want to understand some best practices for managing these complex environments. I am also excited about some of the excellent cognitive projects that I have seen which could soon be in production for our customers. I also expect that some of our more advanced customers will be looking at how cognitive technologies can be incorporated into their current DevOps processes, and how these processes can be expanded into the cloud.
I hope that your 2017 was a good one, and I hope that you have a happy and safe holiday season.
I blog when I have to answer questions that I want to more widely share the answers to. It’s also a good way to remember things before a turkey induced amnesia sets in (it’s a week before U.S. Thanksgiving).
Recently I have seen some questions on being able to get access to the data in an IBM Cloud GitHub project. I had just completed doing a quick internal activity for pulling information out of a GitHub Enterprise repository, so I figured that this would be simple. It was…. and it wasn’t. The IBM Cloud GitHub instance isn’t a GitHub Enterprise deployment, it is a GitLab deployment. The GitLab API is a little different from the GitHub Enterprise API. I managed to find a suitable Python package for using the GitLab API, and if you look at the resulting code, it is pretty simple.
So I have created a simple GitHub project (called IBMCloud_GitLab_CSV) that does a quick CSV export of issues from an IBM Cloud GitHub project. It’s a simple example, written in Python, that you should be able to use and tailor, to fit your specific needs. I use small Python programs like this to pull the open issues from a variety of projects, and then I can share the resulting CSV files with project managers and PowerPoint producers who want to report on these sorts of things.
If you need this type of capability, make sure to read the README file for this project, which instructs you on how to modify the code to point at YOUR IBM Cloud GitHub project, and tells you how to get an access token for yourself (which the program needs, in order to be able to log into your GitHub project).
If you want to improve on this example, or even create some type of generic tool for doing this type of thing, please join the IBMCloud_GitLab_CSV GitHub project and begin contributing to it.
As I have said on earlier blog posts, if I have to answer a question more than once, then it’s probably worthy of a blog post. This week I had one of those situations, and thought that it would be good to share because it highlights a few points at very different levels.
Last week I was answering a survey posted by a Watson Speech-to-Text user, who was complaining about how confusing everything was. I found this a bit surprising, as Speech-to-Text is one of the more simple services for developers to work with. The user claimed that their EasyVSL app was telling them that “Watson wasn’t working”. So I quickly checked the Bluemix status board, and found that nothing indicated a loss of service availability. So I asked some more questions and probed a little deeper.
Watson as an Add-On
I found out that the EasyVSL app has a feature which the user can turn on, which will use the Watson Speech-to-Text service. The user just need to add the service credentials for a valid Watson Speech-to-Text service, and the EasyVSL app will use that Watson service instance to do their speech-to-text processing. Once I saw this, I walked through creating an instance of the Watson Speech-to-Text service for the customer, and showed him how to get his service credentials, which he then supplied to the EasyVSL app. Once this was done, Watson was no longer “broken”.
Up until the past few weeks, I had never seen a product use Watson services in this way – as an add-on to an existing capability, where the CUSTOMER needs to provide the Watson service. In this case, EasyVSL doesn’t worry about paying for Watson services. If their customers see value in them then they use them and pay IBM directly for the services that they use. It is a business/operating model that I am beginning to see now.
Advantages and Disadvantages
This type of model has some positives and negatives associated with it, and I think that it depends on your perspective. As an application provider, it is a positive to be able to let your customers pay for the additional functionality that they want. If they don’t need advanced cognitive capabilities, then why should they pay for them? It also allows you to avoid thinking about how your are going to pass charges along to your customers. If you need to monitor customer usage, track how many times they are using the various Watson API’s, and then figure out some way to pass along the costs associated with them, you will end up spending a fair amount of time and effort developing that monitoring and billing infrastructure (not even considering the adjustments that you will need to make to make your business model work).
From a customer perspective, many of the positives become negatives. While I like to have a choice as a customer, I do not like having to deal with two vendors billing me for a single service. I am also not thrilled with the fact that I need to maintain an instance (or multiple instances) of the Watson services. I am paying money to a vendor so THEY can worry about that stuff. The other downside is that as a customer, it is not always apparent which vendor I need to call when I have a problem. That was the issue that I saw this week – the customer that I had helped had reached out for support from both EasyVSL and IBM – and had not gotten very far with either of them.
Future of Watson as an Add-On
I’m not sure if this pattern will become popular, although my gut tells me that it won’t. Most customers don’t like paying multiple vendors for what they perceive as a single service. However, I do note that cell phones are now largely unsubsidized in the US, with customers buying their own phones, and then paying monthly for services that those phones use. In the past, the phones were “free”, and the cost of the phones was “bundled” into the monthly cell phone rate. So there is precedence for this type of model working.
If you are an app provider, it is critical that you think carefully about your approach to how you use the Watson cognitive services, not only from a technical perspective, but also from a business perspective. The business model that you choose will have an impact on your development, operations, and your customers.
Those of you who read my posts know that I work for IBM, and that I work in the Cloud and Cognitive spaces. Recently I have seen some articles on Multi-Cloud Strategies (like this one from the O’Reilly site), and it puzzles me why so many of the things that I see on this topic seem to miss some of the most basic things. The O’Reilly article simplifies things, and misses some very important points. I don’t blame them, it’s hard to boil down a complex concept into a blog post that can be read in 5 minutes. So after explaining this more than once to someone, I figured it was time to write about it.
Getting Started with Cloud Computing
Cloud computing should be thought of as a tool – a tool to squeeze more computing power from a limited budget. At the highest level it allows a customer to stop buying and maintaining their own sever hardware, and “lease” this capability from a Cloud vendor. Now there are multiple things that you need to account for: security, availability, service availability (what kind of services are available), reliability, cost, etc. Due to some of the current limitations of cloud technologies, not ALL workloads are suitable to be moved to the cloud. There is a lot to understand, and it can be overwhelming.
For this reason, many companies decide to move slowly to the cloud, so they can learn and gain experience as they go. This is a great way to learn about what is important, learn about the limitations and concerns that you should have, and to get some experience with cloud. So they select a single cloud provider, and try a couple of pilot projects, moving some portion of their production computing to the cloud.
This is what I would consider a sane and reasonable approach to starting your journey to the cloud. This approach reduces your risk, and allows you to learn as you go, gaining valuable experience and expertise as your initial projects get deployed. Your team will understand he capabilities of your initial cloud vendor, both positive and negative, and will begin to become more comfortable with the whole cloud paradigm. At this point you are not able to leverage the strengths of various cloud vendors, but you don’t really care because you are in learning mode.
Once these initial pilot projects have been completed, you will then branch out to doing some other projects with your original cloud provider, as well as doing some pilot projects with a second cloud provider. This is the next step on your journey to a true multi-cloud strategy. This second cloud provider will seem quite strange, they may have some different terminology, different billing, and different capabilities. At first your teams will not like them, because they are comfortable with what they already know from your initial cloud provider. Give them time to learn – over time your teams will get a more balanced view of the positives and negatives of the cloud vendors.
At this point you should begin building your own knowledge base of the types of use cases and workloads that work particularly well for each cloud vendor, and you should have some kind of rough order of magnitude way to price workloads that have moved to each cloud vendor. Your team should also have a much better understanding of cloud concepts, and understand the differences in terminology between the cloud vendors.
A True Multi-Cloud Strategy
Once you have branched out to a couple of cloud vendors, branching out and evaluating additional vendors should be easy. Your team will be experienced and will understand some of the key capabilities to look for, the things that are important for your organization. You will be able to intelligently pick and choose between cloud vendors, based on price, capability, reliability, and based on your own experience. You’ll be able to make well informed decisions based on your use case, the workload, and choose the best cloud vendor for that particular job. So let’s look at what the positives and negatives of this approach are.
Time – who wants to wait for two years while your team gets the knowledge that they need? Why not just pick a single cloud vendor and be done with it?
Administration – monitoring and managing one cloud supplier can be a pain – managing 4 or 5 of them can be 4-5 times more painful.
Reduced Risk – your investment portfolio should be well diversified, it reduces the risk you face should one of you investments fail. Your portfolio of cloud technology providers should be diversified for the same reason. It allows you to quickly take advantage of new capabilities and technologies, and gives you FLEXIBILITY. If you remember nothing else from this post – remember this point. It is what all of the other articles that I see miss – and it is the single most important benefit.
Avoiding Vendor Lock In – have a cloud provider that is charging you too much? If you’re familiar with other vendors, moving workloads between cloud providers can be easy (especially if you use container technology).
Best of Breed – giving your business the flexibility to choose a cloud provider that best fits their needs allows you to avoid overpaying for capabilities that might go unused with some cloud providers.
Best Practices – each cloud vendor may be strong in a particular area, but your team can generalize these best practices across ALL of your cloud implementations.
So wrapping things up, I think that a multi-cloud strategy IS important. I strongly believe that the benefits of reduced risk and flexibility far outweigh the costs in terms of time (and patience) and administrative overhead. It also allows your IT teams to focus on your core business, and pushes many of the “bread and butter” IT tasks off onto the cloud provider. This allows you to focus and innovate.
In my role as a Watson and Cloud Adoption Manager, I often talk to customers who are new to the IBM Cloud platform (Bluemix) and the Watson services. Often I will spend a good hour or more talking to customers and answering questions about the best way to get started with the organization of the Bluemix environment, and some of the more operational concerns around cloud and cognitive development. Since I find myself answering these questions over and over, I figured it would be good to get this down in a blog post so I could easily share this information with MORE customers. It also helps me remember all the important things when I do talk on the subject.
Getting Started on Bluemix
Signing up for Bluemix is easy, all you need to do is supply a valid email ID which will be associated with the Bluemix account. This is fine for YOU, but what about your ORGANIZATION? When setting up a Bluemix account for your company or organization, it is best to use a functional ID associated with your company. Just keep in mind that the IBM Cloud will be sending out automated emails to this account (warnings about service usage, services being deprecated, new services available, and other things). So you will want to make sure that you choose an email address that is monitored by someone, so you don’t miss any important notifications.
I KNOW it takes a little longer to set up a functional account within your company, and that it is a pain to go through the paperwork to justify it, but not having your Bluemix account tied to an individual will save you A LOT of time, effort and frustration in the future.
Learning the Cloud Concepts
This is a good spot to pause for a second, and take time to understand some of the basic cloud concepts that will impact how you organize yourself on the IBM Cloud. Looking at this picture may help:
The outermost grey box is my Bluemix Account, and this should be associated with a functional ID. This is the functional ID and email address where you will receive communications from the IBM Cloud team. It is also the account which has access to EVERYTHING underneath it.
Inside of that grey box are three blue boxes. These are the Bluemix Organizations, which represent different functional organizations (or projects) within the larger Bluemix Account. Even though the different organizations reside in the same parent Bluemix Account, the resources within those organizations can be limited to visibility to only resources within that Bluemix Organization.
The green boxes inside of each Bluemix Organization represent Bluemix Spaces. A space can be thought of as an individual development environment, or development area, for the development of a cognitive application.
Finally, at the lowest level, are the various Bluemix Services, indicated with the orange circles. These can be Watson based services (like the Watson Conversation service), infrastructure services (like the OpenWhisk service), or other Bluemix capabilities (like the Weather or Blockchain services). Services essentially “live” within a particular space, and the charges associated with a service are calculated and billed against the Bluemix Account. These charges are broken down by Bluemix Organization, for your own internal billing and tracking purposes.
Getting Organized on the IBM Cloud Platform
We’ll assume that you took my advice, and have a generic corporate ID (like IBMcloud@acme.com) that is your Bluemix account. Now let’s get organized to support the development efforts of your organization on the IBM Cloud platform. I’m also going to assume that you have read the great guide to getting organized on the IBM Cloud Platform. It’s good, and it has more detail than what I cover here. It also doesn’t tell you how YOU should set up YOUR IBM Cloud Platform. I’ll give you a “default” setup that you should use as the basis for your strategy in setting up the IBM Cloud Platform, and I’ll indicate options you have and why you might use those options.
We’ll start with our Bluemix account for our fictional Acme Corporation (for this example, we’re using IBMcloud@acme.com). The first thing that we need to do is to create some Bluemix Organizations for ourselves. Now my fictional company is like a lot of other companies out there, it has some divisions, and these divisions don’t often share responsibilities or development assets. So I will start by giving each of the divisions that are looking at doing cognitive development their own Bluemix Organization.
# Set the Bluemix API endpoint
bluemix api https://api.ng.bluemix.net
# Login to Bluemix, with -u username -p password
bluemix login -u IBMCloudAcct@acme.com -p "mypassword"
# Create new Bluemix Organizations for Delivery, Sales, and Marketing
bluemix iam org-create Acme_Delivery
bluemix iam org-create Acme_Sales
bluemix iam org-create Acme_Marketing
# Print a list of all of the Bluemix Organizations for this Bluemix Account
bluemix iam orgs
The first command sets the API endpoint. The second command gets you logged in. The next three commands create your new organizations, and the last command lists all the Bluemix Organizations for your account. Check out the documentation for Bluemix commands for more details. You will now see a listing of all of your new Bluemix Organizations, along with your default organization. Be sure to provide a unique name for your organizations, and PLEASE establish some kind of naming convention. I used <company_name>_<company_division>, you might want to do this too.
Why set up these separate organizations on the IBM Cloud Platform? For a few reasons:
It helps keep the work organized and segregated. With these in place, and once we get spaces set up, we can limit user access to only certain areas.
It allows us to see usage and billing details by organization. I can see exactly what each organization is using, and how much of my total bill each organization is responsible for.
Also keep in mind that these Bluemix Organizations are just a way to organize your IBM Cloud infrastructure. After reading the guide, you may decide to go with Bluemix Organizations that break up areas based on the roles of your users, or specific projects.
Getting a Subscription
To this point in our setup of the IBM Cloud for the fictional Acme Corporation, we have assumed that you have used a “free”, or “trial”, tier of services. These give you enough resources to figure out what you would like to do on the IBM Cloud, and allowed you to build small demo applications based on the IBM Watson services.
Now since we are talking about doing “real” application development work, we’re going to need more capability than these trial and free versions of the services provide. That means that you will need to spend some amount of money to host these various services and capabilities. Check out this nice overview of the various IBM Cloud Account Types to get an idea of your options. The capabilities and limitations of these options differ, so it is important that you understand them.
For our fictional Acme Corporation, we have decided to go with the popular subscription option. To begin the process of getting a subscription, you will want to follow the directions on how to obtain a subscription on the account types page. Make sure that your subscription is associated with the right Bluemix account (the one with the functional ID, remember that?).
Getting Your Development Environments Right
Now that we have our organizations all set up, and our subscriptions associated with the correct divisions within the fictional Acme Corporation, it’s time to set things up for our development teams. Now I understand that some readers are working for smaller, more nimble organizations. You may be fine with doing things in an ad-hoc manner, and that may actually be the best working model for you. What follows is a broad outline of what larger organizations, seeking a better separation of duties, control of environments, and implementation of a more standard software development lifecycle and DevOps culture will want.
We begin with our Bluemix Account and our Bluemix Organizations. For our purposes here, we will focus on just one of our fictional Acme divisions, the Delivery division. You begin by identifying the projects within the division, and you then begin to create Bluemix Spaces for each project. You should also find out what environments each project will need. Most will need a development environment, a test environment, and a production environment (at a minimum). Some projects will want support for additional environments.
For each project, go out and create the appropriate Bluemix Spaces for the project. Spaces need to have unique names. For the sake of keeping things easily identifiable, unique, and organized, I have followed a naming convention of <Project>_<Environment> (you should probably do the same). So for the Acme Delivery division, we have three projects – called Phoenix, WH, and Dizzi. Each has different services that they use (some are common), and the Dizzi project has no need for a pre-production environment.
So now we use the Bluemix CLI to go and create the spaces that we need (Note: This can also be done interactively from the Bluemix UI – but I know that you are interested in automating this, so I will focus on using the CLI):
# Set the Bluemix API endpoint
bluemix api https://api.ng.bluemix.net
# Login to Bluemix, with -u username -p password -o organization
bluemix login -u IBMCloudAcct@acme.com -p "mypassword" -o Acme_Delivery
# Create new Bluemix Spaces within the Acme_Delivery organization
bluemix iam space-create Phoenix_Dev -o Acme_Delivery
bluemix iam space-create Phoenix_Test -o Acme_Delivery
bluemix iam space-create Phoenix_PreProd -o Acme_Delivery
bluemix iam space-create Phoenix_Prod -o Acme_Delivery
bluemix iam space-create WH_Dev -o Acme_Delivery
bluemix iam space-create WH_Test -o Acme_Delivery
# do the remaining spaces in a similar manner
This results in a Bluemix environment within the Acme_Delivery organization which looks like the diagram below.
You can double-check your results either from the Bluemix CLI (using the “bluemix iam spaces” command), or from the IBM Cloud dashboard. You will notice that there are some small yellow circles within each newly created space. These represent the IBM Cloud services that each project will create within these spaces, in order to implement their project. As a Bluemix administrator, you should not have to do this. The creation of needed services should be left to the individual project teams.
Getting Everyone Else Onboard
So now that you have your Bluemix Organizations, projects and environments (Bluemix Spaces) all set up, you are ready to go. All that you have to do now is get your developers and other stakeholders into the environment. The first step is to get your staff to register for Bluemix and to get their own accounts. I STRONGLY suggest that your force them to register with their company email addresses. It will make things easier for you to administer things as time goes on.
After your team has registered, you can invite them to the correct Bluemix Spaces and Bluemix Organizations. So once again we will use the Bluemix CLI, and send the invitations that we need (Note: This can also be done interactively from the Bluemix UI – but I know that you are interested in automating this, so I will focus on using the CLI):
# Set the Bluemix API endpoint
bluemix api https://api.ng.bluemix.net
# Login to Bluemix, with -u username -p password -o organization
bluemix login -u IBMCloudAcct@acme.com -p "mypassword" -o Acme_Delivery
# Send invitations to the various team members within the Delivery division
# arguments are: USER_NAME ORG_NAME ORG_ROLE SPACE_NAME SPACE_ROLE
bluemix iam account-user-invite email@example.com Acme_Delivery OrgAuditor Phoenix_Dev SpaceDeveloper
bluemix iam account-user-invite firstname.lastname@example.org Acme_Delivery OrgAuditor Phoenix_Dev SpaceDeveloper
bluemix iam account-user-invite email@example.com Acme_Delivery OrgAuditor Phoenix_Test SpaceDeveloper
# do the rest of your users in a similar manner
This gets your users added to the proper organizations and spaces. Some important things to keep in mind when assigning organization roles and space roles to your users.
Give everyone OrgAuditor for their ORG_ROLE. This allows them to see what is going on at the organization level, but doesn’t allow them to change anything. The only exceptions to this are the people who do administration of the organization (your Bluemix Administrator), and handle the billing associated with the organization (often the Bluemix Administrator as well).
Space roles are a bit different.
SpaceManager: This role can invite and manage users, and enable features for a given space. You may want to give this to the development lead for the Development spaces, the test lead for the Test spaces, and your operations leader for the Production spaces.
SpaceDeveloper: This role can create and manage apps and services, and see logs and reports. Give this role to developers in the Development spaces, but you probably don’t want anyone with this role in the Testing and Production spaces. This allows you to maintain the stability of these environments (no code changes).
SpaceAuditor: This role can view logs, reports, and settings for the space. This is used for people who might be interested in the development efforts in a given space – but not requiring access to change anything within the space.
You will probably want to automate this, or provide a self-service capability, so people can easily request access to the environments that they need.
Herding the Cats
At this point we have our Bluemix environments set up, and we have our users added into the Bluemix Organizations and Bluemix Spaces where they need to be, with the access that they require. Our fictional Acme environment looks a little something like this:
Each of our divisions now has the ability to control access to their Bluemix areas, and can effectively isolate their various environments. Each is free to use whatever promotion process and DevOps tooling they prefer, or they can utilize the IBM Cloud Toolchains and the IBM Cloud Continuous Delivery service. Use a DevOps solution which addresses ALL parts of your development process, regardless of where they reside.
There are some important things to consider when thinking about the topic of DevOps in conjunction with the IBM Cloud and the IBM Watson services.
Watson services where you do not provide any training data (Document Conversion, Language Translator, Personality Insights, Tone Analyzer, Text-to-Speech (TTS) and Speech-to-Text (STT) ), can be deployed to additional Bluemix Spaces and do not need to have their data migrated from one environment to the next. Keep in mind that on occasion the versions of the API and underlying services may increment, so you will want to control which API endpoints are used for these services. (Note: If you have customized either Text-to-Speech or Speech-to-Text, then these services will need to be migrated from environment to environment.)
For services like Conversation which support an export/import functionality, it means that you can choose to do your training through the REST API, or interactively through the tooling provided on Bluemix. When you are ready to move an instance from one environment to another, you export from one environment, and then create a new instance in the target environment, and import your data.
For services like Discovery, which do not have an export/import capability, you should resist the temptation to train “by hand”, and instead have a script (or series of scripts) that can be used to train your system. In this way, you can more easily create/recreate an instance of the service.
This should give you enough of an overview, and offer you links to enough information, so that you can begin using the IBM Cloud platform to develop cognitive applications with confidence. Technology changes rapidly, so if you see problems with this basic approach, or have some best practices to share, please reach out to me.
Good Reference Materials
Setting up your Bluemix environment – Not a lot of guidance here, but this documentation page on Bluemix has a lot of the basic information that you will need to understand when setting up a DevOps supported development environment for your cloud and cognitive development.
Bluemix CLI– The reference page for the Bluemix specific CLI. Please use the Bluemix commands and not the Cloud Foundry equivalents, to ensure smooth operation of your Cloud environments.