A/B Test Deployment—Rollout of cloud.ibm.com

5 min read

Risk management and data-driven deployment

Over the past few months, we have been working hard to unite our IBM Cloud IaaS and PaaS consoles to make it easier for you to manage your IBM Cloud resources. We have made tremendous progress in bringing together two very different experiences. The results of this merger are finally clear — an enhanced IBM Cloud platform experience.

During this effort, we also had to ask ourselves, “How do we best roll it out?” In this day of continuous delivery and cloud-native applications, we considered several options. Do we overwrite our existing console and flash-cut over to the new one? Or, do we take a more cautious approach?

In his blog, Etienne Tremel does an excellent job of summarizing six modern deployment strategies. He created the following table to help with deciding which strategy best fits:


deployment strategies

Source: Etienne Tremel, https://thenewstack.io/deployment-strategies/


Our testing method

The deployment strategy we chose is the A/B testing method. In our case, we’ve already been using A/B testing extensively for growth and optimization of our user experience. So, this approach is a very natural fit because it allows us to do the following:

  • Control which users are encouraged to use the new experience. We will take advantage of this approach to further target and refine the audience as we slowly ramp up from a small portion of traffic to 100% within approximately eight weeks.

  • Monitor for any statistically significant differences in behavior. Will this new experience drive a significant increase in conversion? Or, will it lead to a pattern of unexpected failures? With this methodology, we are able to answer these questions and many more. Then, we can use the data to continuously iterate and improve as we prepare for the eventual 100% switch.

Our typical A/B tests have been done within the same deployment of our console. That approach works well when we want one group of users to see a different version of a particular page than the version that another other group sees. In this case, the new experience is significantly different and impacts the entire console, so we’ve chosen to maintain two separate deployments. This is depicted in the following high-level diagram which shows that Group A is sent to the https://console.bluemix.net deployment, and Group B is sent to the new https://cloud.ibm.com deployment.



We’re using the IBM Cloud Kubernetes Service to host the deployments containing the microservices that make up the console. By using this service, it’s very simple to spin up equivalent Kubernetes clusters to host the microservices for the new experience. In addition, we can easily make use of the same build pipeline, Helm Charts, and so on, but with different code branches for the new experience.

One other important thing to note is that both console deployments leverage the same set of IBM Cloud Platform APIs. Even though there have been some significant changes to these APIs as part of platform unification, we’ve taken great care to ensure backward compatibility. This means existing applications that are using these APIs can continue to run without interruption.

Opt in or opt out

A/B tests are typically conducted so that users are not aware of their participation in the experiment. However, we have some unique considerations that make this type of experiment less than ideal. For example, risk management is critical for many of our large clients who are looking to scale operations in preparation for various Q4 activities, including Black Friday and holiday shopping. We recognize that forcing you to use a new experience during the busiest time of the year just isn’t worth the risk and reward. So, we decided to go with an explicit opt-in approach where the choice is up to you. There is no risk to opt in to the new console experience. If you run into difficulties, you can opt out by simply going back to the existing console.

Ready to explore?

The net result is that starting immediately, some of you will see an opt-in button on the console authentication page. And over time, more and more of you will have this option. By clicking the opt-in button, you can use our enhanced console that unifies IaaS, PaaS, and more. With this approach, we can get all the benefits of an A/B testing rollout while reducing risk at the same time.

Can’t wait to try out the new experience, but you don’t see the opt-in option? You can still opt in by going directly to https://cloud.ibm.com.

Finally, we sincerely hope you enjoy the new experience, and we’d love to get your feedback (both good and bad) so that we can learn and better meet your needs. So, please reach out and let us know how it goes!

Check out the new IBM Cloud platform experience

Be the first to hear about news, product updates, and innovation from IBM Cloud