Hybrid Deployments

Getting organized to take your first Bluemix application all the way into Production (Part 2)

Share this post:

In Part 1 of this series we described the IBM Bluemix Garage’s approach to building a Cloud Foundry application and introduced the top ten steps to consider in an MVP Build-Up to ensure the application is production-ready. Below we elaborate and provide practical guidance on the basic steps to consider putting in place to establish a good foundation in preparation to going into production.

 

# Topic
1 Define the RACI matrix
2 Implement appropriate segregation of duties with Organization and Spaces
3 Continuous delivery on Bluemix
4 MVP error handling and alerts notification
5 Bluemix Logs
6 Configure MVP for resilience and scalability
7 Define a release strategy

 

Before going into production, you should also learn about the capabilities that are made available in Bluemix to monitor the platform and the application / services that make up your MVP. Part 3 will further explore the following topics:

 

# Topic
8 Monitor Bluemix Account Usage
9 Monitor Bluemix Incidents and Planned Maintenance that impacts the MVP
10 Monitor MVP Availability and Response Times

 

1 – RACI Matrix

In Bluemix, a Platform as a Service (PaaS), the IBM Operations team will take over some of the responsibilities currently performed by your operations team. Building a RACI matrix is instrumental to:

  • Surface which responsibilities are new and which ones have shifted from you to IBM Operations;
  • Identify members from your application and operations teams that will be responsible to support the applications in production;
  • Learn how your application and operations teams will interact with IBM Operations.

The first step in defining the RACI matrix is demarcating the responsibilities to be owned by the IBM Bluemix Operations team and those owned by your team(s). For your responsibilities, it is necessary to identify the appropriate team(s)/organization(s) and members that will be assigned as Responsible, Accountable, Consulted and Informed. An example is provided below as a starting point for discussion.

 

Areas of responsibilities Details Owner

(Public)

Managing the Bluemix Cloud System –       Applying Bluemix System Maintenance and observing overall System health IBM
–       Monitor incidents, planned maintenance and other events for Bluemix PaaS runtimes and services IBM/Yours
–       Managing user organizations and access rights (dev and support teams) Yours
Determining and developing workloads to run on Bluemix –        Setting up best practices for Bluemix workloads

–        Determining which workloads should be run on Bluemix

–        Developing new workloads or migrating existing workloads

Yours
–       Define workload architecture including runtimes and services to be leverage in Bluemix

–       Define security and integration strategy to on-premises systems

Yours
Customization, automation and integration of Bluemix with existing systems –       Automating tasks using CF APIs and integrating with existing systems (ex: incident management)

–       Integration with on-premises services for metering, etc.

Yours
Deployment and lifecycle management of workloads –       Ongoing maintenance of the applications running on Bluemix including bug fixes, enhancement, performance monitoring/tuning Yours
–       Configuration of Bluemix DevOps services for version control, continuous delivery, auto-scaling, monitoring, etc. Yours
–       Monitoring the resources used by the workloads and the performance and health of the workload Yours
–       End-user support for applications running on Bluemix Yours

 

2 – Segregation of duties with Organization and Spaces

In Bluemix, you can use organizations to enable collaboration among team members and facilitate the logical grouping of project resources. Spaces are generally used to identify and segregate environments: development (DEV), quality analysis (QA), pre-production (PRE-PROD), and production (PROD).

It is important to set up different spaces in Bluemix (e.g. DEV, QA, and PROD) and assign the appropriate team members to each space. This ensures that only the right people have the right access to the right project resources on each space. For example, you likely only want production support to have access to PROD space, while members of the development team have administration privileges in other pre-production environments.

During an MVP Build-Up, we generally define a single organization with the following spaces:

 

Space Team Member (Role) Observation
DEV, QA Developers (Developer Role)

 

Developers are assigned the Bluemix “developer” role which give them permissions to create, modify and delete applications and services on that space.

 

DevOps toolchains are created at the organization level. One of the developers creates the DevOps toolchain for the MVP project and adds other IBM IDs authorized to modify or delete the toolchain to the access control list. At the beginning of a project, other members of the development team contributing to the toolchain’s configuration are added as administrators to the toolchain access control list. Once the MVP moves to production, the access control list might be modified to restrict the administration privileges to the operation and development team members supporting the application in production.

PRE-PROD, PROD Developers (Developer and Auditor Roles)

Operations (Developer Role)

In an MVP, some developers might be assigned responsibilities on Bluemix that would traditionally be assigned to members of the operation team (e.g. deploying new releases, monitoring application availability and performance in PROD).  These developers must be assigned the Bluemix “developer” role in PROD alongside members of the operations team supporting the application in production.

We might or might not configure a PRE-PROD space depending on level of testing and validations required to authorize promotion to PROD.

 

Refer to Managing Organization and Spaces documentation on Bluemix for more details.

 

3 – Continuous Delivery on Bluemix

A single source code base, automated builds, continuous integration, and continuous delivery are all assumed practices when building cloud-based applications. Bluemix provides embedded support to all these and other DevOps practices.

The Git repository(s) and delivery pipeline(s) are configured in the first few days of an MVP Build-Up as the development team begins writing code. The delivery pipeline is initially configured with Build and Deploy (to DEV) stages; each with a single build and deploy corresponding job. Additional stages and jobs are added as the project progresses:

 

Stage Trigger Jobs Description
Build Commit to Git master branch Build

Unit Test

Static and Dynamic Scans

In addition to a build job, the build stage often runs a unit testing job.

Optionally, you can easily add additional test jobs to perform Static and Dynamic app scans

Deploy to DEV Success of previous stage Deploy

Functional Test

In addition to the deploy job, a functional testing job might be added to this stage to ensure new functionally added does not break core functionality already accepted in earlier iterations.
Deploy to QA Manual or success of previous stage Deploy This stage is added when the development team wants to gain more control of which build to expose to product owner for testing and/or if the customer assigns a test team to perform formal testing during the MVP Build-Up.

This stage is often triggered manually during the MVP Build-Up by a member of the development team that selects the build to be promoted from DEV to QA space.

Deploy to PROD Manual Deploy This stage is added towards the end of the MVP Build-Up. The stage is triggered manually by few authorized members of the PROD space. The deploy job can be configured as a zero-downtime deployment (Blue Green deployment).

 

Load and performance testing are generally not in scope of an MVP Build-Up, however third party services (Load Impact and BlazeMeter, for example) are available in the Bluemix catalog and can be added as test jobs to the delivery pipeline. If a pre-production space is created, it is generally configured with the same configuration of production in terms of service plans and runtime memory allocation and instances to run load and performance testing.

The new DevOps Insights Service (currently a Beta Service) allows the configuration of a Gate stage. The service gathers and analyzes the results from unit tests, functional tests, and code coverage tools at specified gates in your deployment process. If your code does not meet or exceed your predefined policies, the deployment is halted, preventing risky changes from being released. To get a feel to how the service works check the Create a toolchain that uses DevOps Insights tutorial.

TIP: Delivery Pipeline Build and Deploy Jobs make available the BUILD_NUMBER environment variable. This variable, however, is not propagated automatically to Bluemix runtimes. To easily identify which build is running in a given space, you may want to take advantage of the VCAP build number and other VCAP variables to differentiate the environment and build numbers in the application. Here is an example of how to set a Cloud Foundry environment variable to be associated with your DevOps Build & Deploy build number:

cf set-env “${CF_APP}” BUILD_NUMBER “${BUILD_NUMBER}”

 

4 – Error handling and alert notification

As in any development project, the appropriate level of error handling should be coded into the MVP application to enable troubleshooting and end-user support. In addition, during the MVP Build-Up, we might leverage services in the Bluemix catalog to proactively send alert notification to development and support teams when failures impacting end-users occur. The goal is to enable the development and support teams to quickly respond to issues and increase end-user satisfaction.

The IBM Alert Notification service is one option in the Bluemix catalog. IBM Alert Notification provides an API that creates alerts in application whenever a minor, major, or critical failure occurs. On Bluemix Public, any component of the MVP project (Bluemix, on-premises, mobile, etc.) can leverage the API to create alerts. The service can be used then as a central hub for all alerts on failures of any of the MVP components. You can then configure the service to send email, text messages to members of the development and supports teams for faster issue resolution. You can manage the alerts across different support teams and manage those team memberships.

See the IBM Alert Notification service in action here.

 

5 – Bluemix Logs for Cloud Foundry Applications

IBM Bluemix logging capabilities are integrated in the platform and data collection is automatically enabled. Bluemix, by default, collects and displays logs for your apps, apps runtimes, and compute runtimes where those apps run.

In Bluemix Public, logs are retained for 7 days by default. Also, logs are not persisted between application crashes. For these reasons, some clients will consider streaming log records to an external log management tool.

There are various ways to send application logs to Bluemix when you have deployed the application in production. Logging for both errors and for internal notes can help users and production support figure out the path of the application as it navigates and performs in the real world. For mobile front ends, an application page dedicated to viewing logs and sending in logs when a crash happens is very useful when it is deployed in Bluemix on the cloud. You may send application logs from either the Bluemix console, Kibana or external log hosts.

Learn more about Bluemix Logs here.

 

 

6 – Resilience and scalability

Bluemix provides capabilities for optimizing the performance and scalability of applications through both vertical (e.g. add memory) and horizontal (number of instances) scaling. You want to appropriately size your application both horizontally and vertically to provide users with good performance and reliability experience at the lowest cost possible.

As a best practice, you should deploy at least 3 runtime instances of your Cloud Foundry application(s) in QA and Production spaces to ensure your application maintains uptime for users.

During an MVP Build-Up, to explore the DevOps capabilities of the Bluemix platform, we configure the Auto-Scaling service on the QA and Production spaces to automatically increase or decrease the number of application instances (keeping 3 at a minimum) dynamically based on the Auto-Scaling policy defined (e.g. memory consumption, throughput and other metric types depending on application type).

Also, whenever possible, write your application to be stateless and not use session data. In some production apps, a session is necessary for user authentication and tokenization of certain endpoint calls. When sessions are used, it is important to use sticky sessions. Sticky sessions ensure that clients are always returned to the same instance when they started an application. Each client session is distributed evenly across the allocated instances.

Learn how to build your application in a multi-instance Bluemix environment here.

Here is a link to a great article on Scaling Applications on Bluemix. Another interesting read can be found here. The article Handle the Unexpected with Bluemix Auto-Scaling article demonstrates how New Relic and BlazeMeter third party services on Bluemix can be leveraged to provide input into scaling your application.

7 – Release Strategy

 

This item is particularly important when the MVP application integrates with new or existing on-premises applications (systems of record) being developed (or maintained) by team(s) other than the MVP Build-Up team.

 

The MVP team follows the Cloud Garage Method, which itself incorporates Lean, agile, and DevOps practices. Often, system of record teams follow more traditional development methodologies and are subject to constraints such pre-scheduled release cadence, dependencies with other on-premises systems, regulatory constraints and so on. The impact of such constraints need to be identified and coordinated among these various teams. It is important to:

 

  • Identify all components that make up the solution: Bluemix applications and services, on-premises components, mobile / device components and so on;
  • Identify which of these components are new or require change to support the MVP;
  • Identify the teams responsible for those components;
  • Identify integration requirements and design; Understand the impact to the MVP application;
  • Define regular checkpoints to verify progress on the various fronts;
  • Define intermediate milestones for integration testing;
  • Understand how on-premises components are deployed (manual / automated); Define sequence of deployment of various components;
  • Coordinate release schedules;

 

As you get into the details of defining the plan to first release the MVP application to production and the dependencies with all components of the solution (Bluemix, on-premises, mobile/device, other cloud providers), consider:

  • What would be the user experience if any of these components are unavailable? Are the components coded to respond gracefully to such situation? Are alerts going to be sent to operations support team?
  • What is the release schedule for the various components?
  • What is the maintenance window for the various components? Is it ever 24 / 7?
  • Is there a particular sequence for deploying the various components?
  • What if there is a need to rollback one of the components?
  • Which teams are responsible for the deployment? Which tools are used for the deployment?
  • How would we go about troubleshooting? Where are the logs for the various components? Which teams are responsible for it?

 

During the MVP Build-Up, even if we do not implement an optimal orchestrated release and support processes, it is important to a) understand the potential impact to the application’s end users, and b) build a backlog to later improve the processes.

What’s next: Monitoring the MVP application

Now that we have considered the basic steps to ensure the MVP production-readiness, in the next blog post we will learn about the capabilities made available on Bluemix to monitor the MVP in production.

Cloud Garage Developer

More Hybrid Deployments stories
May 3, 2019

IBM Garage Method Applied to Moving to and Modernizing with IBM Cloud

The IBM Cloud Garage has expanded our expertise, patterns, and assets to guide our clients’ existing applications to “move-to-cloud” and to better manage their hybrid and multicloud environments by leveraging a variety of IBM Cloud capabilities.

Continue reading

May 1, 2019

What’s Included in the IBM Cloud Developer Tools Version 2.2.0

I’m pleased to announce the latest version of IBM Cloud Developer Tools CLI, which contains some exciting new features.

Continue reading

April 30, 2019

IBM Cloud Garage: Blockchain and New Business Models

The IBM Cloud Garage has played a pivotal role in helping clients uncover the feasibility and potential of blockchain technology.

Continue reading