Managing a continuous delivery process by providing continuity of service

To provide continuity of service, configure a continuous delivery process using IBM UrbanCode Deploy. Using a real scenario, this article describes how to the upgrade components on the staging and production environments without interrupting service.

Ilaria Gorga (ilaria.gorga@it.ibm.com), Software Engineer , IBM

Author1 photoIlaria Gorga has worked as a software engineer in the IBM Tivoli Rome Lab since 2008. She has been on the Tivoli Workload Scheduler quality assurance team since 2010 and focuses on test automation optimization. Ilaria has written several other articles and presented at conferences, such as Eclipse-IT.



Fabio Barillari (fabio.barillari@it.ibm.com), Senior Software Engineer , IBM

Photo of BarillariFabio Barillari is a senior software developer and designer. He works as Quality Assurance Architect at IBM. He defined DevOps processes for IBM Workload Automation.



24 June 2014

Introduction

Learn how to configure IBM® UrbanCode® Deploy to provide continuity of service to your customers. Using a real project, IBM Workload Automation, this article describes an upgrade process that is designed and implemented in a cloud environment to guarantee high availability of the product.

The IBM Workload Automation environment is composed of many sub-environments, which are called offering instances. Each offering instance is composed of two nodes that are virtual images of an operating system: a primary node and a secondary node, as shown in Figure 1.

Workload Automation components are installed on both the machines to provide high availability.

Figure 1. Architecture of the continuous delivery solution
Architecture uses continuous delivery solution

The installed Workload Automation (WA) components are the WA master, which is the server component, and the dynamic workload console (DWC), which is the user interface component.

Each component has its own mechanism to provide high availability.

  • The WA master provides high availability by means of a switch master that it uses to shift responsibility to a second node, the backup master.
  • DWC provides high availability by sharing its data in a common remote database repository.

This article focuses on the upgrade process for the WA master component. When you design an upgrade process, consider the following constraints:

  1. To avoid interruption of service, the upgrade of the components on the primary and secondary nodes cannot occur simultaneously.
  2. The update operations must run on the entire environment. They must synchronize the primary and the secondary node of the same sub environment.

Given Constraint 1, the upgrade process cannot be performed in a single step. In this example, it occurs in four phases:

  • Phase 1: During this phase all secondary nodes are upgraded. Because the secondary nodes are inactive (the primary node is the active node), the upgrade operation does not create a service interruption.
  • Phase 2: During this phase, the primary role is switched from the primary nodes to the secondary nodes. This switch causes the secondary nodes to become the active node.
  • Phase 3: During this phase all the primary nodes are upgraded. After Phase 2, the primary nodes are not active; therefore, this operation does not create a service interruption.
  • Phase 4: In this phase, the primary role is switched back from the secondary nodes to the primary nodes.

Each phase can start only if the previous phase is completed.

Constraint 2 implies that you need to implement a synchronization mechanism between the two nodes of each sub-environment, without affecting all the other sub-environments.

By using this method, if the update process fails on a few environments during the Phase 1, all the remaining environments can continue and can be successfully upgraded.


Implement the solution through UrbanCode Deploy

In the Workload Automation scenario UrbanCode Deploy is used to implement Constraint 1. Four application processes are synchronized by using the version status. To ensure the four processes are processed in the correct sequence, at the end of each phase the process add a status to the version being installed. This version status is required for the next phase.

Constraint 2 is implemented by resource properties that are shared between the primary and the secondary nodes of the same sub-enviroment.

In this scenario, the solution is implemented using these steps:

  • Define the environment.Define all the offering instances as a single UrbanCode Deploy environment and group the nodes of the same sub-environments. The environment is composed of many sub-environments. Each sub-environment is composed of a few nodes. The environment definition uses a naming convention to identify the nodes that belong to the same sub-environment.
  • Define the version status. Configure the UrbanCode Deploy version status that is used to synchronize the application processes.
  • Define the Phase 1 application process. Implement the process to update all the secondary nodes and set the version status to indicate that Phase 1 is complete and Phase 2 can start.
  • Define the Phase 2 application process. Implement the process to switch the responsibility from the primary to the secondary node and set the version status to indicate that Phase 2 is complete and Phase 3 can start.
  • Define the Phase 3 application process. Implement the process to update all the primary nodes and set the version status to indicate that Phase 3 is complete and Phase 4 can start.
  • Define the Phase 4 application process. Implement the process to switch back the responsibility from the secondary to the primary nodes.

Define the environment

The environment is defined in UrbanCode Deploy through a mechanism of automatic population, described in the article "Managing Continuous Delivery on dynamic cloud environment using IBM UrbanCode Deploy".

The environment is composed of two groups: primary nodes and secondary nodes. The first group contains all the agents installed on the primary nodes and the second group contains all the agents installed on the secondary nodes, as shown in Figure 2.

Figure 2. Production environment definition on UrbanCode Deploy
UrbanCode Deploy environment tab

In each group, a set of properties related to the primary role and the secondary role are defined. Set the agent name according to the following naming convention:

 <ENV_TYPE>_<NODE_TYPE>_<ENV_ID>
  • ENV_TYPE: Identifier of the environment type (for example QA or Production)
  • NODE_TYPE: Can be Primary (P) or Secondary (S)
  • ENV_ID: An alphanumeric ID used to identify the particular offering instance.

A primary and secondary node associated to the same environment have the same name except for the NODE_TYPE identifier (P or S). For example, primary and secondary nodes related to the environment 001 are named PROD_P_001 and PROD_S_001.


Define the version status to synchronize the phases of the process

The four phases are implemented through four different UrbanCode Deploy application processes.

To ensure the four processes occur in the correct sequence, at the end of each phase the process add a specific status to the version being installed. The application processes are configured to start only when the version has the correct status, as shown in Figure 3. The entire update process can start only if the version has the appropriate status. In this case, the version status serves as an environment gate.

Figure 3. Version status set at the completion of each phase

This configuration is implemented in the following steps:

  1. Configure the version status.
  2. Configure the environment gates.
  3. Configure the version status constraint in the application processes.
  4. Implement the process to add and remove status to a version.

Configure the version status

To define the version status, click Settings > Plugins > Status as shown in Figure 4.

Figure 4. Version status definition
Version status tab

Configure the environment gates

The entire update process starting at Phase 1 can run on the specific environment only if the environment has the appropriate version status. This constraint is specified at the application level.

Click Application > Configuration > Environmentgates to specify the status required by the specific environment. Environment gates provide a mechanism to ensure that component versions cannot be deployed into environments unless they have the gate-specified status. For example in Figure 5, only environments that have the version status Staging_OK can be used on a production environment.

Figure 5. Environment gates definition
Environment gates for staging and production

Configure the status version constraint in the application processes

At the end of the Phase 1, a new status is added to the component version. The status is <ENVIRONMENT>_Phase1_OK. For example, at the end of the production environment phase, the version status is Production_Phase1_OK.

A component version that has the status <ENVIRONMENT>_Phase1_OK is required to start Phase 2. This constraint is specified at application process level, as shown in Figure 6.

Figure 6. Application process definition
Basic settings for PROD_PHASE1

At the completion of this phase, the state <ENVIRONMENT>_Phase1_OK is removed and the status state <ENVIRONMENT>_Phase2_OK is added to the component version.

In a similar manner, Phase 3 requires <ENVIRONMENT> _Phase2_OK version status. Upon its completion, this status is eliminated and <ENVIRONMENT> _Phase3_OK is added.

The last phase requires the state <ENVIRONMENT> _Phase3_OK. At the end of this phase the version is set with status <ENVIRONMENT>_OK, which indicates the completion of the update on that type of environment.

You can monitor the statuses associated with a version by clicking Component > Versions, as shown in Figure 7.

Figure 7. Status associated to each version
Component versions status panel

Implement the process to add and remove status to a version

To implement the status association to a version, define a process component for each change of status, as shown in Figure 8.

Figure 8. Component process definition
Workflow Define process component for status change

Define the Phase 1 application process

In Phase 1, the upgrade of the master component on all the secondary resources is performed and then the new status of the component version is associated, as shown in Figure 9.

Figure 9. Application process definition
Workflow Update version status

The first step runs the process to update the master component on the secondary nodes. In this process, as soon as the update is successfully completed, a property is set at the agent level to indicate that the current version is successfully installed.

Note: The agent on which this property is set is not the agent installed on the secondary node that has been successfully updated. Rather, it is the agent installed on its relative primary node.

The property name is secondaryVersionLevel and the associated value is the ID of the current component version. The property setting is performed in two component process steps.

The first step is to calculate the name of the primary agent associated with the current secondary node where the process is running. Figure 10 shows the step definition.

Figure 10. Component process step definition
Define a new component process step

The post-processing step SetOutputProperty sets the name calculated in the previous step to a variable that can be used in the next step. Define the variable using the script in Listing 1.

Listing 1. Sample of post-processing script
properties.put("result", "default");
if (properties.get("exitCode") != 0) {
	properties.put(new java.lang.String("Status"), new 		java.lang.String("Failure"));
 }
 else {
 properties.put("Status", "Success");
 scanner.register("^Output", function(lineNumber, line) {
 var temp=line.split("=");
var out=temp[1];
 properties.put("result",out);
 });
 scanner.scan();
   }

The second step sets the agent property that specifies the agent name just calculated, as shown in Figure 11.

Figure 11. Set agent property
Set agent property step

At the end of this step, all of the agents installed on the primary node associated with secondary nodes that have successfully completed the upgrade process, have the property secondaryVersionLevel with the current ID version as the value.

The second step of the application process changes the status to the component version, which is running the relative component process defined in the section, "Definition of version status to synchronize the phases of the process". The steps of the application process related to status changes should run only on the first resource that reaches that point, as shown in Figure 12.

Figure 12. Step properties
Application process step

Definition of Phase 2 application process

In the second phase, the server is switched from the primary node to the secondary node, and then a new status is associated to the component version, as shown in Figure 13.

Figure 13. Application process Phase 2
Workflow Switch and update version status

This process runs on the primary node and should be performed only on the sub-environments where Phase 1 finished successfully.

Ensure that the relative secondary node is updated successfully and that Phase 1 finished with success. Before you run the upgrade operation, verify that the secondaryVersionLevel has the current ID version value. Perform this check by using three component steps, as shown in Figure 14.

Figure 14. Part of the component process
Workflow checks upgrade success
  1. The first step retrieves the value of the property, as shown in Figure 15.
Figure 15. Component process step configuration
Retrieve value of property in component process
  1. The second step verifies that the value of the property matches the ID of the current version and sets the result in the property result with a value of true or false, as shown in Figure 16.
Figure 16. Check property step
Post processing step to check the output
  1. The last step depends on the value of the property set in step 2. If the property is false the process is not performed, as shown in Figure 17.
Figure 17. Switch step definition
Define switch on Edit Properties

Definition of Phase 3 application process

The third phase updates the primary node and sets the correct version status.

Note: The primary nodes are currently the inactive nodes, because the servers are switched to the secondary nodes, as shown in Figure 18.

Figure 18. Application process definition Phase 3
Workflow to switch servers

This process can start only if the following conditions are satisfied:

  • The primary node is not active, for example the server has been switched to the secondary node.
  • The update of the secondary node finished successfully and the property secondaryVersionLevel is set to the current version.

As soon as the upgrade finishes successfully on the primary node, an agent property is set.

The agent on which this property is set is not the agent installed on the primary node that has been successfully updated. Rather, it is the agent installed on its relative secondary node. The property name is primaryVersionLevel and the associated value is the ID of the current component version.


Definition of the Phase 4 application process

During the fourth phase the server is switched back from the secondary node to the primary node.

This operation has to be performed only on the sub-environment where the three previous phases completed successfully. This check is performed verifying that the value of the property is primaryVersionLevel.

At the end of this process the status <Environment>_OK is associated with the version, as shown in Figure 19.

Figure 19. Application process definition Phase 4
Workflow Associate process with environment

Conclusion

This article describes the synchronization at different levels of the upgrade phases of a complex environment. These mechanisms are critical to ensure continuity of service in a production environment. This approach can be extended or generalized for all active-active and active-passive configurations that have similar characteristics.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into DevOps on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • DevOps digest

    Updates on continuous delivery of software-driven innovation.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • IBM evaluation software

    Evaluate IBM software and solutions, and transform challenges into opportunities.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=DevOps
ArticleID=974140
ArticleTitle=Managing a continuous delivery process by providing continuity of service
publish-date=06242014