Migrate Collaborative Lifecycle Management applications to a cluster: Part 1. A high availability approach

This article helps administrators of the Rational solution for Collaborative Lifecycle Management (CLM) migrate their stand-alone setups to a highly available cluster environment. The authors explain the CLM configuration for the high-availability deployment that they used for this article. Then they provide steps and sample code for making backups of the CLM databases available quickly by using a standby server and the IBM DB2 High Availability Disaster Recovery (HADR) feature. That server takes over quickly when the databases are not available on the primary server for any reason.

Murali Dhandapani (muralindia@in.ibm.com), IT Specialist, IBM

author photoMurali Dhandapani is part of the Operations Software Engineering Services team at IBM Software Labs, in India. He is an IBM Certified IT Specialist in System Management and a technical lead for Rational Jazz-based products infrastructure deployment. He is a contributing author for developerWorks.


developerWorks Contributing author
        level

Anil Sharma (anil_sharma@in.ibm.com), Database Administrator, IBM

author photoAnil Sharma is lead database administrator for the Cloud Platform team in the Industry Solutions Group at IBM Software Labs in India. He is an IBM Certified Advanced Database Administrator for IBM DB2 and an IBM Certified Solution Designer for IBM InfoSphere Warehouse.



19 March 2013

Introduction

High availability is a critical requirement for projects that must maintain source code data on a single server. The downtime of the server relates to production downtime and business loss, to a certain extent.

Part 1 of this two-part article explains using a cold standby server for high availability of applications that are part of the Rational solution for Collaborative Lifecycle Management (CLM). These include IBM® Rational Team Concert™, Rational® Quality Manager, and Rational Requirements Composer. The authors also provide instructions for using the High Availability Disaster Recovery (HADR) feature of IBM® DB2® to make the CLM databases highly available. Then they show how to apply this approach while performing any fix pack upgrade on the database servers and in a CLM application failover situation.


Collaborative Lifecycle Management setup example

The diagram in Figure 1 shows the setup used as the basis for this article. The CLM server hosts Rational Team Concert Version 4.0.1, Rational Quality Manager V4.0.1, Rational Requirements Composer 4.0.1, and IBM® WebSphere® Application Server Version 8.0.0.3. As denoted in the diagram, the active CLM server has two IP addresses: 202.12.167.8 is the base IP and 202.12.167.138 is the alias IP, resolving to a public URI. The CLM cold standby server configurations are similar to the active CLM server, but here, the installed applications are in stop states and it do not have any alias IPs.

Figure 1. Collaborative Lifecycle Management with cold standby and DB2 HADR
Flow diagram of the setup

To make this exercise simple, we configured the CLM applications (Rational Team Concert, Rational Quality Manager, and Rational Requirements Composer) on the same WebSphere Application Server, using an application profile. This maintains the same public URI for all three of the applications. See Resources for the link to "System requirements for CLM 2012," which provides the list of supported hardware and software details. On the database server, the JTS (Jazz™ Team Server), CCM, QM, and DW databases are created. See Resources for the link to "Installing the Rational solution for CLM," which provides the details for setting up the database and then deploying and starting the CLM applications on the server.

Notes:

  1. For production use, it is best to have Rational Team Concert, Rational Quality Manager, and Rational Requirements Composer on different servers.
  2. The CLM tools rely heavily on indexing to help with performance. Make sure that the cold standby CLM server contains the same index files. You can either sync those folders to the cold standby server or restore the latest backup copy. See Resources for the link to "Back up the Rational solution for Collaborative Lifecycle Management."

DB2 HADR configuration for CLM databases

DB2 High Availability Disaster Recovery (HADR) is a database replication feature that provides high availability backup for both partial and complete site failure. HADR is designed for quick failover, as well as easy setup and management. It transmits the log records from the primary database server to the standby server. The HADR standby database applies the entire log record to its copy of the database, keeping it synchronized with the primary database server. This operation does not affect the application. The amount of data transferred between the systems is limited to only logged operations. Both systems perform the actual changes to data storage pages independently, which reduces the actual bandwidth required. The standby server is in a continuous roll-forward mode and is always in a state of near-readiness, so that the standby database can take over rapidly.

HADR setup requirements

Setting up the HADR backup requires the following conditions:

  • HADR is a DB2 optimal licensing feature, so you need the Enterprise Server Edition for full use of HADR.
  • The operating systems on both primary and standby DB servers must be the same version.
  • A TCP/IP interface must be enabled between primary and standby DB servers.
  • The DB2 version and level must be the same on both the primary and the standby DB servers.
  • The DB2 software for both the primary and the standby database must have the same bit size.
  • The primary and the standby databases must have the same database name.
  • The amount of space allocated for log files must be same on both the primary and the standby databases.
  • The database table space must be identical on the primary and the standby databases.

Listings 1 and 2 show the details for both the primary and standby servers.

Listing 1. Primary DB server details
Host Name			:islswestest5.in.ibm.com
Primary DB instance		:clm401is
Primary Port			:50001
Database Name			:DW
HADR port for database 'DW'	:64000
Database Name			:QM
HADR port for database 'QM'	:64002
Database Name			:CCM
HADR port for database 'CCM'	:64004
Database Name			:JTS
HADR port for database 'JTS'	:64006
Listing 2. Standby database server details
Host Name			:islrpbeipl112.in.ibm.com
Primary DB instance		:clm401is
Primary Port			:50001
Database Name			:DW
HADR port for database 'DW'	:64001
Database Name			:QM
HADR port for database 'QM'	:64003
Database Name			:CCM
HADR port for database 'CCM'	:64005
Database Name			:JTS
HADR port for database 'JTS'	:64007

Prepare the environment

Configure the HADR database for Archival logging.

  1. Set the required database configuration parameter. The following commands help in creating the log archival for the DW database.
Listing 3. Commands for creating an archival log
$su – clm401is

$db2 connect to DW

 Database Connection Information
 
 Database server        = DB2/AIX64 9.7.4
 SQL authorization ID   = CLM401IS
 Local database alias   = DW

$db2 "update db cfg using LOGRETAIN RECOVERY LOGARCHMETH1 "DISK:/home/clm401is/
archived_logs/DW" LOGINDEXBUILD ON"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W  One or more of the parameters submitted for immediate modification were not 
changed dynamically. For these configuration parameters, all applications must 
disconnect from this database before the changes become effective.

$db2 disconnect DW
DB20000I  The SQL DISCONNECT command completed successfully.
LOGRETAIN
This parameter with value='RECOVERY' indicates that active log files are retained and can be used for roll-forward recovery.
 
LOGARCHMETH1
This parameter with value DISK (followed by fully qualified existing path name) indicates where the log files will be archived.

Note:
As a best practice, have the archive log path on an independent file system, either on a separate NAS or SAN volume.

  1. Repeat the commands for the other databases: JTS, CCM, and QM.
  2. Also, set the LOGINDEXBUILD parameter for all candidate databases: DW, QM, CCM, and JTS. With this parameter set to ON, index creation, recreation, or reorganization operations are logged. The commands in Listing 4 help in setting this parameter for the DW database.
Listing 4. Commands for setting the LOGINDEXBUILD parameter
$su – clm401is

$db2 connect to DW

 Database Connection Information

 Database server        = DB2/AIX64 9.7.4
 SQL authorization ID   = CLM401IS
 Local database alias   = DW

$db2 "update db cfg using LOGINDEXBUILD ON"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 connect reset
DB20000I  The SQL command completed successfully.
  1. Repeat the commands for the other database JTS, CCM, and QM.
  2. Make a full backup of all of the candidate databases, and move the backup images to the standby server. The commands in Listing 5 help in making backup for database DW, JTS, CCM, and QM.
Listing 5. Database backup command
$ db2 "backup database DW to /home/clm401is/db_backup include logs"
Backup successful. The timestamp for this backup image is : 20130124151317

$ db2 "backup database JTS to /home/clm401is/db_backup include logs"
Backup successful. The timestamp for this backup image is : 20130124152301

$ db2 "backup database CCM to /home/clm401is/db_backup include logs"
Backup successful. The timestamp for this backup image is : 20130124151840

$ db2 "backup database QM to /home/clm401is/db_backup include logs"
Backup successful. The timestamp for this backup image is : 20130124151554

Note:
The /home/clm401is/db_backup path is where backup images are kept. The DB instance owner should have read write access to this path.

  1. Copy the DB backup images to the standby server to create a DB clone on that server. The commands in Listing 6 help in transferring the backup file to standby server.
Listing 6. Commands for transferring the backup file to the standby server
$ scp -Cr DW.0.clm401is.NODE0000.CATN0000.20130124151317.001 
clm401is@202.12.167.153:/home/clm401is/db_backup
clm401is@202.12.167.153's password:
DW.0.clm401is.NODE0000.CATN0000.20130124151317.001
100% 3048MB  34.6MB/s   01:28

$ scp -Cr QM.0.clm401is.NODE0000.CATN0000.20130124151554.001 
clm401is@202.12.167.153:/home/clm401is/db_backup
clm401is@202.12.167.153's password:
QM.0.clm401is.NODE0000.CATN0000.20130124151554.001
100% 2091MB  27.2MB/s   01:17

$ scp -Cr CCM.0.clm401is.NODE0000.CATN0000.20130124151840.001 
clm401is@202.12.167.153:/home/clm401is/db_backup
clm401is@202.12.167.153's password:
CCM.0.clm401is.NODE0000.CATN0000.20130124151840.001
100% 1072MB  27.5MB/s   00:39

$ scp -Cr JTS.0.clm401is.NODE0000.CATN0000.20130124152301.001 
clm401is@202.12.167.153:/home/clm401is/db_backup
clm401is@202.12.167.153's password:
JTS.0.clm401is.NODE0000.CATN0000.20130124152301.001
100%  784MB  21.2MB/s   00:37
  1. Log in to the standby server, and use the commands in Listing 7 to restore the transferred databases of the primary server on this server.
Listing 7. Commands to restore the primary databases
$ su – clm401is

$ db2 "restore database DW from /home/clm401is/db_backup"
DB20000I  The RESTORE DATABASE command completed successfully.

$ db2 "restore database QM from /home/clm401is/db_backup"
DB20000I  The RESTORE DATABASE command completed successfully.

$ db2 "restore database CCM from /home/clm401is/db_backup"
DB20000I  The RESTORE DATABASE command completed successfully.

$ db2 "restore database JTS from /home/clm401is/db_backup"
DB20000I  The RESTORE DATABASE command completed successfully.

Note:
Do not use Without Rolling Forward with the restore command. The standby database should always be in a continuous roll-forward mode and always in a state of near readiness, so that the move to the standby database is extremely fast.

  1. Verify whether all the restored databases are in "roll-forward" pending status by using the commands as shown in Listing 8.
Listing 8. Commands to verify status of the restored databases
$ db2 connect to DW
SQL1117N  A connection to or activation of database "DW" cannot be made
because of ROLL-FORWARD PENDING. SQLSTATE=57019

$ db2 connect to QM
SQL1117N  A connection to or activation of database "QM" cannot be made
because of ROLL-FORWARD PENDING. SQLSTATE=57019

$ db2 connect to CCM
SQL1117N  A connection to or activation of database "CCM" cannot be made
because of ROLL-FORWARD PENDING. SQLSTATE=57019

$ db2 connect to JTS
SQL1117N  A connection to or activation of database "JTS" cannot be made
because of ROLL-FORWARD PENDING. SQLSTATE=57019

HADR database setup

In this section, we describe steps for the DB2 HADR configurations between the two database servers, which are islswestest5.in.ibm.com and islrpbeipl112.in.ibm.com in this example.

Configure the database servers

  1. Configure the /etc/hosts file.

    HADR configuration uses host names (hostnames). DB2 then resolves those host names to IP addresses by using the hosts file (/etc/hosts). Manually add the following entries in the host files of both the primary and standby DB servers.
    	202.12.167.139   islswestest5.in.ibm.com islswestest5
    	202.12.167.153   islrpbeipl112.in.ibm.com islrpbeipl112
  2. Configure the /etc/services file.

    DB2 must start a HADR service on both the primary and standby nodes, which transfers logs between primary and standby databases, using the service port provided. Add the following entries to the /etc/services file as root on both the primary and standby database servers:
    	DB2_HADR_DWp    64000/tcp
    	DB2_HADR_DWs    64001/tcp
    	DB2_HADR_QMp    64002/tcp
    	DB2_HADR_QMs    64003/tcp
    	DB2_HADR_CCMp   64004/tcp
    	DB2_HADR_CCMs   64005/tcp
    	DB2_HADR_JTSp   64006/tcp
    	DB2_HADR_JTSs   64007/tcp
    Note:
    Ensure that both primary and standby database servers have identical entries in the /etc/services files. Also, make sure that port numbers used for HADR port entries are unique, compared to other entries in the /etc/services files.
  3. Turn off automatic startup of the database instance on both primary and standby servers. In this example, the instance name is clm401is.
  4. Run the following command on both primary and standby database servers:
    	$ su – clm401is
    	
    	cd sqllib/bin
    	
    	./db2iauto -off clm401is

    Apply the configuration parameters to all databases, both primary and standby
  5. Apply the HADR database configuration parameters for all databases on both primary and secondary database servers. The commands that follow in Listings 9 through 17 help in configuring HADR setting for the DW, QM, CCM and JTS databases, respectively. The command varies with port numbers while configuring it on both primary and secondary database servers.
Listing 9. HADR configuration for the DW database on the primary database server
$db2 "update db cfg for DW using hadr_local_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_local_svc 64000"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_svc 64001"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 update alternate server for database DW using hostname 202.12.167.153 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 10. HADR configuration for the DW database on the standby DB server
$db2 "update db cfg for DW using hadr_local_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_local_svc 64001"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_svc 64000"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for DW using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database DW using hostname 202.12.167.139 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 11. HADR configuration for the QM database on the primary DB server
$db2 "update db cfg for QM using hadr_local_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_local_svc 64002"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_svc 64003"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database QM using hostname 202.12.167.153 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 12. HADR configuration for the QM database on the standby DB server
$db2 "update db cfg for QM using hadr_local_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_local_svc 64003"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_svc 64002"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for QM using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database QM using hostname 202.12.167.139 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 13. HADR configuration for the CCM database on the primary DB server
$db2 "update db cfg for CCM using hadr_local_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_local_svc 64004"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_svc 64005"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database CCM using hostname 202.12.167.153 port 50001	
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 14. HADR configuration for the CCM database on the standby DB server
$db2 "update db cfg for CCM using hadr_local_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_local_svc 64005"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_svc 64004"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for CCM using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database CCM using hostname 202.12.167.139 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 15. HADR configuration for the JTS database on the primary DB server
$db2 "update db cfg for JTS using hadr_local_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_local_svc 64006"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_svc 64007"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database JTS using hostname 202.12.167.153 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
Listing 16. HADR configuration for the JTS database on the standby DB server
$db2 "update db cfg for JTS using hadr_local_host islrpbeipl112.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_host islswestest5.in.ibm.com"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_local_svc 64007"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_svc 64006"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_remote_inst clm401is"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_timeout 120"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$db2 "update db cfg for JTS using hadr_syncmode nearsync"
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 update alternate server for database JTS using hostname 202.12.167.139 port 50001
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

Summary of HADR configuration parameters

hadr_local_host
This Parameter specifies the local host for HADR TCP communication.
 
hadr_remote_host
This parameter specifies the TCP/IP host name or IP address of the remote HADR database server.
 
hadr_local_svc
This parameter specifies the TCP service name or port number for which the local HADR process accepts a connection.
 
hadr_remote_svc
This parameter specifies the TCP service name or port number that will be used by the remote HADR database server.
 
hadr_remote_inst
This parameter specifies the instance name of the remote server.
 
hadr_timeout
This parameter specifies the time (in seconds) that the HADR process waits before considering a communication attempt to have failed.
 
HADR_syncmode
The synchronization mode is the most important HADR configuration parameter. The following types are the three types of synchronizations (we used the NEARSYNC method in this setup):
  • SYNC
    Transactions on the primary server commit only after relevant logs are written to disk on both primary and standby servers.
  • NEARSYNC
    Transactions on the primary server commit only after relevant logs are written to disk on the primary server and received into memory on the standby server.
  • ASYNC
    Transactions on the primary server commit only after relevant logs are written to the local disk and sent to the standby server.

The choice of HADR synchronization mode depends on network speed. For a WAN, ASYNC mode is recommended because transmission latency has no effect on HADR performance. For a LAN, the choice is most often between SYNC and NEARSYNC.

Verify HADR configuration

Use the command in Listing 16 to get HADR configuration for all of the candidate databases. Repeat the command for QM, CCM, and JTS databases on both primary and standby servers. Make sure that all of the HADR configuration parameters are set with correct values.

Listing 17. Verify HADR configuration for each database
$db2 get db cfg for DW|grep "HADR"
HADR database role                                      = PRIMARY
HADR local host name                  (HADR_LOCAL_HOST) = islswestest5.in.ibm.com
HADR local service name                (HADR_LOCAL_SVC) = 64000
HADR remote host name                (HADR_REMOTE_HOST) = islrpbeipl112.in.ibm.com
HADR remote service name              (HADR_REMOTE_SVC) = 64001
HADR instance name of remote server  (HADR_REMOTE_INST) = clm401is
HADR timeout value                       (HADR_TIMEOUT) = 120
HADR log write synchronization mode     (HADR_SYNCMODE) = NEARSYNC
HADR peer window duration (seconds)  (HADR_PEER_WINDOW) = 0

Start HADR on both servers, starting with the standby one

As a best practice, start the standby database server before the primary database server. The following commands help to start the JTS, CCM, QM, and DW databases on the standby and primary database server as shown in Listing 19 and 20, respectively.

Listing 18. Start HADR on the standby DB server
$db2 "start hadr on database DW as standby"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database QM as standby"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database CCM as standby"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database JTS as standby"
DB20000I  The START HADR ON DATABASE command completed successfully.
Listing 19. Start HADR on Primary DB server
$db2 "start hadr on database DW as primary"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database QM as primary"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database CCM as primary"
DB20000I  The START HADR ON DATABASE command completed successfully.

$db2 "start hadr on database JTS as primary"
DB20000I  The START HADR ON DATABASE command completed successfully.

Verify HADR status on primary and standby database servers

Verify HADR status on both primary and standby DB servers, and ensure that they are in sync.

  1. Use the commands that follow for all of the CLM databases. Ensure that the databases are in the Peer state and with correct roles assigned. The listings that follow are examples for the CCM database on primary and standby.
  2. Repeat these commands for remaining CLM databases and verify their status.
Listing 20. Primary DB server HADR status
$db2pd -db CCM -hadr

Database Partition 0 -- Database CCM -- Active -- Up 0 days 00:05:27 -- 
Date 01/28/2013 20:04:28

HADR Information:
Role    State                SyncMode HeartBeatsMissed   LogGapRunAvg (bytes)
PrimaryPeer                 Nearsync 0                  3514

ConnectStatus ConnectTime                           Timeout
Connected     Mon Jan 28 20:00:05 2013 (1359383405) 120

LocalHost                                LocalService
islswestest5.in.ibm.com                  64004

RemoteHost                               RemoteService      RemoteInstance
islrpbeipl112.in.ibm.com                 64005              clm401is

PrimaryFile  PrimaryPg  PrimaryLSN
S0000057.LOG 34         0x000000006BF62B4A

StandByFile  StandByPg  StandByLSN
S0000057.LOG 33         0x000000006BF61A40
Listing 21. Standby DB server HADR status
$db2pd -db CCM -hadr

Database Partition 0 -- Database CCM -- Standby -- Up 0 days 00:04:58 -- 
Date 01/28/2013 20:03:04

HADR Information:
Role    State                SyncMode HeartBeatsMissed   LogGapRunAvg (bytes)
StandbyPeer                 Nearsync 0                  2607

ConnectStatus ConnectTime                           Timeout
Connected     Mon Jan 28 19:58:15 2013 (1359383295) 120

LocalHost                                LocalService
islrpbeipl112.in.ibm.com                 64005

RemoteHost                               RemoteService      RemoteInstance
islswestest5.in.ibm.com                  64004              clm401is

PrimaryFile  PrimaryPg  PrimaryLSN
S0000057.LOG 35         0x000000006BF633C0

StandByFile  StandByPg  StandByLSN         StandByRcvBufUsed
S0000057.LOG 34         0x000000006BF62FDD 0%

HADR test scenarios for CLM applications

The two scenarios that follow explain how to apply this approach in common situations.

Test scenario 1. Planned takeover of database for fix pack upgrade without application downtime

Figure 2 shows that the primary database server is under maintenance. A database fix pack upgrade requires the database to be down, or unavailable. But by using the DB2 HADR configuration, you can avoid the application downtime and complete the upgrade seamlessly. Do not apply a fix pack to a running database.

Figure 2. Collaborative Lifecycle Management setup with primary database server down for maintenance
primary database server is down for maintenance

HADR provides high availability during a fix pack upgrade by using role exchange. During an upgrade, you switch roles, making the standby server the primary one. We do the role switching from the standby server when the databases are in Peer state. The TAKEOVER command fails if the databases are in any other state.

  1. Verify the alternative server hostname and port setting for the database. The following command helps to check those details from the primary database server.
Listing 22. List all database directory
$ db2 list database directory

 System Database Directory

 Number of entries in the directory = 4

Database 1 entry:

 Database alias                       = DW
 Database name                        = DW
 Local database directory             = /home/clm401is
 Database release level               = d.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = 202.12.167.153
 Alternate server port number         = 50001

Database 2 entry:

 Database alias                       = QM
 Database name                        = QM
 Local database directory             = /home/clm401is
 Database release level               = d.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = 202.12.167.153
 Alternate server port number         = 50001

Database 3 entry:

 Database alias                       = JTS
 Database name                        = JTS
 Local database directory             = /home/clm401is
 Database release level               = d.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = 202.12.167.153
 Alternate server port number         = 50001

Database 4 entry:

 Database alias                       = CCM
 Database name                        = CCM
 Local database directory             = /home/clm401is
 Database release level               = d.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = 202.12.167.153
 Alternate server port number         = 50001
  1. Run the deactivate command from the standby database server for each database, and stop the instance of the database.
Listing 23. Database deactivate and stop command
$ db2 deactivate db DW

$ db2 deactivate db QM

$ db2 deactivate db CCM

$ db2 deactivate db JTS

$ db2stop

Important:
In a HADR environment, the fix pack upgrade should be done on the standby server first, and then on the primary server.

  1. Apply the fix pack or perform any planned activity on the standby database server (islrpbeipl112.in.ibm.com in this example).
  2. After completing the activity, start the database instance on the standby server and start HADR for all databases as STANDBY, as shown in Listing 24.
Listing 24. Database start and HADR start command
$ su – clm401is

$ db2start

$ db2 start HADR on database DW as STANDBY
DB20000I  The START HADR ON DATABASE command completed successfully.

$ db2 start HADR on database JTS as STANDBY
DB20000I  The START HADR ON DATABASE command completed successfully.

$ db2 start HADR on database QM as STANDBY
DB20000I  The START HADR ON DATABASE command completed successfully.

$ db2 start HADR on database CCM as STANDBY
DB20000I  The START HADR ON DATABASE command completed successfully.

Now you need to perform the fix pack upgrade on the primary database server.

  1. First, run the following database takeover command from the standby database server for each database. This makes the standby database server to become primary and vice versa.
Listing 25. Database takeover command
$ db2 takeover hadr on database DW
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database JTS
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database CCM
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database QM
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.
  1. From the standby database server, verify that the role of the database has become primary for each database.
Listing 26. Database role verification command
$ db2pd -db CCM -HADR

Database Partition 0 -- Database CCM -- Active -- Up 5 days 23:55:31 -- 
Date 02/05/2013 15:00:18

HADR Information:
Role    State                SyncMode HeartBeatsMissed   LogGapRunAvg (bytes)
Primary Peer                 Nearsync 0                  271351

ConnectStatus ConnectTime                           Timeout
Connected     Fri Feb  1 17:15:44 2013 (1359719144) 120

LocalHost                                LocalService
islrpbeipl112.in.ibm.com                 64005

RemoteHost                               RemoteService      RemoteInstance
islswestest5.in.ibm.com                  64004              clm401is

PrimaryFile  PrimaryPg  PrimaryLSN
S0000086.LOG 3363       0x0000000089C63DE5

StandByFile  StandByPg  StandByLSN
S0000086.LOG 3363       0x0000000089C63C85

Now the application server should be using the new primary database server, islrpbeipl112.in.ibm.com, without much interruption.

  1. Verify that there is no active database connection on the previous primary database server (islswestest5.in.ibm.com) and that the previous standby database server (islrpbeipl112.in.ibm.com) has become primary, with active connections as shown in Listing 28.
Listing 27. List active database command and its output from islswestest5.in.ibm.com
$ db2 list active databases

                           Active Databases

Database name                              = JTS
Applications connected currently           = 0
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00001/

Database name                              = DW
Applications connected currently           = 0
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00004/

Database name                              = CCM
Applications connected currently           = 0
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00002/

Database name                              = QM
Applications connected currently           = 0
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00003/
Listing 28. List active database command and its output from islrpbeipl112.in.ibm.com
$ db2 list active databases

                           Active Databases

Database name                              = DW
Applications connected currently           = 0
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00001/

Database name                              = JTS
Applications connected currently           = 2
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00004/

Database name                              = QM
Applications connected currently           = 3
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00002/

Database name                              = CCM
Applications connected currently           = 3
Database path                              = /home/clm401is/clm401is/NODE0000/SQL00003/
  1. Now repeat Steps 2, 3, and 4 on the previous primary database server (in this example, that is islswestest5.in.ibm.com).

Now, the databases on the previous primary database server (islswestest5.in.ibm.com) will have the standby role for all the databases. To bring the database role as primary and to make the application server use this database server (islswestest5.in.ibm.com), run the takeover commands in Listing 29. This brings the setup back to its original state without interrupting the application, which instantly starts accessing the databases from islswestest5.in.ibm.com.

Listing 29. Database takeover command
$ db2 takeover hadr on database DW
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database JTS
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database CCM
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

$ db2 takeover hadr on database QM
DB20000I  The TAKEOVER HADR ON DATABASE command completed successfully.

Test scenario 2. Planned or unplanned downtime for the CLM application server

  1. When the application server goes down, if it is planned downtime, remove the alias IP from the islswestest14.in.ibm.com server, and create the alias on islswestest3.in.ibm.com, as shown in Figure 3.
Figure 3. Collaborative Lifecycle Management setup with primary CLM application server downtime
network diagram

In case of unplanned downtime, create the alias IP on the cold standby server. However, ensure proper planned downtime for the application server while you enable it back on the primary server to avoid IP conflict issues. The cold standby server is already installed and configured with the CLM application. The JTS, CCM, and QM configuration files are pointing to the primary database server in their respective teamserver.properties files.

Note:
To avoid IP conflict, add the IP alias to the servers temporarily.

  1. Create the alias IP 202.12.167.138 on the cold standby server, and start the CLM application server as shown in Listing 30.
Listing 30. Configure alias IP and start CLM
$ ifconfig en0 202.12.167.138 netmask 255.255.255.0 alias
$ cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin
$ ./startServer.sh server1

Note:
The ifconfig command temporarily adds the alias IP on the cold standby server.

  1. Now run the https://islswestest4.in.ibm.com:9443/jts/setup, using the cold standby server to ensure that the application is able to connect to the Java Team Server (JTS) database. Because all of the CLM databases (JTS, CCM, and QM) were already in use by the other CLM server, which is currently down, you need to remove the lock id, as explained in Figure 4, for each database.
Figure 4. Database lock error notice
screen capture of the error message
  1. Log in to the cold standby server, and stop the CLM application server.
  2. Run the lock reset commands for all of the databases, and then start the CLM application server again, using the commands shown in Listing 31.
Listing 31. Reset the lock ID
$ cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin

$ ./stopServer.sh server1

$ cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/JazzTeamServer/server

$ ./repotools-jts.sh -resetRepoLockId

$ ./repotools-ccm.sh – resetRepoLockId

$ ./repotools-qm.sh –resetRepoLockId

$ cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin

$ ./startServer.sh server1

Now you should be able to access the CLM application server by using this URL without any problems:
https://islswestest4.in.ibm.com:9443/jts/admin

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=861192
ArticleTitle=Migrate Collaborative Lifecycle Management applications to a cluster: Part 1. A high availability approach
publish-date=03192013