Collocation of IBM.Application with DB2 HADR Primay Node Through Shadow Equivalency

Body

This blog sent in by Daniel Yeap, Advisory support engineer for NetWork Assurance in ITSM

There are times when there is a need to have a resource to be collocated with DB2 HADR primary node. 

 

Most of us would thought that a DependsOn relationship that comes with implicit collocation between the source and target resources could have handled this request easily.

 

Yes, 'DependsOn' could do the job easily, but only in idle situations (i.e. perfect takeover).

 

Consider this...
(1) You performed a HADR takeover.
(2) The new primary failed to go online properly and was declared Failed Offline.
(3) The old primary was ordered online and did that successfully.

 

You would see such opstates:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

From the example above, you can see at least 2 problems:
(i) The HADR resource group (db2_db2inst1_db2inst1_TESTDB-rg) is LOCKED to prevent any further automation (nodes are not in PEER CONNECTED state) and to avoid any data integrity issue.
(ii) Even though HADR primary is ONLINE on node clnode1, the target resource (in this example, TestApp3) is OFFLINE on both nodes.

 

Problem (ii) happened because of the lock on resource group 'db2_db2inst1_db2inst1_TESTDB-rg'. It has caused TSAMP to put a StartInhibitedBecauseSuspended control on resource TestApp3 that prevents it from being started due to unsatisfactory start of its dependent resource (HADR).

 

Even though DB2 HADR primary is actually up and running on node clnode1, the LOCK request has caused resource 'TestApp3' to be unavailable. The outcome might not be satisfactory to business operation.

The question now is "How do we fix this?"

 

This is where 'shadow equivalency' comes into play!

For more information about shadow equivalency or shadow resource, you can refer to http://www-01.ibm.com/support/knowledgecenter/SSRM2X_4.1.0.2/com.ibm.samp.doc_4.1.0.2/sampugbaug_usingshadows.html?lang=en

Instead of creating a 'DependsOn' relationship to the HADR resource directly, we can create a shadow equivalency that consists of shadow resources that monitor the operation state of the HADR pair (ONLINE to represent PRIMARY; OFFLINE to represent STANDBY).

 

Once the shadow equivalency is created, a 'DependsOn' relationship can be created between the resource that needs to be collocated with the HADR primary node (Eg. TestApp3) and the shadow equivalency.

 

In a nutshell, the steps are:
(1) Create a script that cater for the start, stop and monitor operations of the shadow resources.
NOTE: The "start" and "stop" actions do not need to be applied because shadow resource will only use the script for monitoring.

 

Eg.
 

root@clnode1 test]# cat hadr.sh  case $1 in     start)    exit 0;;     stop)    exit 0;;     monitor)         if [[ $(ps -ef | grep db2sysc | grep -v grep | wc -l) -eq 0 ]]         then             exit 2         fi         DB2_TABLE="TESTDB"         DB2_CMD="/home/db2inst1/sqllib/bin/db2 get db cfg for $DB2_TABLE"         HADR_ROLE_PARAM="'HADR database role'"         DB2_NODE_ROLE="$DB2_CMD | grep -i $HADR_ROLE_PARAM"         HADR_ROLE=$(eval $DB2_NODE_ROLE | awk -F'=[ \t]*' '{ print $2 }')         if [[ $HADR_ROLE = 'PRIMARY' ]]         then             logger "hadr.sh[monitor]: 1 [PRIMARY]"             exit 1         else             if [[ $HADR_ROLE = 'STANDBY' ]]             then                 logger "hadr.sh[monitor]: 2 [STANDBY]"                 exit 2                         else                 logger "hadr.sh[monitor]: 0 [UNKNOWN]"                 exit 0             fi         fi;; esac  [root@clnode1 test]#

 

(2) Create the shadow resources with the "mkrsrc" command.
Eg.
mkrsrc IBM.Application Name="shadow_hadr_1" ResourceType=0 NodeNameList="{'clnode1'}" UserName="root" StartCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh start" StopCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh stop" MonitorCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh monitor" MonitorCommandPeriod=5 RunCommandsSync=1

mkrsrc IBM.Application Name="shadow_hadr_2" ResourceType=0 NodeNameList="{'clnode2'}" UserName="root" StartCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh start" StopCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh stop" MonitorCommand="/usr/sbin/rsct/sapolicies/test/hadr.sh monitor" MonitorCommandPeriod=5 RunCommandsSync=1

 

(3) Create an equivalency with a SelectFromPolicy attribute value of NoControl.
Eg.
mkequ -p A,NoControl hadr-equ IBM.Application:shadow_hadr_1:clnode1,shadow_hadr_2:clnode2

 

(4) Create the 'DependsOn' relationship between the resource that needs to be collocated with the HADR primary node (Eg. TestApp3) and the shadow equivalency.
Eg.
mkrel -p dependson -S IBM.Application:TestApp3 -G IBM.Equivalency:hadr-equ test2hadr

After all the above steps are executed, you can test the takeover and it will work smoothly irregardless of any hiccups with the HADR pair!

Eg.