There are procedures to be followed in order to get the geographically dispersed DB2® pureScale® cluster (GDPC) installed, and up and running.
On site A, designate hostA1, hostA2, hostB1, and hostB2 as members where hostB1 is the shared disk member and hostB2 is the tiebreaker member. During install the tiebreaker disk must be set up using one of the LUNs. This is temporary and can be changed later. For the following, an option is to use hdiskA2.
The file system that the db2setup command creates for the shared instance metadata is initially a non-replicated GPFS file system. This is converted later to a replicated file system across the sites.
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -list
-tiebreaker
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -set -tiebreaker –majority
Configuring quorum device for domain 'db2domain_20110224005525' ...
Configuring quorum device for domain 'db2domain_20110224005525' was successful.
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -list
-tiebreaker
The current quorum device is of type Majority Node Set.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsnsd –X
Disk name NSD volume ID Device Devtype Node name Remarks
--------------------------------------------------------
gpfs1nsd 091A33584D65F2F6 /dev/hdiskA1 hdisk hostA1 pr=yes
root@hostA1:/opt/IBM/db2/V10.1/bin> su – db2inst1
db2inst1@hostA1:/home/db2inst1> db2stop force
02/24/2011 01:24:16 0 0 SQL1064N DB2STOP processing was successful.
02/24/2011 01:24:19 1 0 SQL1064N DB2STOP processing was successful.
02/24/2011 01:24:21 3 0 SQL1064N DB2STOP processing was successful.
02/24/2011 01:24:22 2 0 SQL1064N DB2STOP processing was successful.
SQL1064N DB2STOP processing was successful.
db2inst1@hostA1:/home/db2inst1> exit
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cfs -stop –all
All specified hosts have been stopped successfully.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 hostA1 down
2 hostA2 down
3 hostA3 down
4 hostB1 down
5 hostB2 down
6 hostB3 down
Disable SCSI-3 PR by issuing this command:
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmchconfig usePersistentReserve=no
Verifying GPFS is stopped on all nodes ...
mmchconfig: Processing the disks on node hostA1.torolab.ibm.com
mmchconfig: Processing the disks on node hostA2.torolab.ibm.com
mmchconfig: Processing the disks on node hostA3.torolab.ibm.com
mmchconfig: Processing the disks on node hostB1.torolab.ibm.com
mmchconfig: Processing the disks on node hostB2.torolab.ibm.com
mmchconfig: Processing the disks on node hostB3.torolab.ibm.com
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all affected nodes. This
is an asynchronous process.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsnsd -X
Disk name NSD volume ID Device Devtype Node name Remarks
--------------------------------------------------------
gpfs1nsd 091A33584D65F2F6 /dev/hdiskA1 hdisk hostA1
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsconfig
Configuration data for cluster db2cluster_20110224005554.torolab.ibm.com:
-----------------------------------------------------------
clusterName db2cluster_20110224005554.torolab.ibm.com
clusterId 655893150084494058
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxFilesToCache 10000
pagepool 256M
verifyGpfsReady yes
assertOnStructureError yes
worker1Threads 150
sharedMemLimit 2047M
usePersistentReserve no
failureDetectionTime 35
leaseRecoveryWait 35
tiebreakerDisks gpfs1nsd
[hostA1]
psspVsd no
adminMode allToAll
File systems in cluster db2cluster_20110224005554.torolab.ibm.com:
------------------------------------------------------------------
/dev/db2fs1
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -list
-hostfailuredetectiontime
The host failure detection time is 4 seconds.
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -set -option hostfailuredetectiontime
-value 16
The host failure detection time has been set to 16 seconds.
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -list
-hostfailuredetectiontime
The host failure detection time is 16 seconds.
root@T:/path containing db2_install. /db2_install
Default directory for installation of products - /opt/IBM/db2/V10.1
***********************************************************
Do you want to choose a different directory to install [yes/no] ?
no
Specify one of the following keywords to install DB2 products.
ESE_DSF
Enter "help" to redisplay product names.
Enter "quit" to exit.
***********************************************************
ESE_DSF
DB2 installation is being initialized.
Total number of tasks to be performed: 46
Total estimated time for all tasks to be performed: 2850 second(s)
Task #1 start
...
Task #46 end
The execution completed successfully.
For more information see the DB2 installation log at /tmp/db2_install.log.nnnnnnnn.
root@hostA1>/var/db2/db2ssh/db2locssh display_config
version = 1
time_delta = 20 second(s)
debug_level = 2
db2sshid = db2inst1
gdkit_path = /opt/IBM/db2/V10.1/lib64/gskit/
fips_mode = on
/var/db2/db2ssh/db2locssh reset_config
/var/db2/db2ssh/db2locssh set_gskit_path /opt/IBM/db2/V10.1/lib64/gskit/
/var/db2/db2ssh/db2locssh set_db2sshid db2inst1
root@T>/var/db2/db2ssh/db2locssh display_config
version = 1
time_delta = 20 second(s)
debug_level = 2
db2sshid = db2inst1
gdkit_path = /opt/IBM/db2/V10.1/lib64/gskit/
fips_mode = on
/var/db2/db2ssh/db2locssh generate_keys
hostA1:
root@hostA1.priv
root@hostA1.pub
root@hostA2.pub
root@hostA3.pub
root@hostB1.pub
root@hostB2.pub
root@hostB3.pub
root@T.pub
hostB1:
root@hostB1.priv
root@hostB1.pub
root@hostB2.pub
root@hostB3.pub
root@hostA1.pub
root@hostA2.pub
root@hostA3.pub
root@T.pub
T:
root@T.priv
root@T.pub
root@hostA1.pub
root@hostA2.pub
root@hostA3.pub
root@hostB1.pub
root@hostB2.pub
root@hostB3.pub
root@T>/var/db2/db2ssh/db2locssh root@hostA1 hostname
hostA1
root@T>/var/db2/db2ssh/db2locssh root@hostB1 hostname
hostB1
root@T>/var/db2/db2ssh/db2locssh root@hostT hostname
hostT
root@hostA1>/var/db2/db2ssh/db2locssh root@T hostname
T
root@hostB1>/var/db2/db2ssh/db2locssh root@T hostname
T
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cfs -set -tiebreaker –majority
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cfs -list
-tiebreaker
The current quorum device is of type Majority Node Set.
root@T> preprpnode hostA1 hostA2 hostB1 hostB2 hostA3 hostB3
root@hostA1:/opt/IBM/db2/V10.1/bin> lsrpnode
Name OpState RSCTVersion
hostB2 Online 3.1.2.2
hostB3 Online 3.1.2.2
hostA3 Online 3.1.2.2
hostB1 Online 3.1.2.2
hostA2 Online 3.1.2.2
hostA1 Online 3.1.2.2
root@hostA1:/opt/IBM/db2/V10.1/bin> /home/db2inst1/sqllib/bin/db2cluster -cm -add -host T
Adding node 'T' to the cluster ...
Trace spooling could not be enabled on the local host.
Adding node 'T' to the cluster was successful.
root@hostA1:/opt/IBM/db2/V10.1/bin> lsrpnode
Name OpState RSCTVersion
T Online 3.1.2.2
hostB3 Online 3.1.2.2
hostB2 Online 3.1.2.2
hostB1 Online 3.1.2.2
hostA3 Online 3.1.2.2
hostA2 Online 3.1.2.2
hostA1 Online 3.1.2.2
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsnode
GPFS nodeset Node list
------------- ------------------------------------------------
db2cluster_20110224005554 hostA1 hostA2 hostA3 hostB1 hostB2 hostB3
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmaddnode T:quorum-client
Thu Feb 24 01:49:38 EST 2011: mmaddnode: Processing node T.torolab.ibm.com
mmaddnode: Command successfully completed
mmaddnode: Warning: Not all nodes have proper GPFS license designations.
mmaddnode: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsnode
===============================================================
| Warning: |
| This cluster contains nodes that do not have a proper GPFS license |
| designation. This violates the terms of the GPFS licensing agreement. |
| Use the mmchlicense command and assign the appropriate GPFS licenses |
| to each of the nodes in the cluster. For more information about GPFS |
| license designation, see the Concepts, Planning, and Installation Guide. |
===============================================================
GPFS nodeset Node list
------------- ----------------------------------------------
db2cluster_20110224005554 hostA1 hostA2 hostA3 hostB1 hostB2 hostB3 T
root@T:/opt/IBM/db2/V10.1/bin> ./db2cluster -cfs -add –license
The license for the shared file system cluster has been successfully added.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsnode
GPFS nodeset Node list
------------- --------------------------------------------------
db2cluster_20110224005554 hostA1 hostA2 hostA3 hostB1 hostB2 hostB3 T
echo "example text" > /var/mmfs/etc/ignoreStartupMount
If
SCSI-3 PR is required, you can turn on the SCSI-3 PR flags by running
the following commands:
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsconfig
Configuration data for cluster db2cluster_20110224005554.torolab.ibm.com:
----------------------------------------------------------
clusterName db2cluster_20110224005554.torolab.ibm.com
clusterId 655893150084494058
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxFilesToCache 10000
pagepool 256M
verifyGpfsReady yes
assertOnStructureError yes
worker1Threads 150
sharedMemLimit 2047M
usePersistentReserve no
failureDetectionTime 35
leaseRecoveryWait 35
[T]
unmountOnDiskFail yes
[common]
[hostA1]
psspVsd no
adminMode allToAll
File systems in cluster db2cluster_20110224005554.torolab.ibm.com:
------------------------------------------------------------------
/dev/db2fs1
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 hostA1 down
2 hostA2 down
3 hostA3 down
4 hostB1 down
5 hostB2 down
6 hostB3 down
7 T down
root@hostA1:/> /usr/lpp/mmfs/bin/mmchconfig failureDetectionTime=30
Verifying GPFS is stopped on all nodes ...
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
root@hostA1:/> /usr/lpp/mmfs/bin/mmchconfig leaseRecoveryWait=25
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
root@hostA1:/opt/IBM/db2/V10.1/bin> ping hostA1-ib0
PING hostA1-ib0.torolab.ibm.com (10.1.1.1): 56 data bytes
64 bytes from 10.1.1.1: icmp_seq=0 ttl=255 time=0 ms
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmchconfig subnets=10.1.1.0
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
root@hostA1:/opt/IBM/db2/V10.1/bin> /usr/lpp/mmfs/bin/mmlsconfig
Configuration data for cluster db2cluster_20110224005554.torolab.ibm.com:
--------------------------------------------------------
clusterName db2cluster_20110224005554.torolab.ibm.com
clusterId 655893150084494058
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxFilesToCache 10000
pagepool 256M
verifyGpfsReady yes
assertOnStructureError yes
worker1Threads 150
sharedMemLimit 2047M
usePersistentReserve no
failureDetectionTime 30
leaseRecoveryWait 25
[T]
unmountOnDiskFail yes
[common]
subnets 10.1.1.0
[hostA1]
psspVsd no
adminMode allToAll
File systems in cluster db2cluster_20110224005554.torolab.ibm.com:
------------------------------------------------------------------
/dev/db2fs1
root@hostA1> /usr/lpp/mmfs/bin/mmchcluster -p hostA1 -s hostB1
root@hostA1:/> /usr/lpp/mmfs/bin/mmlscluster
GPFS cluster information
========================
GPFS cluster name: db2cluster_20110224005554.torolab.ibm.com
GPFS cluster ID: 655893150084494058
GPFS UID domain: db2cluster_20110224005554.torolab.ibm.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
Primary server: hostA1.torolab.ibm.com
Secondary server: hostB1.torolab.ibm.com
root@hostA1:/> lscomg
Name Sensitivity Period Priority Broadcast SourceRouting NIMPathName NIMParameters Grace MediaType UseForNodeMembership
CG1 4 1.6 1 Yes Yes 60 1 (IP) 1
CG2 4 1.6 1 Yes Yes 60 1 (IP) 1
root@hostA1:/> chcomg –x r CG1
root@hostA1:/> chcomg –x r CG2
root@hostA1:/> lscomg
Name Sensitivity Period Priority Broadcast SourceRouting NIMPathName NIMParameters Grace MediaType UseForNodeMembership
CG1 4 1.6 1 Yes No 60 1 (IP) 1
CG2 4 1.6 1 Yes No 60 1 (IP) 1
Note that if at anytime the db2cluster -cm -delete -domain/create domain commands are run to recreate the TSA domain, then LSR needs to be disabled again.
root:/> cat /etc/hosts
10.1.1.1 hostA1-ib0.torolab.ibm.com hostA1-ib0
10.1.1.2 hostA2-ib0.torolab.ibm.com hostA2-ib0
10.1.1.3 hostA3-ib0.torolab.ibm.com hostA3-ib0
10.1.1.4 hostB1-ib0.torolab.ibm.com hostB1-ib0
10.1.1.5 hostB2-ib0.torolab.ibm.com hostB2-ib0
10.1.1.6 hostB3-ib0.torolab.ibm.com hostB3-ib0
9.26.82.1 hostA1.torolab.ibm.com hostA1
9.26.82.2 hostA2.torolab.ibm.com hostA2
9.26.82.3 hostA3.torolab.ibm.com hostA3
9.26.82.4 hostB1.torolab.ibm.com hostB1
9.26.82.5 hostB2.torolab.ibm.com hostB2
9.26.82.6 hostB3.torolab.ibm.com hostB3
9.23.1.12 T
After the cluster has been installed and is running, set up GPFS replication.