Technical Blog Post
Abstract
Adding a storage group in pureScale : A practical example
Body
Adding a storage group in DB2 is done by very simple command below.
db2 "create stogroup testgroup on '/path1','/path2'"
As a DBA, you wait for new disks to be attached and ready from system administrator and just run the command.
On pureScale, you should better to understand that has more things behind the scene related to TSA (Tivoli System Automation) and GPFS (The General Parallel File System).
In short, that command does interface with TSA to add the resources accordingly.
Before this, you may also need to work on GPFS file system level commands by yourself, not leaning on system administrator.
I will show you
- how it looks like on this blog
- what should we do to verify the result
- and a sample error situation you may encounter without the proper TSA license.
( NOTE : Refer the user name whether 'db2inst1' or 'root' for every sample command line. .)
1. Firstly, let me introduce my test environment.
2 hosts and co-located member/CF for each.
myhost4- member 0 / CF 128
myhost3- member 1 / CF 129
DB2 instance name : db2inst1
DB name : SAMPLE
AIX 7.1
DB2 : V10.5
I have one default storage group on a GPFS file system '/db2fs01'.
db2inst1@myhost3:/home/db2inst1>db2pd -d sample -storagepaths
Database Member 1 -- Database SAMPLE -- Active -- Up 0 days 00:00:16 -- Date 2016-12-23-11.01.20.091461
Storage Group Configuration:
Address SGID Default DataTag Name
0x0A00020012301CA0 0 Yes 0 IBMSTOGROUP
Storage Group Statistics:
Address SGID State Numpaths NumDropPen
0x0A00020012301CA0 0 0x00000000 1 0
Storage Group Paths:
Address SGID PathID PathState PathName
0x0A0002001235E000 0 0 InUse /db2fs01
db2inst1@myhost3:/home/db2inst1>db2pd -d sample -storagegroups
Database Member 1 -- Database SAMPLE -- Active -- Up 0 days 00:32:56 -- Date 2016-12-23-11.34.00.401244
Storage Group Configuration:
Address SGID Default DataTag Name
0x0A00020012301CA0 0 Yes 0 IBMSTOGROUP
Storage Group Statistics:
Address SGID State Numpaths NumDropPen
0x0A00020012301CA0 0 0x00000000 1 0
Storage Group Paths:
Address SGID PathID PathState PathName
0x0A0002001235E000 0 0 InUse /db2fs01
'/db2fs01' is the GPFS file system that was created during pureScale installation.
And it's added to HA and TSA configuration.
db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0
db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
'- Online IBM.Application:db2mnt-db2fs01-rs
|- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
'- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ
2. Creating a new GPFS file system.
My poor test system has only 3 disks.
-Local hdisk0 for 'rootvg' for each host.
-Two disks hdisk1-2 attached to both system.
db2inst1@myhost3:/home/db2inst1>lspv
hdisk0 00f8c6622ea150a2 rootvg active
hdisk1 00f8c6624fdd9c4f None
hdisk2 00f8c6624fddb4a6 gpfs1nsd
root@myhost3:/>ssh myhost4 lspv
hdisk0 00f8c6622ecf89a6 rootvg active
hdisk1 00f8c6624fdd9c4f None
hdisk2 00f8c6624fddb4a6 gpfs1nsd
hdisk2 is being already used for the existing GPFS file system '/db2fs01' and hdisk1 is being used for TSA tiebreaker.
Therefore I'm going to remove hdisk1 from TSA tiebreaker and utilize this for adding storage group for our test temporarily.
( NOTE : Do not follow this steps of changing TSA tiebreaker. It's just for my specific poor test environment. Not for yours.)
db2inst1@myhost3:/home/db2inst1>db2cluster -cm -list -TIEBREAKER
The current quorum device is of type Disk with the following specifics: DEVICE=/dev/hdisk1.
root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -set -tiebreaker -majority
Configuring quorum device for domain 'db2domain_20160114122841' ...
Configuring quorum device for domain 'db2domain_20160114122841' was successful.
root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -list -tiebreaker
The current quorum device is of type Majority Node Set.
Created the new GPFS file system using hdisk1.
root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cfs -create -filesystem testfs -disk /dev/hdisk1 -mount /testfs
File system 'testfs' has been successfully created.
root@myhost3:/>lspv
hdisk0 00f8c6622ea150a2 rootvg active
hdisk1 00f8c6624fdd9c4f gpfs2nsd <=== disk for 'testfs'
hdisk2 00f8c6624fddb4a6 gpfs1nsd
root@myhost3:/>chown -R db2inst1:db2adm /testfs
root@myhost3:/>df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 9.75 2.33 77% 11798 3% /
/dev/hd2 19.25 14.61 25% 47342 2% /usr
/dev/hd9var 9.75 8.69 11% 7071 1% /var
/dev/hd3 19.25 13.24 32% 9692 1% /tmp
/dev/hd1 14.50 12.05 17% 2364 1% /home
/dev/hd11admin 0.25 0.25 1% 5 1% /admin
/proc - - - - - /proc
/dev/hd10opt 19.25 10.85 44% 23903 1% /opt
/dev/livedump 0.25 0.25 1% 4 1% /var/adm/ras/livedump
/dev/db2fs1 200.00 144.88 28% 6275 4% /db2fs01
/dev/testfs 50.00 49.40 2% 4136 4% /testfs <===== added
At this stage, there is no change in HA and TSA resource information as the file system is not added for DB2 database usage.
db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0
db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
'- Online IBM.Application:db2mnt-db2fs01-rs
|- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
'- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ
3. Creating new storage group.
Now creating new storage group using new GPFS file system '/testfs'.
Oops !!! Error. Why ?? There is no typo. But the message says "The path "/testfs" does not exist or is not valid".
How can it be possible ?
db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>db2 "create stogroup testgroup on '/testfs'"
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL1051N The path "/testfs" does not exist or is not valid. SQLSTATE=57019
Don't be panic. DB2 diag log file will say something.
Hum, sounds like errors during adding a resource group to TSA and a kind of license issue.
2016-12-23-12.01.21.945741+660 E3510A809 LEVEL: Error
PID : 23986266 TID : 5913 PROC : db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB : SAMPLE
APPHDL : 1-100 APPID: *N1.db2inst1.161223003935
AUTHID : DB2INST1 HOSTNAME: myhost3
EDUID : 5913 EDUNAME: db2agent (SAMPLE) 1
FUNCTION: <0>, <0>, <0>, probe:1416
RETCODE : ECF=0x90000545=-1879046843=ECF_SQLHA_CREATE_DEPENDENCY_FAILED
Add dependency failed
DATA #1 : String, 65 bytes
libsqlha: sqlhaAddResourceGroup() call error from wrapper library
DATA #2 : Hex integer, 4 bytes
0x00000000
DATA #3 : String, 96 bytes
Line # : 7099---2621-309 Command not allowed as daemon does not have a valid license.
; FFDC ID:
DATA #4 : signed integer, 4 bytes
309
Indeed, TSA license has the problem.
db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>samlicm -s
Error: Product license is invalid and needs to be upgraded.
root@myhost4:/testfs>samlicm -s
Error: Product license is invalid and needs to be upgraded.
I don't want to think too much here. No matter what the cause is, let's apply TSA license accordingly on all hosts.
root@myhost3:/home/db2inst1>samlicm -i sam32.lic
root@myhost3:/home/db2inst1>samlicm -s
Product: IBM Tivoli System Automation for Multiplatforms 3.2
Creation date: Wed Aug 19 00:00:01 EST 2009
Expiration date: Thu Dec 31 00:00:01 EST 2037
root@myhost4:/home/db2inst1>samlicm -i sam32.lic
root@myhost4:/home/db2inst1>samlicm -s
Product: IBM Tivoli System Automation for Multiplatforms 3.2
Creation date: Wed Aug 19 00:00:01 EST 2009
Expiration date: Thu Dec 31 00:00:01 EST 2037
OK. TSA license is good now. Running again.
Sigh~~. Same error again ?
db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>db2 "create stogroup testgroup on '/testfs'"
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL1051N The path "/testfs" does not exist or is not valid. SQLSTATE=57019
2016-12-23-12.13.25.246219+660 E5990A809 LEVEL: Error
PID : 23986266 TID : 28272 PROC : db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB : SAMPLE
APPHDL : 1-114 APPID: *N1.db2inst1.161223011112
AUTHID : DB2INST1 HOSTNAME: myhost3
EDUID : 28272 EDUNAME: db2agent (SAMPLE) 1
FUNCTION: <0>, <0>, <0>, probe:1416
RETCODE : ECF=0x90000545=-1879046843=ECF_SQLHA_CREATE_DEPENDENCY_FAILED
Add dependency failed
DATA #1 : String, 65 bytes
libsqlha: sqlhaAddResourceGroup() call error from wrapper library
DATA #2 : Hex integer, 4 bytes
0x00000000
DATA #3 : String, 96 bytes
Line # : 7099---2621-309 Command not allowed as daemon does not have a valid license.
; FFDC ID:
DATA #4 : signed integer, 4 bytes
309
We may assume here that TSA processes may need to be restarted to recognize the applied license.
There are multiple ways to restart TSA. Here I choose to enter and exit 'CM(TSA)' maintenance mode.
db2stop
db2stop instance on myhost4
db2stop instance on myhost3
(by root)
/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -enter -maintenance
/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -exit -maintenance
(DB2 Instance user)
db2start instance on myhost4
db2start instance on myhost3
db2start
Now added successfully. And you can see the new entries in HA/TSA resource information.
db2inst1@myhost3:/home/db2inst1>db2 "create stogroup testgroup on '/testfs'"
DB20000I The SQL command completed successfully.
db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0
A05000600000000,MO,/testfs,SAMPLE,0,1,0 <====
db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
'- Online IBM.Application:db2mnt-db2fs01-rs
|- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
'- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.ResourceGroup:db2mnt-testfs-rg Nominal=Online
'- Online IBM.Application:db2mnt-testfs-rs
|- Online IBM.Application:db2mnt-testfs-rs:myhost3
'- Online IBM.Application:db2mnt-testfs-rs:myhost4
4. Then the final quiz !! What will happen on HA/TSA resource information when dropping the storagegroup .
Now we can easily guess and let's see.
db2inst1@myhost3:/home/db2inst1>db2 drop stogroup testgroup
DB20000I The SQL command completed successfully.
db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0
db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
'- Online IBM.Application:db2mnt-db2fs01-rs
|- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
'- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ
Yes, the information has gone.
Hope you could sense a little relationship between DB2 storage operation and TSA/GPFS.
Merry Christmas !!
UID
ibm11139998