Adding a storage group in pureScale : A practical example

Body

Adding a storage group in DB2 is done by very simple command below.
db2 "create stogroup testgroup on '/path1','/path2'"

As a DBA, you wait for new disks to be attached and ready from system administrator and just run the command.

On pureScale, you should better to understand that has more things behind the scene related to TSA (Tivoli System Automation) and GPFS (The General Parallel File System).
In short, that command does interface with TSA to add the resources accordingly.
Before this, you may also need to work on GPFS file system level commands by yourself, not leaning on system administrator.
I will show you
- how it looks like on this blog
- what should we do to verify the result
- and a sample error situation you may encounter without the proper TSA license.

( NOTE : Refer the user name whether 'db2inst1' or 'root' for every sample command line. .)

1. Firstly, let me introduce my test environment.

2 hosts and co-located member/CF for each.

myhost4- member 0 / CF 128
myhost3- member 1 / CF 129
DB2 instance name : db2inst1
DB name : SAMPLE
AIX 7.1
DB2 : V10.5

I have one default storage group on a GPFS file system '/db2fs01'.

db2inst1@myhost3:/home/db2inst1>db2pd -d sample -storagepaths

Database Member 1 -- Database SAMPLE -- Active -- Up 0 days 00:00:16 -- Date 2016-12-23-11.01.20.091461

Storage Group Configuration:
Address            SGID  Default  DataTag    Name
0x0A00020012301CA0 0     Yes      0          IBMSTOGROUP

Storage Group Statistics:
Address            SGID  State      Numpaths  NumDropPen
0x0A00020012301CA0 0     0x00000000 1         0         

Storage Group Paths:
Address            SGID  PathID    PathState    PathName
0x0A0002001235E000 0     0         InUse        /db2fs01

db2inst1@myhost3:/home/db2inst1>db2pd -d sample -storagegroups

Database Member 1 -- Database SAMPLE -- Active -- Up 0 days 00:32:56 -- Date 2016-12-23-11.34.00.401244

Storage Group Configuration:
Address            SGID  Default  DataTag    Name
0x0A00020012301CA0 0     Yes      0          IBMSTOGROUP

Storage Group Statistics:
Address            SGID  State      Numpaths  NumDropPen
0x0A00020012301CA0 0     0x00000000 1         0         

Storage Group Paths:
Address            SGID  PathID    PathState    PathName
0x0A0002001235E000 0     0         InUse        /db2fs01

'/db2fs01' is the GPFS file system that was created during pureScale installation.
And it's added to HA and TSA configuration.

db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0

db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
        '- Online IBM.Application:db2mnt-db2fs01-rs
                |- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
                '- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ

2. Creating a new GPFS file system.
 
My poor test system has only 3 disks.
-Local hdisk0 for 'rootvg' for each host.
-Two disks hdisk1-2 attached to both system.

db2inst1@myhost3:/home/db2inst1>lspv
hdisk0          00f8c6622ea150a2                    rootvg          active      
hdisk1          00f8c6624fdd9c4f                    None                        
hdisk2          00f8c6624fddb4a6                    gpfs1nsd

root@myhost3:/>ssh myhost4 lspv
hdisk0          00f8c6622ecf89a6                    rootvg          active      
hdisk1          00f8c6624fdd9c4f                    None                        
hdisk2          00f8c6624fddb4a6                    gpfs1nsd    

hdisk2 is being already used for the existing GPFS file system '/db2fs01' and hdisk1 is being used for TSA tiebreaker.
Therefore I'm going to remove hdisk1 from TSA tiebreaker and utilize this for adding storage group for our test temporarily.
( NOTE : Do not follow this steps of changing TSA tiebreaker. It's just for my specific poor test environment. Not for yours.)

db2inst1@myhost3:/home/db2inst1>db2cluster -cm -list -TIEBREAKER
The current quorum device is of type Disk with the following specifics: DEVICE=/dev/hdisk1.

root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -set -tiebreaker -majority
Configuring quorum device for domain 'db2domain_20160114122841' ...
Configuring quorum device for domain 'db2domain_20160114122841' was successful.

root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -list -tiebreaker
The current quorum device is of type Majority Node Set.

Created the new GPFS file system using hdisk1.

root@myhost3:/>/opt/IBM/db2/V10.5FP6/bin/db2cluster -cfs -create -filesystem testfs -disk /dev/hdisk1 -mount /testfs
File system 'testfs' has been successfully created.

root@myhost3:/>lspv
hdisk0          00f8c6622ea150a2                    rootvg          active      
hdisk1          00f8c6624fdd9c4f                    gpfs2nsd    <=== disk for 'testfs'                   
hdisk2          00f8c6624fddb4a6                    gpfs1nsd

root@myhost3:/>chown -R db2inst1:db2adm /testfs

root@myhost3:/>df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           9.75      2.33   77%    11798     3% /
/dev/hd2          19.25     14.61   25%    47342     2% /usr
/dev/hd9var        9.75      8.69   11%     7071     1% /var
/dev/hd3          19.25     13.24   32%     9692     1% /tmp
/dev/hd1          14.50     12.05   17%     2364     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt      19.25     10.85   44%    23903     1% /opt
/dev/livedump      0.25      0.25    1%        4     1% /var/adm/ras/livedump
/dev/db2fs1      200.00    144.88   28%     6275     4% /db2fs01
/dev/testfs       50.00     49.40    2%     4136     4% /testfs   <===== added


At this stage, there is no change in HA and TSA resource information as the file system is not added for DB2 database usage.

db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0

db2inst1@myhost3:/home/db2inst1>lssam |grep mnt
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
        '- Online IBM.Application:db2mnt-db2fs01-rs
                |- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
                '- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ

3. Creating new storage group.

Now creating new storage group using new GPFS file system '/testfs'.
Oops !!! Error. Why ?? There is no typo. But the message says "The path "/testfs" does not exist or is not valid".
How can it be possible ?

db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>db2 "create stogroup testgroup on '/testfs'"
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
SQL1051N  The path "/testfs" does not exist or is not valid.  SQLSTATE=57019

Don't be panic. DB2 diag log file will say something.
Hum, sounds like errors during adding a resource group to TSA and a kind of license issue.

2016-12-23-12.01.21.945741+660 E3510A809            LEVEL: Error
PID     : 23986266             TID : 5913           PROC : db2sysc 1
INSTANCE: db2inst1             NODE : 001           DB   : SAMPLE
APPHDL  : 1-100                APPID: *N1.db2inst1.161223003935
AUTHID  : DB2INST1             HOSTNAME: myhost3
EDUID   : 5913                 EDUNAME: db2agent (SAMPLE) 1
FUNCTION: <0>, <0>, <0>, probe:1416
RETCODE : ECF=0x90000545=-1879046843=ECF_SQLHA_CREATE_DEPENDENCY_FAILED
          Add dependency failed
DATA #1 : String, 65 bytes
libsqlha: sqlhaAddResourceGroup() call error from wrapper library
DATA #2 : Hex integer, 4 bytes
0x00000000
DATA #3 : String, 96 bytes
Line # : 7099---2621-309 Command not allowed as daemon does not have a valid license.
; FFDC ID:
DATA #4 : signed integer, 4 bytes
309

Indeed, TSA license has the problem.

db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>samlicm -s
Error: Product license is invalid and needs to be upgraded.

root@myhost4:/testfs>samlicm -s
Error: Product license is invalid and needs to be upgraded.

I don't want to think too much here. No matter what the cause is, let's apply TSA license accordingly on all hosts.

root@myhost3:/home/db2inst1>samlicm -i sam32.lic
root@myhost3:/home/db2inst1>samlicm -s
Product: IBM Tivoli System Automation for Multiplatforms 3.2
Creation date: Wed Aug 19 00:00:01 EST 2009
Expiration date: Thu Dec 31 00:00:01 EST 2037

root@myhost4:/home/db2inst1>samlicm -i sam32.lic
root@myhost4:/home/db2inst1>samlicm -s
Product: IBM Tivoli System Automation for Multiplatforms 3.2
Creation date: Wed Aug 19 00:00:01 EST 2009
Expiration date: Thu Dec 31 00:00:01 EST 2037

OK. TSA license is good now. Running again.
Sigh~~. Same error again ?

db2inst1@myhost3:/home/db2inst1/sqllib/db2dump/DIAG0001>db2 "create stogroup testgroup on '/testfs'"
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
SQL1051N  The path "/testfs" does not exist or is not valid.  SQLSTATE=57019

2016-12-23-12.13.25.246219+660 E5990A809            LEVEL: Error
PID     : 23986266             TID : 28272          PROC : db2sysc 1
INSTANCE: db2inst1             NODE : 001           DB   : SAMPLE
APPHDL  : 1-114                APPID: *N1.db2inst1.161223011112
AUTHID  : DB2INST1             HOSTNAME: myhost3
EDUID   : 28272                EDUNAME: db2agent (SAMPLE) 1
FUNCTION: <0>, <0>, <0>, probe:1416
RETCODE : ECF=0x90000545=-1879046843=ECF_SQLHA_CREATE_DEPENDENCY_FAILED
          Add dependency failed
DATA #1 : String, 65 bytes
libsqlha: sqlhaAddResourceGroup() call error from wrapper library
DATA #2 : Hex integer, 4 bytes
0x00000000
DATA #3 : String, 96 bytes
Line # : 7099---2621-309 Command not allowed as daemon does not have a valid license.
; FFDC ID:
DATA #4 : signed integer, 4 bytes
309

We may assume here that TSA processes may need to be restarted to recognize the applied license.
There are multiple ways to restart TSA. Here I choose to enter and exit 'CM(TSA)' maintenance mode.

db2stop
db2stop instance on myhost4
db2stop instance on myhost3
(by root)
/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -enter -maintenance
/opt/IBM/db2/V10.5FP6/bin/db2cluster -cm -exit -maintenance

(DB2 Instance user)
db2start instance on myhost4
db2start instance on myhost3
db2start

Now added successfully. And you can see the new entries in HA/TSA resource information.
 

db2inst1@myhost3:/home/db2inst1>db2 "create stogroup testgroup on '/testfs'"
DB20000I  The SQL command completed successfully.

db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO                     
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0
A05000600000000,MO,/testfs,SAMPLE,0,1,0   <====

db2inst1@myhost3:/home/db2inst1>lssam |grep mnt                             
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
        '- Online IBM.Application:db2mnt-db2fs01-rs
                |- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
                '- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.ResourceGroup:db2mnt-testfs-rg Nominal=Online
        '- Online IBM.Application:db2mnt-testfs-rs
                |- Online IBM.Application:db2mnt-testfs-rs:myhost3
                '- Online IBM.Application:db2mnt-testfs-rs:myhost4

4. Then the final quiz !! What will happen on HA/TSA resource information when dropping the storagegroup .
   Now we can easily guess and let's see.

db2inst1@myhost3:/home/db2inst1>db2 drop stogroup testgroup
DB20000I  The SQL command completed successfully.

db2inst1@myhost3:/home/db2inst1>db2hareg -dump |grep MO    
A05000600000000,MO,/db2fs01, ,0,5,0
A05000600000000,MO,/db2fs01,SAMPLE,0,8,0

db2inst1@myhost3:/home/db2inst1>lssam |grep mnt            
Online IBM.ResourceGroup:db2mnt-db2fs01-rg Nominal=Online
        '- Online IBM.Application:db2mnt-db2fs01-rs
                |- Online IBM.Application:db2mnt-db2fs01-rs:myhost3
                '- Online IBM.Application:db2mnt-db2fs01-rs:myhost4
Online IBM.Equivalency:db2mnt-db2fs01-rg_group-equ

Yes, the information has gone.
Hope you could sense a little relationship between DB2 storage operation and TSA/GPFS.

Merry Christmas !!