Chris's AIX Blog

In this post I’ll demonstrate how to create a Cluster Aware AIX (CAA) cluster on AIX 7.1. I’ll create a simple two-node CAA cluster. I’ll then check the cluster status and, finally, remove the cluster.

 

Before I start, I make sure that both nodes have entries in /etc/hosts on both systems.

 

node1[/] > grep node /etc/hosts

172.29.134.169  node1

172.29.152.160  node2

 

node2[/] > grep node /etc/hosts

172.29.134.169  node1

172.29.152.160  node2

 

I verify that cluster services are not active on both nodes.

 

node1[/tmp/cg] > lscluster -i

Cluster services are not active.

 

node2[/] >  lscluster -i

Cluster services are not active.

 

I’ll now create a cluster named mycluster. My CAA repository disk is named hdisk1.

 

node1[/] > mkcluster -n mycluster -m node1,node2 -r hdisk1

 

My two node CAA cluster is now operational.

 

node1[/] > lscluster -c

Cluster query for cluster mycluster returns:

Cluster uuid: 2a476f06-be50-11e0-9781-66da93d16b18

Number of nodes in cluster = 2

        Cluster id for node node2 is 1

        Primary IP address for node node2 is 172.29.152.160

        Cluster id for node node1 is 2

        Primary IP address for node node1 is 172.29.134.169

Number of disks in cluster = 0

Multicast address for cluster is 228.29.134.169

 

I can immediately execute commands on both nodes using the clcmd utility.

 

node1[/] > clcmd uptime

 

-------------------------------

NODE node1

-------------------------------

  11:16PM   up   1:16,  1 user,  load average: 1.10, 1.18, 1.36

 

 

-------------------------------

NODE node2

-------------------------------

  11:16PM   up 45 mins,  1 user,  load average: 0.97, 0.75, 0.64

 

Prior to cluster creation, the disks on my system were named as follows:

 

node1[/] > lspv

hdisk1          00f6048892b78f9f                    None

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

After cluster creation, hdisk1 is renamed to caa_private0 and is assigned to the CAA repository volume group named caavg_private.

 

node1[/] > lspv

caa_private0    00f6048892b78f9f                    caavg_private   active

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

node1[/] > lsvg caavg_private

VOLUME GROUP:       caavg_private            VG IDENTIFIER:  00f6048800004c0000000131930c1af3

VG STATE:           active                   PP SIZE:        64 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      799 (51136 megabytes)

MAX LVs:            256                      FREE PPs:       784 (50176 megabytes)

LVs:                6                        USED PPs:       15 (960 megabytes)

OPEN LVs:           2                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

node1[/] > lsvg -l caavg_private

caavg_private:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

caalv_private1      boot       1       1       1    closed/syncd  N/A

caalv_private2      boot       1       1       1    closed/syncd  N/A

caalv_private3      boot       4       4       1    open/syncd    N/A

fslv00              jfs2       4       4       1    open/syncd    /clrepos_private1

fslv01              jfs2       4       4       1    closed/syncd  /clrepos_private2

powerha_crlv        boot       1       1       1    closed/syncd  N/A

 

node1[/tmp/cg] > df -g

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.25      0.07   74%    10118    37% /

/dev/hd2           2.50      0.19   93%    45192    48% /usr

/dev/hd9var        0.38      0.02   95%     5995    47% /var

/dev/hd3           0.50      0.38   24%      187     1% /tmp

/dev/hd1           0.12      0.12    1%       11     1% /home

/dev/hd11admin      0.12      0.12    1%        5     1% /admin

/proc                 -         -    -         -     -  /proc

/dev/hd10opt       0.38      0.13   65%     8021    21% /opt

/dev/livedump      0.25      0.25    1%        4     1% /var/adm/ras/livedump

/aha                  -         -    -        37     1% /aha

/dev/fslv00        0.25      0.25    2%       15     1% /clrepos_private1

 

Several services are started during cluster creation. In particular you will notice the following subsystems active (lssrc –a output).

 

cld              caa              7340104      active

solidhac         caa              8912986      active

solid            caa              8519794      active

clconfd          caa              9306188      active

cthags           cthags           9830450      active

ctrmc            rsct             3342554      active

IBM.DRM          rsct_rm          8716462      active

IBM.ServiceRM    rsct_rm          9240724      active

IBM.StorageRM    rsct_rm          8061146      active

IBM.MgmtDomainRM rsct_rm          8454200      active

 

node1[/tmp/cg] > lssrc -g caa

Subsystem         Group            PID          Status

 clcomd           caa              6553810      active

 cld              caa              6619252      active

 solidhac         caa              8650856      active

 solid            caa              8781824      active

 clconfd          caa              10092598     active

 

Now I will remove the cluster.

 

node1[/] > rmcluster -n mycluster

 

rmcluster: Removed cluster shared disks are automatically renamed to names such

        as hdisk10, [hdisk11, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, you must manually

        rename them by removing them from the ODM database and then running

        the cfgmgr command to recreate them with default names.  For example:

        rmdev -l cldisk1 -d

        rmdev -l cldisk2 -d

        cfgmgr

 

CAA cluster services are no longer active.

 

node1[/] > lscluster -c

Cluster services are not active.

 

And caa_private0 is renamed back to hdisk1 and the caavg_private volume group is removed also.

 

node1[/] > lspv

 

hdisk1          00f6048892b78f9f                    None

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

node1[/] > df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4          524288    139136   74%    10097    37% /

/dev/hd2         5242880    405088   93%    45192    48% /usr

/dev/hd9var       786432     88040   89%     5986    35% /var

/dev/hd3         1048576    806528   24%      186     1% /tmp

/dev/hd1          262144    261352    1%       11     1% /home

/dev/hd11admin     262144    261416    1%        5     1% /admin

/proc                  -         -    -         -     -  /proc

/dev/hd10opt      786432    279912   65%     8021    21% /opt

/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump

/aha                   -         -    -        32     1% /aha

node1[/] >

 

 

 

 

While attending the IBM Power Systems Symposium this week, I learnt that starting with AIX 7.1 (and AIX 6.1 TL6) JFS2 logging is disabled during a mksysb restore. You may be familiar with disabling JFS2 logs, if not, take a look at this IBM technical note:

 

https://www-304.ibm.com/support/docview.wss?uid=isg3T1012643

 

I’ve been unable to find any official documentation from IBM that mentions this new enhancement to the mksysb restore process. However, when I checked my own AIX 7.1 system I found the following statement/code in the /usr/lpp/bosinst/bi_main script:

 

# Disable jfs2 logs for temp mounts -- (def#751604 performance)

if [[ "$2" = "TRUE" ]]; then

   mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

else

   mount -v $LVTYPE -o log=/dev/$LOGDEV,nomanager /dev/$LV $MROOT$FS || $ERROR

fi

 

If I compare this against my AIX 5.3 system, I don’t find any reference to this enhancement.

 

AIX 7.1:

[/usr/lpp/bosinst] > oslevel -s

7100-00-03-1115

[/usr/lpp/bosinst] # grep log=NULL bi_main

bi_main:         mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

 

AIX 5.3:

[/usr/lpp/bosinst] > oslevel -s

5300-11-03-1013

[/usr/lpp/bosinst] # grep log=NULL bi_main

[/usr/lpp/bosinst] #

 

This change is clever. I mean why would we ever need JFS2 logging during a restore? By disabling logging, the restore process will be faster. Simple, really.

Did you know that you can connect to and query the new solidDB database used by Cluster Aware AIX (CAA)?

 

You can connect to the solidDB instance and issue SQL queries against the database. In the following example, I issued two queries: one to list the tables in the database and another to query information relating to the nodes in the cluster. The formatting in the second query is a mess, but you get the idea.

 

root@node1:/> /opt/cluster/solidDB/bin/solsql -x pwdfile:/etc/cluster/dbpass 'tcp localhost 2188' caa

IBM solidDB SQL Editor (teletype) - Version: 6.5.0.3 Build 2010-10-04

(c) Solid Information Technology Ltd. 1993, 2010

Connected to 'tcp localhost 2188'.

Execute SQL statements terminated by a semicolon.

Exit by giving command: exit;

 

list tables;

RESULT

------

Catalog: CAA

  Schema: CAA

    TABLES:

    -------

    CLUSTERS

    NODES

    REPOSNAMESPACE

    REPOSSTORES

    SHAREDDISKS

    INTERFACES

    INTERFACE_ATTRS

   PARENT_CHILD

    ENTITIES

 

1 rows fetched.

 

 

select * from clusters;

          CLUSTER_ID CLUSTER_NAME                      ETYPE             ESUBTYPE              GLOB_ID UUID                   VERIFIED_STATUS               ESTATE    VERSION_OPERATING      VERSION_CAPABLE MULTICAST

          ---------- ------------                      -----             --------              ------- ----                   ---------------               ------    -----------------      --------------- ---------

                   1 SIRCOL_UNKNOWN               4294967296                   32           4294967297 00000000-0000-0000-0000-000000000000                NULL                    1                    1                    1 0

                   2 node2                        4294967296                   32           4294967296 4145b106-a7ab-11e0-bb35-00145e5286d0                NULL                    1                    1                    1 0

2 rows fetched.

 

exit

 

quit;

IBM solidDB SQL Editor exiting.

root@node1:/>

 

The new Redbook on IBM PowerHA 7.1 SystemMirror for AIX has some more information on and examples of using the solsql tool in a PowerHA 7.1 (with CAA) environment. Refer to section 8.3.2 CAA commands and log files:

 

http://www.redbooks.ibm.com/abstracts/sg247845.html?Open

 

You can find more information on the solsql command here:

 

http://publib.boulder.ibm.com/infocenter/soliddb/v6r5/topic/com.ibm.swg.im.soliddb.admin.doc/doc/solsql.commands.html

 

This little tip was passed on to me by a friendly IBM hardware engineer many years ago.

 

When entering a capacity on demand (CoD) code into a Power system, you can tell how many processors and how much memory will be activated, just by looking at the code you’ve given by IBM.

 

For example, the following codes, when entered for the appropriate Power system, will enable 4 processors (POD) and 64GB of memory (MOD). I can also tell* that once the VET code is entered, this system will be licensed for PowerVM Enterprise Edition (2C28).

 

System Type: 9117    Serial Number: 06-B719P Type Activation Code Posted Date (MM/DD/YYYY)  VET   F2FB60ERC4FF3DB50CA1F00002C28004177   04/30/2010   POD   FA94E589F2067F0C54680000000400419B   04/30/2010   MOD   B154E7097C231A85821200000064004102   04/30/2010

 

Activation type definitions
POD:	CUoD Processor Activation Code
MOD:	CUoD Memory Activation Code
TCOD:	On/Off CoD Enablement Code
TMOD:	On/Off CoD Memory Enablement Code
VET:	Virtualization Technology Code
	(PowerVM, Enterprise Enablement, WWPN, Active Memory Expansion)
STDP:	Standard Trial CoD Processor Activation Code
STDM:	Standard Trial CoD Memory Activation Code
STME:	Standard Trial Active Memory Expansion Code
EXCP:	Exception Trial CoD Processor Activation Code
EXCM:	Exception Trial CoD Memory Activation Code
USTA:	Utility CoD Enablement Code
USTO:	Utility CoD Termination Code
URPT:	Utility CoD Reporting Code
PAID:	Utility CoD PrePaid Code

0000 = Reset to base defaults (No PowerVM)
1xxx = Express
2c00 = Standard
2c20 = Enterprise

*Refer to the following link for more information on “How to tell which PowerVM Editions feature was ordered”: https://www-304.ibm.com/support/docview.wss?uid=isg3T1010860

 

You can obtain your CoD codes from the following website: http://www-912.ibm.com/pod/pod

 

 

======

 

UPDATE 12/9/2011: Script from Chris-R that collects CoD information for a system from the IBM CoD Website. Thanks Chris-R, nice job!

 

Done!

$ ./activations.pl 9117 10 06ebp
02 processors activated on 05/23/2011
024 GB memory activated on 05/23/2011
02 processors activated on 03/11/2011
016 GB memory activated on 03/11/2011
055 GB memory activated on 03/02/2011
06 processors activated on 02/22/2011
096 GB memory activated on 02/22/2011
01 processors activated on 01/12/2011
016 GB memory activated on 01/12/2011
06 processors activated on 03/17/2010
072 GB memory activated on 03/17/2010
19 processors activated on 03/16/2010
228 GB memory activated on 03/16/2010
================
TOTAL CPU=36 RAM=519168MB (507GB)

$cat activations.pl
#!/usr/bin/perl
#
# PoC Script to go out on the net to the IBM POD site and tally up activations for a given i or p Series machine
# Chris-R rutherfc@gmail.com
# v1.0 9/9/2011
#
# todo - add logic for processor deactivation - who uses that? IBM dont even publish the code on the pod site ;)
#
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("mozilla 8.0");
# $ua->proxy(['http'], 'http://proxy:8080/');
$ua->timeout(10);
use HTTP::Request::Common qw(POST);
if ($#ARGV != 2) {
print "usage: $0 MODEL XX XXXXX e.g $0 9119 83 9f6bf\n";
exit;
}
($model, $serial1, $serial2) = @ARGV;

##### main #####
get('http://www-912.ibm.com/pod/pod',"$serial2.htm");
html2txt("$serial2.htm","$serial2.txt");
total("$serial2.txt");
exit;

sub get # fakes a mozilla browser, fills in the CGI form and snags the returned page to a local html file
{
my $req = (POST $_[0],
["system_type" => $model,
"system_serial_number1" => $serial1,
"system_serial_number2" => $serial2 ]);
$request = $ua->request($req);
$activations = $request->content;
open(POD,">$_[1]");
print POD $activations;
close(POD);
}

sub html2txt # strips out the crap and converts the table to a local txt file to parse
{
open(HTML,"<$_[0]");
open(TXT,">$_[1]");
while (<HTML>) {
if (/<\/table>/) {$f = 0;};
if (/<th>Posted Date \(MM/) {$f = 1;};
if ($f == 1) {
# poor mans HTML::TableExtract - excuse my sed like perl....
s/<tr align="center">/:/g;
s/<[^>][^>]*>//g;
s/ //g;
s/\n//g;
s/:/\n/g;
print TXT $_;
};
};
close(TXT);
close(HTML);
}
sub total # totals up the de/activations to get totals
{
open(TXT,"<$_[0]");
$[ = 1; $\ = "\n";# set array base & output record separator
while (<TXT>) {
($code,$hex,$date) = split(' ', $_, -1);
if (/POD/) {
$p = substr($hex, 27, 2);
print $p . ' processors activated on ' . $date;
$pt = $pt + $p;
};
if (/MOD/) {
$r = substr($hex, 26, 3);
print $r . ' GB memory activated on ' . $date;
$rt = $rt + $r;
};
if (/RMEM/) {
$r = substr($hex, 27, 2);
print $r . ' GB memory activated on ' . $date;
$rt = $rt - $r;
};
};
print '================';
print 'TOTAL CPU=' . $pt . ' RAM=' . $rt*1024 . 'MB (' . $rt . 'GB)';
close(TXT);
}
</TXT></tr></th></HTML>