Chris's AIX Blog

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

 

"Hello Chris,
 
So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.
 
So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup
 
# lsnim -l netbackup
netbackup:
   class       = resources
   type        = file_res
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /tmp/netbackup
   alloc_count = 0
   server      = master
   dest_dir    = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on. 

Can I actually push files out by themselves? Thanks"

 

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

 

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value     Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value     Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

 

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

 

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm
 
Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really! 

 

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

 

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

First, I'll create a new file_res NIM resource, called soe_conf.

 

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf     resources       file_res

# lsnim -l soe_conf
soe_conf:
   class       = resources
   type        = file_res
   dest_dir    = /usr/local/etc
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /usr/local/etc
   alloc_count = 0
   server      = master

 

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   file_res       = soe_conf
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = not running
   Mstate         = currently running
   cpuid          = 00F94F584C00
   control        = master

 

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

 

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

 

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

 

root@MASTER /# nim -o cust AIXmig

 

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

The file_res resource, is automatically deallocated from the NIM client.

 

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = customization is being performed
   Mstate         = currently running
   cpuid          = 00F94F584C00
   Cstate_result  = success
     
If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable). 
   
  
   

Do you use SSL with nimsh on AIX? No? Well, you might want to consider it. If you regularly use LPM to migrate AIX partitions from one server to another, you may have found that, on occasion, your NIM master has trouble communicating with its NIM clients afterwards. This is by design, as nimsh uses the NIM clients cpuid to authenticate with the NIM master. During an LPM operation, the cpuid of the NIM client changes and its possible the NIM master may reject the client as a result. This problem can occur even when CPU validation is disabled on the NIM master.

 

In the example below, I’ve LPM’ed a NIM client (750lpar1) to another server. Immediately afterwards, I’m able to execute a NIM command against the NIM client, from the NIM master (750lpar4). At this point the NIM client is configured with standard nimsh authentication i.e. no SSL.

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@750lpar1]/var/adm/ras# echo HELLO > /var/adm/ras/nim.installp

 

NIM MASTER:

[root@750lpar4]/ # nim -o change -a validate_cpuid=no master

[root@750lpar4]/ # lsnim -l master | grep -i cpu

   validate_cpuid      = no

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00                << Notice that the cpuid is different.

 

[root@750lpar4]/ # nim -o showlog 750lpar1

HELLO

 

The cpuid is cached by the nimsh daemon, so the previous system id is retained in memory and passed to the NIM master, which allows the operation to complete successfully. But, if I restart the nimsh daemon, on the NIM client, I find that the NIM master is no longer able to communicate with the client.

 

[root@750lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@750lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160610.

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

0042-001 nim: processing error encountered on "master":

   0042-006 m_mkbosi: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar4]/ # nim -o showlog 750lpar1

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar1]/var/adm/ras# grep match nimsh.log | grep CD

Wed Apr 27 20:51:12 2016        error: local value passed, '00F603CD4C00', does not match environment value '00F627664C00'

 

One way to work around this problem is to use the procedure outlined at the following link:

 

http://www.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.osdevice/lpm_nim.htm

 

“Migrating a NIM client by using LPM

 

When Live Partition Mobility (LPM) is used to move a machine from one physical server to another and the machine is defined as a Network Installation Management (NIM) client, the NIM administrator must update the cpuid attribute for the NIM client to reflect the new hardware value after the LPM migration completes. To update the cpuid attribute, complete the following steps:

 

On the NIM client, acquire the new cpuid ID by running the following command:

 

uname –a

 

On the NIM master, run the following command:

 

nim -o change -a cpuid=cpuid client”

 

However, there is a better way. Using nimsh, with SSL-enabled authentication, will prevent the checking of cpuid during nimsh service handling.  This is considered the recommended choice of operation (since the client/server can agree upon identity using the certificate information passed during the ssl handshake). Once the certificate is in place, the NIM master will disregard any cpuid validation and instead rely on the success of the SSL-handshake. This configuration will work well with LPM. If using standard nimsh, the limitation of cpuid updating would still apply (because NIM has no way of automatically updating a client once the value has changed).

 

Both the NIM master and client must be configured to support SSL-enabled authentication. To configure SSL-enabled authentication, we can use ‘nimconfig -c’ on the NIM master.

 

[root@750lpar4]/ # oslevel -s

7100-04-01-1543

 

[root@750lpar4]/ # nimconfig -c

0513-029 The tftpd Subsystem is already active.

Multiple instances are not supported.

NIM_MASTER_HOSTNAME=750lpar4

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Target "all" is up to date.

Generating a 1024 bit RSA private key

.................++++++

....................................++++++

writing new private key to '/ssl_nimsh/keys/rootkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

Getting Private key

Generating a 1024 bit RSA private key

...++++++

....++++++

writing new private key to '/ssl_nimsh/keys/clientkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

Generating a 1024 bit RSA private key

.....................++++++

..........++++++

writing new private key to '/ssl_nimsh/keys/serverkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

 

[root@750lpar4]/ # lsnim -l | grep -i ssl

   ssl_support         = yes

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /ssl_nimsh/certs/client.pem

certname= /ssl_nimsh/certs/client.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

certname= /tftpboot/server.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

To enable SSL nimsh on the NIM client, we can use the ‘nimclient –c’ command. You can fall-back to non-SSL nimsh, with ‘nimclient –C’.

 

[root@750lpar1]/# oslevel -s

7100-04-01-1543

 

[root@750lpar1]/var/adm/ras# nimclient -c

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Received 2784 Bytes in 0.0 Seconds

0513-044 The nimsh Subsystem was requested to stop.

0513-077 Subsystem has been changed.

0513-059 The nimsh Subsystem has been started. Subsystem PID is 14942422.

 

[root@750lpar1]/var/adm/ras# cat nimsh.log

Fri Apr 29 23:10:45 2016        /usr/sbin/nimsh: NIM Service Handler started from SRC

Fri Apr 29 23:10:45 2016        no environment value for NIM_SECONDARY_PORT

Fri Apr 29 23:10:45 2016        value for hostname is 750lpar1

Fri Apr 29 23:10:45 2016        value for netaddr is: 10.1.50.31

Fri Apr 29 23:10:45 2016        value for route is net,-hopcount,0,,0 and gateway is 10.1.50.1

Fri Apr 29 23:10:45 2016        value for netif is en0

Fri Apr 29 23:10:45 2016        value for netmask is 255.255.255.0

Fri Apr 29 23:10:45 2016        obtained master's hostname: NIM_MASTER_HOSTNAME=750lpar4

 

Fri Apr 29 23:10:45 2016        obtained master's id: NIM_MASTERID=00F603CD4C00

 

Fri Apr 29 23:10:45 2016        value for machine id is 00F603CD4C00

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libssl.so.0.9.8 from /usr/lib/libssl.a ..

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libcrypto.so.0.9.8 from /usr/lib/libcrypto.a ..

 

Now I can LPM the NIM client to another server and even if I restart nimsh, the master can still communicate with the client. All thanks to SSL.

 

NIM MASTER:

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160616.

 

[root@750lpar1]/var/adm/ras# tail -f nimsh.log

Fri Apr 29 23:13:33 2016        passing OpenSSL setting of 1

Fri Apr 29 23:13:33 2016        set symbol table

Fri Apr 29 23:13:33 2016         cert filename discovered: /ssl_nimsh/certs/750lpar4.0

Fri Apr 29 23:13:33 2016        ** OpenSSL FIPS mode enabled successfully

Fri Apr 29 23:13:33 2016        seed_prng

Fri Apr 29 23:13:33 2016        Loading certificates..

Fri Apr 29 23:13:33 2016        Loading private key file..

Fri Apr 29 23:13:33 2016        create BIO

Fri Apr 29 23:13:33 2016        - SSL Connection verified successfully -

Fri Apr 29 23:13:33 2016        sending ack to client

Fri Apr 29 23:13:33 2016        setting descriptors to include 2nd port

Fri Apr 29 23:13:33 2016        command to exec  __  /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_showlog" "-alocation=/var/adm/ras/nim.installp"

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

 

+---------------------------------------------------------------------+

                System Backup Image Space Information

              (Sizes are displayed in 1024-byte blocks.)

+---------------------------------------------------------------------+

 

Required = 9457741 (9237 MB)    Available = 51653640 (50444 MB)

 

Creating information file (/image.data) for rootvg.

..etc..

 

 

 

Using NIM to install clients configured with SSL authentication

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_install_ssl.htm

 

Using the certificate viewing file

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/sample_files_certificate.htm%23sample_files_certificate?lang=en

 

 

 

 

 

Is that a versioned WPAR? Use the lswpar command and the –G option to determine the WPAR type. Yeah, I could use clogin and then run oslevel, but this is faster!

 

[root@gibopvc1]/ # lswpar -G p8wpar1

=================================================================

p8wpar1 - Active

=================================================================

Type:                    S

RootVG WPAR:             yes

Owner:                   root

Hostname:                p8wpar1

WPAR-Specific Routing:   no

Virtual IP WPAR:

Directory:               /wpars/p8wpar1

Start/Stop Script:

Auto Start:              no

Private /usr:            yes

Checkpointable:          no

Application:

 

OStype:                  1024

Cross-WPAR IPC:          yes

Architecture:            none

UUID:                    418c1da6-bceb-4b4e-b872-b1c6701da0b6

 

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

 

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

 

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

   0042-061 m_rmpdir: the "liveupdaterte" resource is currently

        allocated for client use

 

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 1

   server      = master

 

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

 

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

 

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# nim -o remove liveupdaterte

#

 

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

 

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

 

 

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

 

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

 

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. lparaix1nim.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo deallocate -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015).  This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs.  Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

 

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

 

 Major Advantages of using HTTP during NIM Management:

 

• All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
• Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
• Easy to consume by NIM or other products that currently use the client/server model of NFS.
• Able to extend the end-product to support additional protocols (context driven).”

 

 

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources.  The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901.  When active, NIM clients attempt file access and/or scripting customization requests from nimhttp.  If http access fails or is denied, a failover attempt at NFS client access occurs.  Future support will include options to remove NFS client attempts altogether.”

 

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user.  First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

 

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/.  This path location is defined as the document_root and cannot be modified at this time.  Future enhancements will support multiple document_root paths.  The document root path is not limited in depth and may contain many sub-directories.  Client requests are able to traverse the path setting by using the enable_directory_listing option.  If set to “no”, all files being served must reside in the current working directory of document_root.”

 

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable.  To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server.  The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM.  Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

 

# nimconfig –c”

 

I tested this functionality during the AIX 7.2 Early Ship Program.

 

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

 

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

 

NIM MASTER:

 

# lslpp -l | grep nim

  bos.sysmgt.nim.master      7.2.0.0  COMMITTED  Network Install Manager -

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager –

 

NIM CLIENT:

 

# lssrc -s nimsh

Subsystem         Group            PID          Status

 nimsh            nimclient        6554064      active

 

# lslpp -l | grep nim

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager -

 

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

 

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

 

# lssrc -s nimhttp

Subsystem         Group            PID          Status

 nimhttp                           6685178      active

 

# ps -ef | grep nimhttp

    root  6685178  4194712   0   Nov 10      -  0:00 /usr/sbin/nimhttpd –v

 

# ls -ltr /httpd.conf

-rw-r--r--    1 root     system         1159 Nov 05 15:31 /httpd.conf

 

 

# cat /httpd.conf

 

#

#---------------------

# http service defines

#---------------------

#

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

#

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

#

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

#

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#

#

#---------------------

# http configuration

#---------------------

#

document_root=/export/nim/

# Designates the directory to serve files from.

#

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

#

enable_proxy=no

# Enable the web service to act as a proxy server.

#

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

#

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

#

 

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdaterte               resources       lpp_source

 

# lsnim -l liveupdaterte

liveu:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r--    1 root     system      2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r--    1 root     system          688 Oct 31 09:33 .toc

 

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

 

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

 

# lslpp -l bos.liveupdate.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

Path: /etc/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

 

# tail -f nimhttp.log

 

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015        socks: Copied string

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015        Request Type is

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015        socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:23 2015        Request Type is GET

Mon Nov 16 13:51:23 2015        Mime not supported

Mon Nov 16 13:51:23 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015        Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015        Total length sent is 509

Mon Nov 16 13:51:23 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015        Total length sent is 72

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015        Total length sent is 755

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Total length sent is 2574336

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

 

 

“Example:  Commands supporting nimhttp

 

If using push operation (From NIM master)

 

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

 

If using pull operation (From NIM client)

 

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

One of my customers was configuring a new AIX 5.3 Versioned WPAR when they came across a very interesting issue. I thought I’d share the experience here, just in case anyone else comes across the problem. We configured the VWPAR to host an old application. The setup was relatively straight forward, restore the AIX 5.3 mksysb into the VWPAR and export the data disk from the Global into the VWPAR, import the volume group and mount the file systems. Job done!  However, we noticed some fairly poor performance during application load tests. After some investigation we discovered that disk I/O performance was worse in the VWPAR than on the source LPAR.  The question was, why?

 

We initially suspected the customers SAN and/or the storage subsystem, but both of these came back clean with no errors or configuration issues.  In the end, the problem was related to a lack of ODM attributes in the PdAt object class, which prevented the VWPAR disk from using the correct queue depth setting.

 

Let me explain by demonstrating the problem and the workaround.

 

First, let’s add a new disk to a VWPAR. This will be used for a data volume group and file system. The disk in question is hdisk3.

 

# uname -W

0

 

# lsdev -Cc disk

hdisk0 Available  Virtual SCSI Disk Drive

hdisk1 Available  Virtual SCSI Disk Drive

hdisk2 Defined    Virtual SCSI Disk Drive

hdisk3 Available  Virtual SCSI Disk Drive  <<<<<<

 

We set the disk queue depth to an appropriate number, in this case 256.

Note: This value will differ depending on the storage subsystem type, so check with your storage team and/or vendor for the best setting for your environment.

 

# chdev -l hdisk3 -a queue_depth=256

hdisk3 changed

 

Using the lsattr command, we verify that the queue depth attribute is set correctly in both the ODM and the AIX kernel.

 

# lsattr -El hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

We can also use kdb to verify the setting in the kernel. Remember at this stage, we are concentrating on hdisk3, which is referenced with a specific kernel device address in kdb.

 

# echo scsidisk | kdb

           START              END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME            ADDRESS             STATE   CMDS_OUT  CURRBUF             LOW

hdisk0          0xF1000A01505DC000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk0          0xF1000A01C0148000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01505D4000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk3          0xF1000A01C014C000  0x00000001  0x0000    0x0000000000000000  0x0  <<<<<<<

 

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x100;

 

From the output above, we can see that the queue depth is correctly i.e. set to 0x100 in Hex (256 in decimal).

 

Next, we export hdisk3 to the VWPAR using the chwpar command. The disk, as expected, enters a Defined state in the Global environment. It is known as hdisk1 in the VWPAR.

 

# chwpar -D devname=hdisk3 p8wpar1

 

# lswpar -D p8wpar1 | head -2 ; lswpar -D p8wpar1 | grep hdisk

Name     Device Name      Type    Virtual Device  RootVG  Status

-------------------------------------------------------------------

p8wpar1  hdisk3           disk    hdisk1          no      EXPORTED <<<<<<

p8wpar1  hdisk2           disk    hdisk0          yes     EXPORTED

 

[root@gibopvc1]/ # lsdev -Cc disk

hdisk0 Available  Virtual SCSI Disk Drive

hdisk1 Available  Virtual SCSI Disk Drive

hdisk2 Defined    Virtual SCSI Disk Drive

hdisk3 Defined    Virtual SCSI Disk Drive

 

In the VWPAR, we run cfgmgr to discover the disk. We create a new data volume group (datavg) and file system (datafs) for application use (note: the steps to create the VG and FS are not shown below). This is for demonstration purposes only; the customer imported the data volume groups on their system.

 

# clogin p8wpar1

*******************************************************************************

*                                                                             *

*  Welcome to AIX Version 5.3!                                                *

*                                                                             *

*  Please see the README file in /usr/lpp/bos for information pertinent to    *

*  this release of the AIX Operating System.                                  *

*                                                                             *

*******************************************************************************

Last login: Sun Apr 26 21:27:03 2015 on /dev/Global from

 

# uname -W

11

 

# lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

 

# cfgmgr ; lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

hdisk1          none                                None

 

# lsvg

rootvg

datavg

 

# lsvg -l datavg

datavg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

datalv              jfs2       1024    1024    1    open/syncd    /datafs

loglv00             jfs2log    1       1       1    open/syncd    N/A

 

We perform a very simple I/O test in the /datafs file system. We write/create a 1GB file and time the execution. We noticed immediately that the task took longer than expected.

 

# cd /datafs

# time lmktemp Afile 1024M

Afile

 

real    0m7.22s <<<<<<<<<<<<<<< SLOW?

user    0m0.04s

sys     0m1.36s

 

We ran the iostat command, from the Global environment, and noticed that “serv qfull” was constantly non-zero (very large numbers) for hdisk3. Essentially the hdisk queue was full all the time. This was bad and unexpected, given the queue depth setting of 256!

 

# iostat -DTRl 1

 

System configuration: lcpu=8 drives=4 paths=8 vdisks=2

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:53

hdisk0           0.0   4.1K   1.0   0.0    4.1K   0.0   0.0    0.0    0.0     0    0   1.0   0.3    0.3    0.3     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:53

hdisk2           1.0  12.3K   3.0   0.0   12.3K   0.0   0.0    0.0    0.0     0    0   3.0   0.5    0.3    0.6     0    0   0.4    0.0    0.9    0.0   0.0   3.0  03:11:53

hdisk3         100.0 140.5M 1072.0   0.0  140.5M   0.0   0.0    0.0    0.0     0    0 1072.0   0.9    0.8   24.0     0    0 176.7  107.5  279.8  152.0   0.0 1072.0  03:11:53

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk0           0.0   4.1K   1.0   4.1K   0.0    1.0   0.3    0.3    0.3     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk3         100.0 154.0M 1175.0   0.0  154.0M   0.0   0.0    0.0    0.0     0    0 1175.0   0.8    0.8    1.8     0    0 161.9  108.0  217.7  303.0   1.0 1175.0  03:11:54

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk3         100.0 150.3M 1147.0   0.0  150.3M   0.0   0.0    0.0    0.0     0    0 1147.0   0.9    0.8   10.6     0    0 165.9  108.5  239.4  304.0   1.0 1147.0  03:11:55

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk3         100.0 155.5M 1186.0   0.0  155.5M   0.0   0.0    0.0    0.0     0    0 1186.0   0.8    0.8    1.8     0    0 161.6  106.8  217.8  307.0   1.0 1186.0  03:11:56

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk3          67.0 102.2M 780.0   0.0  102.2M   0.0   0.0    0.0    0.0     0    0 780.0   0.9    0.8   11.8     0    0 166.1  106.3  232.8   53.0   0.0 779.0  03:11:57

 

Now comes the interesting part. With a little help from our friends in IBM support, using kdb we found that the queue depth was reported as being set to 1 in the kernel and not 256! You’ll also notice here that the hdisk name has changed from hdisk3 to hdisk1. This happened as a result of exporting hdisk3 to the VWPAR. The disk is known as hdisk1 in the VWPAR (not hdisk3) but the kernel address is the same.

 

# echo scsidisk | kdb

           START              END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME            ADDRESS             STATE   CMDS_OUT  CURRBUF             LOW

hdisk0          0xF1000A01505DC000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk0          0xF1000A01C0148000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01505D4000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01C014C000  0x00000001  0x0000    0x0000000000000000  0x0 <<<<

 

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x1;                 <<<< WRONG QUEUE DEPTH!

 

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

The kdb output above proved that the queue depth was set to 1 in the VWPAR. Even though the ODM still had the attribute set to 256 (in both the Global and VWPAR environments).

 

We discovered that this behaviour was a bug.

 

IV63665: CAN'T SET THE QUEUE_DEPTH IN A VERSION WPAR.

http://www-01.ibm.com/support/docview.wss?uid=isg1IV63665

 

 

APAR status

 

    Closed as fixed if next.

 

Error description

 

    When a  customer run the following in a version

    WPAR such as 5.2 or 5.3. The disk in the kernel

    will not have the queue depth set in kernel.

 

    1. chdev -P -l hdisk5 -a queue_depth=8

    2.  odmget CuAt | grep -p hdisk5 | less

 

    CuAt:

          name = "hdisk5"

          attribute = "queue_depth"

          value = "8"

          type = "R"

          generic = "UD"

          rep = "nr"

          nls_index = 30

 

    3.  stopwpar -N <wpar_name>

 

    The customer then will see that the performance

    is still the same. When a perfpmr is sent in

    from the global lpar you will see that the

    queue_depth is set to 1 for all of the disk

    exported to the wpar.

 

Local fix

 

    N/A

 

Problem summary

 

    When a  customer run the following in a version

    WPAR such as 5.2 or 5.3. The disk in the kernel

    will not have the queue depth set in kernel.

 

Fortunately, IBM support was able to provide us with a workaround. The first step was to add the missing vparent PdAt entry to the ODM in the Global environment.

 

# cat addodm_pdat_for_vparent.txt

PdAt:

      uniquetype = "wio/common/vparent"

      attribute = "naca_1_spt"

      deflt = "1"

      values = "1"

      width = ""

      type = "R"

      generic = ""

      rep = "n"

      nls_index = 0

 

# odmadd addodm_pdat_for_vparent.txt

 

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

        uniquetype = "wio/common/vparent"

        attribute = "naca_1_spt"

        deflt = "1"

        values = "1"

        width = ""

        type = "R"

        generic = ""

        rep = "n"

        nls_index = 0

 

We did the same in the VWPAR.

 

# clogin p8wpar1

# uname -W

11

 

# odmget PdAt | grep -p "wio/common/vparent"

#

# odmadd addodm_pdat_for_vparent.txt

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

        uniquetype = "wio/common/vparent"

        attribute = "naca_1_spt"

        deflt = "1"

        values = "1"

        width = ""

        type = "R"

        generic = ""

        rep = "n"

        nls_index = 0

 

In the VWPAR, we removed the hdisk and then discovered it again, ensuring that the queue depth attribute was set to 256 in the ODM.

 

# uname –W

11

# rmdev -dl hdisk1

hdisk1 deleted

# cfgmgr

# lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

hdisk1          none                                None

# lsattr -El hdisk1 –a queue_depth

queue_depth     256              Queue DEPTH                True

# odmget CuAt | grep -p queue

CuAt:

        name = "hdisk1"

        attribute = "queue_depth"

        value = "256"

        type = "R"

        generic = "UD"

        rep = "nr"

        nls_index = 12

 

Back in the Global environment we checked that the queue depth was set correctly in the kernel. And it was!

 

# uname -W

0

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x100;

 

We re-ran the simple I/O test and immediately found that the test ran faster and the hdisk queue (for hdisk3, as shown by iostat from the Global environment) was no longer full. Subsequent application load tests showed much better performance.

 

# time lmktemp Afile 1024M

Afile

 

real    0m3.15s <<<< BETTER!

user    0m0.03s

sys     0m1.60s

 

 

# iostat -DTRl 1

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:14

hdisk0           0.0   4.1K   1.0   4.1K   0.0    1.0   0.2    0.2    0.2     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:14

hdisk2           0.0  20.5K   5.0  20.5K   0.0    5.0   0.2    0.2    0.4     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   5.0  03:20:14

hdisk3          86.0 280.6M 2144.0   0.0  280.6M   0.0   0.0    0.0    0.0     0    0 2144.0   1.2    0.3    2.9     0    0   0.0    0.0    0.0    0.0   4.0   0.0  03:20:14

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:15

hdisk0           0.0   8.2K   2.0   0.0    8.2K   0.0   0.0    0.0    0.0     0    0   2.0   1.2    0.6    1.9     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:15

hdisk2           0.0   4.1K   1.0   0.0    4.1K   0.0   0.0    0.0    0.0     0    0   1.0   0.6    0.6    0.6     0    0   0.0    0.0    0.0    0.0   0.0   1.0  03:20:15

hdisk3         100.0 327.0M 2495.0   0.0  327.0M   0.0   0.0    0.0    0.0     0    0 2495.0   1.3    0.8   10.0     0    0   0.0    0.0    0.0    0.0   2.0   0.0  03:20:15

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk3         100.0 354.3M 2703.0   0.0  354.3M   0.0   0.0    0.0    0.0     0    0 2703.0   2.1    0.9   16.1     0    0   0.0    0.0    0.0    0.0   5.0   0.0  03:20:16

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk3          32.0 111.7M 852.0   0.0  111.7M   0.0   0.0    0.0    0.0     0    0 852.0   1.3    0.9    2.8     0    0   0.0    0.0    0.0    0.0   1.0   0.0  03:20:17

 

 

Please Note: kdb will not work inside a WPAR. If you attempt to run it, you’ll receive the following error message. Run kdb from the Global environment instead.

 

# clogin p8wpar1

# kdb

The specified kernel file is a 64-bit kernel

open: A file or directory in the path name does not exist.

cannot open /dev/pmem

 

Last year I was working for a customer that was upgrading several AIX 5.3 systems to AIX 6.1. The migrations were successful for the most part, but we did encounter one issue that took a little time to resolve.

 

The customer was using nimadm to migrate. This process worked fine, however, on a couple of systems a strange error was encountered after the migration. The LPAR was booted into AIX 6.1 and everything came up fine. The applications were started and users started accessing the system.

 

It was several days later, when the AIX administrator attempted to configure new storage on the AIX system, when the first sign of trouble appeared.  He had asked his Storage guy to assign a couple of new disks to his LPAR (via NPIV/VFC). As soon as the Storage admin had completed the assignment, the AIX admin ran cfgmgr to detect and configure the new hdisks. Immediately, cfgmgr reported the following error:

 

Method error (/usr/lib/methods/cfgscsidisk):

        0514-023 The specified device does not exist in the

                 customized device configuration database.

 

Initially, the AIX team suspected there was some fault with either the storage device or the zoning of the disk. Both of these items were checked and doubled-checked and were found to be OK. Our next step was to run cfgmgr again, but this time we wanted a greater level of detail captured. To do this we used the following environment variable to force cfgmgr to be ‘more verbose’.

 

# export CFGLOG="cmd,meth,lib,verbosity:9"

 

We ran cfgmgr and went to the /var/adm/ras/cfglog file to view the results with the alog command. However, we noticed that the cfglog file had a size of zero (0) and contained no data.

 

# cd /var/adm/ras

# ls –l cfglog

-rw-r-----    1 root     system      0 May 16 13:22 cfglog

 

We decided to recreate the cfglog alog file and run mkdev again to reproduce the disk configuration error.

 

# rm cfglog

# echo "Create cfglog `date`"|alog -t cfg

# mkdev -l hdisk0

Method error (/usr/lib/methods/cfgscsidisk):

        0514-023 The specified device does not exist in the

                 customized device configuration database.

 

This time we found some useful data in the cfglog file.

 

# alog -t cfg -o

MS 31981804 28835876 /usr/lib/methods/cfgscsidisk -l hdisk39

M4 31981804 Parallel mode = 0

M4 31981804 Get CuDv for hdisk39

M4 31981804 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 31981804 Get parent CuDv, name=fscsi0

M4 31981804 ..is_mpio_capable()

M4 31981804 Device is MPIO

M4 31981804 ..get_paths()

M4 31981804 Getting CuPaths for name='hdisk39'

M4 31981804 Found 1 paths

M0 31981804 cfgcommon.c 225 mpio_init error, rc=23

MS 28835892 31981568 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 28835892 Parallel mode = 0

M4 28835892 Get CuDv for hdisk0

M4 28835892 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 28835892 Get parent CuDv, name=fscsi0

M4 28835892 ..is_mpio_capable()

M4 28835892 Device is MPIO

M4 28835892 ..get_paths()

M4 28835892 Getting CuPaths for name='hdisk0'

M4 28835892 Found 2 paths

M0 28835892 cfgcommon.c 225 mpio_init error, rc=23

MS 25690326 27328608 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 25690326 Parallel mode = 0

M4 25690326 Get CuDv for hdisk0

M4 25690326 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 25690326 Get parent CuDv, name=fscsi0

M4 25690326 ..is_mpio_capable()

M4 25690326 Device is MPIO

M4 25690326 ..get_paths()

M4 25690326 Getting CuPaths for name='hdisk0'

M4 25690326 Found 2 paths

M0 25690326 cfgcommon.c 225 mpio_init error, rc=23

 

The configuration method was attempting to configure a disk device type of htcvspmpio (which was correct) but it was unable to configure the device paths (mpio_init error rc=23). We suspected that the system was missing some sort device driver support for the type of storage in use.

 

Cutting a very long story short, we determined, with the help of the IBM AIX support team, that the issue stemmed from “old” AIX installation media used to create the AIX 6.1 TL6 SP5 SPOT and lppsource on the NIM master. Old AIX 6.1 media was originally used (several years ago) to create the NIM resources and was gradually updated over time, all the way up to TL6 SP5.

 

IBM support identified that the older install media contained a liblpp.a file that was missing the necessary PdPathAt ODM files. Newer install media contained a fix to add the appropriate entries to the bos.rte.cfgfiles. e.g.

                                                                      

SPOT created using OLD install media.

======================================

# ar xv /export/spot/spotaix610605_OLD/usr/lpp/bos/liblpp.a bos.rte.cfgfiles   

x - bos.rte.cfgfiles                                                   

                                                                        

# grep PdPathAt bos.rte.cfgfiles

#                                       

 

SPOT created using NEW install media.

======================================

# ar xv /export/spot/spotaix610605_NEW/usr/lpp/bos/liblpp.a bos.rte.cfgfiles   

x - bos.rte.cfgfiles                                                   

                                                                        

# grep PdPathAt bos.rte.cfgfiles                                       

/usr/lib/objrepos/PdPathAt              v4preserve                     

/usr/lib/objrepos/PdPathAt.vc           v4preserve 

 

We recreated the SPOT and lppsource resources using newer media on the NIM master. We were then able to migrate the AIX 5.3 LPAR to 6.1 without encountering the issues faced previously.

 

My team and I have recently been trying to stream line our AIX disaster recovery process. We’ve been looking for ways to reduce our overall recovery time. Several ideas were tossed around such as a) using a standby DR LPAR with AIX already installed and using rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy (with the –O flag for a device reset) to clone rootvg to an alternate disk which is then replicated to DR. These methods may work but are cumbersome to administer and (in the case of alt_disk_copy) require additional (permanent) resources on every production system. With over 120 production instances of AIX, the disk space requirements start to add up.

 

So far we’ve concluded that the best way to achieve our goal is by using SAN replicated rootvg volumes at our DR site.

 

Our current DR process relies on recovery of AIX systems from mksysb images from a NIM master. All our data (non-rootvg) LUNs are already replicated to our DR site. The aim was to change the process and ‘recover’ our AIX images using replicated rootvg LUNs. This will reduce our overall recovery time at DR (which is crucial if we are to meet the proposed recovery time objectives set by our business). Based on current IBM documentation we were relatively comfortable with the proposed approach. The following IBM developerWorks article (originally published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported. More easily switch AIX environments from one system to another and help achieve higher availability and reduced down time. These scenarios also allow for fast deployment of new systems using cloning.”

 

The document focuses on fully virtualised environments that utilise shared processors and VIO servers. One area where this document is currently lacking in information is the use of NPIV and virtual fibre channel adapters in a DR scenario. We reached out to our contacts in the AIX development space and asked the following question:

 

“Hoping you can help us find some statements regarding support for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?
 
The following IBM developerWorks article discusses VSCSI and we are looking for something similar for NPIV.
 
http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html
 
My guess is that restrictions similar to those for physical FC adapters will apply here? But I'm hoping that given the adapters are virtual the limitations may be relaxed.
 
Are you aware of any statement regarding support (or not) for booting from another system using a disk subsystem image of rootvg replicated to another disk subsystem when using NPIV? And what, if any, additional requirements/restrictions may apply when using NPIV?”

We received the following responses:

“There are some additional considerations when using NPIV for booting from a replicated rootvg.  With NPIV the client partitions has virtual Fibre Channel adapter ports, but has physical access to the actual (physical) disk devices.  There may be an increased chance of needing to update the boot list via the Open Firmware SMS menu.  Since the clients have access to the actual disks, you have the possibility of running multipathing software besides AIX MPIO.   If you are using a multipathing software to manage the NPIV attached disks besides AIX MPIO, then you should contact the vendor that provided the software to check their support statement, 

Since one or more of the physical devices will change when booting from an NPIV replicated rootvg, it is recommend to set the ghostdev attribute.  The ghostdev attribute will trigger when it detects the AIX image is booting from either a different partition or server.  Ghostdev attribute should not trigger during LPM operations (Live Partition Mobility).  Once triggered, ghostdev will clear the customized ODM database.  This will cause detected devices to be discovered as new devices (with default settings), and avoid the issue with missing/stale device entries in ODM.  Since ghostdev does clear the entire customized ODM database, this will require you import your data (non-rootvg) volume groups again, and perform any (device) attribute customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1".  Ghostdev must be set before the rootvg is replicated. 

Similar with virtual devices, the client partition is booting an existing rootvg where the hardware may be different.  It's possible some applications have dependency on tracking the actual physical devices (instead of data on the disks).  For example PowerHA, may keep track of a disk for a cluster health checks.  If  you do have applications that have a dependency on tracking physical devices, then additional setup (of those applications) may be required after the first boot from the replicated rootvg. 

We do have multiple customers using NPIV for such scenarios.  I believe most of them worked with IBM Lab Based Services to assist with implementing such a configuration, and some of the customers required some custom scripts to further customize their system after booting from the replicated rootvg.  Those customers set the ghostdev attribute, and had custom scripts to import their data (non-rootvg) volume groups, and update PowerHA to point to the new health check disk. 

You should get support for such an NPIV setup with IBM as long as you follow the considerations listed in the WhitePaper.”

 

“Development has approved using NPIV to do this for one customer. Below are more detailed requirements for this DR strategy using NPIV. If a Disaster Recovery (DR) environment not using PowerHA Enterprise Edition  is used, then we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and as well as the limitations of such a DR deployment.  Deployments as detailed in the white paper are supported by IBM.  However note that such a deployment has many manually instituted responsibilities on the customer to setup and maintain such an environment. IBM expects that the customer carefully manage these manual steps without any mistakes.  The White paper does not currently cover using NPIV in such a DR scenario.

 

We have the following guidelines regarding the configuration, which includes NPIV as an option: All of the system configuration should be virtualized, with the possible exception of disk devices when using NPIV. If NPIV is used then AIX MPIO must be used as the multi-pathing solution.  If multi-pathing software is used besides AIX MPIO, then the vendor of that software must be contacted regarding a support statement. Install AIX (at least minimum required TLs/SPs for desired AIX version) and software stack (Middleware and applications) on primary systems, which is compatible with systems at both sites. Primary and secondary sites should be using systems with similar hardware, same microcode levels, and same VIOS levels.

 

Many manual steps are needed to setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then discover the unique identification on the primary site and map the disks to the corresponding replication disks. Map the same appropriately on VIOS on secondary site. Level of VIOS should support attribute to open the secondary devices passively. This setting needs to be setup correctly on the VIOS on secondary site. Operating environment should not have subnet dependencies. Manage the replication relationships accurately. Manually may need to switch the secondary disks to primary node. Raw disk usage may cause problems. Some middle ware products may bypass Operating system and use the disk directly. They might have their own restrictions for this environment.  (e.g. anything that is device location code or storage LUN unique ID dependant may have issues when the cloned image is restarted on the secondary system with replicated storage).

 

Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system. If the "ghostdev" attribute is not set, then the booting from the alternate site will result in devices in ODM showing up in "Defined" or "Missing" state.

 

After a failover to secondary site, may need to reset the boot device list for each LPAR before the boot of the LPAR using SMS menus of the firmware Note that this is not an exhaustive list of issues. Refer to the white paper and study the same as it applies to the environment. So as long as they are using MPIO, you're OK.  If not using MPIO and some OEM storage, then the storage vendor must also support it.”

While these responses indicate that this form of recovery is supported by IBM, we were still looking to IBM for clarity on the support position. It has been noted that other IBM customers have had mixed responses when contacting AIX support for feedback and assistance with this type of DR procedure. And it’s not hard to see why when you read statements like this from the “Supported Methods of Duplicating an AIX System” document:

 

“Unsupported Methods

1. Using a bitwise copy of a rootvg disk to another disk.

This bitwise copy can be a one-time snapshot copy such as flashcopy, from one disk to another, or a continuously-updating copy method, such as Metro Mirror.

While these methods will give you an exact duplicate of the installed AIX operating system, the copy of the OS may not be bootable. A typical scenario where this is tried is when one system is a production host and there is a desire to create a duplicate system at a disaster recovery site in a remote location.


2. Removing the rootvg disks from one system and inserting into another.

This also applies to re-zoning SAN disks that contain the rootvg so another host can see them and attempt to boot from them.


Why don't these methods work?

The reason for this is there are many objects in an AIX system that are unique to it; Hardware location codes, World-Wide Port Names, partition identifiers, and Vital Product Data (VPD) to name a few. Most of these objects or identifiers are stored in the ODM and used by AIX commands.

If a disk containing the AIX rootvg in one system is copied bit-for-bit (or removed), then inserted in another system, the firmware in the second system will describe an entirely different device tree than the AIX ODM expects to find, because it is operating on different hardware. Devices that were previously seen will show missing or removed, and usually the system will typically fail to boot with LED 554 (unknown boot disk).”

 

So, as a secondary objective, we have been working closely with our local IBM representatives to obtain some surety from IBM that our proposed DR strategy for AIX is fully supported by both the AIX development and support teams.

 

With that in mind I’ll provide an overview of our new DR approach and hope that it offers others insight to alternative method for recovery and also to assist IBM in further understanding what some of the “larger” AIX customers are looking for in terms of simplified AIX disaster recovery.

 

What follows is a detailed description of our IBM AIX, PowerVM/Power Systems environment, the proposed recovery steps and other items for consideration.

 

Our Environment:

 

·        -  AIX 5.3*, 6.1 & 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.

·         - VIOS 2.2.1.3 – both production and DR.

·        -  VIOS physical FC adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735) – Firmware level: 2.00X7.

·         - All client LPARs utilise Virtual FC adapters (NPIV) for disk storage.

·         - Production: POWER7 795 (9119-FHB). Firmware level: AH730_078.

·         - DR: POWER6 595 (9119-FHA). Firmware level: EH350_120.

·         - HDS VSP storage – at both production and DR sites.

·         - AIX MPIO only – HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is not installed.

·         - Production: Dual VIOS.

·         - DR: Single VIOS.

·         - Production: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.

·         - DR: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a single VIOS. i.e. single path only.

·         - Please refer to the following table for a summary of the environment details.

 

Our Recovery Procedure:

 

1.       - Change the sys0 ghostdev attribute value to 1 on the source production AIX system. Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system.

2.       - Take note of the source systems rootvg hdisk PVID.

3.       - Select the source production rootvg LUN for replication on the HDS VSP.

4.       - Replicate the LUN from the production site to the DR HDS VSP.

5.       - In a DR test, suspend HDS replication from production to DR.

6.       - Assign the replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the WWPN of the virtual FC adapter on the DR LPAR.

7.       - Attempt to boot the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to update the boot list i.e. select the correct boot disk, check for the same PVID as the source host.

8.       - Once the LPAR has successfully booted, the AIX administrator would configure the necessary devices i.e. import data volume groups, configure network interfaces, etc. This may also be scripted for execution during the first boot process.

9.       - Please refer to the following diagrams for a visual representation of the proposed process.

 

Some Notes/Caveats:

 

The following is a list of items that we understand are possible limitations and issues with our new DR process.

 

·         - Booting from replicated rootvg disks may fail for several reasons, such as, a) there is unexpected corruption in the replicated LUN image due to rootvg not being quiesced during replication or b) there is a unidentified issue with the AIX system that is only apparent the next time the system is booted; this could be mis-configuration by the administrator or some other unforseen problem.

 

·         - In the event that an LPAR fails to boot via a replicated rootvg LUN, a backup method is available for recovery. Switching back to manual NIM mksysb restore provides a sufficient backup should the replicated rootvg be unusable.

 

·         - If the "ghostdev" attribute is not set, then booting from the DR site will result in devices in the ODM showing up in a "Defined" or "Missing" state.

 

·        -  Once a DR test is completed, the DR LPAR should be de-activated immediately so that SAN disk replication can be restarted between production and DR. Failure to perform this step may result in the DR LPAR failing as a result of file system corruption.

 

·         - At present we are using AIX MPIO only. There is discussion of using HDLM in the future. We will contact HDS for a support statement regarding booting from replicated rootvg LUNs with HDLM installed.

 

·         - The ghostdev attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.

 

So far all of our testing has been successful. We verified that we could replicate an SOE rootvg image of AIX 6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk. Based on these tests there doesn’t appear to be anything stopping us from using this method for DR purposes. The following table outlines the different versions of AIX we tested and the results.

 

Once the system was booted we needed to perform some post boot configuration tasks. These tasks were handled by two scripts that were called from /etc/inittab. On the source system we installed the new scripts (in /etc) and added new entries to the /etc/inittab file. These scripts only run if the systemid matches that of the DR systemid.  Note: Only partial contents of each script are shown below…but you get the idea.

 

# mkitab -i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# lsitab -a | grep DR

AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

 

AIX_DRconfig.ksh:

 

#!/bin/ksh

#set -xv

#################################################################

#

# This script will configure the AIX environment for DR purposes.

# This script will only run if the systemid matches the systemid of the DR system.

# aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'

#

# 0211D11C1 = DR POWER System

#

#################################################################

 

MYNAME=$(basename $0)

DR_SYSTEMID=0211D11C1

LSATTR_SYSTEMID_DR=`lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`

 

if [ "$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID"  ]

then

   #Set the DR hostname, IP address, netmask and gateway.

   #mktcpip -h <hostname> -a <IP to use in DR> -m <network mask> -i en0 -g <gateway IP>

   #en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0 10.2.2.10,255.255.255.0

 

   dr_hostname=`cat /usr/local/dr/dr_hostname.txt`

   dr_defgw=`cat /usr/local/dr/dr_defgw.txt`

   dr_intf=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $1}'`

   dr_ip=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $2}'`

   dr_netm=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $3}'`

 

   echo "$MYNAME: Setting DR hostname and IP address."

   mktcpip -h $dr_hostname -a $dr_ip -m $dr_netm -i $dr_intf -g $dr_defgw

 

   #Configure IP aliases

   #chdev -l en0 -a alias4=10.1.1.10,255.255.255.0

   #Remove IP aliases

   #chdev -l en0 -a delalias4=10.1.1.10,255.255.255.0

 

   for i in `cat /usr/local/dr/dr_en0.txt | awk -F: '{print $4}'`

   do

     chdev -l $dr_intf -a alias4=$i

   done

 

   for inet in `cat dr_inet.txt | grep -v en0`

   do

       for i in `cat /usr/local/dr/dr_$inet.txt | awk -F: '{print $4}'`

       do

         chdev -l $inet -a alias4=$i

       done

   done

 

   #Configure the AIX environment for DR.

   echo "$MYNAME: Configure the AIX environment for DR."

 

   echo

   #Delete the MDC DNS entries

   echo "$MYNAME: Configuring DNS for DR."

   namerslv -d -i 10.1.6.21

   namerslv -d -i 10.1.4.21

   namerslv -d -i 10.1.5.21

 

   #Add the DR DNS entry

   namerslv -a -i 10.1.7.38

 

   echo

   echo "$MYNAME: Configuring bootlist for DR."

   bootlist -m normal -o

   bootlist -m normal hdisk0

   bootlist -m normal -o

   echo

   echo "$MYNAME: Configuring /etc/inittab for DR."

   lsitab -a | grep nim

   chitab "nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console 2>&1 #   inform nim we're running"

   lsitab -a | grep nim

   echo

 

else

   MYNAME=$(basename $0)

   echo "$MYNAME: The systemid $LSATTR_SYSTEMID_DR does not match the expected DR systemid of $DR_SYSTEMID."

   echo "$MYNAME: This script should only be executed at DR."

   echo "$MYNAME: If you are not booting the system at the DR site, then you can ignore this message."

   echo "$MYNAME: No changes have been perfomed. Script is exiting."

fi

 

 

 

 

 

The ghostdev attribute essentially provides us with a clean ODM and allows the system to discover new devices and build the ODM from scratch. If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554) because of a new device tree and/or missing devices.  You might be able to recover from this situation (without restoring from mksysb) by performing the steps outlined on pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for the presentation).

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf

This little tip was passed on to me by a friendly IBM hardware engineer many years ago.

 

When entering a capacity on demand (CoD) code into a Power system, you can tell how many processors and how much memory will be activated, just by looking at the code you’ve given by IBM.

 

For example, the following codes, when entered for the appropriate Power system, will enable 4 processors (POD) and 64GB of memory (MOD). I can also tell* that once the VET code is entered, this system will be licensed for PowerVM Enterprise Edition (2C28).

 

System Type: 9117    Serial Number: 06-B719P Type Activation Code Posted Date (MM/DD/YYYY)  VET   F2FB60ERC4FF3DB50CA1F00002C28004177   04/30/2010   POD   FA94E589F2067F0C54680000000400419B   04/30/2010   MOD   B154E7097C231A85821200000064004102   04/30/2010

 

Activation type definitions
POD:	CUoD Processor Activation Code
MOD:	CUoD Memory Activation Code
TCOD:	On/Off CoD Enablement Code
TMOD:	On/Off CoD Memory Enablement Code
VET:	Virtualization Technology Code
	(PowerVM, Enterprise Enablement, WWPN, Active Memory Expansion)
STDP:	Standard Trial CoD Processor Activation Code
STDM:	Standard Trial CoD Memory Activation Code
STME:	Standard Trial Active Memory Expansion Code
EXCP:	Exception Trial CoD Processor Activation Code
EXCM:	Exception Trial CoD Memory Activation Code
USTA:	Utility CoD Enablement Code
USTO:	Utility CoD Termination Code
URPT:	Utility CoD Reporting Code
PAID:	Utility CoD PrePaid Code

0000 = Reset to base defaults (No PowerVM)
1xxx = Express
2c00 = Standard
2c20 = Enterprise

*Refer to the following link for more information on “How to tell which PowerVM Editions feature was ordered”: https://www-304.ibm.com/support/docview.wss?uid=isg3T1010860

 

You can obtain your CoD codes from the following website: http://www-912.ibm.com/pod/pod

 

 

======

 

UPDATE 12/9/2011: Script from Chris-R that collects CoD information for a system from the IBM CoD Website. Thanks Chris-R, nice job!

 

Done!

$ ./activations.pl 9117 10 06ebp
02 processors activated on 05/23/2011
024 GB memory activated on 05/23/2011
02 processors activated on 03/11/2011
016 GB memory activated on 03/11/2011
055 GB memory activated on 03/02/2011
06 processors activated on 02/22/2011
096 GB memory activated on 02/22/2011
01 processors activated on 01/12/2011
016 GB memory activated on 01/12/2011
06 processors activated on 03/17/2010
072 GB memory activated on 03/17/2010
19 processors activated on 03/16/2010
228 GB memory activated on 03/16/2010
================
TOTAL CPU=36 RAM=519168MB (507GB)

$cat activations.pl
#!/usr/bin/perl
#
# PoC Script to go out on the net to the IBM POD site and tally up activations for a given i or p Series machine
# Chris-R rutherfc@gmail.com
# v1.0 9/9/2011
#
# todo - add logic for processor deactivation - who uses that? IBM dont even publish the code on the pod site ;)
#
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("mozilla 8.0");
# $ua->proxy(['http'], 'http://proxy:8080/');
$ua->timeout(10);
use HTTP::Request::Common qw(POST);
if ($#ARGV != 2) {
print "usage: $0 MODEL XX XXXXX e.g $0 9119 83 9f6bf\n";
exit;
}
($model, $serial1, $serial2) = @ARGV;

##### main #####
get('http://www-912.ibm.com/pod/pod',"$serial2.htm");
html2txt("$serial2.htm","$serial2.txt");
total("$serial2.txt");
exit;

sub get # fakes a mozilla browser, fills in the CGI form and snags the returned page to a local html file
{
my $req = (POST $_[0],
["system_type" => $model,
"system_serial_number1" => $serial1,
"system_serial_number2" => $serial2 ]);
$request = $ua->request($req);
$activations = $request->content;
open(POD,">$_[1]");
print POD $activations;
close(POD);
}

sub html2txt # strips out the crap and converts the table to a local txt file to parse
{
open(HTML,"<$_[0]");
open(TXT,">$_[1]");
while (<HTML>) {
if (/<\/table>/) {$f = 0;};
if (/<th>Posted Date \(MM/) {$f = 1;};
if ($f == 1) {
# poor mans HTML::TableExtract - excuse my sed like perl....
s/<tr align="center">/:/g;
s/<[^>][^>]*>//g;
s/ //g;
s/\n//g;
s/:/\n/g;
print TXT $_;
};
};
close(TXT);
close(HTML);
}
sub total # totals up the de/activations to get totals
{
open(TXT,"<$_[0]");
$[ = 1; $\ = "\n";# set array base & output record separator
while (<TXT>) {
($code,$hex,$date) = split(' ', $_, -1);
if (/POD/) {
$p = substr($hex, 27, 2);
print $p . ' processors activated on ' . $date;
$pt = $pt + $p;
};
if (/MOD/) {
$r = substr($hex, 26, 3);
print $r . ' GB memory activated on ' . $date;
$rt = $rt + $r;
};
if (/RMEM/) {
$r = substr($hex, 27, 2);
print $r . ' GB memory activated on ' . $date;
$rt = $rt - $r;
};
};
print '================';
print 'TOTAL CPU=' . $pt . ' RAM=' . $rt*1024 . 'MB (' . $rt . 'GB)';
close(TXT);
}
</TXT></tr></th></HTML>

 

 

 

 

 

While attending the IBM Power Systems Symposium this week, I learnt that starting with AIX 7.1 (and AIX 6.1 TL6) JFS2 logging is disabled during a mksysb restore. You may be familiar with disabling JFS2 logs, if not, take a look at this IBM technical note:

 

https://www-304.ibm.com/support/docview.wss?uid=isg3T1012643

 

I’ve been unable to find any official documentation from IBM that mentions this new enhancement to the mksysb restore process. However, when I checked my own AIX 7.1 system I found the following statement/code in the /usr/lpp/bosinst/bi_main script:

 

# Disable jfs2 logs for temp mounts -- (def#751604 performance)

if [[ "$2" = "TRUE" ]]; then

   mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

else

   mount -v $LVTYPE -o log=/dev/$LOGDEV,nomanager /dev/$LV $MROOT$FS || $ERROR

fi

 

If I compare this against my AIX 5.3 system, I don’t find any reference to this enhancement.

 

AIX 7.1:

[/usr/lpp/bosinst] > oslevel -s

7100-00-03-1115

[/usr/lpp/bosinst] # grep log=NULL bi_main

bi_main:         mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

 

AIX 5.3:

[/usr/lpp/bosinst] > oslevel -s

5300-11-03-1013

[/usr/lpp/bosinst] # grep log=NULL bi_main

[/usr/lpp/bosinst] #

 

This change is clever. I mean why would we ever need JFS2 logging during a restore? By disabling logging, the restore process will be faster. Simple, really.

IBM made some announcements today relating to their latest POWER7 server offerings. The new line of systems includes new entry level systems and the highly anticipated high-end system, the POWER7 795!  They also officially outlined some of the new features available in AIX 7.1. You can review the details here. I’ve discussed some of these new features here and here. The official AIX 7.1 announcement details are available here.

 

The announcement got me thinking about my recent customer engagements and why some have chosen to deploy AIX into their IBM POWER environments, while others are considering a Linux on POWER solution.

 

I’ve found that it usually comes down to a skills decision more than anything else. Most customers are happy to either continue working with AIX (if they are existing AIX users) or migrate from another UNIX OS to AIX. I’ve seen very few customers actually migrate to Linux on POWER, but I’ve worked with several that have seriously considered it. Those that have chosen to deploy Linux are doing so purely because they have in-house Linux skills. They are concerned that migrating to AIX may be too big a jump for their technical staff. I find this thinking interesting, as most of the customers I’ve dealt with who run other UNIX OS’s like Tru64, Solaris or HP-UX are more than happy to migrate to AIX. They believe the move is relatively minor and doesn’t require massive re-training of their UNIX admins. I tend to agree.

 

For me, AIX is my preferred “Enterprise class” UNIX Operating System. Notice I’m prefacing this with the words Enterprise class. Don’t get me wrong, I have worked with Linux systems in both small and large customer environments. It is a great OS. But I’ve found that it really only fits into environments that have a relatively small number of users and where significant downtime can be tolerated for things like operating system maintenance. This doesn’t fit the Enterprise class of UNIX server OS’s that I’m thinking of here. When I contemplate the word Enterprise, I think of servers and operating systems that can respond to business demands in terms of performance, reliability, stability and availability. An Enterprise UNIX can provide all of these things without compromise. Linux can offer performance and reliability (in my opinion). However, from what I’ve seen, it lacks features & functions in the areas of stability and availability. AIX on the other hand ticks all the boxes. Again, this is just my opinion based on my experiences with both AIX and Linux in the Enterprise landscape. Others will no doubt have their own experiences that may or may not match my own.

 

So when I’m designing an Enterprise UNIX server environment for a customer, I always start with an AIX on POWER base. If the customer wants Linux, sure I can look at that too, but I strongly recommend AIX as my preferred choice for large systems. Most of my customers are running relatively large SAP/Oracle systems. AIX on POWER is a great combination for large Enterprise systems. If you need to deploy large database systems that must service tens of thousands of users (like a big SAP system), then I believe AIX is the perfect OS on which to provide a platform for these large scale systems.

 

AIX is a very mature and powerful UNIX OS. It has been a major player in the UNIX server market for over 20 years (as shown below). Some people are just not aware of how mature, robust and stable the AIX OS has become over the years. There are many impressive aspects of the OS in the areas of performance, scalability, reliability, management and administration.

 

 

Just looking at some of the administration capabilities built into AIX are enough for me to always recommend AIX over Linux (or any other UNIX OS), when it comes to large Enterprise servers.

 

For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler. This is an interactive tool that is part of the AIX OS. Almost all tasks that an AIX administrator may need to perform can be executed using this tool. It is a text-based tool (there is also an X interface but I recommend sticking with the text-based menus). Everything it does, it does through standard AIX commands and Korn shell functions. This feature is especially useful when you need to automate a repetitive task; you can have SMIT create the proper command-line sequence, and you can then use those commands in your own script. My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.

 

The AIX Logical Volume Manager (LVM) is built into the OS, for free. AIX LVM helps UNIX system administrators manage their storage in a very flexible manner. LVM allows logical volumes to span multiple physical volumes. Data on logical volumes appears to be contiguous to the user, but might not be contiguous on the physical volume. This allows file systems, paging space, and other logical volumes to be resized or relocated, span multiple physical volumes, and have their contents replicated for greater flexibility and availability. It provides capabilities for mirroring data across disks, migrating data across disks & storage subsystems, expand/shrink filesystems and more.......all of which can be performed dynamically.....no downtime required. The concept, implementation and interface to the AIX LVM is one of a kind. All of its features support the ‘continuous availability’ philosophy.

 

One of the biggest reasons that I love AIX over Linux is the mksysb. It’s built into the OS and allows you to create a bootable image of your AIX system. This image can be used to restore a broken AIX system or for cloning other systems. The cloning feature is truly amazing. You can take an image created on a low-end system and deploy it on any POWER system, all the way up to the high-end POWER boxes. This simplifies the installation and cloning processing when you need to install and manage many AIX LPARs. By using an SOE mksysb image you can deploy consistent AIX images across your Enterprise POWER server environment.

 

This brings me to another wonderful feature of AIX, the Network Installation Manager (NIM). NIM is powerful network installation tool (comparable to Linux Kickstart). Using NIM you can backup/restore, update and upgrade one or more AIX systems either individually or simultaneously. This can all be achieved over a network connection, removing the need for handling physical installation media forever.

 

Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates. Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows.

 

When it comes to upgrading the OS to a new release of AIX, the nimadm utility can assist the administrator greatly in this task. The nimadm utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a NIM client's rootvg and migrate the disk to a newer version or release of AIX. All of this can be done without disruption to the client (there is no outage required to perform the migration). After the migration is finished, the only downtime required will be a single scheduled reboot of the system.

 

AIX 6.1 introduced new capability that most UNIX operating systems are still working on. Concurrent Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS that can provide continuous availability, even if it needs to be patched. AIX now has the ability to update certain kernel components and kernel extensions in place, without needing a system reboot. In addition, concurrent updates can be removed from the system without needing a reboot. Can you do this with other UNIX OSs?

 

And that’s just some of the features that make AIX the only UNIX OS that I recommend for Enterprise systems. There are many more tools and features that I couldn’t live without (like alt_disk_install, savevg, installp, WPARs, etc, the list goes on). If you are new to AIX and you are considering what your next UNIX OS should be, then I recommend you take a very close look at AIX.

 

And finally, the support provided by IBM, is first class. Whenever I’ve needed assistance with an AIX issue or query, I have always received timely, professional and useful advice. On the rare occasions where I’ve uncovered a new bug, IBM AIX support have always been quick to provide me with an interim fix to resolve or workaround a problem. That’s the sort of support you’d expect for an Enterprise UNIX OS, isn’t it? What’s the support like from your current UNIX (or Linux) OS vendor?

 

Linux is still a viable UNIX operating system. However, I think it’s more suited to certain workloads like small to medium size mail, web and other utility servers and services. AIX, however, would be my platform of choice for my 10TB Oracle database running SAP ERP, not just for performance reasons, but primarily because of the system administration features of AIX that allow me to support and manage the system without impacting my customers or enforcing reboots/outages whenever I need to change something on the system.

 

All IBM need to do is create their own Linux distribution (Blue Linux perhaps?) that has all the features of AIX built in and then I’m sold! But why would they? We already have AIX.

 

Follow me on Twitter: http://twitter.com/cgibbo

 

I like lsof.

 

Whenever I’m building a new AIX system I always make sure to install it. I really like the fact that I can quickly list processes that are connected to TCP and UDP ports on my system. For example, to check for the current SSH connections on my system I can run lsof and check port 22 (SSH). Immediately I have a good idea of the existing SSH sessions/connections. I can also check to see if the SSH server (sshd daemon) is running and listening (LISTEN) on my AIX partition.

 

# lsof -i tcp:22

Value of I :77   np:0

COMMAND    PID     USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

sshd    340040     root    3u  IPv4 0xf100020002e4a3b0 0t277394  TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd    385184 u0008904    3u  IPv4 0xf100020002e4a3b0 0t277394  TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd    409808     root    3u  IPv4 0xf100020001c48bb0      0t0  TCP *:ssh (LISTEN)

 

But sometimes I work on systems that don’t have lsof installed. It may not be practical or appropriate for me to install it either. So I have to find another tool (or tools) that will do something similar.

 

Of course, I could use netstat to check that a server daemon was listening on a particular TCP port and view any established connections. But this doesn’t give me the associated process id’s.

 

$ netstat -a | grep -i ssh

tcp4       0      0  *.ssh                  *.*            LISTEN

tcp4       0     48  aix01.ssh      172.29.131.16.50284    ESTABLISHED

 

Fortunately, the rmsock command can provide that information. So if I wanted to find the process id for the sshd daemon that is listening on my system I’d do the following. First I need to find the socket id using netstat*.

 

# netstat -@aA | grep -i ssh | grep LIST | grep Global

Global  f1000700049303b0 tcp4       0      0  *.ssh     *.*                LISTEN

 

Then I can use rmsock to discover the process id associated with the sockect. In this case it’s PID 282700.

 

$ rmsock f1000200003e9bb0 tcpcb

The socket 0x3e9808 is being held by proccess 282700 (sshd).

 

Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket. Note that the second argument of rmsock is the protocol. It's tcpcb in this example to indicate that the protocol is TCP. The results of the command are also logged to /var/adm/ras/rmsock.log.

 

# tail /var/adm/ras/rmsock.log

socket 0xf100020001c45008 held by process 434420 (writesrv) can't be removed.

socket 0xf100020000663008 held by process 418040 (java) can't be removed.

socket 0xf1000200012ad008 held by process 418040 (java) can't be removed.

socket 0xf100020000dec008 held by process 163840 (inetd) can't be removed.

socket 0xf100020000deb008 held by process 163840 (inetd) can't be removed.

socket 0xf10002000016f808 held by process 192554 (snmpdv3ne) can't be removed.

socket 0xf100020001c51808 held by process 442596 (dtlogin) can't be removed.

socket 0xf1000200012a4008 held by process 418040 (java) can't be removed.

socket 0xf100020000666008 held by process 315640 (java) can't be removed.

socket 0xf100020000deb808 held by process 163840 (inetd) can't be removed.

 

The following links have more information on the rmsock command:

 

http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg21264632

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/rmsock.htm

http://www.ibmsystemsmag.com/print/print.aspx?print_page=%2Faix%2Ftipstechniques%2F6666printp1.aspx&string_referer=/aix/tipstechniques/6666p1.aspx

http://aixblogs.blogspot.com/2009/03/using-netstat-and-rmsock-to-identify.html

 

 

*Note: In my example I specified the @ symbol with the netstat command. I also grep’ed for the string Global. You may have to do the same if you have WPARs running on your system. In my case I have two active WPARs who both have their own sshd process. My Global environment also has an sshd process. So in total there are three sshd daemons that I can view from the Global environment. By specifiying the @ symbol with netstat, I can quickly determine which process belongs to the Global environment and those that exist within each WPAR.

 

# netstat -aA | grep -i ssh | grep LISTEN

f1000700049303b0 tcp4       0      0  *.ssh              *.*                LISTEN

f100070000198bb0 tcp4       0      0  *.ssh              *.*                LISTEN

f100070000194bb0 tcp4       0      0  *.ssh              *.*                LISTEN

 

# netstat -@aA | grep -i ssh | grep LISTEN

Global  f1000700049303b0 tcp4       0      0  *.ssh       *.*                LISTEN

wpar1   f100070000198bb0 tcp4       0      0  *.ssh       *.*                LISTEN

wpar2   f100070000194bb0 tcp4       0      0  *.ssh       *.*                LISTEN

 

OK, it’s official. The AIX 7 Open Beta is now available for testing!

 

An overview of the program is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml

 

Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.

 

I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.

 

The AIX 7 beta code is available for download from here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/download.shtml

 

The user forum, where you can provide feedback and ask questions, is here:

 

https://www.ibm.com/developerworks/forums/forum.jspa?forumID=2255                        

 

I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/library.shtml

 

The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.

 

The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:

 

A.    AIX 5.2 Workload Partitions for AIX 7.

B.    Removal of WPAR local storage device restrictions.

C.    Domain support in Role Based Access Control (RBAC).

D.    Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.

E.    NIM thin server support for IPv6 and NFSv4.

 

I’m interested in testing A, B and D. Here’s why.

 

A.  AIX 5.2 Workload Partitions for AIX 7.

AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.

 

B.  Removal of WPAR local storage device restrictions.

AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?

 

D.  Etherchannel enhancements in 802.3ad mode.

There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I’m interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it’s not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.

 

And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:

 

Installing the AIX Open Beta Driver

The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.

 

Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the ‘mksysb’.

 

1.     Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:

• Type 1 and press Enter to have English during install.

• Type 1 to continue with the install.

• Type 1 to Start Install Now with Default Setting.

 

2.     The system will start installing the AIX 7.0 BETA.

 

3.     Upon completion of the install, the system will reboot. You can then login as “root”, no password is required.

 

Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:

 

·         The Open Beta code is being delivered via an “mksysb” install image. Migration installation is not supported with the open beta driver.

 

·         The open beta driver does not support IBM Systems Director Agent.

 

·         When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.

 

·         The image is known to install without issues on an 8 GB disk.

 

·         oslevel output shows V7BETA.

 

By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17^th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it’s great if you just want to test something on an AIX system.

 

I’ll report back once I’ve got my AIX 7 Open Beta system up and running!

If you are a system administrator that is responsible for managing AIX systems that run SAP, then you’ve probably had an experience similar to the following?

 

OK, so one day my SAP Basis administrator contacts me and says “I can’t start saposcol......can you please reboot the system?” I quickly reply, “Was there an error message when trying to restart saposcol?”. He replies, “No”. Again I return very quickly, “OK, have you checked to see if there are any shared memory segments left for saposcol?”.  Just as quick, he replies “How do I do that?”

 

Together we try starting saposcol and what we find is that it thinks it’s already running (as shown below, PID 327924). But there is no such process!

 

$ saposcol -l

-l

09:37:24 22.01.2009   LOG: Effective User Id is root

***********************************************************************

* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

* Usage:  saposcol -l: Start OS Collector

*         saposcol -k: Stop  OS Collector

*         saposcol -d: OS Collector Dialog Mode

*         saposcol -s: OS Collector Status

* The OS Collector (PID 327924) is already running .....

************************************************************************

$ ps -fp 327924

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

So we try to stop saposcol. This of course fails, as there is no PID 327924 running!

 

$ saposcol -k

 

Setting Stop Flag :

09:19:29 22.01.2009   LOG: ==== Stop Flag was set by saposcol (kill_collector()).

 

09:19:29 22.01.2009   LOG: ====  The collection process will stop as soon as possible

 

********************

can't kill process 327924.

kill: No such process

ERROR:No reaction from collecting process 327924.

Please kill collecting process.

 

My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?

 

Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.

 

http://www.saptechies.com/os-collector-saposcol/

7. Question: How do I remove the shared memory key of saposcol?

Answer: Sometimes it may be necessary to remove the shared memory key of saposcol (see point 6). Caution: Please be very careful! This procedure should be performed only after checking that saposcol is really not running (see point 4) and only in cases when other options (see point 6) really do not work! For this, execute command “ipcs -ma” and note the line that contains saposcol key “4dbe”. You need the shared memory ID. After that, execute command ipcrm -m ID. Now the command “saposcol -s” should show that saposcol is not running and that the shared memory is not attached. The shared memory key will be created automatically by the saposcol when the collector is next started: “saposcol -l”.

 

So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.

 

There are several process id’s ‘attached’ to this segment. However, only one of them actually exists (PID 2293794).

 

# ipcs -ma | grep 4dbe

m   2097156 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      0   1870188 467164 2293794  9:39:43  9:39:43  8:12:43

 

# ps -fp 1870188

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 467164

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 2293794

     UID     PID    PPID   C    STIME    TTY  TIME CMD

  sapadm 2293794       1   0 19:26:41      -  1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01

 

I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.

 

# ipcrm -m 2097156

# ipcs -ma | grep 4dbe

 

We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.

 

# ipcs -ma | grep 4dbe

m   3145732 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      1   1870188 2293818 2207892 10:14:25 10:14:25 10:12:47

 

# ps -ef | grep oscol

  sapadm 2064616       1   0 10:12:53      -  0:00 saposcol -l

   

# /usr/sap/SAP/SYS/exe/run/saposcol -s

**************************************************************

Collector Versions :

  running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

  dialog  : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

Shared Memory       : attached

Number of records   : 17640

Active Flag         : active (01)

Operating System    : AIX aix01 3 5 00C01C704C00

Collector PID       : 2064616 (001F80E8)

Collector           : running

Start time coll.    : Thu Jan 22 10:12:53 2009

 

Current Time        : Thu Jan 22 10:17:46 2009

 

Last write access   : Thu Jan 22 10:17:38 2009

 

Last Read  Access   : Thu Jan 22 10:16:16 2009

 

Collection Interval : 10 sec (next delay).

Collection Interval : 10 sec (last ).

Status              : free

Collect Details     : required

Refresh             : required

 

Header Extention Structure

Number of x-header      Records : 1

Number of Communication Records : 60

Number of free Com.     Records : 60

Resulting offset to 1.data rec. : 61

 

Trace level             : 2

 

Collector in IDLE - mode ? : NO

  become idle after 300 sec without read access.

  Length of Idle Interval  : 60 sec

  Length of norm.Interval  : 10 sec

**************************************************************

 

Problem solved and no reboot required.....this is AIX after all!  :)