Chris's AIX Blog

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

 

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

 

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

   0042-061 m_rmpdir: the "liveupdaterte" resource is currently

        allocated for client use

 

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 1

   server      = master

 

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

 

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

 

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# nim -o remove liveupdaterte

#

 

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

 

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

 

 

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

 

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

 

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. lparaix1nim.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo deallocate -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

NIM boot crash - 888-102-700-0c5.

 

Could be a problem with /etc/exports on NIM master. Permissions on /etc/exports should be 644. Perform these steps on the NIM master:

- Reset the NIM client:

 

# nim -Fo reset <client>                                               

# nim -o deallocate -a subclass=all <client>                           

 

- Recreate the /etc/exports file:

# cd /etc                                                              

# cp exports exports.save                                              

# exportfs -ua                                                         

# vi /etc/exports  ==> remove unwanted entries.

 

- Cleanup and restart NFS daemons.

# stopsrc -g nfs                                                       

# rm -rf sm sm.bak state rmtab xtab

# startsrc -g nfs                                                      

# exportfs -va                                                         

 

- Retry the NIM install.

# smit nim_bosinst                  

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015).  This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs.  Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

 

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

 

 Major Advantages of using HTTP during NIM Management:

 

• All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
• Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
• Easy to consume by NIM or other products that currently use the client/server model of NFS.
• Able to extend the end-product to support additional protocols (context driven).”

 

 

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources.  The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901.  When active, NIM clients attempt file access and/or scripting customization requests from nimhttp.  If http access fails or is denied, a failover attempt at NFS client access occurs.  Future support will include options to remove NFS client attempts altogether.”

 

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user.  First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

 

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/.  This path location is defined as the document_root and cannot be modified at this time.  Future enhancements will support multiple document_root paths.  The document root path is not limited in depth and may contain many sub-directories.  Client requests are able to traverse the path setting by using the enable_directory_listing option.  If set to “no”, all files being served must reside in the current working directory of document_root.”

 

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable.  To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server.  The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM.  Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

 

# nimconfig –c”

 

I tested this functionality during the AIX 7.2 Early Ship Program.

 

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

 

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

 

NIM MASTER:

 

# lslpp -l | grep nim

  bos.sysmgt.nim.master      7.2.0.0  COMMITTED  Network Install Manager -

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager –

 

NIM CLIENT:

 

# lssrc -s nimsh

Subsystem         Group            PID          Status

 nimsh            nimclient        6554064      active

 

# lslpp -l | grep nim

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager -

 

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

 

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

 

# lssrc -s nimhttp

Subsystem         Group            PID          Status

 nimhttp                           6685178      active

 

# ps -ef | grep nimhttp

    root  6685178  4194712   0   Nov 10      -  0:00 /usr/sbin/nimhttpd –v

 

# ls -ltr /httpd.conf

-rw-r--r--    1 root     system         1159 Nov 05 15:31 /httpd.conf

 

 

# cat /httpd.conf

 

#

#---------------------

# http service defines

#---------------------

#

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

#

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

#

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

#

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#

#

#---------------------

# http configuration

#---------------------

#

document_root=/export/nim/

# Designates the directory to serve files from.

#

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

#

enable_proxy=no

# Enable the web service to act as a proxy server.

#

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

#

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

#

 

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdaterte               resources       lpp_source

 

# lsnim -l liveupdaterte

liveu:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r--    1 root     system      2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r--    1 root     system          688 Oct 31 09:33 .toc

 

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

 

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

 

# lslpp -l bos.liveupdate.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

Path: /etc/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

 

# tail -f nimhttp.log

 

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015        socks: Copied string

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015        Request Type is

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015        socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:23 2015        Request Type is GET

Mon Nov 16 13:51:23 2015        Mime not supported

Mon Nov 16 13:51:23 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015        Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015        Total length sent is 509

Mon Nov 16 13:51:23 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015        Total length sent is 72

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015        Total length sent is 755

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Total length sent is 2574336

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

 

 

“Example:  Commands supporting nimhttp

 

If using push operation (From NIM master)

 

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

 

If using pull operation (From NIM client)

 

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

 

I’ve been working with a customer recently on an issue with nimadm. They were attempting to migrate a system from AIX 5.3 to 7.1 using nimadm. The NIM client AIX level was 5.3 TL12 SP4 and the NIM master was running AIX 7.1 TL1 SP1.

 

lpar1 : / # oslevel -s

5300-12-04-1119

 

root@nim1 : / # oslevel -s

7100-01-01-1141

 

They were running the following nimadm command:

 

# nimadm -j nimadmvg -c lpar1 -s spotaix710101 -l lpp_sourceaix710101 -d hdisk1 -Y

 

The nimadm operation would always fail at phase 11:

 

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

 

We also found the init_multibos error in the /var/adm/ras/alt_disk_inst.log file on the NIM client:

 

Tue Nov 22 14:48:12 EETDT 2011

cmd: /ALT_MIG_SPOT/sbin/alt_disk_copy -j -M 7.1 -P3 -d hdisk1

Verifying altinst_rootvg...

alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.

Cleaning up.

 

Given that the error appeared to be related to init_multibos, we assumed the failure was due to some multibos checks being performed by alt_disk_copy on the client. The client system did not have an existing multibos standby instance. So, we tried two things: First we created a standby instance on the client (multibos –s –X) and re-tried the nimadm operation. This failed. Next we removed the standby instance (multibos –R) and re-tried the nimadm operation. This worked and the client then migrated to AIX 7.1 successfully. We re-tried the same operations (i.e. create standby instance, remove standby instance & nimadm) several times and each worked as expected.

 

So it appeared that the unofficial work around to this problem would be to create and then remove a standby multibos instance prior to the nimadm migrate. However, the customer has over 200 LPARs that they need to migrate to AIX 7.1. If possible they would really rather avoid this extra step in the AIX 7.1 migration plan. We’ve made contact with IBM support and are hoping they can assist us in identifying the root cause of the issue and provide us with an official solution to the problem.

 

And just yesterday we hit the same problem when migrating from AIX 6.1 to 7.1 using nimadm. I’ll update my blog with any progress we make with this problem. In the meantime, our unofficial work around will get us “out of hot water”!

 

UPDATE (14/12/2011): The simple fix is to remove the /bos_inst directory before attempting the AIX migration. i.e.

 

# rm –r /bos_inst

 

In my nimadm article I wrote:

• If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos enabled systems. Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of rootvg and reboot the LPAR. For example:

# multibos –sXp         # multibos –sX         # shutdown –Fr

 

Confirm the legacy LV names are now in use that is, not bos_.

# lsvg -l rootvg | grep hd | grep open         hd6           paging     80      160     2    open/syncd    N/A         hd8           jfs2log    1       2       2    open/syncd    N/A         hd4           jfs2       1       2       2    open/syncd    /         hd2           jfs2       7       14      2    open/syncd    /usr         hd3           jfs2       16      32      2    open/syncd    /tmp         hd1           jfs2       1       2       2    open/syncd    /home         hd9var        jfs2       8       16      2    open/syncd    /var         hd7           sysdump    8       8       1    open/syncd    N/A         hd7a          sysdump    8       8       1    open/syncd    N/A         hd10opt       jfs2       8       16      2    open/syncd    /opt

 

Remove the old multibos instance.

# multibos -R

 

Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst directory. If this directory exists the nimadm operation will most likely fail.

 

 

 

 

 

Here’s a problem I’ve encountered more than once.

 

Someone installs SAP onto an AIX system and decides to use TCP port 3901 as an SAP service port. This is the same port used by nimsh. In some rare cases, nimsh may not be active on the LPAR, which makes it easy for the SAP installation to hijack port 3901. If nimsh is active, the person installing SAP may consciously stop nimsh and use port 3901 for SAP anyway. Hopefully that doesn’t happen. Hopefully, they will talk to the AIX administrator and discuss the best way forward. Hopefully...

 

In either case, if the port is taken by SAP, nimsh will no longer work. If you love using NIM as much as I do, this is a real problem! We could revert back to using rsh but no-one will do this anymore because of concerns around security. And rightfully so!

 

The ports used by nimsh (3901 and 3902) are registered to Internet Assigned Number Authority (IANA).  These port numbers appear in the /etc/services file.

 

nimsh                   3901/tcp               # NIM Service Handler

nimsh                   3901/udp               # NIM Service Handler

nimaux                  3902/tcp                # NIMsh Auxiliary Port

nimaux                  3902/udp                # NIMsh Auxiliary Port

 

Considering these port numbers are registered with IANA, we can usually persuade our SAP colleagues to change their SAP installation to use a different port number. However, depending on the skills/experience of the SAP resource, one of two things usually happens 1) They take an outage, re-install SAP and choose a different port number or 2) The more experienced/confident SAP basis resource will take an outage and modify the instance to use a different port: without reinstalling SAP.

 

Perhaps SAP need to include a warning in their install notes, advising customers not to use port 3901 on AIX systems (i.e. best practice)?

 

Now, if you must change nimsh to use a different port number, it is possible. But not recommended.

 

To do this, you must change the /etc/services file on the NIM master and the NIM client to reflect the same port numbers for nimsh. This will work until the NIM master or the NIM client have their services file overwritten by way of install or fileset updates. After which, the default values for nimsh will be reinstated.

 

You would also need to change the services file on all of your NIM clients. Every time you performed a NIM fileset update, you would need to remember to change the /etc/services file again. This is painful and bound to catch someone out eventually!

 

In the following example I’ll demonstrate how to change the port number used by nimsh.

 

We start with a typical nimsh configuration using port 3901. On the NIM client, nimsh is listening on port 3901.

nimaix / # lsof -i tcp:3901

Value of I :77   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2883740 root    4u  IPv6 0xf1000e0006f843b0      0t0  TCP *:nimsh (LISTEN)

 

On the NIM master we can connect to the NIM client on port 3901.

 

nimmast / # nim -o lslpp nimaix | head -10

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  Firefox.base.rte          2.0.0.20  COMMITTED  Firefox Web Browser

  ICU4C.rte                  6.1.4.0  COMMITTED  International Components for

                                                 Unicode

                             6.1.4.1  COMMITTED  International Components for

                                                 Unicode

                             6.1.6.0  COMMITTED  International Components for

                                                 Unicode

...

 

The /etc/services file on both the NIM client and the master have the same port number for nimsh.

 

nimmast / # grep nimsh /etc/services

nimsh                    3901/tcp               # NIM Service Handler

nimsh                    3901/udp               # NIM Service Handler

 

nimaix / # grep nimsh /etc/services

nimsh                    3901/tcp               # NIM Service Handler

nimsh                    3901/udp               # NIM Service Handler

 

To change the port number for nimsh, first we select an unused port number, in this case 39011.

 

nimaix / # lsof -i tcp:39011

Value of I :77   np:0

 

We update the /etc/services file on the master and the client to reflect the new port number.

 

nimaix / # grep nimsh /etc/services

#nimsh                   3901/tcp               # NIM Service Handler

#nimsh                   3901/udp               # NIM Service Handler

nimsh                   39011/tcp               # NIM Service Handler

nimsh                   39011/udp               # NIM Service Handler

 

nimmast / # grep nimsh /etc/services

#nimsh                   3901/tcp               # NIM Service Handler

#nimsh                   3901/udp               # NIM Service Handler

nimsh                   39011/tcp               # NIM Service Handler

nimsh                   39011/udp               # NIM Service Handler

 

We stop and start nimsh on the client and confirm that nimsh is now listening on the new port number, 39011.

 

nimaix / # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

nimaix / # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 2883742.

 

nimaix / # lsof -i tcp:39011

Value of I :77   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2883742 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

 

From the NIM master we connect to the NIM client using the nim command.

 

nimmast / # nim -o lslpp nimaix | head -10

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  Firefox.base.rte          2.0.0.20  COMMITTED  Firefox Web Browser

  ICU4C.rte                  6.1.4.0  COMMITTED  International Components for

                                                 Unicode

                             6.1.4.1  COMMITTED  International Components for

                                                 Unicode

                             6.1.6.0  COMMITTED  International Components for

                                                 Unicode

....

 

We can confirm that we have connected to the NIM client on port 39011 by looking at the output from lsof and netstat. There is a TCP session established between the master and the client on port 39011.

 

nimmast / # lsof -i tcp:39011

Value of I :93   np:0

COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

m_showlog 430080 root    8u  IPv4 0xf1000700047693b0      0t5  TCP nimmast:1023->nimaix:nimsh (ESTABLISHED)

 

 

nimmast / # netstat -na | grep 39011

tcp4       0      0  172.29.144.167.1023    172.29.152.241.39011   ESTABLISHED

 

nimaix / # lsof -itcp:39011

Value of I :75   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2293836 root    0u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    1u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    2u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

nimsh   2293836 root    5u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2883742 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

nimsh   2883742 root    5u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

 

If the /etc/services file on either the NIM client or master is incorrect we would see error messages similar to the following:

 

   0042-006 m_lslpp: (From_Master) connect A remote host refused an attempted connect operation.

 

nconn: connect() failed, errno is 79nimaix: A remote host refused an attempted connect operation.

 

Would I recommend this approach? No. This is not a good, permanent solution to the problem.

 

Do I think IBM would support this configuration? No.

 

Why am I writing about it then? Because it might help someone caught in a bind. You could use this method temporarily until you’ve sorted out your SAP port problem.

Do you use SSL with nimsh on AIX? No? Well, you might want to consider it. If you regularly use LPM to migrate AIX partitions from one server to another, you may have found that, on occasion, your NIM master has trouble communicating with its NIM clients afterwards. This is by design, as nimsh uses the NIM clients cpuid to authenticate with the NIM master. During an LPM operation, the cpuid of the NIM client changes and its possible the NIM master may reject the client as a result. This problem can occur even when CPU validation is disabled on the NIM master.

 

In the example below, I’ve LPM’ed a NIM client (750lpar1) to another server. Immediately afterwards, I’m able to execute a NIM command against the NIM client, from the NIM master (750lpar4). At this point the NIM client is configured with standard nimsh authentication i.e. no SSL.

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@750lpar1]/var/adm/ras# echo HELLO > /var/adm/ras/nim.installp

 

NIM MASTER:

[root@750lpar4]/ # nim -o change -a validate_cpuid=no master

[root@750lpar4]/ # lsnim -l master | grep -i cpu

   validate_cpuid      = no

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00                << Notice that the cpuid is different.

 

[root@750lpar4]/ # nim -o showlog 750lpar1

HELLO

 

The cpuid is cached by the nimsh daemon, so the previous system id is retained in memory and passed to the NIM master, which allows the operation to complete successfully. But, if I restart the nimsh daemon, on the NIM client, I find that the NIM master is no longer able to communicate with the client.

 

[root@750lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@750lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160610.

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

0042-001 nim: processing error encountered on "master":

   0042-006 m_mkbosi: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar4]/ # nim -o showlog 750lpar1

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar1]/var/adm/ras# grep match nimsh.log | grep CD

Wed Apr 27 20:51:12 2016        error: local value passed, '00F603CD4C00', does not match environment value '00F627664C00'

 

One way to work around this problem is to use the procedure outlined at the following link:

 

http://www.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.osdevice/lpm_nim.htm

 

“Migrating a NIM client by using LPM

 

When Live Partition Mobility (LPM) is used to move a machine from one physical server to another and the machine is defined as a Network Installation Management (NIM) client, the NIM administrator must update the cpuid attribute for the NIM client to reflect the new hardware value after the LPM migration completes. To update the cpuid attribute, complete the following steps:

 

On the NIM client, acquire the new cpuid ID by running the following command:

 

uname –a

 

On the NIM master, run the following command:

 

nim -o change -a cpuid=cpuid client”

 

However, there is a better way. Using nimsh, with SSL-enabled authentication, will prevent the checking of cpuid during nimsh service handling.  This is considered the recommended choice of operation (since the client/server can agree upon identity using the certificate information passed during the ssl handshake). Once the certificate is in place, the NIM master will disregard any cpuid validation and instead rely on the success of the SSL-handshake. This configuration will work well with LPM. If using standard nimsh, the limitation of cpuid updating would still apply (because NIM has no way of automatically updating a client once the value has changed).

 

Both the NIM master and client must be configured to support SSL-enabled authentication. To configure SSL-enabled authentication, we can use ‘nimconfig -c’ on the NIM master.

 

[root@750lpar4]/ # oslevel -s

7100-04-01-1543

 

[root@750lpar4]/ # nimconfig -c

0513-029 The tftpd Subsystem is already active.

Multiple instances are not supported.

NIM_MASTER_HOSTNAME=750lpar4

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Target "all" is up to date.

Generating a 1024 bit RSA private key

.................++++++

....................................++++++

writing new private key to '/ssl_nimsh/keys/rootkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

Getting Private key

Generating a 1024 bit RSA private key

...++++++

....++++++

writing new private key to '/ssl_nimsh/keys/clientkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

Generating a 1024 bit RSA private key

.....................++++++

..........++++++

writing new private key to '/ssl_nimsh/keys/serverkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

 

[root@750lpar4]/ # lsnim -l | grep -i ssl

   ssl_support         = yes

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /ssl_nimsh/certs/client.pem

certname= /ssl_nimsh/certs/client.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

certname= /tftpboot/server.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

To enable SSL nimsh on the NIM client, we can use the ‘nimclient –c’ command. You can fall-back to non-SSL nimsh, with ‘nimclient –C’.

 

[root@750lpar1]/# oslevel -s

7100-04-01-1543

 

[root@750lpar1]/var/adm/ras# nimclient -c

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Received 2784 Bytes in 0.0 Seconds

0513-044 The nimsh Subsystem was requested to stop.

0513-077 Subsystem has been changed.

0513-059 The nimsh Subsystem has been started. Subsystem PID is 14942422.

 

[root@750lpar1]/var/adm/ras# cat nimsh.log

Fri Apr 29 23:10:45 2016        /usr/sbin/nimsh: NIM Service Handler started from SRC

Fri Apr 29 23:10:45 2016        no environment value for NIM_SECONDARY_PORT

Fri Apr 29 23:10:45 2016        value for hostname is 750lpar1

Fri Apr 29 23:10:45 2016        value for netaddr is: 10.1.50.31

Fri Apr 29 23:10:45 2016        value for route is net,-hopcount,0,,0 and gateway is 10.1.50.1

Fri Apr 29 23:10:45 2016        value for netif is en0

Fri Apr 29 23:10:45 2016        value for netmask is 255.255.255.0

Fri Apr 29 23:10:45 2016        obtained master's hostname: NIM_MASTER_HOSTNAME=750lpar4

 

Fri Apr 29 23:10:45 2016        obtained master's id: NIM_MASTERID=00F603CD4C00

 

Fri Apr 29 23:10:45 2016        value for machine id is 00F603CD4C00

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libssl.so.0.9.8 from /usr/lib/libssl.a ..

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libcrypto.so.0.9.8 from /usr/lib/libcrypto.a ..

 

Now I can LPM the NIM client to another server and even if I restart nimsh, the master can still communicate with the client. All thanks to SSL.

 

NIM MASTER:

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160616.

 

[root@750lpar1]/var/adm/ras# tail -f nimsh.log

Fri Apr 29 23:13:33 2016        passing OpenSSL setting of 1

Fri Apr 29 23:13:33 2016        set symbol table

Fri Apr 29 23:13:33 2016         cert filename discovered: /ssl_nimsh/certs/750lpar4.0

Fri Apr 29 23:13:33 2016        ** OpenSSL FIPS mode enabled successfully

Fri Apr 29 23:13:33 2016        seed_prng

Fri Apr 29 23:13:33 2016        Loading certificates..

Fri Apr 29 23:13:33 2016        Loading private key file..

Fri Apr 29 23:13:33 2016        create BIO

Fri Apr 29 23:13:33 2016        - SSL Connection verified successfully -

Fri Apr 29 23:13:33 2016        sending ack to client

Fri Apr 29 23:13:33 2016        setting descriptors to include 2nd port

Fri Apr 29 23:13:33 2016        command to exec  __  /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_showlog" "-alocation=/var/adm/ras/nim.installp"

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

 

+---------------------------------------------------------------------+

                System Backup Image Space Information

              (Sizes are displayed in 1024-byte blocks.)

+---------------------------------------------------------------------+

 

Required = 9457741 (9237 MB)    Available = 51653640 (50444 MB)

 

Creating information file (/image.data) for rootvg.

..etc..

 

 

 

Using NIM to install clients configured with SSL authentication

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_install_ssl.htm

 

Using the certificate viewing file

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/sample_files_certificate.htm%23sample_files_certificate?lang=en

 

 

 

 

 

If you have the AIX iSCSI Software Initiator (iscsi0) configured on your system, you may find that nmon reports some “funny” information under the Disk Adapter (a) and Disk I/O (d) screens, such as:

 

┌─topas_nmon──4=Top-by-RAM-use───Host=aix1 Refresh=2 secs───15:09.41──────────────────────────┐

│ Disk-Adapter-I/O ─────────────────────────────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type                                  │

│vscsi1          0.0      0.0      0.0 KB/s     0.0 4     Virtual SCSI Client A                         │

│vscsi0          0.0      0.0      0.0 KB/s     0.0 4     Virtual SCSI Client A                         │

│Adapter name "not available" returned by AIX API                                                       │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 12    Virtual SCSI Client A                         │

│vscsi2          0.0      0.0      0.0 KB/s     0.0 12    Virtual SCSI Client A                         │

│TOTALS  5 adapters       0.0      0.0 KB/s     0.0 32    TOTAL(MB/s)=0.0                               │

│ Disk-KBytes/second-(K=1024,M=1024*1024) ──────────────────────────────────────────────────────────────│

│Disk        Size  Free Disk  Disk    Volume      Disk                                                  │

│ Name         GB    GB Paths Adapter Group       Description                                           │

│hdisk11       29     0   1    vscsi1 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk1        50    20   1    vscsi1 rootvg        Virtual SCSI Disk Drive                             │

│hdisk9        25     0   1    vscsi1 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk3        50    34   1    vscsi1 datavg        Virtual SCSI Disk Drive                             │

│hdisk2        50    34   1    vscsi0 datavg        Virtual SCSI Disk Drive                             │

│hdisk8        25     0   1    vscsi0 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk10       29     0   1    vscsi0 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk0        50    20   1    vscsi0 rootvg        Virtual SCSI Disk Drive                             │

│hdisk6         0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk16        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk17        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk12        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk7         0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk13        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk14        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk15        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk4        50     0   1  not available projvg        MPIO NetApp iSCSI Disk                         │

│hdisk5       750     0   1  not available ipscratchvg   MPIO NetApp iSCSI Disk                         │

│hdisk18      104    54   2    vscsi2 projectsvg    Virtual SCSI Disk Drive                             │

│hdisk19      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk20      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk21      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk22      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk23      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk24      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk25      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk26      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk27      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk28      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk29      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│Totals(MB/s) Read=0.0   Write=0.0   Size(GB)=2354  Free(GB)=161                                        │

│───────────────────────────────────────────────────────────────────────────────────────────────────────│

 

I don’t know if there are any plans to fix this but I wanted to share this information in case anyone else encounters this issue. iSCSI is not all that popular on AIX systems (perhaps this is a good thing?), so it may not impact enough people to warrant a quick fix.

The nohup command in AIX has the ability to “nohup” an existing process and protect if from hangup signals. Using the -p flag, an existing, running process can be “nohup’ed” and told to ignore SIGHUPs.

 

"Why is this useful?" I hear you ask.

 

Well, I'll often start a "long running" process on a system at work, like a backup for instance. I'll sometimes forget to use the nohup command, so that my long running job will ignore any SIGHUPs and continue running, say for example if my ssh session to the server dies. The nohup command also generates a nohup.out file to capture any messages (errors or otherwise) generated by the process e.g.

 

$ nohup ./processname &

$ ls –l nohup.out

-rw-------    1 cgibson  staff  0 May 26 19:42 nohup.out

 

In older releases of AIX, if I forgot to use nohup, I had two choices. I could either stop the process and start it again with nohup  (often not desirable or possible) or I could leave my ssh session logged in on my work PC and hope that it didn't power down or crash or that my ssh connection did not drop over night.

 

Fortunately with more recent releases of AIX, this is not a problem anymore. If I forget to “nohup” a process, I can use the -p flag to inform the already running process that it should ignore all hangup (SIGHUP) signals. This will allow the process to continue running if my ssh session (terminal) dies and will also capture any output from the process into the nohup.out file, so I don’t lose any important output from the process.

 

Here’s an example. I’m at work, it's 5pm and I've started a "long running" process named mycmd. This was started via an ssh (Putty) session from my PC.

 

$ ./mycmd

 

; Partial output from topas showing mycmd running.

Name      PID      CPU%  

mycmd    823432    98.8       

 

 

; proctree output displaying PIDs associcated with mycmd and the associated ssh session (tty).

 

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

I've walked out the door and I'm now on my way home for the evening.

 

Oh no! I've forgotten to start the process with nohup! I don't want to restart my process but I want to protect it from SIGHUPs, while it executes after hours.

 

No problem, I'll use nohup to modify the running process, so I can rest assured it will continue to run, even if my ssh connection drops over night. From home I start a new ssh session on the server and run  nohup against the running process:

 

$ nohup -p 823432

 

At work the next morning, I discover that my original Putty session has indeed dropped! But, my process is still happily running! :)

 

; Partial output from topas showing mycmd still running.

 

Name      PID      CPU%  

mycmd    823432    99.3  

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

You might find this useful if you ever forget to “nohup” a process and don’t want to restart it again.

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nohup.htm

 

 

 

For those who weren't aware.....starting with AIX 5.3 TL7, quorum changes are allowed online without having to varyoff and varyon the volume group. This also means no reboot is required when changing quorum for rootvg. Of course this also applies to AIX 6.1.