Chris's AIX Blog

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

I LPM'ed an AIX LPAR from Frame A to Frame B. The LPAR id before the LPM was 99 and after the LPM it's still 99. And the system id has changed, as expected. Good.

 

Before migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,017268B60 Hardware system identifier False

 

After migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,0157F5B16 Hardware system identifier False

 

Great, that's exactly what I wanted.

 

Hang on a minute! What kind of sorcery is this?!

 

root@lpar1 / # errpt | grep sys

60AFC9E5   0405132517 I O sys0           Partition ID changed since last boot.

 

root@lpar1 / # errpt -aN sys0

---------------------------------------------------------------------------

LABEL:          PARTID_CHANGE

IDENTIFIER:     60AFC9E5

 

Date/Time:       Wed Apr  5 13:25:53 EET 2017

Sequence Number: 81372

Machine Id:      000157F5B16

Node Id:         lpar1

Class:           O

Type:            INFO

WPAR:            Global

Resource Name:   sys0

 

Description

Partition ID changed since last boot.

 

Whaddaya mean my partiton ID changed since last boot?! It hasn't! I know this for a fact! Look at the uname output above!!

 

Well, I'm 100% correct, the partition id HAS NOT CHANGED! But the system id has....and this is the cause of the "informational" message in the AIX error report.

 

AIX uses the output from 'uname -f' to detect if the system AND/OR partition id have changed since last boot.

 

So, in my case, the partition id (99 dec/63 hex) did not change but (as you'd expect with LPM) the system id (shown in hex below) has changed, hence the message "Partition ID changed since last boot".

 

Before LPM.

 

root@lpar1 / # uname -f

80000BA02DC00063

 

After LPM.

 

root@lpar1 / # uname -f

800016B466F00063

 

From the uname man page:

 

-F Displays a system identification string comprised of hexadecimal characters. This identification string is the same for all partitions on a particular system.

                                                     

-f Similar to the F flag, except that the partition number is also used in the calculation of this string. The resulting identification string is unique for each partition on a particular system.

 

So, no problem here. Nothing to see, nothing to do. Move along! 

AIX has a new “critical volume group” capability which will monitor for the loss or failure of a volume group. You can apply this to any volume group, including rootvg. If applied to rootvg, then you can monitor for the loss of the root volume group.

 

This feature may be useful if your AIX LPAR experiences a loss of SAN connectivity e.g. total loss of access to SAN storage and/or all SAN switches. Typically, when this happens, AIX will continue to run, in memory for a period of time and will not immediately crash. Often you can still log on to the AIX system but if you attempt to write a file you’ll see in I/O error. But even then the system may (potentially) remain up. When the SAN issue is resolved the AIX system may continue running, with file systems in read-only mode (or not, it depends) but to really resolve the issue you would still need to reboot the AIX LPAR in order for it regain access to its disks. This can result in the need to run fsck against file systems. Note that the behaviour you encounter will be impacted by a variety of factors, such as length and type of outage. As always, your mileage may vary!

 

You can encounter this behaviour with both VSCSI and NPIV SAN booted LPARs. This new AIX VG option, which caters for the scenario described above, is not enabled by default. From the chvg man page:

 

“-r y | n                      Changes the critical volume group (VG) option of the volume group.

 

n                                  Disables the critical VG option of the volume group.

 

y                                  Enables the critical VG option of the volume group. If the volume group is set

to the critical VG, any I/O request failure starts the Logical Volume Manager

(LVM) metadata write operation to check the state of the disk before

returning the I/O failure. If the critical VG option is set to rootvg and if the

volume group losses access to quorum set of disks (or all disks if quorum

disabled), instead of moving the VG to an offline state, the node is crashed

and a message is displayed on the console.”

 

PowerHA also caters for and supports this now....and should already be enabled by default. You want this feature enabled for your HA clusters so that they respond appropriately to loss of the root volume group and initiate a failover.

 

smitty sysmirror -> Custom Cluster Configuration -> Events -> System Events -> Change/Show Event Response (smitty cm_change_show_sys_event)

 

                     Change/Show Event Response

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                        [Entry Fields]

* Event Name            ROOTVG                 +

 

* Response              Log event and reboot   +

* Active                Yes                    +

 

"Exploitation of LVM rootvg failure monitoring

 

AIX LVM has recently added the capability to change a volume group to be a known as critical volume group. Though PowerHA has allowed critical volume groups in the past, that

only applied to non-operating system/data volume groups. PowerHA v7.2 now also takes advantage of this functionality specifically for rootvg. If the volume group is set to the critical VG, any I/O request failure starts the Logical Volume Manager (LVM) metadata write operation to check the state of the disk before returning the I/O failure. If the critical VG option is set to rootvg and if the volume group losses access to quorum set of disks (or all disks if quorum is disabled), instead of moving the VG to an offline state, the node is crashed and a message is displayed on the console. You can set and validate rootvg as a critical volume group by executing the commands shown below. The command has to run once since we are using the CAA distributed command clcmd.

 

# clcmd chvg -r y rootvg

# clcmd lsvg rootvg |grep CRIT

DISK BLOCK SIZE: 512 CRITICAL VG: yes

DISK BLOCK SIZE: 512 CRITICAL VG: yes"

 

To test this new feature in my lab, I simulated a disk "failure" or accidental unmapping/removal of a rootvg disk from an LPAR.

 

On the AIX LPAR, prior to disk failure simulation, I turn on the “CRITICAL VG” option for rootvg.

 

# oslevel -s

7200-00-00-0000

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    no

 

# chvg -r y rootvg

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    yes

 

On the VIOS, I unmap the rootvg disk from the corresponding vhost adapter:

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

$ lu -list | grep volume-A | grep Lab

volume-AIX71LabImage-b~ 10240       7790        acc7e518f3f561d4719fcecfff88b0f1

volume-AIX71LabImage-b~ 1024        1021        81823bdb90537250b95b3b2f35bb55f9

 

$ lu -unmap -luudid acc7e518f3f561d4719fcecfff88b0f1

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

On the AIX LPAR, I attempt to create (write) a file in /tmp (which resides in rootvg):

 

# touch /tmp/mynewfile

 

The LPAR stops responding immediately. I can no longer connect or login to it.

 

I then remap the disk and restart the LPAR. The AIX error report shows that the system halted due a critical VG going offline.

 

$ lu -map -luudid acc7e518f3f561d4719fcecfff88b0f1 -vadapter vhost30

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

# errpt -a

...

LABEL:          KERNEL_PANIC

IDENTIFIER:     225E3B63

 

Date/Time:       Wed Mar  1 14:27:51 AEDT 2017

Sequence Number: 215

Machine Id:      00F94F584C00

Node Id:         aix72lab

Class:           S

Type:            TEMP

WPAR:            Global

Resource Name:   PANIC

 

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

 

        Recommended Actions

        PERFORM PROBLEM DETERMINATION PROCEDURES

 

Detail Data

ASSERT STRING

 

PANIC STRING

Critical VG Force off, halting.

 

This feature is available with AIX 6.1 or later.

 

IV52743: ADD CRITICAL VG SUPPORT FOR ROOTVG APPLIES TO AIX 7100-03

http://www-01.ibm.com/support/docview.wss?uid=isg1IV52743

 

Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.

 

[root@750lpar4]/ # nim -o lslpp 750lpar9

 

The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:

 

…

Thu Jan 12 14:31:49 2017        Loading certificates..

Thu Jan 12 14:31:49 2017        Loading private key file..

Thu Jan 12 14:31:49 2017        create BIO

…

 

NIM master: 750lpar4

7100-04-03-1543

 

NIM client: 750lpar9

7100-04-02-1614

 

[root@750lpar4]/ # lsnim -l 750lpar9

750lpar9:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar9 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

The root cause of the problem become apparent when we ran truss against the nim –o command.

 

[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9

 

[root@750lpar4]/ # cat truss.lsnim.out

…

13959372: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m   e r r o r :   N o   s u c h

13959372: f i l e   o r   d i r e c t o r y

…

19267612: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m   e r r o r :   N o   s u

19267612: c h   f i l e   o r   d i r e c t o r y

 

The required shared library object files were missing on the NIM master.

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found

 

We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.

 

[root@750lpar4]/usr/lib # slibclean

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x    1 root     system       724913 Jan 18 09:08 libssl.so

-rwxr-xr-x    1 root     system      3031337 Jan 18 09:08 libcrypto.so

 

After that, the nim –o commands started working again.

 

[root@750lpar4]/usr/lib # nim -o showlog 750lpar9

HELLO

 

So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run.  But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.

 

I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.

 

Here’s one good reason to setup NIMSH over SSL.

 

NIMSH, SSL and LPM

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en

 

The following link is a great reference guide for configuring NIMSH over SSL.

 

NIMSH over SSL

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024615

 

 

 

Recently I've come across an odd issue at two different customers. I thought I'd share the experience, in case others also come across this strange behaviour.

 

In both cases they reported j2pg high CPU usage.

 

Similar to this...

 

And, in both cases, we discovered many /usr/sbin/update processes running. Unexpectedly.

 

When we stopped these processes, j2pg's CPU consumption dropped to nominal levels.

 

The j2pg process is responsible for, among other things, flushing data to disk and is called by the sync(d) process.

 

The /usr/sbin/update command is just a script that calls sync in a loop. Its purpose it to "..periodically update the super block., ..execute a sync subroutine every 30 seconds. This action ensures the file system is up-to-date in the event of a system crash".

 

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds5/update.htm

 

# cat /usr/sbin/update

...BLAH...

PATH=/usr/bin:/usr/sbin:/etc

 

while true

do

sync

sleep 30

done &

exit 0

 

Because of the large number of /usr/sbin/update (sync) processes (in some cases over 150 of them), j2pg was constantly kept busy, assisting with flushing data to disk.

 

It appears that the application team (in both cases) was attempting to perform some sort of SQL "update" but due to an issue with their shell environment/PATH setting they were calling /usr/sbin/update instead of the intended update (or UPDATE) command. And yes, a non-root user can call /usr/sbin/update - no problem. So, in the "ps -ef" output we found processes that looked similar to this:

 

fred 50791260 1 0 Jan 09 - 0:04 /usr/bin/bsh /usr/sbin/update prd_ctl_q_and_p.af_data_obj set as_of_dt=2016-12-09 00:00:00.000000 where DATA_OBJ_NM like %LOANIQ%20161209%

 

# ls -ltr /usr/sbin/update

-r-xr-xr-x    1 bin      bin             943 Aug 17 2011  /usr/sbin/update

The application teams were directed to fix their scripts to prevent them calling /usr/sbin/update and instead call the correct command.

 

And here’s some information (more than you’ll probably ever need to know) about j2pg on AIX.

 

"j2pg - Kernel process integral to processing JFS2 I/O requests.

The kernel thread is responsible of managing I/Os in JFS2 filesystems,
so it is normal to see it running in case of lot of I/Os or syncd.

And we could see that j2pg runs syncHashList() very often.The sync is
done in syncHashList(). In syncHashList(), all inodes are extracted from hash
list. And whether the inode needs to synchronize or not is then judged
by iSyncNeeded().

** note that a sync() call will cause the system to scan *all* the
memory currently used for filecaching to see which pages are dirty
and have to be synced to disk

Therefore, the cause of j2pg having this spike is determined by the
two calls that were being made (iSyncNeeded ---> syncHashList). What is
going on here is a flush/sync of the JFS2 metadata to disk. Apparently
some program went recursively through the filesystem accessing files
forcing the inode access timestamp to change. These changes would have
to propogated to the disk.

Here's a few reasons why j2pg would be active and consume high CPU:
1. If there several process issuing sync then the j2pg process will be
very active using cpu resources.
2. If there is file system corruption then the j2pg will use more cpu
resources.
3. If the storage is not running data fast enough then the j2pg process
will be using high amount of cpu resources.

j2pg will get started for any JFS2 dir activity.
Another event that can cause j2pg activity, is syncd.
If the system experiences a lot of JFS2 dir activity, the j2pg process
will also be active handling the I/O.
Since syncd flushes I/O from real memory to disk, then any JFS2 dir's
with files in the buffer will also be hit."

"Checking the syncd...

From data, we see:
$ grep -c sync psb.elfk
351 << this is high
$ grep sync psb.elfk | grep -c oracle
348 << syncd called by Oracle user only

It appears that the number of sync which causes j2pg
to run is causing spikes.

We see:
/usr/sbin/syncd 60

J2pg is responsible for flushing data to disk and is
usually called by the syncd process. If you have a
large number of sync processes running on the system,
that would explain the high CPU for j2pg

The syncd setting determines the frequency with which
the I/O disk-write buffers are flushed.

The AIX default value for syncd as set in /sbin/rc.boot
is 60. It is recommended to change this value to 10.

This will cause the syncd process to run more often
and not allow the dirty file pages to accumulate,
so it runs more frequently but for shorter period of
time. If you wish to make this permanent then edit
the /sbin/rc.boot file and change to the 60 to 10.

You may consider mounting all of the
non-rootvg file systems with the 'noatime' option.
This can be done without any outage:

However selecting a non-peak production hours is better:

Use the commands below: For Example:
# mount -o remount,noatime /oracle
Then use chfs to make is persistent:
# chfs -a options=noatime /oracle

- noatime -
Turns off access-time updates. Using this option can
improve performance on file systems where a large
number of files are read frequently and seldom updated.
If you use the option, the last access time for a file
cannot be determined. If neither atime nor noatime is
specified, atime is the default value."

 

AIX 7.2 requires 2GB of memory to boot, but this minimum is not enforced in the LPAR profile except by Live Update (to ensure we'll be able to boot the surrogate LPAR). You can check your Minimum Memory setting in your LPARs profile by running the lparstat command (as shown below).

 

# lparstat -i | grep Memory
Online Memory                              : 4096 MB
Maximum Memory                             : 8192 MB
Minimum Memory                             : 2048 MB

 

If your partition does not meet the minimum memory profile requirement, you’ll receive the following error message when you perform a live update preview (with geninstall –p –k).

 

Checking lpar minimal memory size:

------------------------------------------

Required memory size: 2048 MB

Current memory size: 1024 MB

1430-119 FAILED: the lpar minimal memory size is not sufficient

 

You’ll need to change partition profile so that the minimum memory setting is, at least, 2048 (2GB) and then stop & start the partition for the profile update to take effect.

 

The /var/adm/ras/liveupdate/logs/lvupdlog log file will also contain error messages indicating the problem:

 

LVUPD 11/28/2016-18:52:00.716 DEBUG lvupdate_utils32.c - 6713 - lvup_check_minmem_size: Partition minimum memory size (1024 MB) on p8tlc-lvup is lower than the minimum memory size required (2048 MB).

 

LVUPD 11/28/2016-18:52:00.716 ERROR lvupdate_utils32.c - 8647 - lvup_preview: ERROR(s) while checking the current mimimal memory size against the computed required size.

 

 

 

 

Enabling TCP large send offload (LSO), on AIX, is generally considered a good thing. It helps with overall network performance.

 

Taking advantage of networking large-send large-receive

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Taking%20advantage%20of%20networking%20large-send%20large-receive

 

TCP large send offload

“This option improves the performance on 10 Gigabit Ethernet and faster adapters for workloads that manage data streaming (such as file transfer protocol (FTP), RCP, tape backup, and similar bulk data movement applications). The virtual Ethernet adapter and shared Ethernet adapter (SEA) devices are exceptions, where the large send offload option is disabled by default due to inter operability problems with the Linux or IBM i operating system. Enabling Large Send and other performance features can be done in AIX and virtual Ethernet adapter or SEA environments.”

 

https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.performance/tcp_large_send_offload.htm

 

So, is LSO enabled in my AIX partition?

 

No, it’s not!

 

# ifconfig en0

en0: flags=1e084863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.50.191 netmask 0xffffff00 broadcast 10.1.50.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

# netstat -p tcp | grep -i large

                0 large sends

                0 bytes sent using largesend

                0 bytes is the biggest largesend

 

Enable it now with chdev!

 

# chdev -l en0 -a mtu_bypass=on

en0 changed

 

# ifconfig en0

en0: flags=1e084863,4c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

        inet 10.1.50.191 netmask 0xffffff00 broadcast 10.1.50.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

I can now see large send packet activity.

 

# netstat -p tcp | grep -i large

                1205 large sends

        19687632 bytes sent using largesend

        17376 bytes is the biggest largesend

 

netstat -p protocol

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.performance/nestat_p_protocol.htm

 

You can use the HMC lssvcevents command to view DLPAR event history. In the following example, I’m looking for “Add processor” related DLPAR events, over the last 30 days, for a specific LPAR.

 

$ lssvcevents -t console -d 30 | egrep 'Add processor|DLPAR' | grep s82861p22_nim

 

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

 

Search for all DLPAR events over the last 30 days.

 

$ lssvcevents -t console -d 30 | grep -i dlpar

time=12/01/2016 00:52:27,text=HSCE2211 User name hscroot: DLPAR Remove processor resources from partition s82861p22_nim  succeeded on managed system Server-8286-42A-SN214F55V

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

 

HMC Manual Reference Pages  - LSSVCEVENTS (1)

http://www.ibm.com/support/knowledgecenter/TI0003N/p8edm/lssvcevents.html

 

Here’s some handy tips and tricks for managing WPARs in your AIX environment.

 

1. How to share a global file system with a WPAR.

 

- In the Global.

 

root@750lpar11]/ # uname -W

0

 

- Global File System that you want to share with the WPAR (750wpar2).

 

[root@750lpar11]/ # df -g /cg

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/cglv          1.98      0.30   85%       64     1% /cg

 

[root@750lpar11]/ # lswpar

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar1  A      S     750wpar1  /wpars/750wpar1  yes

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

- Create "namefs" file system from the Global. /cg file system will be mounted under /stuff in the WPAR.

 

[root@750lpar11]/ # crfs -v namefs -A yes -d /cg -m /wpars/750wpar2/stuff -u 750wpar2

 

[root@750lpar11]/ # lsfs /wpars/750wpar2/stuff

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/cg             --         /wpars/750wpar2/stuff  namefs --      --         no   no

 

root@750lpar11]/ # grep -p /wpars/750wpar2/stuff /etc/filesystems

/wpars/750wpar2/stuff:

        dev             = /cg

        vfs             = namefs

        mount           = true

        type            = 750wpar2

        account         = false

 

- Mount the file system, in the Global.

 

[root@750lpar11]/ # mount /wpars/750wpar2/stuff

 

root@750lpar11]/ # mount | grep stuff

         /cg              /wpars/750wpar2/stuff namefs Jun 23 10:27 rw

 

- In the WPAR.

 

[root@750wpar2]/stuff # uname -W

8

[root@750wpar2]/stuff # df -g /stuff

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

Global             1.98      0.30   85%       64     1% /stuff

 

2. Don't remove the ipv6 address from the global LPAR or else you may not be able to start your WPARs.

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6=''

lo0 changed

 

[root@gibopvc1]/ # stopwpar -Fv p8wpar1

Stopping workload partition 'p8wpar1'.

Stopping workload partition subsystem 'cor_p8wpar1'.

0513-044 The cor_p8wpar1 Subsystem was requested to stop.

Shutting down all workload partition processes.

WPAR='p8wpar1' CID=1

sysV mq ID=2097153 key=0x4107001c uid=0 gid=9

sysV sem ID=5242893 key=0x62023457 uid=0 gid=0

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1'.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

startwpar: 0960-231 ATTENTION: '/usr/lib/wpars/loadwpar' failed with return code 13.

startwpar: 0960-244 Error loading workload partition.

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1'.

Return Status = FAILURE.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6='::1'

lo0 changed

[root@gibopvc1]/ #

[root@gibopvc1]/ #

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

en1 net default: gateway 10.1.1.10

Exporting workload partition devices.

Exporting workload partition kernel extensions.

Starting workload partition subsystem 'cor_p8wpar1'.

0513-059 The cor_p8wpar1 Subsystem has been started. Subsystem PID is 9896106.

Verifying workload partition startup.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

3. Follow this procedure if you are unable to clogin to a WPAR because root logins are not allowed.

 

[root@750lpar11]/ GLOBAL # uname -W

0

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

Remote logins are not allowed for this account.

 

[root@750lpar11]/ GLOBAL # chroot /wpars/750wpar1 /usr/bin/ksh

750lpar11 : / # chuser rlogin=true root

750lpar11 : / # exit

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

750wpar1 : / # oslevel -s

5300-12-09-1341

750wpar1 : / # uname -W

4

 

4. How back up your WPAR to a NIM master.

 

Backing up WPAR to NIM master over NFS mounted file system.

--------------------------------------------------------------------------------------

 

[root@750lpar11]/ GLOBAL # mount nimmast:/export/images/wpars /mnt

[root@750lpar11]/ GLOBAL # savewpar -if /mnt/750wpar1-savewpar-image

Creating information file (/image.data) for 750wpar1-savewpar-image.

Creating list of files to back up

Backing up 92766 files......................

92766 of 92766 files backed up (100%)

0512-038 savewpar: Backup Completed Successfully.

 

############################################################################

 

Backing up WPAR with NIM.

--------------------------------------

On the NIM master, check the Global LPAR is already configured as a NIM client.

 

# nim -o showlog 750lpar11

BEGIN:Thu Jan 30 17:25:47 2014:013017254714

Command line is:

/usr/sbin/installp -acNgXY -e /var/adm/ras/nim.installp -f \

/tmp/.workdir.12583118.12255434_1/.genlib.installp.list.12255434-d \

/tmp/_nim_dir_11862198/mnt0

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

openssh.man.en_US           6.0.0.6100      USR         APPLY       SUCCESS

 

END:Thu Jan 30 17:25:51 2014:013017255114

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

Check that the WPAR hostname/IP can be resolved from the NIM master.

 

# host 750wpar2

750wpar2 is 10.1.50.44

 

# host 10.1.50.44

750wpar2 is 10.1.50.44

 

Define the WPAR in the NIM database. Where 750lpar11 is the Global partition NIM client name and 750wpar2 is the WPAR name.

 

# nim -o define -t wpar -a mgmt_profile1="750lpar11 750wpar2" -a if1="find_net 750wpar2 0" 750wpar2

 

Verify that the NIM master can now query the status of the WPAR.

 

# nim -o lswpar 750wpar2

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

   manages        = 750wpar2

 

# lsnim -l 750wpar2

750wpar2:

   class         = machines

   type          = wpar

   connect       = shell

   platform      = chrp

   if1           = 10_1_50 750wpar2 0

   mgmt_profile1 = 750lpar11 750wpar2

   Cstate        = ready for a NIM operation

   prev_state    = managed system defined but not yet created

   Mstate        = currently running

 

Backup the WPAR from the NIM master.

 

# nim -o define -t savewpar -a server=master -a location=/export/mksysb/cg/750wpar2-savewpar-image -a source=750wpar2 -a mk_image=yes 750wpar2-savewpar-image

 

Creating list of files to back up.

.

Backing up 64560 files..........

 

64560 of 64560 files (100%)

0512-038 savewpar: Backup Completed Successfully.

  

 

5. Moving a Rootvg Versioned Workload Partition (VWPAR) to Another Disk

 

"I tested in the lab, for a versioned wpar you can use it more like a regular LPAR - extend a disk into rootvg, mirror on it, unmirror from the old one, reduce out and rmdev.  No bosboot or bootlist needed.  There is no bootset in a versioned wpar for some reason.”

Example -  5.3 versioned wpar, rootvg with rootvg on hdisk11 from global env.

global environment:
 

add a new disk (hdisk10) to the vwpar:

# chwpar -D devname=hdisk10 rootvg=yes 53rvg

chwpar: 0960-721 The workload partition's rootvg must be extended with the new disk(s). Please run extendvg inside the workload partition prior to stopping or rebooting it.

In the versioned wpar:

# clogin 53rvg
# cfgmgr
# extendvg rootvg hdisk1
# mirrorvg rootvg hdisk1
# unmirrorvg rootvg hdisk0
# reducevg rootvg hdisk0
# rmdev -dl hdisk0
# exit (go back to global env)

Now in the global deallocate the old rootvg disk.

# chwpar -K -D devname=hdisk11 53rvg

Now you can shut down and start the wpar, it will boot from the other disk.

# stopwpar 53rvg
# startwpar 53rvg

Apparently bootset commands don't work with a versioned rootvg wpar:

# lswpar -Br 53rvg
Name   Device Name      Virtual Device  RootVG  Bootset
--------------------------------------------------------

# chwpar -B devname=hdisk10 rootvg=yes 53rvg
chwpar: 0960-808 Cannot add bootsets to versioned workload partitions.
 

 

6.Moving a Rootvg Workload Partition (WPAR) to Another Disk

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024257

 

Technote (FAQ)

 

Question

I have a rootvg WPAR that is on one disk, is there a method to move it to a new disk?

 

Answer

There may be an occasion where you have created a rootvg WPAR on a specific disk and you want to move the entire WPAR to another disk. One example might be that the original disk is from an older storage enclosure, and you wish to move the WPAR to newly purchased storage, connected to the system.

 

You can do this by means of an alternate bootset. Similar to how using the alt_disk_copy command in a global LPAR will create a copy of rootvg on another disk, an alternate bootset is a copy of a WPAR's rootvg on another disk.

 

The example in this technote will use a rootvg wpar that is on a single disk (hdisk11), and has private /opt and /usr filesystems (AKA a "detached" WPAR). This was initially created using these options:

# mkwpar -D devname=hdisk11 rootvg=yes -l -n rootvgwpar
# startwpar rootvgwpar 

1. List the bootsets for the rootvg WPAR (-Br) we are interested in:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk11      hdisk0      yes   0 

So we can see that hdisk11 is being used for the rootvg in the WPAR, as internal hdisk0.

2a. Now we allocate a new unused disk to the WPAR, and set a bootset on it in the same action:

# chwpar -B devname=hdisk9 rootvgwpar
Creating a bootset for WPAR rootvgwpar. Please wait... 

After this is finished we can see that a new bootset has been created on hdisk9:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk1      no   1
rootvgwpar  hdisk11      hdisk0      yes   0 

2b. If we log in to the WPAR we'll see a rootvg and an alternate rootvg:

# clogin rootvgwpar

# lspv
hdisk0     c7a96733e1ebfc      rootvg      active
hdisk1     c7a9673433f0a7      altvg.1 

3. From the global environment again, we can set the bootlist for the WPAR to boot from the bootset on the new disk:

# chwpar -b bootlist=1 rootvgwpar 
check that it was set properly:

# lswpar -b rootvgwpar
Name     Bootlist
---------------------
rootvgwpar ف 

4. Reboot the wpar on the new bootset.

# rebootwpar rootvgwpar

Stopping workload partition rootvgwpar.
Stopping workload partition subsystem cor_rootvgwpar.
0513-044 The cor_rootvgwpar Subsystem was requested to stop.
stopwpar: 0960-261 Waiting up to 600 seconds for workload partition to halt.
Shutting down all workload partition processes.
Unmounting all workload partition file systems.

Starting workload partition rootvgwpar.
Mounting all workload partition file systems.
Loading workload partition.
Exporting workload partition devices.
Exporting workload partition kernel extensions.
Starting workload partition subsystem cor_rootvgwpar.
0513-059 The cor_rootvgwpar Subsystem has been started. Subsystem PID is 8323226.
Verifying workload partition startup.

If we look at the bootsets we'll see that the 2nd disk now has rootvg officially on it:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes  1
rootvgwpar  hdisk11      hdisk1      no   ـ 

That bootset also believes hdisk9 is "hdisk0" for it, and the other disk is hdisk1. Notice the bootset ID has not changed, bootset 0 is still on (global) disk hdisk11 and bootset 1 on (global) disk hdisk9.

If we log into the WPAR we can see this:

# lspv
hdisk0     c7a9673433f0a7      rootvg      active
hdisk1     c7a96733e1ebfc      altvg.0 

The volume group "altvg.0" is our original rootvg.

5. At this point if everything is looking good, we can remove the original bootset from the global environment:

# chwpar -K -B bootset=0 rootvgwpar

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes   1

# lswpar -D rootvgwpar | grep disk
rootvgwpar  disk   hdisk9      yes   EXPORTED
rootvgwpar  disk   hdisk11      no    EXPORTED

6. Now we can remove the original rootvg disk:

# chwpar -K -D devname=hdisk11 rootvgwpar 

7. Remember to run cfgmgr to put the disk in an Available state again so it can be used by another WPAR or volume group:

# lsdev -xc disk -l hdisk11
hdisk11 Defined 00-01-02 MPIO IBM 2076 FC Disk

# cfgmgr

# lsdev -xc disk -l hdisk11
hdisk11 Available 00-01-02 MPIO IBM 2076 FC Disk

 

 

 

 

When troubleshooting Fibre Channel adapter issues on AIX (or VIOS), I often use the fcstat command to assist me in the process. The command may display error numbers similar to the following:

 

$ fcstat fcs0

 

Error opening device: /dev/fscsi0

errno: 00000045

 

The error number (errno) displayed can (in some cases) be used to identify the root cause of a problem. In the example above, the error number is 45. This number is in hex. If we convert this to decimal, the number is 69. Now, if we look for 69 in the /usr/include/errno.h file (on AIX), we discover that this error number relates to a "Network is down" event.

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

 

# echo "ibase=16; 45"|bc

69

 

# grep 69 /usr/include/errno.h

#define ENETDOWN        69      /* Network is down */

 

The AIX error report (errpt or errlog on VIOS) also tells me that there's some type of link error. This helps me focus my investigation toward the most likely problem area. In this case, I suspect either a physical link or cable problem between my FC adapter and the SAN switch.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

7BFEEA1F   1116121816 T H fcs0           LINK ERROR

 

# errpt -aN fcs0 | grep -p Desc

...

Description

LINK ERROR

 

We can use the same process for other errors displayed by the fcstat command. For example:

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000046

 

# echo "ibase=16; 46"|bc

70

 

# grep 70 /usr/include/errno.h

#define ENETUNREACH     70      /* Network is unreachable */

 

This particular error may direct us to the solution outlined in the following tech note:

 

Fibre channel workspace takes over 2 minutes to load but returns no data

http://www-01.ibm.com/support/docview.wss?uid=swg21600064

 

 

fcstat Command (AIX 6.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.2)

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds2/fcstat.htm

 

 

 

There may be better ways to do this, and if there are, please let me know. But lately I've been "hacking around" with cloud-init on AIX and trying to make it behave the way I want it to. There were two problems I faced and solved.

 

My first challenge. The AIX /etc/hosts file isn't updated with the IP address and hostname of the new AIX VM after deployment from PowerVC.

To work-around this niggle*, I added the following short but effective shell script to the Activation Input in the PowerVC GUI. 

 

#!/bin/ksh
myip=`lsattr -El en0 -a netaddr | awk '{print $2}'`
myname=`hostname`
echo Adding $myip $myname to /etc/hosts >> /tmp/mycloud.log
/usr/bin/hostent -a $myip -h $myname
date >> /tmp/mycloud.log

 

 

 

It's ugly but hey, it does the job!

 

And the second niggle. In my customers lab environment, where there's no DNS at all. Everything is /etc/hosts only. Every time they deployed an AIX VM, there was a significant delay for ssh sessions, even with netsvc.conf set to hosts=local4, the dodgy**, default, resolv.conf (search localdomain) forced DNS first....and ssh connections would hang for a minute or so before a login prompt appeared....waiting on name resolution to complete.

 

Some people told me that I could change "manage_resolv_conf" to false in my cloud-init config file and this would prevent the resolv.conf file from being managed (over-written) by cloud-init. But changing that option did nothing. And I really didn't want a resolv.conf file at all anyway!

 

What I really wanted was for cloud-init to deploy the AIX VM and to NOT create an /etc/resolv.conf file. But how? Well, I managed to fudge it***. I made a change to the aix.py python script. With this change, the script now writes out an /etc/resolv.conf.cloud file instead. This works OK.

 

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cat /etc/resolv.conf.cloud
search localdomain

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # mkdir cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cd cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg/
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp ../aix.py .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # vi aix.py

 

Change    resolve_conf_fn = "/etc/resolv.conf"
To        resolve_conf_fn = "/etc/resolv.conf.cloud"

 

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # diff aix.py /tmp/cg/aix.py
35c35
<     resolve_conf_fn = "/etc/resolv.conf.cloud"
---
>     resolve_conf_fn = "/etc/resolv.conf"

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # python -m compileall .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp -p * ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cd ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # rm -r cg

 

Now the customer can connect to new AIX VMs immediately after deployment.

 

# ls -ltr /etc/resolv.conf.cloud
-rw-r--r--    1 root     system           19 Aug 03 12:47 /etc/resolv.conf.cloud

 

It's a hack but it was fun. 

 

And I guess the next time I install the latest release of cloud-init for AIX, I'll need to modify the script again. But I'm OK with this, as I expect the newer release may actually provide me with a fix to each of the problems I faced.

 

*niggle: To cause slight but persistent annoyance, discomfort, or anxiety.

**dodgy: dishonest or unreliable, of low quality.

***fudge it: put together clumsily. Perhaps an alteration of fadge "make suit, fit".

 

My colleague in Lab Services, Ben Swinney, recently configured the Virtual HMC on his Apple Macbook. He was kind enough to write up his experience, so that others could also run the vHMC on their Mac. Me included! 

You can find Ben’s instructions here:

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/vHMC_running_on_Apple_hardware.pdf

Thanks Ben!

 

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

 

"Hello Chris,
 
So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.
 
So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup
 
# lsnim -l netbackup
netbackup:
   class       = resources
   type        = file_res
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /tmp/netbackup
   alloc_count = 0
   server      = master
   dest_dir    = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on. 

Can I actually push files out by themselves? Thanks"

 

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

 

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value     Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value     Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

 

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

 

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm
 
Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really! 

 

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

 

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

First, I'll create a new file_res NIM resource, called soe_conf.

 

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf     resources       file_res

# lsnim -l soe_conf
soe_conf:
   class       = resources
   type        = file_res
   dest_dir    = /usr/local/etc
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /usr/local/etc
   alloc_count = 0
   server      = master

 

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   file_res       = soe_conf
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = not running
   Mstate         = currently running
   cpuid          = 00F94F584C00
   control        = master

 

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

 

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

 

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

 

root@MASTER /# nim -o cust AIXmig

 

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

The file_res resource, is automatically deallocated from the NIM client.

 

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = customization is being performed
   Mstate         = currently running
   cpuid          = 00F94F584C00
   Cstate_result  = success
     
If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable). 
   
  
   

Do you use SSL with nimsh on AIX? No? Well, you might want to consider it. If you regularly use LPM to migrate AIX partitions from one server to another, you may have found that, on occasion, your NIM master has trouble communicating with its NIM clients afterwards. This is by design, as nimsh uses the NIM clients cpuid to authenticate with the NIM master. During an LPM operation, the cpuid of the NIM client changes and its possible the NIM master may reject the client as a result. This problem can occur even when CPU validation is disabled on the NIM master.

 

In the example below, I’ve LPM’ed a NIM client (750lpar1) to another server. Immediately afterwards, I’m able to execute a NIM command against the NIM client, from the NIM master (750lpar4). At this point the NIM client is configured with standard nimsh authentication i.e. no SSL.

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@750lpar1]/var/adm/ras# echo HELLO > /var/adm/ras/nim.installp

 

NIM MASTER:

[root@750lpar4]/ # nim -o change -a validate_cpuid=no master

[root@750lpar4]/ # lsnim -l master | grep -i cpu

   validate_cpuid      = no

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00                << Notice that the cpuid is different.

 

[root@750lpar4]/ # nim -o showlog 750lpar1

HELLO

 

The cpuid is cached by the nimsh daemon, so the previous system id is retained in memory and passed to the NIM master, which allows the operation to complete successfully. But, if I restart the nimsh daemon, on the NIM client, I find that the NIM master is no longer able to communicate with the client.

 

[root@750lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@750lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160610.

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

0042-001 nim: processing error encountered on "master":

   0042-006 m_mkbosi: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar4]/ # nim -o showlog 750lpar1

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar1]/var/adm/ras# grep match nimsh.log | grep CD

Wed Apr 27 20:51:12 2016        error: local value passed, '00F603CD4C00', does not match environment value '00F627664C00'

 

One way to work around this problem is to use the procedure outlined at the following link:

 

http://www.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.osdevice/lpm_nim.htm

 

“Migrating a NIM client by using LPM

 

When Live Partition Mobility (LPM) is used to move a machine from one physical server to another and the machine is defined as a Network Installation Management (NIM) client, the NIM administrator must update the cpuid attribute for the NIM client to reflect the new hardware value after the LPM migration completes. To update the cpuid attribute, complete the following steps:

 

On the NIM client, acquire the new cpuid ID by running the following command:

 

uname –a

 

On the NIM master, run the following command:

 

nim -o change -a cpuid=cpuid client”

 

However, there is a better way. Using nimsh, with SSL-enabled authentication, will prevent the checking of cpuid during nimsh service handling.  This is considered the recommended choice of operation (since the client/server can agree upon identity using the certificate information passed during the ssl handshake). Once the certificate is in place, the NIM master will disregard any cpuid validation and instead rely on the success of the SSL-handshake. This configuration will work well with LPM. If using standard nimsh, the limitation of cpuid updating would still apply (because NIM has no way of automatically updating a client once the value has changed).

 

Both the NIM master and client must be configured to support SSL-enabled authentication. To configure SSL-enabled authentication, we can use ‘nimconfig -c’ on the NIM master.

 

[root@750lpar4]/ # oslevel -s

7100-04-01-1543

 

[root@750lpar4]/ # nimconfig -c

0513-029 The tftpd Subsystem is already active.

Multiple instances are not supported.

NIM_MASTER_HOSTNAME=750lpar4

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Target "all" is up to date.

Generating a 1024 bit RSA private key

.................++++++

....................................++++++

writing new private key to '/ssl_nimsh/keys/rootkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

Getting Private key

Generating a 1024 bit RSA private key

...++++++

....++++++

writing new private key to '/ssl_nimsh/keys/clientkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

Generating a 1024 bit RSA private key

.....................++++++

..........++++++

writing new private key to '/ssl_nimsh/keys/serverkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

 

[root@750lpar4]/ # lsnim -l | grep -i ssl

   ssl_support         = yes

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /ssl_nimsh/certs/client.pem

certname= /ssl_nimsh/certs/client.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

certname= /tftpboot/server.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

To enable SSL nimsh on the NIM client, we can use the ‘nimclient –c’ command. You can fall-back to non-SSL nimsh, with ‘nimclient –C’.

 

[root@750lpar1]/# oslevel -s

7100-04-01-1543

 

[root@750lpar1]/var/adm/ras# nimclient -c

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Received 2784 Bytes in 0.0 Seconds

0513-044 The nimsh Subsystem was requested to stop.

0513-077 Subsystem has been changed.

0513-059 The nimsh Subsystem has been started. Subsystem PID is 14942422.

 

[root@750lpar1]/var/adm/ras# cat nimsh.log

Fri Apr 29 23:10:45 2016        /usr/sbin/nimsh: NIM Service Handler started from SRC

Fri Apr 29 23:10:45 2016        no environment value for NIM_SECONDARY_PORT

Fri Apr 29 23:10:45 2016        value for hostname is 750lpar1

Fri Apr 29 23:10:45 2016        value for netaddr is: 10.1.50.31

Fri Apr 29 23:10:45 2016        value for route is net,-hopcount,0,,0 and gateway is 10.1.50.1

Fri Apr 29 23:10:45 2016        value for netif is en0

Fri Apr 29 23:10:45 2016        value for netmask is 255.255.255.0

Fri Apr 29 23:10:45 2016        obtained master's hostname: NIM_MASTER_HOSTNAME=750lpar4

 

Fri Apr 29 23:10:45 2016        obtained master's id: NIM_MASTERID=00F603CD4C00

 

Fri Apr 29 23:10:45 2016        value for machine id is 00F603CD4C00

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libssl.so.0.9.8 from /usr/lib/libssl.a ..

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libcrypto.so.0.9.8 from /usr/lib/libcrypto.a ..

 

Now I can LPM the NIM client to another server and even if I restart nimsh, the master can still communicate with the client. All thanks to SSL.

 

NIM MASTER:

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160616.

 

[root@750lpar1]/var/adm/ras# tail -f nimsh.log

Fri Apr 29 23:13:33 2016        passing OpenSSL setting of 1

Fri Apr 29 23:13:33 2016        set symbol table

Fri Apr 29 23:13:33 2016         cert filename discovered: /ssl_nimsh/certs/750lpar4.0

Fri Apr 29 23:13:33 2016        ** OpenSSL FIPS mode enabled successfully

Fri Apr 29 23:13:33 2016        seed_prng

Fri Apr 29 23:13:33 2016        Loading certificates..

Fri Apr 29 23:13:33 2016        Loading private key file..

Fri Apr 29 23:13:33 2016        create BIO

Fri Apr 29 23:13:33 2016        - SSL Connection verified successfully -

Fri Apr 29 23:13:33 2016        sending ack to client

Fri Apr 29 23:13:33 2016        setting descriptors to include 2nd port

Fri Apr 29 23:13:33 2016        command to exec  __  /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_showlog" "-alocation=/var/adm/ras/nim.installp"

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

 

+---------------------------------------------------------------------+

                System Backup Image Space Information

              (Sizes are displayed in 1024-byte blocks.)

+---------------------------------------------------------------------+

 

Required = 9457741 (9237 MB)    Available = 51653640 (50444 MB)

 

Creating information file (/image.data) for rootvg.

..etc..

 

 

 

Using NIM to install clients configured with SSL authentication

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_install_ssl.htm

 

Using the certificate viewing file

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/sample_files_certificate.htm%23sample_files_certificate?lang=en

 

 

 

 

 

Is that a versioned WPAR? Use the lswpar command and the –G option to determine the WPAR type. Yeah, I could use clogin and then run oslevel, but this is faster!

 

[root@gibopvc1]/ # lswpar -G p8wpar1

=================================================================

p8wpar1 - Active

=================================================================

Type:                    S

RootVG WPAR:             yes

Owner:                   root

Hostname:                p8wpar1

WPAR-Specific Routing:   no

Virtual IP WPAR:

Directory:               /wpars/p8wpar1

Start/Stop Script:

Auto Start:              no

Private /usr:            yes

Checkpointable:          no

Application:

 

OStype:                  1024

Cross-WPAR IPC:          yes

Architecture:            none

UUID:                    418c1da6-bceb-4b4e-b872-b1c6701da0b6

 

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

 

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

 

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

   0042-061 m_rmpdir: the "liveupdaterte" resource is currently

        allocated for client use

 

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 1

   server      = master

 

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

 

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

 

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# nim -o remove liveupdaterte

#

 

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

 

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

 

 

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

 

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

 

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. lparaix1nim.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo deallocate -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015).  This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs.  Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

 

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

 

 Major Advantages of using HTTP during NIM Management:

 

• All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
• Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
• Easy to consume by NIM or other products that currently use the client/server model of NFS.
• Able to extend the end-product to support additional protocols (context driven).”

 

 

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources.  The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901.  When active, NIM clients attempt file access and/or scripting customization requests from nimhttp.  If http access fails or is denied, a failover attempt at NFS client access occurs.  Future support will include options to remove NFS client attempts altogether.”

 

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user.  First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

 

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/.  This path location is defined as the document_root and cannot be modified at this time.  Future enhancements will support multiple document_root paths.  The document root path is not limited in depth and may contain many sub-directories.  Client requests are able to traverse the path setting by using the enable_directory_listing option.  If set to “no”, all files being served must reside in the current working directory of document_root.”

 

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable.  To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server.  The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM.  Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

 

# nimconfig –c”

 

I tested this functionality during the AIX 7.2 Early Ship Program.

 

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

 

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

 

NIM MASTER:

 

# lslpp -l | grep nim

  bos.sysmgt.nim.master      7.2.0.0  COMMITTED  Network Install Manager -

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager –

 

NIM CLIENT:

 

# lssrc -s nimsh

Subsystem         Group            PID          Status

 nimsh            nimclient        6554064      active

 

# lslpp -l | grep nim

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager -

 

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

 

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

 

# lssrc -s nimhttp

Subsystem         Group            PID          Status

 nimhttp                           6685178      active

 

# ps -ef | grep nimhttp

    root  6685178  4194712   0   Nov 10      -  0:00 /usr/sbin/nimhttpd –v

 

# ls -ltr /httpd.conf

-rw-r--r--    1 root     system         1159 Nov 05 15:31 /httpd.conf

 

 

# cat /httpd.conf

 

#

#---------------------

# http service defines

#---------------------

#

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

#

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

#

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

#

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#

#

#---------------------

# http configuration

#---------------------

#

document_root=/export/nim/

# Designates the directory to serve files from.

#

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

#

enable_proxy=no

# Enable the web service to act as a proxy server.

#

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

#

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

#

 

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdaterte               resources       lpp_source

 

# lsnim -l liveupdaterte

liveu:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r--    1 root     system      2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r--    1 root     system          688 Oct 31 09:33 .toc

 

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

 

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

 

# lslpp -l bos.liveupdate.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

Path: /etc/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

 

# tail -f nimhttp.log

 

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015        socks: Copied string

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015        Request Type is

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015        socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:23 2015        Request Type is GET

Mon Nov 16 13:51:23 2015        Mime not supported

Mon Nov 16 13:51:23 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015        Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015        Total length sent is 509

Mon Nov 16 13:51:23 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015        Total length sent is 72

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015        Total length sent is 755

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Total length sent is 2574336

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

 

 

“Example:  Commands supporting nimhttp

 

If using push operation (From NIM master)

 

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

 

If using pull operation (From NIM client)

 

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

I enjoy it when I open my email in the morning and find a new message with a subject line of “weird one….”! I immediately prepare myself for whatever challenge awaits. Fortunately I do delight in helping others with their AIX challenges so I usually open these emails first and start to diagnose and troubleshoot the problem!

 

This week I was contacted by someone that was having a little trouble with a mksysb backup on one of their AIX systems.

 

“Hi Chris,

This one has me stumped, any ideas? I’ll have to log a call I think as I’m not sure why this is happening. I run a mksysb and it just backs up 4 files! I also can’t do an alt_disk_copy that also fails.

My /etc/exclude.rootvg is empty.

# cat /etc/exclude.rootvg
# mksysb -i /mksysb/aixlpar1-mksysb

Creating information file (/image.data) for rootvg.

Creating list of files to back up.

Backing up 4 files

4 of 4 files (100%)
0512-038 mksysb: Backup Completed Successfully.

# lsmksysb -f /mksysb/aixlpar1-mksysb
New volume on /mksysb/aixlpar1-mksysb:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Wed Oct 21 22:12:04 EST 2015
Files are backed up by name.
The user is root.
5911 ./bosinst.data
11 ./tmp/vgdata/rootvg/image.info
11837 ./image.data
270567 ./tmp/vgdata/rootvg/backup.data
The total size is 288326 bytes.
The number of archived files is 4.”
 

Yep, that is a weird one! J I replied with the following tips.

 

“Hi,

A couple of things to check and try.....

1. Check the mount command is OK. i.e.

# ls -ltr /usr/sbin/mount
-r-sr-xr-x 3 root system 67040 Aug 18 15:52 /usr/sbin/mount << Not zero size?

# mount

2. Check the find command is OK i.e.

# ls -ltr /usr/bin/find
-r-xr-xr-x 1 bin bin 58608 Aug 18 15:44 /usr/bin/find << Not symlink'ed?

3. Delete the /etc/exclude.rootvg file. Try mksysb again. Does it work?

4. Riun mkszfile and check image.data. Are fs data stanzas listed for each file system?

5. Try with verbose and debug enabled on mksysb/mkszfile commands.

# cd /usr/bin
# cp -p mksysb mksysb.orig
# cp -p mkszfile mkszfile.orig

# vi mksysb

; search for main

#################### main ##########################

#
# set up environment
#
PATH=/usr/bin:/usr/sbin:/sbin:/etc: ; export PATH
export ODMDIR="/etc/objrepos"
NAME=`/usr/bin/basename $0`
PLATFORM=`/usr/sbin/bootinfo -a` # Needed for IA64 (value 4)

Add the following:

#################### main ##########################

set -x
for F in $(typeset +f); do typeset -ft $F; done

#
# set up environment
#

; Do the same (above) for mkszfile.

; Save the files and run mkszfile and mksysb with script (to capture the output).

# script /tmp/mksysb.out
# mkszfile
# mksysb -i -v /mksysb/aixlpar-mksysbimg
# exit”
 

A short time later I received another email stating that the problem had been resolved! Curiously, the find command had been (somehow) destroyed on this system.

 

“LLLLLLLLLLLEEEEEEEEEEEGGGGGGGGGGGGEEEEEEEEEEENNNNNNNNNNNDDDDDDDDDDDDDDDDD!!!!!!!!!!!!!!!!!!!!!!!!

 

/usr/bin/find was a zero size file!!! WTF!

 

I just copied the file from another LPAR and it worked perfect.

 

THANKS!!!!”

 

I thought I’d share this information here, just in case anyone else comes across a similar problem in the future.

In my previous post on AIX Live Updates I discussed how to use the geninstall command to perform a non-disruptive (ifix) update on an AIX system. In this post I wanted to show you how to perform the same task using NIM.

 

NIM can be used to start an AIX Live Update operation on a target machine (NIM client) either from a NIM master or from the NIM client itself (with nimclient).

 

Note: The AIX Live Update operation started by NIM calls the hmcauth command during the cust operation to authenticate to the NIM client with the HMC by using the HMC passwd file. The NIM master is responsible for obtaining password information from the HMC (using ssh).  Without it, NIM clients will not have the password information necessary when running hmcauth as part of the NIM client operation.  So, we must first define an hmc object in NIM and create the password file (used when accessing the console.)  Once this required step has been completed, all clients using NIM live_update have the ability to pass the proper hmc login credentials when configuring 'hmcauth'.

 

First, I need to install the dsm.core fileset and configure SSH keys between the NIM master and the HMC.

 

# lslpp -l dsm.core

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  dsm.core                  7.2.0.0  COMMITTED  Distributed Systems Management

                                               Core

 

# dpasswd -f /etc/ibm/sysmgt/dsm/config/my_password_file -U hscroot

 

# dkeyexch -f /etc/ibm/sysmgt/dsm/config/my_password_file -I hmc -H hsc02

 

Then define a HMC and Managed System NIM object on the NIM master.

 

# nim -o define -t hmc -a if1="find_net hsc02 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/my_password_file hsc02

 

# nim -o define -t cec -a hw_type=8286 -a hw_model=42A -a hw_serial=214F58V -a mgmt_source=hsc02 S824

 

The NIM client must either be defined with or updated to include the Managed System name (Management Source) and LPAR id number.

 

# smit nim_chmac

 

               Change/Show Characteristics of a Machine

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                  [Entry Fields]

  Machine Name                                       [AIXmig]

* Hardware Platform Type                             [chrp]                +

* Kernel to use for Network Boot                     [64]                  +

  Machine Type                                        standalone

  Network Install Machine State                       currently running

  Network Install Control State                       ready for a NIM operation

  Primary Network Install Interface

    Network Name                                      net1

    Host Name                                        [AIXmig]

    Network Adapter Hardware Address                 [0]

    Network Adapter Logical Device Name              [ent]

    Cable Type                                        N/A                  +

    Network Speed Setting                            []                    +

    Network Duplex Setting                           []                    +

  IPL ROM Emulation Device                           []                    +/

  VLAN Tag Priority (0 to 7)                         []                    #

  VLAN Tag Identifier (0 to 4094)                    []                    #

  CPU Id                                             [00F94F584C00]

  Communication Protocol used by client              [nimsh]               +

  NFS Client Reserved Ports                          []                    +

  Comments                                           []

 

  Managing System Information

    LPAR Options

      Identity                                       [88]

      Management Source                              [S824]                                                                                                                                      

 

 

# lsnim -l AIXmig

AIXmig:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = net1 AIXmig 0

   cable_type1    = N/A

   mgmt_profile1  = hsc02 88 S824               <<< LPARD id 88, Mgmt Src S824

   Cstate         = ready for a NIM operation

   prev_state     = ready for a NIM operation

   Mstate         = currently running

   cpuid          = 00F94F584C00

   Cstate_result  = success

 

 

I also need to configure an lpp_source for the ifix location (on the NIM master) and the Live Update data file (on the NIM master). This file can reside on the NIM client if you wish but I’ve chosen to manage all the resources on the NIM master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdatefix       resources       lpp_source

 

# lsnim -l liveupdatefix

liveupdatefix:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/ifix

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/ifix

total 72

-rw-r-----    1 root     system        35625 Oct 15 14:50 dummy.150813.epkg.Z

 

# lsnim -t live_update_data

liveupdate_AIXmig     resources       live_update_data

 

# lsnim -l liveupdate_AIXmig

liveupdate_AIXmig:

   class       = resources

   type        = live_update_data

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/lvupdate.data

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/

total 16

drwxr-xr-x    2 root     system          256 Oct 15 14:54 ifix

-r--r-----    1 root     system         4289 Oct 15 15:04 lvupdate.data

 

# tail -20 /nim/lvup/lvupdate.data

 

#         Users need not provide redundant options such as "-a -U -C and -o"

#         in the trc_option field for trace stanza.

#         Do not add a trace stanza to the lvupdate.data file unless you

#         want the live update commands to be traced.

#

 

general:

        mode = automated

        kext_check = no

 

disks:

        nhdisk  = hdisk0

        mhdisk  = hdisk1

        tohdisk =

        tshdisk =

 

hmc:

        lpar_id  = 88

        management_console = 10.1.50.30

        user = hscroot

 

Now I can perform a preview of the live update operation, from the NIM master. The preview operation will be run on the NIM client called AIXmig.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 595

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffers.

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.

Validating live update input data.

 

With the preview looking good, I can perform the actual update, again starting it from the NIM master.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 644

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffer

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be a

Validating live update input data.

 

Non-interruptable live update operation begins in 10 seconds.

....................................

Initializing live update on original LPAR.

 

Validating original LPAR environment.

 

Beginning live update operation on original LPAR.

 

Requesting resources required for live update.

............

Notifying applications of impending live update.

 

Creating rootvg for boot of surrogate.

................................................................

Starting the surrogate LPAR.

........................................

Creating mirror of original LPAR's rootvg.

................................................

Moving workload to surrogate LPAR.

....

        Blackout Time started.

................................................................................................................................

        Blackout Time end.

 

Workload is running on surrogate LPAR.

....................................

Shutting down the Original LPAR.

................

The live update operation succeeded.

 

If you want, you could initiate the live update from the NIM client using the nimclient command. All the resources reside on the NIM master, but the NIM client starts the operation, not the NIM master.

 

# nimclient -o cust -a live_update=yes -a lpp_source=liveupdatefix -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a filesets=dummy.150813.epkg.Z

 

 

AIX Live Update

Starting with AIX Version 7.2, the AIX operating system provides the AIX Live Update function which eliminates downtime associated with patching the AIX operating system. Previous releases of AIX required systems to be rebooted after an interim fix was applied to a running system. This new feature allows workloads to remain active during a Live Update operation and the operating system can use the interim fix immediately without needing to restart the entire system. In the first release of this feature, AIX Live Update will allow customers to install interim fixes (ifixes) only. Ultimately it may be possible to use this function to install AIX Service Packs (SPs) and Technology Levels (TLs) without a reboot.

 

IBM delivers kernel fixes in the form of ifixes to resolve issues that are reported by customers. If a fix changes the AIX kernel or loaded kernel extensions that cannot be unloaded, the host logical partition (LPAR) must be rebooted. To address this issue, AIX Version 7.1, and earlier, provided concurrent update-enabled ifixes that allowed deployment of some limited kernel fixes to a running LPAR. Unfortunately not all ifixes could be delivered as “concurrent update-enabled”. The AIX Live Update solution is not constrained by the same limitations as in the case of concurrent update enabled ifixes. The AIX 7.2, Live Update feature will allow customers to install ifixes without needing to reboot their AIX systems, avoiding downtime for their mission critical, production workloads.

 

This article (in the link below) will discuss the high-level concepts relating to AIX Live Updates and then provide a real example of how to use the tool to patch a live AIX system. I was fortunate enough to take part in an Early Ship Program (ESP) for AIX 7.2. During the ESP I had the opportunity to test the AIX Live Update feature. I’ll share my experience using this tool in the example that follows.

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/AIXLiveUpdateblog.pdf

 

IBMs PowerVP tool became available in November 2013. It was designed to provide Power Systems administrators with performance information in an enhanced visual format. The aim was to accelerate the identification of performance bottlenecks so that performance analysts could make better decisions based on more detailed and comprehensive data from POWER7 (and POWER8) systems. PowerVP presents both System (frame) and Partition level views of performance data. This has not been possible in the past using any single tool. Administrators would typically need to use many different tools and interfaces to obtain a single, system-wide performance view across an entire CEC and to drill down to all individual partitions.

 

The tool was originally developed for IBM internal use only (known as Sleuth) which helped the IBM development team with rapid development of prototype technology and performance analysis. After a brief demonstration of the tool during an internal, invitation only, event for customers at IBM Austin, almost all of the customer attendees requested that the tool be made available for use outside of IBM.

 

In this post I will briefly discuss how to quickly install and configure PowerVP in an AIX environment. I will start by discussing how to install the PowerVP GUI on a Windows laptop and then cover to how to install the PowerVP agents on an AIX and/or VIOS partition. I’ll then show you how to monitor your system and collect system wide metrics for an entire frame by recording and playing back your PowerVP sessions.

 

I’ve written about PowerVP a few times on my blog. First I discussed the challenge I had installing a PowerVP service pack, then how to display VCPU affinity and finally how to collect PowerVP System level data from the AIX command line. Each of these posts referred to version 1.1.2. of PowerVP. In June of 2015, IBM released PowerVP version 1.1.3. I upgraded by lab environment to this version recently and found that the product and its interface had changed quite a bit. I will discuss the changes and new features at the end of this post.

 

To begin, let’s download, extract and install the latest version of PowerVP. Customers that are entitled (PowerVM Enterprise Edition customers) can download the PowerVP software directly from the IBM Entitled Systems Support (ESS) website.

 

http://www-304.ibm.com/servers/eserver/ess/index.wss

 

 

You can be forgiven for thinking that the latest version of PowerVP isn’t available in ESS but if you look closely and expand everything out, you’ll find v1.1.3 is listed for download.

 

 

 

Once you’ve downloaded the software you’ll end up with a PowerVP package that is named something similar to ESD_-_PowerVP_Standard_Edition_v_1.1.3_62015.zip. Extract the zip file and you’ll discover the following directory structure:

 

 

To install the PowerVP GUI for Windows, simply run the PowerVP.exe from the Windows folder.  Choose PowerVP Client GUI and click next.

 

 

When prompted, select to install Liberty for PowerVP as part of the GUI installation. The IBM PowerVP Redbook explains why:

 

“Starting with Version 1.1.3 PowerVP has a web based GUI. It is packaged in the Web Application Archive (WAR) format and it must be deployed onto an application server. By default, PowerVP GUI uses IBM WebSphere® Application Server Liberty Core.  Liberty profile is a new server profile of IBM WebSphere Application Server V8.5. Liberty profile provides all features required to run the PowerVP, it is lightweight, has a small footprint and fast startup time. PowerVP and a configured Liberty profile are packaged into a compressed file. This provides for an easy and efficient distribution and a simplified installation procedure. Because the new PowerVP GUI is web based it is now possible that a single instance of this GUI be accessed by multiple users using web browsers. This eliminates the need to install a console for each PowerVP user and avoids the potential overhead generated by additional performance data requests initiated from multiple consoles. PowerVP users can connect to the web GUI using web browsers. Users must be able to connect to the ports on which the application server is listening. Default port numbers are 9080 for HTTP traffic and 9443 for HTTPS traffic. Port numbers can be changed during the installation process.”

 

 

The following diagram, also from the Redbook, provides a visual representation as to where Liberty fits in with the new GUI, the System and Partition level agents.

 

 

Once the GUI is installed on your Windows desktop, the next step is to extract the PowerVP agent for AIX/VIOS. To do this you, once again, run the PowerVP.exe installer. Select PowerVP Server Agents and click next.

 

 

 

Select AIX/VIOS and click next.

 

 

When prompted for a System Level Agent Hostname or IP Address, you can enter anything here as it is ignored. All we want to do here is extract the installation software, not connect to an agent. I entered localhost, even though this is not where the PowerVP agent will reside. Click next.

 

 

Once the install process is complete you’ll find the extracted AIX/VIOS installation filesets in the C:\Program Files (x86)\IBM\PowerVP\PowerVP_Installation\PowerVP_Agent_Installation_Instructions\AIX directory.

 

 

You can now transfer these files to your AIX and/or VIOS system of choice, essentially wherever you’d like to install and run the PowerVP server agent and any partitions you’d like to monitor as a partition level agent. Many customers have chosen to install the PowerVP System level agent on their VIOS. This seems like a logical place to install it as these systems are typically always up and available. Ensure that you copy the powervp.1.1.30.bff fileset and the the GSKit filesets to the destination system, as both are needed for installation. Of course, you should download and install the latest fixes for PowerVP from the IBM Fix Central site as well.

 

http://www-933.ibm.com/support/fixcentral/

 

 

 

The agent installation on AIX is very simple. Make sure that your hardware and system firmware support PowerVP before you install it. The IBM Redbook, IBM PowerVP Introduction and Technical Overview REDP-5112-00, has a comprehensive list of supported systems and minimum requirements.

 

http://www.redbooks.ibm.com/abstracts/redp5112.html?Open

 

If you attempt to install PowerVP on a POWER system that does not have the correct firmware level, you will see the following message in the /var/log/powervp.log file.

 

# cat /var/log/powervp.log
2014-02-14 15:28:01.844 Error - MSG0047: MSG0205: System does not support the required interfaces. Firmware version 7.7 or greater is required.. [syslet.C:6090]

 

Ensure that Allow performance information collection is ticked on the LPARs properties. If this box is not checked the System level agent will fail to start with the following error.

 

 

 

 

 

2015-07-23 04:57:42.634 Error - syslet- Gpci_tools- Partition is not authorized to collect system-wide performance metrics. Enable the 'Allow performance information collection' partition property using the Hardware Management Console.. [syslet.C:11470]

 

To install the agent on an AIX or VIOS partition, simply copy the filesets to the system and use installp to install both the GSKit and powervp filesets. The GSKit  filesets are required for SSL support with PowerVP. Even if you don’t plan on using SSL with PowerVP, these filesets must be installed when powervp.rte is installed, regardless.

 

 

$ ioslevel

2.2.3.3

$ r oem

# installp -agXd . powervp.rte

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

…

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

GSKit8.gskcrypt64.ppc.rte   8.0.50.42       USR         APPLY       SUCCESS

GSKit8.gskssl64.ppc.rte     8.0.50.42       USR         APPLY       SUCCESS

powervp.rte                 1.1.3.0         USR         APPLY       SUCCESS

powervp.rte                 1.1.3.0         ROOT        APPLY       SUCCESS

powervp.rte                 1.1.3.2         USR         APPLY       SUCCESS

[root@gibopvc2]/tmp/cg/pvp # lslpp -l GSK\*

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  GSKit8.gskcrypt64.ppc.rte

                           8.0.50.42  COMMITTED  IBM GSKit Cryptography Runtime

  GSKit8.gskssl64.ppc.rte  8.0.50.42  COMMITTED  IBM GSKit SSL Runtime With

                                                 Acme Toolkit

[root@gibopvc2]/tmp/cg/pvp #

[root@gibopvc2]/tmp/cg/pvp # lslpp -l powervp.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  powervp.rte                1.1.3.2  APPLIED    PowerVP 1.1.3.2 Fixpack for

                                                 AIX

 

Path: /etc/objrepos

  powervp.rte                1.1.3.0  COMMITTED  PowerVP for AIX

 

With the agent installed successfully, you can simply start the PowerVP agent. No further configuration is required at this point. The agent will run as a System level agent and allow you to connect to it with the PowerVP client GUI.

 

# /etc/rc.d/rc2.d/SPowerVP

# cat /var/log/powervp.log

2015-07-19 04:52:21.911 S824VIO2 - running in partition server mode.

2015-07-19 04:52:23.019 Server running, waiting for clients to request data.

 

However, if you wanted to configure this agent as a Partition level agent, you need to run the PowerVP iconfig tool to point the Partition agent at an existing System level agent. For example, we could configure the newly installed agent to communicate with an existing System level agent at IP address 10.1.50.59. Then we start the agent using the SPowerVP script. We then confirm that the agent has registered with the System level agent by reviewing the output in the /var/log/powervp.log file on the client partition.

 

[root@gibopvc2]/ # oslevel -s

7100-03-04-1441

 

[root@gibopvc2]/tmp/cg/pvp # /opt/ibm/powervp/iconfig Listen="* 13000" SystemLevelAgent=10.1.50.59

 

[root@gibopvc2]/tmp/cg/pvp # grep SystemLevelAgent /etc/opt/ibm/powervp/powervp.conf

#   sample interval from the system level agent. (See SystemLevelAgent

# SystemLevelAgent [hostname-or-address [port [ssl]]]

#   connect to the system level agent. Up to two SystemLevelAgents

SystemLevelAgent 10.1.50.59

 

[root@gibopvc2]/tmp/cg/pvp # /etc/rc.d/rc2.d/SPowerVP

[root@gibopvc2]/ # cat /var/log/powervp.log

2015-07-19 04:52:21.911 GIBOPVC2 - running in partition server mode.

2015-07-19 04:52:23.019 Server running, waiting for clients to request data.

2015-07-19 04:52:23.211 Host GIBOPVC2 successfully registered with syslet server 10.1.50.59.

 

Now that the agent is installed we can connect to it with the PowerVP GUI.  You can start the GUI by doubling clicking on the PowerVP icon on your Windows desktop. This starts the Liberty server, opens your web browser and connects you to the PowerVP interface.

 

 

You should see the following messages as the PowerVP GUI server is started on your Windows desktop/laptop.

 

 

Note: Please ensure that Java is in your path on your Windows machine. If it is not PowerVP will fail to start and you may be presented with an error stating that “javaw” cannot be found. You can check if Java is in the path by opening a DOS prompt and entering a java command For example:

 

C:\Users\ChrisG>java -version

java version "1.7.0"

Java(TM) SE Runtime Environment (build pwi3270sr9-20150417_01(SR9))

IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 x86-32 20150406_242981 (JIT enabled, A

OT enabled)

J9VM - R26_Java726_SR9_20150406_1443_B242981

JIT  - tr.r11_20150401_88894

GC   - R26_Java726_SR9_20150406_1443_B242981

J9CL - 20150406_242981)

JCL - 20150414_02 based on Oracle 7u79-b14

 

If java isn’t found from the DOS command line, then the following link provides some tips on how to resolve this problem.

 

https://www.java.com/en/download/help/path.xml

 

To connect to the System level agent, click on New Connection and enter the IP address or hostname of the partition running the System leve agent, followed by the user name and password of the root user (or padmin if running the agent on a VIOS). Then click on connect.

 

 

 

 

You will be presented with the PowerVP main panel. From here you can start exploring each of the main views available, such as System Topology, Node Drill Down and Partition Drill Down.

 

 

The System Topology view shows the hardware topology of the system we are connected to in the current session. In this view, we can see the topology of a POWER8 S824 with two processor modules. We can see each node has two chips/sockets. We can also see numbers in the boxes which indicate how busy each of the chips are on the system. The lines between the nodes show the traffic on the SMP fabric between each node. If you select Toggle Buses button, PowerVP GUI will show lines between the processor module boxes and processor nodes which represent buses. The Toggle Affinity button is intended to show affinity where every partition has a different colour.

 

 

The Node Drill Down view appears when you click on one of the nodes and allows you to see the resources being consumed by the partitions running on the system. In this view we can see this processor module has 12 cores/processors. We can also see lines showing the buses between the chips. We can also see the memory controllers and the PHB buses which shows traffic to and from our remote I/O.  We also see connecting to the other processor module; this is the SMP connections to other nodes and shows traffic.

 

 

The Partition Drill Down view allows us to drill down on resources being used by a specific partition that we clicked on.  This view opens in a new tab in our web browser. In this view, we see CPU, Memory, Disk Iops, and Ethernet being consumed. We can also get an idea of cache and memory affinity (under Detailed LSU Breakdown). 

 

 

The main panel also provides you with a view of processor utilisation for each LPAR on the system. You can easily sort LPARs based on utilisation to quickly understand which LPARs are consuming the most (or least) CPU across a single system.

 

 

Overall system processor utilisation is available from the main panel also.  This view provides a graph of total processor utilisation, over time, for the entire POWER system. Directly above this graph useful information is displayed for items such as clock frequency, total cores, platform (AIX, Linux, VIOS or IBM i), system model/serial number and sample rate.

 

 

One very useful feature of PowerVP is the ability to record and playback your PowerVP sessions. By clicking on the Start Recording button, PowerVP will start to record your session to your local machine (in my case, my Windows laptop). I can then load this recording at later date for playback inside the PowerVP GUI.

 

 

 

 

A new feature, in version 1.1.3, now allows you to run the VIOS advisor (part) directly from the PowerVP GUI. When you connect to a System level agent on a VIOS, you will be presented with the VIOS Performance Advisor panel in the GUI. You can configure PowerVP to run the VIOS advisor at a particular time or you can run it on demand. You can also retrieve previously created VIOS advisor reports from the GUI. This is a very nice feature.

 

 

When I clicked on the Run Advisor button I noticed, on my VIOS, that a new topas_nmon and part process were started. The process ran for 10 minutes (default) and then a new tar file was created in /opt/ibm/powervp/advisor.

 

# pwd

/opt/ibm/powervp/advisor

# ls -ltr

total 672

-rw-r--r--    1 root     system          369 Jul 23 14:11 ioscli.log

-rw-r--r--    1 root     staff        337920 Jul 23 14:22 s824vio2_150723_14_11_34.tar

#

# ps -ef | grep topas_nmon

    root 14614550        1   0 15:55:20      -  0:00 /usr/bin/topas_nmon  -X -s 15 -c 40 -t -w 4 -l 150 -I 0.1 -ytype=advisor -o ./s824vio2_150723_15_55_20/ -youtput_dir=/opt/ibm/powervp/advisor/./s824vio2_150723_15_55_20/ -ystart_time=15:55:20,Jul23,2015

 

# ps -ef | grep part

    root 16908532  7077922   0 15:55:19      -  0:00 /usr/bin/perl /usr/ios/utils/part -i 10

#

 

A new tab was automatically opened in my web browser which showed me the VIOS advisor report for my VIOS. Impressive stuff!

 

 

Please refer to the PowerVP Redbook for more information how to configure and use this option.

 

Several features and functions have changed since the last release of PowerVP. Here is the short list of important changes I’ve encountered so far:

 

1. The new PowerVP web interface does not support the older level of PowerVP agents. You will need to update both the System level and Partition level agents to v1.1.3 in order for the new version to function.

 

2. The old PowerVP GUI was previously installed on Windows as a PowerVP.exe application. This has been replaced by a laucnh-powervp.bat file (a shortcut will be created on your desktop). This starts the Liberty server for the GUI. You must select to install Liberty for this file to be installed. The following screenshot lists the contents of my PowerVP GUI Installation directory for my Windows laptop.

 

 

I also came across this useful tip in the Redbook. PowerVP can record large amounts of data when recording is enabled. So you should make sure you have sufficient space available to store recorded data on your local machine. It is recommended to increase the sample rate from the default of 1 second to reduce the amount of data collected during recording. The sample rate can be changed by editing the/etc/opt/ibm/powervp/powervp.conf file and changing SampleInterval to a larger value. You need only change the sample interval on the System level agent (the Partition level agents pick up the sample interval from the System level agent). Once you’ve modified the powervp.conf file you must restart the PowerVP system level agent (syslet on AIX).

 

The aim of this post was to help you quickly install and configure PowerVP in your AIX environment. I encourage the reader to review the available PowerVP material from IBM, in particular the PowerVP Redbook, to learn more about the features and functions of the tool. This tool, finally, provides Power System administrators with a single method of obtaining some important performance data in their POWER7/POWER8 systems environment.

 

PowerVP version 1.1.3 installation tips

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W1d4862277dc8_48f6_9e32_135a9e1d39b7/page/PowerVP%20version%201.1.3%20installation%20tips

 

PowerVP Overview

http://www-01.ibm.com/support/knowledgecenter/POWER7/p7ecu/p7ecu_intro_powervp.htm?cp=POWER7%2F1-8-3-10

 

One of my customers was configuring a new AIX 5.3 Versioned WPAR when they came across a very interesting issue. I thought I’d share the experience here, just in case anyone else comes across the problem. We configured the VWPAR to host an old application. The setup was relatively straight forward, restore the AIX 5.3 mksysb into the VWPAR and export the data disk from the Global into the VWPAR, import the volume group and mount the file systems. Job done!  However, we noticed some fairly poor performance during application load tests. After some investigation we discovered that disk I/O performance was worse in the VWPAR than on the source LPAR.  The question was, why?

 

We initially suspected the customers SAN and/or the storage subsystem, but both of these came back clean with no errors or configuration issues.  In the end, the problem was related to a lack of ODM attributes in the PdAt object class, which prevented the VWPAR disk from using the correct queue depth setting.

 

Let me explain by demonstrating the problem and the workaround.

 

First, let’s add a new disk to a VWPAR. This will be used for a data volume group and file system. The disk in question is hdisk3.

 

# uname -W

0

 

# lsdev -Cc disk

hdisk0 Available  Virtual SCSI Disk Drive

hdisk1 Available  Virtual SCSI Disk Drive

hdisk2 Defined    Virtual SCSI Disk Drive

hdisk3 Available  Virtual SCSI Disk Drive  <<<<<<

 

We set the disk queue depth to an appropriate number, in this case 256.

Note: This value will differ depending on the storage subsystem type, so check with your storage team and/or vendor for the best setting for your environment.

 

# chdev -l hdisk3 -a queue_depth=256

hdisk3 changed

 

Using the lsattr command, we verify that the queue depth attribute is set correctly in both the ODM and the AIX kernel.

 

# lsattr -El hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

We can also use kdb to verify the setting in the kernel. Remember at this stage, we are concentrating on hdisk3, which is referenced with a specific kernel device address in kdb.

 

# echo scsidisk | kdb

           START              END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME            ADDRESS             STATE   CMDS_OUT  CURRBUF             LOW

hdisk0          0xF1000A01505DC000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk0          0xF1000A01C0148000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01505D4000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk3          0xF1000A01C014C000  0x00000001  0x0000    0x0000000000000000  0x0  <<<<<<<

 

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x100;

 

From the output above, we can see that the queue depth is correctly i.e. set to 0x100 in Hex (256 in decimal).

 

Next, we export hdisk3 to the VWPAR using the chwpar command. The disk, as expected, enters a Defined state in the Global environment. It is known as hdisk1 in the VWPAR.

 

# chwpar -D devname=hdisk3 p8wpar1

 

# lswpar -D p8wpar1 | head -2 ; lswpar -D p8wpar1 | grep hdisk

Name     Device Name      Type    Virtual Device  RootVG  Status

-------------------------------------------------------------------

p8wpar1  hdisk3           disk    hdisk1          no      EXPORTED <<<<<<

p8wpar1  hdisk2           disk    hdisk0          yes     EXPORTED

 

[root@gibopvc1]/ # lsdev -Cc disk

hdisk0 Available  Virtual SCSI Disk Drive

hdisk1 Available  Virtual SCSI Disk Drive

hdisk2 Defined    Virtual SCSI Disk Drive

hdisk3 Defined    Virtual SCSI Disk Drive

 

In the VWPAR, we run cfgmgr to discover the disk. We create a new data volume group (datavg) and file system (datafs) for application use (note: the steps to create the VG and FS are not shown below). This is for demonstration purposes only; the customer imported the data volume groups on their system.

 

# clogin p8wpar1

*******************************************************************************

*                                                                             *

*  Welcome to AIX Version 5.3!                                                *

*                                                                             *

*  Please see the README file in /usr/lpp/bos for information pertinent to    *

*  this release of the AIX Operating System.                                  *

*                                                                             *

*******************************************************************************

Last login: Sun Apr 26 21:27:03 2015 on /dev/Global from

 

# uname -W

11

 

# lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

 

# cfgmgr ; lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

hdisk1          none                                None

 

# lsvg

rootvg

datavg

 

# lsvg -l datavg

datavg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

datalv              jfs2       1024    1024    1    open/syncd    /datafs

loglv00             jfs2log    1       1       1    open/syncd    N/A

 

We perform a very simple I/O test in the /datafs file system. We write/create a 1GB file and time the execution. We noticed immediately that the task took longer than expected.

 

# cd /datafs

# time lmktemp Afile 1024M

Afile

 

real    0m7.22s <<<<<<<<<<<<<<< SLOW?

user    0m0.04s

sys     0m1.36s

 

We ran the iostat command, from the Global environment, and noticed that “serv qfull” was constantly non-zero (very large numbers) for hdisk3. Essentially the hdisk queue was full all the time. This was bad and unexpected, given the queue depth setting of 256!

 

# iostat -DTRl 1

 

System configuration: lcpu=8 drives=4 paths=8 vdisks=2

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:53

hdisk0           0.0   4.1K   1.0   0.0    4.1K   0.0   0.0    0.0    0.0     0    0   1.0   0.3    0.3    0.3     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:53

hdisk2           1.0  12.3K   3.0   0.0   12.3K   0.0   0.0    0.0    0.0     0    0   3.0   0.5    0.3    0.6     0    0   0.4    0.0    0.9    0.0   0.0   3.0  03:11:53

hdisk3         100.0 140.5M 1072.0   0.0  140.5M   0.0   0.0    0.0    0.0     0    0 1072.0   0.9    0.8   24.0     0    0 176.7  107.5  279.8  152.0   0.0 1072.0  03:11:53

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk0           0.0   4.1K   1.0   4.1K   0.0    1.0   0.3    0.3    0.3     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:54

hdisk3         100.0 154.0M 1175.0   0.0  154.0M   0.0   0.0    0.0    0.0     0    0 1175.0   0.8    0.8    1.8     0    0 161.9  108.0  217.7  303.0   1.0 1175.0  03:11:54

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:55

hdisk3         100.0 150.3M 1147.0   0.0  150.3M   0.0   0.0    0.0    0.0     0    0 1147.0   0.9    0.8   10.6     0    0 165.9  108.5  239.4  304.0   1.0 1147.0  03:11:55

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:56

hdisk3         100.0 155.5M 1186.0   0.0  155.5M   0.0   0.0    0.0    0.0     0    0 1186.0   0.8    0.8    1.8     0    0 161.6  106.8  217.8  307.0   1.0 1186.0  03:11:56

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:11:57

hdisk3          67.0 102.2M 780.0   0.0  102.2M   0.0   0.0    0.0    0.0     0    0 780.0   0.9    0.8   11.8     0    0 166.1  106.3  232.8   53.0   0.0 779.0  03:11:57

 

Now comes the interesting part. With a little help from our friends in IBM support, using kdb we found that the queue depth was reported as being set to 1 in the kernel and not 256! You’ll also notice here that the hdisk name has changed from hdisk3 to hdisk1. This happened as a result of exporting hdisk3 to the VWPAR. The disk is known as hdisk1 in the VWPAR (not hdisk3) but the kernel address is the same.

 

# echo scsidisk | kdb

           START              END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME            ADDRESS             STATE   CMDS_OUT  CURRBUF             LOW

hdisk0          0xF1000A01505DC000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk0          0xF1000A01C0148000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01505D4000  0x00000002  0x0000    0x0000000000000000  0x0

hdisk1          0xF1000A01C014C000  0x00000001  0x0000    0x0000000000000000  0x0 <<<<

 

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x1;                 <<<< WRONG QUEUE DEPTH!

 

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

 

The kdb output above proved that the queue depth was set to 1 in the VWPAR. Even though the ODM still had the attribute set to 256 (in both the Global and VWPAR environments).

 

We discovered that this behaviour was a bug.

 

IV63665: CAN'T SET THE QUEUE_DEPTH IN A VERSION WPAR.

http://www-01.ibm.com/support/docview.wss?uid=isg1IV63665

 

 

APAR status

 

    Closed as fixed if next.

 

Error description

 

    When a  customer run the following in a version

    WPAR such as 5.2 or 5.3. The disk in the kernel

    will not have the queue depth set in kernel.

 

    1. chdev -P -l hdisk5 -a queue_depth=8

    2.  odmget CuAt | grep -p hdisk5 | less

 

    CuAt:

          name = "hdisk5"

          attribute = "queue_depth"

          value = "8"

          type = "R"

          generic = "UD"

          rep = "nr"

          nls_index = 30

 

    3.  stopwpar -N <wpar_name>

 

    The customer then will see that the performance

    is still the same. When a perfpmr is sent in

    from the global lpar you will see that the

    queue_depth is set to 1 for all of the disk

    exported to the wpar.

 

Local fix

 

    N/A

 

Problem summary

 

    When a  customer run the following in a version

    WPAR such as 5.2 or 5.3. The disk in the kernel

    will not have the queue depth set in kernel.

 

Fortunately, IBM support was able to provide us with a workaround. The first step was to add the missing vparent PdAt entry to the ODM in the Global environment.

 

# cat addodm_pdat_for_vparent.txt

PdAt:

      uniquetype = "wio/common/vparent"

      attribute = "naca_1_spt"

      deflt = "1"

      values = "1"

      width = ""

      type = "R"

      generic = ""

      rep = "n"

      nls_index = 0

 

# odmadd addodm_pdat_for_vparent.txt

 

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

        uniquetype = "wio/common/vparent"

        attribute = "naca_1_spt"

        deflt = "1"

        values = "1"

        width = ""

        type = "R"

        generic = ""

        rep = "n"

        nls_index = 0

 

We did the same in the VWPAR.

 

# clogin p8wpar1

# uname -W

11

 

# odmget PdAt | grep -p "wio/common/vparent"

#

# odmadd addodm_pdat_for_vparent.txt

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

        uniquetype = "wio/common/vparent"

        attribute = "naca_1_spt"

        deflt = "1"

        values = "1"

        width = ""

        type = "R"

        generic = ""

        rep = "n"

        nls_index = 0

 

In the VWPAR, we removed the hdisk and then discovered it again, ensuring that the queue depth attribute was set to 256 in the ODM.

 

# uname –W

11

# rmdev -dl hdisk1

hdisk1 deleted

# cfgmgr

# lspv

hdisk0          00f94f58a0b98ca2                    rootvg          active

hdisk1          none                                None

# lsattr -El hdisk1 –a queue_depth

queue_depth     256              Queue DEPTH                True

# odmget CuAt | grep -p queue

CuAt:

        name = "hdisk1"

        attribute = "queue_depth"

        value = "256"

        type = "R"

        generic = "UD"

        rep = "nr"

        nls_index = 12

 

Back in the Global environment we checked that the queue depth was set correctly in the kernel. And it was!

 

# uname -W

0

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

    ushort queue_depth   = 0x100;

 

We re-ran the simple I/O test and immediately found that the test ran faster and the hdisk queue (for hdisk3, as shown by iostat from the Global environment) was no longer full. Subsequent application load tests showed much better performance.

 

# time lmktemp Afile 1024M

Afile

 

real    0m3.15s <<<< BETTER!

user    0m0.03s

sys     0m1.60s

 

 

# iostat -DTRl 1

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:14

hdisk0           0.0   4.1K   1.0   4.1K   0.0    1.0   0.2    0.2    0.2     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:14

hdisk2           0.0  20.5K   5.0  20.5K   0.0    5.0   0.2    0.2    0.4     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   5.0  03:20:14

hdisk3          86.0 280.6M 2144.0   0.0  280.6M   0.0   0.0    0.0    0.0     0    0 2144.0   1.2    0.3    2.9     0    0   0.0    0.0    0.0    0.0   4.0   0.0  03:20:14

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:15

hdisk0           0.0   8.2K   2.0   0.0    8.2K   0.0   0.0    0.0    0.0     0    0   2.0   1.2    0.6    1.9     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:15

hdisk2           0.0   4.1K   1.0   0.0    4.1K   0.0   0.0    0.0    0.0     0    0   1.0   0.6    0.6    0.6     0    0   0.0    0.0    0.0    0.0   0.0   1.0  03:20:15

hdisk3         100.0 327.0M 2495.0   0.0  327.0M   0.0   0.0    0.0    0.0     0    0 2495.0   1.3    0.8   10.0     0    0   0.0    0.0    0.0    0.0   2.0   0.0  03:20:15

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:16

hdisk3         100.0 354.3M 2703.0   0.0  354.3M   0.0   0.0    0.0    0.0     0    0 2703.0   2.1    0.9   16.1     0    0   0.0    0.0    0.0    0.0   5.0   0.0  03:20:16

 

Disks:                     xfers                                read                                write                                  queue                    time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

                 %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv

                 act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull

hdisk1           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk0           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk2           0.0   0.0    0.0   0.0    0.0    0.0   0.0    0.0    0.0     0    0   0.0   0.0    0.0    0.0     0    0   0.0    0.0    0.0    0.0   0.0   0.0  03:20:17

hdisk3          32.0 111.7M 852.0   0.0  111.7M   0.0   0.0    0.0    0.0     0    0 852.0   1.3    0.9    2.8     0    0   0.0    0.0    0.0    0.0   1.0   0.0  03:20:17

 

 

Please Note: kdb will not work inside a WPAR. If you attempt to run it, you’ll receive the following error message. Run kdb from the Global environment instead.

 

# clogin p8wpar1

# kdb

The specified kernel file is a 64-bit kernel

open: A file or directory in the path name does not exist.

cannot open /dev/pmem