Chris's AIX Blog

Before I describe the sudo problem I hit, I will tell you that the root cause of the problem is evident in the following output:

 

# mount

  node       mounted        mounted over    vfs       date        options

-------- ---------------  ---------------  ------ ------------ ---------------

         /dev/hd4         /                jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd2         /usr             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd9var      /var             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd3         /tmp             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd11admin   /admin           jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/mksysblv    /export/mksysb   jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/fslv00      /export/nim      jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/hd1         /home            jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd10opt     /opt             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/swlv        /software        jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/fslv01      /tftpboot        jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/livedump    /var/adm/ras/livedump jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/fixelv      /fixes           jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/lpmautolv   /lpmauto         jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/promonlv    /promon          jfs2   Aug 14 12:00 rw,log=INLINE

         /aha             /aha             ahafs  Aug 14 12:00 rw

 

The output above is from the mount command on an AIX host.

 

Can you see any problems that might impact sudo's ability to run?

 

No?

 

Problem Description:

 

A customer was attempting to use sudo to run some commands on an AIX host. Each time, sudo would fail with the following error message:

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

The only information we had was that the host had been rebooted at some point in the last day or so.

 

We spent some time checking and verifying a bunch of things (specifically making sure that the /etc/sudoers file was intact and did not contain any bad entries) but none of them provided any real clues as to the root cause of the problem.

 

Problem Debugging and Investigation

 

We enabled debug for sudo. This involved creating an /etc/sudo.conf file with the following entries:

 

# vi sudo.conf

Debug sudo /var/log/sudo_debug.log all@debug

Debug sudoers.so /var/log/sudo_debug.log all@debug

 

Then, as the non-root user, we attempted to run sudo again. This generated some useful data in the /var/log/sudo_debug.log file. You'll notice we grep'ed for the string tty specifically, given that this was part of the sudo error message provided to us.

 

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

# grep tty /var/log/sudo_debug.log

Aug 10 16:20:26 sudo[14352830] -> get_process_ttyname @ ./ttyname.c:417

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 10 16:20:26 sudo[14352830] <- get_process_ttyname @ ./ttyname.c:444 := (null)

Aug 10 16:20:26 sudo[14352830] -> sudo_get_ttysize_v1 @ ./ttysize.c:59

Aug 10 16:20:26 sudo[14352830] -> get_ttysize_ioctl @ ./ttysize.c:38

Aug 10 16:20:26 sudo[14352830] <- get_ttysize_ioctl @ ./ttysize.c:44 := 0

Aug 10 16:20:26 sudo[14352830] <- sudo_get_ttysize_v1 @ ./ttysize.c:75

Aug 10 16:20:26 sudo[14352830] searching for tty time stamp record @ timestamp_lock() ./timestamp.c:603

Aug 10 16:20:26 sudo[14352830] appending new tty time stamp record @ timestamp_lock() ./timestamp.c:611

Aug 10 16:20:26 sudo[14352830] tty time stamp position is 112 @ timestamp_lock() ./timestamp.c:617

Aug 10 16:20:26 sudo[14352830] -> tty_present @ ./tgetpass.c:360

Aug 10 16:20:26 sudo[14352830] <- tty_present @ ./tgetpass.c:361 := false

Aug 10 16:20:26 sudo[14352830] no tty present and no askpass program specified @ tgetpass() ./tgetpass.c:107

 

The following information was of great interest to us! It appeared to be that some information was "missing" from the /proc file system on the host?

 

# grep tty /var/log/sudo_debug.log | grep unable

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 18 11:19:59 sudo[21889328] unable to resolve tty via /proc/21889328/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

 

Upon further investigation, we found that /proc did, in fact, appear to be completely empty!! This was not expected or normal on a modern AIX system.

 

# ls -ltr /proc

total 0

 

And, after running the mount command, we discovered that, yes, the /proc file system was NOT mounted!

 

# mount | grep proc

#

 

Resolution

 

Next, we attempted to mount /proc. But we couldn't, as it was not a known file system? Curious indeed.

 

# mount /proc

mount: 0506-334 /proc is not a known file system.

 

The /proc file system stanza was missing from /etc/filesystems. So, we added it back in and mounted the file system successfully.

 

 

# grep -p proc /etc/filesystems

#                                  << No /proc entry found in /etc/filesystems!

 

# grep -p proc /etc/filesystems

/proc:

        dev       = /proc

        vol       = "/proc"

        mount     = true

        check     = false

        free      = false

        vfs       = procfs

 

# mount /proc

# mount | grep proc

         /proc            /proc            procfs Aug 18 11:31 rw

             

# ls -ltr /proc | head -10

total 0

dr-xr-xr-x    8 root     system            0 Aug 10 16:21 sys

-r--r--r--    1 root     system           91 Aug 10 16:21 version

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 0

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 1

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 260

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 65798

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 131336

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 196874

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 262412

 

As soon as /proc was mounted again, the non-root user could run sudo without any issues.

 

$ sudo -l

Password:

User cgibson may run the following commands on 750lpar4:

    (ALL) /usr/sbin/slibclean

$

 

The /proc file system is different from other file systems. It does not reside on disk. The /proc file system resides in AIX memory and is referred to as a virtual file system. The files maintained in /proc represent the running processes on the system.

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012015

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.files/proc.htm

 

Using PowerSC RTC to monitor for the deletion of a specific file.

 

One of my customers was concerned that someone or something was randomly deleting the /etc/niminfo on several of his AIX hosts. We discussed several ways in which he could track down who or what was removing this file. Obviously, the AIX audit subsystem would be an ideal method of capturing this event. However, PowerSC Real Time Compliance (RTC) can also monitor and capture this type of event.

 

“The PowerSC™ Real Time Compliance feature continuously monitors enabled AIX® systems to ensure that they are configured consistently and securely.

 

The PowerSC Real Time Compliance feature works with the PowerSC Compliance Automation and AIX Security Expert policies to provide notification when compliance violations occur or when a monitored file is changed. When the security configuration policy of a system is violated, the PowerSC Real Time Compliance feature sends an email or a text message to alert the system administrator.”

 

Here’s a quick start guide on configuring PowerSC RTC on AIX.

 

1. PowerSC is included with the AIX Enterprise Edition offering. Download the PowerSC software from IBM Entitled Systems Support (ESS) website.

 

 

The package name is ESD_-_PowerSC_Standard_Edition_v1.1.5_122016.tar.gz.

 

2. Install the following filesets on the AIX host.

 

# lslpp -L powerscStd\*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  powerscStd.license         7.1.1.5    C     F    PowerSC Standard Edition

  powerscStd.rtc.rte         1.1.5.0    C     F    Real-Time Compliance

 

# oslevel -s

7100-04-04-1717

 

3. Configure RTC.

 

# smit RTC

>Configure Real-Time Compliance Subsystem

 

                  Configure Real-Time Compliance subsystem

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Email Address (comma separated)                    [root@lpar9]

  Alert Information Level                            [1]            +#

  Alert Style                                        [once]         +

  Alert Email Subject                                []

  Debug                                              [off]          +

 

Or you can use /usr/sbin/mkrtc.

 

# lssrc -s rtcd

Subsystem         Group            PID          Status

 rtcd                              11075602     active

 

4. Configure the /etc/security/rtc/rtcd.conf and rtcd_policy.conf files.

 

/etc/security/rtc/rtcd.conf:

 

infolevel: 3

alertStyle: event

locallogfile: /var/log/rtc.log

 

Add /etc/niminfo to rtcd_policy.conf file.

 

# tail -2 /etc/security/rtc/rtcd_policy.conf

 

/etc/niminfo:

        eventtype = modFile,modFileAttr

 

Restart RTC.

 

# stopsrc -s rtcd

 

# startsrc -s rtcd

 

5. Test RTC monitoring. e.g If someone deletes the niminfo file, a report is sent via email to root.

 

# rm /etc/niminfo

 

# mail

Mail [5.2 UCB] [AIX 5.X]  Type ? for help.

"/var/spool/mail/root": 2 messages 2 new

>N  1 RTC@lpar9.mel  Thu Aug  3 12:46  35/880  "PowerSC Real-Time Alert - 75"

?

Message  1:

From root Thu Aug  3 12:46:25 2017

Date: Thu, 3 Aug 2017 12:46:25 GMT

To: root@lpar9.meldemo.au.ibm.com

From: RTC@lpar9.meldemo.au.ibm.com

Subject: PowerSC Real-Time Alert - lpar9

 

The following event(s) has occurred:

BEGIN_EVENT_INFO

Hostname      : lpar9

Filename      : /etc/niminfo

Time          : Thu Aug  3 12:46:25 2017

Sequence Num  : 2

Process ID    : 15991252

User Info     : userName=root, loginName=root, groupName=system

Program Name  : rm

Event         : AHAFS_MODFILE_REMOVE

STACK_TRACE

[14D70]

ahafs_evprods+970

aha_process_vnop+194

vnop_remove+3A4

kunlinkat+530

.svc_instr

IPRA.$rm+800

IPRA.$rm_main+100

main+C4

__start+68

END_EVENT_INFO

 

The event information is also captured in the RTC log file (/var/log/rtc.log) on the host.

 

# grep -p REMOVE /var/log/rtc.log | head -50

The following event(s) has occurred:

BEGIN_EVENT_INFO

Hostname      : lpar9

Filename      : /etc/niminfo

Time          : Thu Aug  3 12:46:25 2017

Sequence Num  : 2

Process ID    : 15991252

User Info     : userName=root, loginName=root, groupName=system

Program Name  : rm

Event         : AHAFS_MODFILE_REMOVE

STACK_TRACE

[14D70]

ahafs_evprods+970

aha_process_vnop+194

vnop_remove+3A4

kunlinkat+530

.svc_instr

IPRA.$rm+800

IPRA.$rm_main+100

main+C4

__start+68

END_EVENT_INFO

 

 

IBM Knowledge Center PowerSC Real Time Compliance

https://www.ibm.com/support/knowledgecenter/en/SSTQK9_1.1.5/com.ibm.powersc.se/realtimecompliance.htm

 

 

The AIX minidump facility was introduced with AIX 5.3 TL3. A mini dump is a small compressed dump that is stored to NVRAM when a system crashes or a dump is initiated, and then written to the AIX error log on reboot. It can be used to see some of the system’s state and do some debugging when a full dump is not available. It can also be used to get a quick snapshot of a crash without having to transfer the entire dump from the crashed system to IBM support.

 

Please refer to the following, official guide, on "How to examine a minidump in AIX".

 

"Using this crash stack IBM support personnel can then search through the database to find what the fault may mean".

"The RAS effort mentioned..is part of an ongoing effort by AIX to increase stability and to make more information available for troubleshooting when a problem occurs. The ability to look at minidump data has helped solve many issues that would otherwise go unresolved".

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010709

 

Of course, minidumps have their limitations and do not replace the need for a full system dump in many cases.

 

"Limitations of minidumps
A minidump is of limited or no use in situations where a server has hung. In this situation we can use mdmprpt to see what was running on each cpu. Practically speaking a full dump is usually needed to determine how and why a server has hung."

 

Here are some examples of how I've used minidump to assist me (and IBM support) in diagnosing the root cause of a system crash.

 

The first example is from a customer that found that one of their AIX partitions would crash when they ran optmem DPO (Dynamic Platform Optimiser) against one of their new POWER8 E880s. Immediately after optmem ran, one specific LPAR would crash. The LPAR reference LED code would show "888 102 300 C20". This LPAR was installed with AIX 7.1 TL3 SP4. When we tried to restart the partition, it would crash (and dump) several times before it would start successfully. Once the partition booted successfully, we noticed that there were several system dumps (SYSDUMP) and minidumps (COMPRESSED MINIMAL DUMP) shown in the AIX error report.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

A6DF45AA   0411163117 I O RMCdaemon      The daemon is started.

2BFA76F6   0411163117 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   0411163117 T O errdemon       ERROR LOGGING TURNED ON

192AC071   0411162917 T O errdemon       ERROR LOGGING TURNED OFF

A6DF45AA   0411162517 I O RMCdaemon      The daemon is started.

2BFA76F6   0411162517 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   0411162517 T O errdemon       ERROR LOGGING TURNED ON

192AC071   0411162217 T O errdemon       ERROR LOGGING TURNED OFF

A6DF45AA   0411161117 I O RMCdaemon      The daemon is started.

67145A39   0411161017 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411161017 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411161017 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR

9DBCFDEE   0411161017 T O errdemon       ERROR LOGGING TURNED ON

A6DF45AA   0411155917 I O RMCdaemon      The daemon is started.

67145A39   0411155817 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411155817 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411155817 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR

9DBCFDEE   0411155817 T O errdemon       ERROR LOGGING TURNED ON

A6DF45AA   0411155717 I O RMCdaemon      The daemon is started.

67145A39   0411155717 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411155717 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411155717 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR            << CRASH!

9DBCFDEE   0411155717 T O errdemon       ERROR LOGGING TURNED ON

6AEB31F5   0410143317 I H sysplanar0     Platform Resource Reassignment.              << DPO

 

I used the minidump reporting tool (mdmprpt) to perform a quick analysis of the cause of the crash, by reviewing the "Stack Trace" information. The mdmprpt tool has a couple of options.

 

 

# mdmprpt -?

Usage:

mdmprpt [[-l seq_no] [-i filename]] | [ -b filename ] [ -F ] [-r]

 

Process error log entries from the supplied file(s).

-l seq_no Format the minidump at the specified sequence number in the error log.

-i filename  Uses the error log file specified by the filename param.

-b filename  Uses the binary file specified by the filename param.

-F prints only dump failure information.

-r print the raw minidump, without formatting.

 

If no parameters are specified, the most recently logged minidump is shown. You may not specify -b with -i or -l.

 

Typically, the most common usage is to simply run mdmprpt and redirect the output to a file for review.

 

# mdmprpt > /tmp/minidumpreport.out

 

The most relevant section of the report is the "Stack Trace" section(s).

 

# vi /tmp/minidumpreport.out

 

MINIDUMP VERSION 4D33

***************************************************

64-bit Kernel, 57 Entries

 

Last Error Log Entry:

         Error ID: 9D035E4D      Resource Name: SYSVMM

         Detail Data: 0000000000000000 00007FFFFFFFD080 00000007000000C7 0000000000000115

 

Symptom Information:

        Crash Location: [00000000000ADE98] trc_generate_ea+78

        Component: COMP  Exception Type: 277

 

Data From CPU #1 (Faulting CPU)

***************************************************

...

Stack Trace:

[00000000000ADE98] trc_generate_ea+78

        0000000000000002 F1000A002033C000 C0C0C0C0C0C0C0C1 8000002841900000

        00000000025BBC00 00000000000034C8 00000000009F3998 0000000003322000

[00000000000ACF64] trc_inmem_recor+144

        0000000000000000 F1000815B0052D00 0000000000007800 000000000000F000

 ...

 

In this case, the stack trace showed messages relating to trc_generate and trc_inmem_recor. Using this information, I was able to find several hits (both internally to IBM and using my preferred internet search tool) for the potential cause of the crash. The problem was a known issue and related to the following APAR.

 

IV69116: AIX CAN CRASH IBM POWER8 E880 (9119-MHE) WITH > THAN 128 CORES APPLIES TO AIX 7100-03

http://www-01.ibm.com/support/docview.wss?uid=isg1IV69116

 

 

We verified our findings with IBM support (who performed their own internal search) and we concluded that an update would be required. We chose to update the AIX system to AIX 7.1 TL4 SP3, and the problem went away.

 

In the next example, the customer had updated one of their AIX partitions to AIX 7.1 TL4 SP3 (previously on AIX 7.1 TL4 SP1). This LPAR also housed a single, AIX 5.3 versioned WPAR. The update was successful but whenever they ran the stopwpar command, the entire partition would crash (dump). So, once again, I employed the mdmprpt utiliy to check the stack traces.

 

MINIDUMP VERSION 4D33

***************************************************

64-bit Kernel, 48 Entries

 

Last Error Log Entry:

         Error ID: 9D035E4D      Resource Name: SYSVMM

         Detail Data: 000000000A000000 00000000C000C400 F1000000C01ED500 0000000000000086

 

Symptom Information:

        Crash Location: [F1000000C012B968] pcmUserGetDevIn+E8

        Component: COMP  Exception Type: 134

...

Stack Trace:

[F1000000C012B968] pcmUserGetDevIn+E8

       000000002FF44EAC 00000000DEADBEEF 00000000A28F47CB 0000000000000005

       F1000A000036A500 0000000000010000 F1000000C01D3000 000000002FEF56E0

[F1000000C01D1808] sddUserInterfac+588

       0000000000000002 00000000DEADBEEF 00000000A28F47CB 0000000000000088

...

 

The report showed us pcmUserGetDevIn and sddUserInterfac in the stack trace. Both are related to the IBM sddpcm device driver. They appeared to be the likely culprits. Searching for these presented us with a couple of potential reasons for the crash, such as too many paths configured for a sddpcm device (which we checked and confirmed was far less than the maximum of 32). So, my next question was, what version of sddpcm was installed? We found that the customer had (unexpectedly) two different versions of sddpcm installed. In the Global LPAR, the sddpcm version was 2.6.9.0 (the latest) and in the vWPAR, 2.6.6.0. During the TL update, the customer had updated sddpcm to the latest level available, but had forgotten to update sddpcm inside the vWPAR as well. We then discovered, that if they simply ran "pcmpath query device" from inside the vWPAR, this would crash the partition. The latest sddpcm readme file contained the following information "5557  Fix for AIX 7.1 crash during pcmpath query device"! This was indeed the cause of the system crash. After the customer updated sddpcm, inside the vWPAR, the system was stable and the problem was resolved.

 

If you encounter a system crash in the future, and there's minidump data available, why not consider using the minidump reporting tool to analyse the issue? It might just help speed up the root cause analysis of the problem.

 

 

 

The latest release of the VIOS provides a way to automatically backup your virtual device configuration whenever it changes. From the IBM Knowledge Center for POWER8:

 

https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_viosbr.htm

 

"The viosbr command automatically creates a backup, whenever there are any configuration changes. This functionality is known as the autoviosbr backup. It is triggered every hour, and checks if there are any configuration changes, or any other changes. If it detects any changes, a backup is created. Otherwise, no action is taken. The backup files resulting from the autoviosbr backup are located under the default path /home/padmin/cfgbackups with the names autoviosbr_SSP.<cluster_name>.tar.gz for cluster level and autoviosbr_<hostname>.tar.gz for node level. The cluster-level backup file is present only in the default path of the database node.

 

The -autobackup flag is provided for the autoviosbr backup functionality. By default, autoviosbr backup is enabled on the system. To disable the autoviosbr backup, use the stop parameter and to enable it you can use the start parameter. When the autoviosbr backupis disabled, no autoviosbr related tar.gz file is generated.

 

To check if the autoviosbr backup file, present in the default path is up to date, you can use the status parameter. To access the cluster-level backup file on any node of the cluster, use the save parameter. This action is necessary as the cluster-level backup file is present in the default path of the database node only.

 

If the node is a part of cluster, you can use the -type flag to specify the parameter. The parameter can be either cluster or node, depending on if it is a cluster-level or a node-level backup.

Flags""

 

On your VIOS, under oem_setup_env (as root), you'll find the following new entry in root's crontab:

 

# crontab -l | grep autoviosbr

0 * * * * /usr/ios/sbin/autoviosbr -start 1>/dev/null 2>/dev/null

 

This entry will check for any configuration changes and generate a new backup if necessary. Here's an example from my VIOS, running 2.2.5.10.

 

$ ioslevel

2.2.5.10

 

I create a new virtual optical device, which should trigger a new backup the next time the autoviosbr script runs (once per hour). Prior to creating the device, viosbr shows the autobackup status as Complete (no changes). I ensure that autobackup is configured by stopping and starting it with the viosbr command.

 

$ viosbr -autobackup stop -type node

Autobackup stopped successfully.

 

$ viosbr -autobackup start -type node

Autobackup started successfully.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

$ mkvdev -fbo -vadapter vhost34

vtopt13 Available

 

Immediately after the vtopt device is created, the autobackup status displays as Pending (something has changed but has not yet been backed up).

 

$ viosbr -autobackup status -type node

Node configuration changes:Pending.

 

The autoviosbr file is created in /home/padmin/cfgbackups.

 

$ r oem

oem_setup_env

 

# cd /home/padmin/cfgbackups

# ls -tlr

total 72

-rw-r--r--    1 root     staff         12189 May 10 14:00 autoviosbr_s824vio2.tar.gz

 

On the hour, the autoviosbr script runs, notices that the configuration has changed and generates a new viosbr backup file. The viosbr autobackup status changes to Complete.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

# ls -ltr

total 88

-rw-r--r--    1 root     staff         12218 May 10 15:00 autoviosbr_s824vio2.tar.gz

 

What's inside the tar file?

 

# cp /home/padmin/cfgbackups/autoviosbr_s824vio2.tar.gz /tmp/cg/

# cd /tmp/cg

# gzip -d autoviosbr_s824vio2.tar.gz

# tar -xvf autoviosbr_s824vio2.tar

x ArtexRules_MTM8286-42A02214F58VP2.xml, 20549 bytes, 41 media blocks.

x autoviosbr_s824vio2, 198554 bytes, 388 media blocks.

# ls -ltr

total 888

-rw-r--r--   1 root staff  198554 May 10 15:00 autoviosbr_s824vio2

-rw-r--r--   1 root staff  20549 May 10 15:00 ArtexRules_MTM8286-42A02214F58VP2.xml

-rw-r--r--   1 root staff  225280 May 10 15:00 autoviosbr_s824vio2.tar

 

The ArtexRules file contains device specific configuration attributes.

 

# head ArtexRules_MTM8286-42A02214F58VP2.xml

<?xml version="1.0" encoding="UTF-8"?>

<Profile origin="get" version="3.0.0" date="2017-05-10T04:10:54Z">

 <Catalog id="devParam.disk.fcp.mpioosdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioosdisk"/>

  </Parameter>

 </Catalog>

 <Catalog id="devParam.disk.fcp.mpioapdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioapdisk"/>

#

 

The autoviosbr file contains virtual device configuration information. In this file I can confirm that the configuration of my new virtual optical device (vtopt13) has been backed up.

 

# grep vtopt13 autoviosbr_s824vio2

                <name>vtopt13</name>

                        <vtdName>vtopt13</vtdName>

                        <value>vtopt13</value>

                                          

# vi autoviosbr_s824vio2

...

        <virtDevOPT>

                <name>vtopt13</name>

                <state>AVAILABLE</state>

                <locCode>U8286.42A.214F58V-V2-C101-L2</locCode>

                <unique_type>virtual_target/vtdev/fbopt</unique_type>

                <type>BDOPFBOPT</type>

                <fbOpt>

                </fbOpt>

                <CuAtcount>2</CuAtcount>

                <CuAt name="LogicalUnitAddr" value="0x8200000000000000" action="FALSE" />

                <CuAt name="udid" value="19cc3cb76ad94674c3" action="TRUE" />

        </virtDevOPT>

...

 

If you want to force a backup to run, before the next hour, manually run the following as root (oem_setup_env):

 

$ oem_setup_env

# /usr/ios/sbin/autoviosbr -start

 

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

I LPM'ed an AIX LPAR from Frame A to Frame B. The LPAR id before the LPM was 99 and after the LPM it's still 99. And the system id has changed, as expected. Good.

 

Before migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,017268B60 Hardware system identifier False

 

After migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,0157F5B16 Hardware system identifier False

 

Great, that's exactly what I wanted.

 

Hang on a minute! What kind of sorcery is this?!

 

root@lpar1 / # errpt | grep sys

60AFC9E5   0405132517 I O sys0           Partition ID changed since last boot.

 

root@lpar1 / # errpt -aN sys0

---------------------------------------------------------------------------

LABEL:          PARTID_CHANGE

IDENTIFIER:     60AFC9E5

 

Date/Time:       Wed Apr  5 13:25:53 EET 2017

Sequence Number: 81372

Machine Id:      000157F5B16

Node Id:         lpar1

Class:           O

Type:            INFO

WPAR:            Global

Resource Name:   sys0

 

Description

Partition ID changed since last boot.

 

Whaddaya mean my partiton ID changed since last boot?! It hasn't! I know this for a fact! Look at the uname output above!!

 

Well, I'm 100% correct, the partition id HAS NOT CHANGED! But the system id has....and this is the cause of the "informational" message in the AIX error report.

 

AIX uses the output from 'uname -f' to detect if the system AND/OR partition id have changed since last boot.

 

So, in my case, the partition id (99 dec/63 hex) did not change but (as you'd expect with LPM) the system id (shown in hex below) has changed, hence the message "Partition ID changed since last boot".

 

Before LPM.

 

root@lpar1 / # uname -f

80000BA02DC00063

 

After LPM.

 

root@lpar1 / # uname -f

800016B466F00063

 

From the uname man page:

 

-F Displays a system identification string comprised of hexadecimal characters. This identification string is the same for all partitions on a particular system.

                                                     

-f Similar to the F flag, except that the partition number is also used in the calculation of this string. The resulting identification string is unique for each partition on a particular system.

 

So, no problem here. Nothing to see, nothing to do. Move along! 

AIX has a new “critical volume group” capability which will monitor for the loss or failure of a volume group. You can apply this to any volume group, including rootvg. If applied to rootvg, then you can monitor for the loss of the root volume group.

 

This feature may be useful if your AIX LPAR experiences a loss of SAN connectivity e.g. total loss of access to SAN storage and/or all SAN switches. Typically, when this happens, AIX will continue to run, in memory for a period of time and will not immediately crash. Often you can still log on to the AIX system but if you attempt to write a file you’ll see in I/O error. But even then the system may (potentially) remain up. When the SAN issue is resolved the AIX system may continue running, with file systems in read-only mode (or not, it depends) but to really resolve the issue you would still need to reboot the AIX LPAR in order for it regain access to its disks. This can result in the need to run fsck against file systems. Note that the behaviour you encounter will be impacted by a variety of factors, such as length and type of outage. As always, your mileage may vary!

 

You can encounter this behaviour with both VSCSI and NPIV SAN booted LPARs. This new AIX VG option, which caters for the scenario described above, is not enabled by default. From the chvg man page:

 

“-r y | n                      Changes the critical volume group (VG) option of the volume group.

 

n                                  Disables the critical VG option of the volume group.

 

y                                  Enables the critical VG option of the volume group. If the volume group is set

to the critical VG, any I/O request failure starts the Logical Volume Manager

(LVM) metadata write operation to check the state of the disk before

returning the I/O failure. If the critical VG option is set to rootvg and if the

volume group losses access to quorum set of disks (or all disks if quorum

disabled), instead of moving the VG to an offline state, the node is crashed

and a message is displayed on the console.”

 

PowerHA also caters for and supports this now....and should already be enabled by default. You want this feature enabled for your HA clusters so that they respond appropriately to loss of the root volume group and initiate a failover.

 

smitty sysmirror -> Custom Cluster Configuration -> Events -> System Events -> Change/Show Event Response (smitty cm_change_show_sys_event)

 

                     Change/Show Event Response

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                        [Entry Fields]

* Event Name            ROOTVG                 +

 

* Response              Log event and reboot   +

* Active                Yes                    +

 

"Exploitation of LVM rootvg failure monitoring

 

AIX LVM has recently added the capability to change a volume group to be a known as critical volume group. Though PowerHA has allowed critical volume groups in the past, that

only applied to non-operating system/data volume groups. PowerHA v7.2 now also takes advantage of this functionality specifically for rootvg. If the volume group is set to the critical VG, any I/O request failure starts the Logical Volume Manager (LVM) metadata write operation to check the state of the disk before returning the I/O failure. If the critical VG option is set to rootvg and if the volume group losses access to quorum set of disks (or all disks if quorum is disabled), instead of moving the VG to an offline state, the node is crashed and a message is displayed on the console. You can set and validate rootvg as a critical volume group by executing the commands shown below. The command has to run once since we are using the CAA distributed command clcmd.

 

# clcmd chvg -r y rootvg

# clcmd lsvg rootvg |grep CRIT

DISK BLOCK SIZE: 512 CRITICAL VG: yes

DISK BLOCK SIZE: 512 CRITICAL VG: yes"

 

To test this new feature in my lab, I simulated a disk "failure" or accidental unmapping/removal of a rootvg disk from an LPAR.

 

On the AIX LPAR, prior to disk failure simulation, I turn on the “CRITICAL VG” option for rootvg.

 

# oslevel -s

7200-00-00-0000

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    no

 

# chvg -r y rootvg

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    yes

 

On the VIOS, I unmap the rootvg disk from the corresponding vhost adapter:

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

$ lu -list | grep volume-A | grep Lab

volume-AIX71LabImage-b~ 10240       7790        acc7e518f3f561d4719fcecfff88b0f1

volume-AIX71LabImage-b~ 1024        1021        81823bdb90537250b95b3b2f35bb55f9

 

$ lu -unmap -luudid acc7e518f3f561d4719fcecfff88b0f1

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

On the AIX LPAR, I attempt to create (write) a file in /tmp (which resides in rootvg):

 

# touch /tmp/mynewfile

 

The LPAR stops responding immediately. I can no longer connect or login to it.

 

I then remap the disk and restart the LPAR. The AIX error report shows that the system halted due a critical VG going offline.

 

$ lu -map -luudid acc7e518f3f561d4719fcecfff88b0f1 -vadapter vhost30

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

# errpt -a

...

LABEL:          KERNEL_PANIC

IDENTIFIER:     225E3B63

 

Date/Time:       Wed Mar  1 14:27:51 AEDT 2017

Sequence Number: 215

Machine Id:      00F94F584C00

Node Id:         aix72lab

Class:           S

Type:            TEMP

WPAR:            Global

Resource Name:   PANIC

 

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

 

        Recommended Actions

        PERFORM PROBLEM DETERMINATION PROCEDURES

 

Detail Data

ASSERT STRING

 

PANIC STRING

Critical VG Force off, halting.

 

This feature is available with AIX 6.1 or later.

 

IV52743: ADD CRITICAL VG SUPPORT FOR ROOTVG APPLIES TO AIX 7100-03

http://www-01.ibm.com/support/docview.wss?uid=isg1IV52743

 

Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.

 

[root@750lpar4]/ # nim -o lslpp 750lpar9

 

The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:

 

…

Thu Jan 12 14:31:49 2017        Loading certificates..

Thu Jan 12 14:31:49 2017        Loading private key file..

Thu Jan 12 14:31:49 2017        create BIO

…

 

NIM master: 750lpar4

7100-04-03-1543

 

NIM client: 750lpar9

7100-04-02-1614

 

[root@750lpar4]/ # lsnim -l 750lpar9

750lpar9:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar9 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

The root cause of the problem become apparent when we ran truss against the nim –o command.

 

[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9

 

[root@750lpar4]/ # cat truss.lsnim.out

…

13959372: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m   e r r o r :   N o   s u c h

13959372: f i l e   o r   d i r e c t o r y

…

19267612: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m   e r r o r :   N o   s u

19267612: c h   f i l e   o r   d i r e c t o r y

 

The required shared library object files were missing on the NIM master.

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found

 

We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.

 

[root@750lpar4]/usr/lib # slibclean

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x    1 root     system       724913 Jan 18 09:08 libssl.so

-rwxr-xr-x    1 root     system      3031337 Jan 18 09:08 libcrypto.so

 

After that, the nim –o commands started working again.

 

[root@750lpar4]/usr/lib # nim -o showlog 750lpar9

HELLO

 

So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run.  But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.

 

I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.

 

Here’s one good reason to setup NIMSH over SSL.

 

NIMSH, SSL and LPM

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en

 

The following link is a great reference guide for configuring NIMSH over SSL.

 

NIMSH over SSL

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024615

 

 

 

Recently I've come across an odd issue at two different customers. I thought I'd share the experience, in case others also come across this strange behaviour.

 

In both cases they reported j2pg high CPU usage.

 

Similar to this...

 

And, in both cases, we discovered many /usr/sbin/update processes running. Unexpectedly.

 

When we stopped these processes, j2pg's CPU consumption dropped to nominal levels.

 

The j2pg process is responsible for, among other things, flushing data to disk and is called by the sync(d) process.

 

The /usr/sbin/update command is just a script that calls sync in a loop. Its purpose it to "..periodically update the super block., ..execute a sync subroutine every 30 seconds. This action ensures the file system is up-to-date in the event of a system crash".

 

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds5/update.htm

 

# cat /usr/sbin/update

...BLAH...

PATH=/usr/bin:/usr/sbin:/etc

 

while true

do

sync

sleep 30

done &

exit 0

 

Because of the large number of /usr/sbin/update (sync) processes (in some cases over 150 of them), j2pg was constantly kept busy, assisting with flushing data to disk.

 

It appears that the application team (in both cases) was attempting to perform some sort of SQL "update" but due to an issue with their shell environment/PATH setting they were calling /usr/sbin/update instead of the intended update (or UPDATE) command. And yes, a non-root user can call /usr/sbin/update - no problem. So, in the "ps -ef" output we found processes that looked similar to this:

 

fred 50791260 1 0 Jan 09 - 0:04 /usr/bin/bsh /usr/sbin/update prd_ctl_q_and_p.af_data_obj set as_of_dt=2016-12-09 00:00:00.000000 where DATA_OBJ_NM like %LOANIQ%20161209%

 

# ls -ltr /usr/sbin/update

-r-xr-xr-x    1 bin      bin             943 Aug 17 2011  /usr/sbin/update

The application teams were directed to fix their scripts to prevent them calling /usr/sbin/update and instead call the correct command.

 

And here’s some information (more than you’ll probably ever need to know) about j2pg on AIX.

 

"j2pg - Kernel process integral to processing JFS2 I/O requests.

The kernel thread is responsible of managing I/Os in JFS2 filesystems,
so it is normal to see it running in case of lot of I/Os or syncd.

And we could see that j2pg runs syncHashList() very often.The sync is
done in syncHashList(). In syncHashList(), all inodes are extracted from hash
list. And whether the inode needs to synchronize or not is then judged
by iSyncNeeded().

** note that a sync() call will cause the system to scan *all* the
memory currently used for filecaching to see which pages are dirty
and have to be synced to disk

Therefore, the cause of j2pg having this spike is determined by the
two calls that were being made (iSyncNeeded ---> syncHashList). What is
going on here is a flush/sync of the JFS2 metadata to disk. Apparently
some program went recursively through the filesystem accessing files
forcing the inode access timestamp to change. These changes would have
to propogated to the disk.

Here's a few reasons why j2pg would be active and consume high CPU:
1. If there several process issuing sync then the j2pg process will be
very active using cpu resources.
2. If there is file system corruption then the j2pg will use more cpu
resources.
3. If the storage is not running data fast enough then the j2pg process
will be using high amount of cpu resources.

j2pg will get started for any JFS2 dir activity.
Another event that can cause j2pg activity, is syncd.
If the system experiences a lot of JFS2 dir activity, the j2pg process
will also be active handling the I/O.
Since syncd flushes I/O from real memory to disk, then any JFS2 dir's
with files in the buffer will also be hit."

"Checking the syncd...

From data, we see:
$ grep -c sync psb.elfk
351 << this is high
$ grep sync psb.elfk | grep -c oracle
348 << syncd called by Oracle user only

It appears that the number of sync which causes j2pg
to run is causing spikes.

We see:
/usr/sbin/syncd 60

J2pg is responsible for flushing data to disk and is
usually called by the syncd process. If you have a
large number of sync processes running on the system,
that would explain the high CPU for j2pg

The syncd setting determines the frequency with which
the I/O disk-write buffers are flushed.

The AIX default value for syncd as set in /sbin/rc.boot
is 60. It is recommended to change this value to 10.

This will cause the syncd process to run more often
and not allow the dirty file pages to accumulate,
so it runs more frequently but for shorter period of
time. If you wish to make this permanent then edit
the /sbin/rc.boot file and change to the 60 to 10.

You may consider mounting all of the
non-rootvg file systems with the 'noatime' option.
This can be done without any outage:

However selecting a non-peak production hours is better:

Use the commands below: For Example:
# mount -o remount,noatime /oracle
Then use chfs to make is persistent:
# chfs -a options=noatime /oracle

- noatime -
Turns off access-time updates. Using this option can
improve performance on file systems where a large
number of files are read frequently and seldom updated.
If you use the option, the last access time for a file
cannot be determined. If neither atime nor noatime is
specified, atime is the default value."

 

AIX 7.2 requires 2GB of memory to boot, but this minimum is not enforced in the LPAR profile except by Live Update (to ensure we'll be able to boot the surrogate LPAR). You can check your Minimum Memory setting in your LPARs profile by running the lparstat command (as shown below).

 

# lparstat -i | grep Memory
Online Memory                              : 4096 MB
Maximum Memory                             : 8192 MB
Minimum Memory                             : 2048 MB

 

If your partition does not meet the minimum memory profile requirement, you’ll receive the following error message when you perform a live update preview (with geninstall –p –k).

 

Checking lpar minimal memory size:

------------------------------------------

Required memory size: 2048 MB

Current memory size: 1024 MB

1430-119 FAILED: the lpar minimal memory size is not sufficient

 

You’ll need to change partition profile so that the minimum memory setting is, at least, 2048 (2GB) and then stop & start the partition for the profile update to take effect.

 

The /var/adm/ras/liveupdate/logs/lvupdlog log file will also contain error messages indicating the problem:

 

LVUPD 11/28/2016-18:52:00.716 DEBUG lvupdate_utils32.c - 6713 - lvup_check_minmem_size: Partition minimum memory size (1024 MB) on p8tlc-lvup is lower than the minimum memory size required (2048 MB).

 

LVUPD 11/28/2016-18:52:00.716 ERROR lvupdate_utils32.c - 8647 - lvup_preview: ERROR(s) while checking the current mimimal memory size against the computed required size.

 

 

 

 

Enabling TCP large send offload (LSO), on AIX, is generally considered a good thing. It helps with overall network performance.

 

Taking advantage of networking large-send large-receive

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Taking%20advantage%20of%20networking%20large-send%20large-receive

 

TCP large send offload

“This option improves the performance on 10 Gigabit Ethernet and faster adapters for workloads that manage data streaming (such as file transfer protocol (FTP), RCP, tape backup, and similar bulk data movement applications). The virtual Ethernet adapter and shared Ethernet adapter (SEA) devices are exceptions, where the large send offload option is disabled by default due to inter operability problems with the Linux or IBM i operating system. Enabling Large Send and other performance features can be done in AIX and virtual Ethernet adapter or SEA environments.”

 

https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.performance/tcp_large_send_offload.htm

 

So, is LSO enabled in my AIX partition?

 

No, it’s not!

 

# ifconfig en0

en0: flags=1e084863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.50.191 netmask 0xffffff00 broadcast 10.1.50.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

# netstat -p tcp | grep -i large

                0 large sends

                0 bytes sent using largesend

                0 bytes is the biggest largesend

 

Enable it now with chdev!

 

# chdev -l en0 -a mtu_bypass=on

en0 changed

 

# ifconfig en0

en0: flags=1e084863,4c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

        inet 10.1.50.191 netmask 0xffffff00 broadcast 10.1.50.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

I can now see large send packet activity.

 

# netstat -p tcp | grep -i large

                1205 large sends

        19687632 bytes sent using largesend

        17376 bytes is the biggest largesend

 

netstat -p protocol

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.performance/nestat_p_protocol.htm

 

You can use the HMC lssvcevents command to view DLPAR event history. In the following example, I’m looking for “Add processor” related DLPAR events, over the last 30 days, for a specific LPAR.

 

$ lssvcevents -t console -d 30 | egrep 'Add processor|DLPAR' | grep s82861p22_nim

 

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

 

Search for all DLPAR events over the last 30 days.

 

$ lssvcevents -t console -d 30 | grep -i dlpar

time=12/01/2016 00:52:27,text=HSCE2211 User name hscroot: DLPAR Remove processor resources from partition s82861p22_nim  succeeded on managed system Server-8286-42A-SN214F55V

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

 

HMC Manual Reference Pages  - LSSVCEVENTS (1)

http://www.ibm.com/support/knowledgecenter/TI0003N/p8edm/lssvcevents.html

 

Here’s some handy tips and tricks for managing WPARs in your AIX environment.

 

1. How to share a global file system with a WPAR.

 

- In the Global.

 

root@750lpar11]/ # uname -W

0

 

- Global File System that you want to share with the WPAR (750wpar2).

 

[root@750lpar11]/ # df -g /cg

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/cglv          1.98      0.30   85%       64     1% /cg

 

[root@750lpar11]/ # lswpar

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar1  A      S     750wpar1  /wpars/750wpar1  yes

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

- Create "namefs" file system from the Global. /cg file system will be mounted under /stuff in the WPAR.

 

[root@750lpar11]/ # crfs -v namefs -A yes -d /cg -m /wpars/750wpar2/stuff -u 750wpar2

 

[root@750lpar11]/ # lsfs /wpars/750wpar2/stuff

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/cg             --         /wpars/750wpar2/stuff  namefs --      --         no   no

 

root@750lpar11]/ # grep -p /wpars/750wpar2/stuff /etc/filesystems

/wpars/750wpar2/stuff:

        dev             = /cg

        vfs             = namefs

        mount           = true

        type            = 750wpar2

        account         = false

 

- Mount the file system, in the Global.

 

[root@750lpar11]/ # mount /wpars/750wpar2/stuff

 

root@750lpar11]/ # mount | grep stuff

         /cg              /wpars/750wpar2/stuff namefs Jun 23 10:27 rw

 

- In the WPAR.

 

[root@750wpar2]/stuff # uname -W

8

[root@750wpar2]/stuff # df -g /stuff

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

Global             1.98      0.30   85%       64     1% /stuff

 

2. Don't remove the ipv6 address from the global LPAR or else you may not be able to start your WPARs.

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6=''

lo0 changed

 

[root@gibopvc1]/ # stopwpar -Fv p8wpar1

Stopping workload partition 'p8wpar1'.

Stopping workload partition subsystem 'cor_p8wpar1'.

0513-044 The cor_p8wpar1 Subsystem was requested to stop.

Shutting down all workload partition processes.

WPAR='p8wpar1' CID=1

sysV mq ID=2097153 key=0x4107001c uid=0 gid=9

sysV sem ID=5242893 key=0x62023457 uid=0 gid=0

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1'.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

startwpar: 0960-231 ATTENTION: '/usr/lib/wpars/loadwpar' failed with return code 13.

startwpar: 0960-244 Error loading workload partition.

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1'.

Return Status = FAILURE.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6='::1'

lo0 changed

[root@gibopvc1]/ #

[root@gibopvc1]/ #

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

en1 net default: gateway 10.1.1.10

Exporting workload partition devices.

Exporting workload partition kernel extensions.

Starting workload partition subsystem 'cor_p8wpar1'.

0513-059 The cor_p8wpar1 Subsystem has been started. Subsystem PID is 9896106.

Verifying workload partition startup.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

3. Follow this procedure if you are unable to clogin to a WPAR because root logins are not allowed.

 

[root@750lpar11]/ GLOBAL # uname -W

0

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

Remote logins are not allowed for this account.

 

[root@750lpar11]/ GLOBAL # chroot /wpars/750wpar1 /usr/bin/ksh

750lpar11 : / # chuser rlogin=true root

750lpar11 : / # exit

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

750wpar1 : / # oslevel -s

5300-12-09-1341

750wpar1 : / # uname -W

4

 

4. How back up your WPAR to a NIM master.

 

Backing up WPAR to NIM master over NFS mounted file system.

--------------------------------------------------------------------------------------

 

[root@750lpar11]/ GLOBAL # mount nimmast:/export/images/wpars /mnt

[root@750lpar11]/ GLOBAL # savewpar -if /mnt/750wpar1-savewpar-image

Creating information file (/image.data) for 750wpar1-savewpar-image.

Creating list of files to back up

Backing up 92766 files......................

92766 of 92766 files backed up (100%)

0512-038 savewpar: Backup Completed Successfully.

 

############################################################################

 

Backing up WPAR with NIM.

--------------------------------------

On the NIM master, check the Global LPAR is already configured as a NIM client.

 

# nim -o showlog 750lpar11

BEGIN:Thu Jan 30 17:25:47 2014:013017254714

Command line is:

/usr/sbin/installp -acNgXY -e /var/adm/ras/nim.installp -f \

/tmp/.workdir.12583118.12255434_1/.genlib.installp.list.12255434-d \

/tmp/_nim_dir_11862198/mnt0

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

openssh.man.en_US           6.0.0.6100      USR         APPLY       SUCCESS

 

END:Thu Jan 30 17:25:51 2014:013017255114

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

Check that the WPAR hostname/IP can be resolved from the NIM master.

 

# host 750wpar2

750wpar2 is 10.1.50.44

 

# host 10.1.50.44

750wpar2 is 10.1.50.44

 

Define the WPAR in the NIM database. Where 750lpar11 is the Global partition NIM client name and 750wpar2 is the WPAR name.

 

# nim -o define -t wpar -a mgmt_profile1="750lpar11 750wpar2" -a if1="find_net 750wpar2 0" 750wpar2

 

Verify that the NIM master can now query the status of the WPAR.

 

# nim -o lswpar 750wpar2

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

   manages        = 750wpar2

 

# lsnim -l 750wpar2

750wpar2:

   class         = machines

   type          = wpar

   connect       = shell

   platform      = chrp

   if1           = 10_1_50 750wpar2 0

   mgmt_profile1 = 750lpar11 750wpar2

   Cstate        = ready for a NIM operation

   prev_state    = managed system defined but not yet created

   Mstate        = currently running

 

Backup the WPAR from the NIM master.

 

# nim -o define -t savewpar -a server=master -a location=/export/mksysb/cg/750wpar2-savewpar-image -a source=750wpar2 -a mk_image=yes 750wpar2-savewpar-image

 

Creating list of files to back up.

.

Backing up 64560 files..........

 

64560 of 64560 files (100%)

0512-038 savewpar: Backup Completed Successfully.

  

 

5. Moving a Rootvg Versioned Workload Partition (VWPAR) to Another Disk

 

"I tested in the lab, for a versioned wpar you can use it more like a regular LPAR - extend a disk into rootvg, mirror on it, unmirror from the old one, reduce out and rmdev.  No bosboot or bootlist needed.  There is no bootset in a versioned wpar for some reason.”

Example -  5.3 versioned wpar, rootvg with rootvg on hdisk11 from global env.

global environment:
 

add a new disk (hdisk10) to the vwpar:

# chwpar -D devname=hdisk10 rootvg=yes 53rvg

chwpar: 0960-721 The workload partition's rootvg must be extended with the new disk(s). Please run extendvg inside the workload partition prior to stopping or rebooting it.

In the versioned wpar:

# clogin 53rvg
# cfgmgr
# extendvg rootvg hdisk1
# mirrorvg rootvg hdisk1
# unmirrorvg rootvg hdisk0
# reducevg rootvg hdisk0
# rmdev -dl hdisk0
# exit (go back to global env)

Now in the global deallocate the old rootvg disk.

# chwpar -K -D devname=hdisk11 53rvg

Now you can shut down and start the wpar, it will boot from the other disk.

# stopwpar 53rvg
# startwpar 53rvg

Apparently bootset commands don't work with a versioned rootvg wpar:

# lswpar -Br 53rvg
Name   Device Name      Virtual Device  RootVG  Bootset
--------------------------------------------------------

# chwpar -B devname=hdisk10 rootvg=yes 53rvg
chwpar: 0960-808 Cannot add bootsets to versioned workload partitions.
 

 

6.Moving a Rootvg Workload Partition (WPAR) to Another Disk

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024257

 

Technote (FAQ)

 

Question

I have a rootvg WPAR that is on one disk, is there a method to move it to a new disk?

 

Answer

There may be an occasion where you have created a rootvg WPAR on a specific disk and you want to move the entire WPAR to another disk. One example might be that the original disk is from an older storage enclosure, and you wish to move the WPAR to newly purchased storage, connected to the system.

 

You can do this by means of an alternate bootset. Similar to how using the alt_disk_copy command in a global LPAR will create a copy of rootvg on another disk, an alternate bootset is a copy of a WPAR's rootvg on another disk.

 

The example in this technote will use a rootvg wpar that is on a single disk (hdisk11), and has private /opt and /usr filesystems (AKA a "detached" WPAR). This was initially created using these options:

# mkwpar -D devname=hdisk11 rootvg=yes -l -n rootvgwpar
# startwpar rootvgwpar 

1. List the bootsets for the rootvg WPAR (-Br) we are interested in:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk11      hdisk0      yes   0 

So we can see that hdisk11 is being used for the rootvg in the WPAR, as internal hdisk0.

2a. Now we allocate a new unused disk to the WPAR, and set a bootset on it in the same action:

# chwpar -B devname=hdisk9 rootvgwpar
Creating a bootset for WPAR rootvgwpar. Please wait... 

After this is finished we can see that a new bootset has been created on hdisk9:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk1      no   1
rootvgwpar  hdisk11      hdisk0      yes   0 

2b. If we log in to the WPAR we'll see a rootvg and an alternate rootvg:

# clogin rootvgwpar

# lspv
hdisk0     c7a96733e1ebfc      rootvg      active
hdisk1     c7a9673433f0a7      altvg.1 

3. From the global environment again, we can set the bootlist for the WPAR to boot from the bootset on the new disk:

# chwpar -b bootlist=1 rootvgwpar 
check that it was set properly:

# lswpar -b rootvgwpar
Name     Bootlist
---------------------
rootvgwpar ف 

4. Reboot the wpar on the new bootset.

# rebootwpar rootvgwpar

Stopping workload partition rootvgwpar.
Stopping workload partition subsystem cor_rootvgwpar.
0513-044 The cor_rootvgwpar Subsystem was requested to stop.
stopwpar: 0960-261 Waiting up to 600 seconds for workload partition to halt.
Shutting down all workload partition processes.
Unmounting all workload partition file systems.

Starting workload partition rootvgwpar.
Mounting all workload partition file systems.
Loading workload partition.
Exporting workload partition devices.
Exporting workload partition kernel extensions.
Starting workload partition subsystem cor_rootvgwpar.
0513-059 The cor_rootvgwpar Subsystem has been started. Subsystem PID is 8323226.
Verifying workload partition startup.

If we look at the bootsets we'll see that the 2nd disk now has rootvg officially on it:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes  1
rootvgwpar  hdisk11      hdisk1      no   ـ 

That bootset also believes hdisk9 is "hdisk0" for it, and the other disk is hdisk1. Notice the bootset ID has not changed, bootset 0 is still on (global) disk hdisk11 and bootset 1 on (global) disk hdisk9.

If we log into the WPAR we can see this:

# lspv
hdisk0     c7a9673433f0a7      rootvg      active
hdisk1     c7a96733e1ebfc      altvg.0 

The volume group "altvg.0" is our original rootvg.

5. At this point if everything is looking good, we can remove the original bootset from the global environment:

# chwpar -K -B bootset=0 rootvgwpar

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes   1

# lswpar -D rootvgwpar | grep disk
rootvgwpar  disk   hdisk9      yes   EXPORTED
rootvgwpar  disk   hdisk11      no    EXPORTED

6. Now we can remove the original rootvg disk:

# chwpar -K -D devname=hdisk11 rootvgwpar 

7. Remember to run cfgmgr to put the disk in an Available state again so it can be used by another WPAR or volume group:

# lsdev -xc disk -l hdisk11
hdisk11 Defined 00-01-02 MPIO IBM 2076 FC Disk

# cfgmgr

# lsdev -xc disk -l hdisk11
hdisk11 Available 00-01-02 MPIO IBM 2076 FC Disk

 

 

 

 

When troubleshooting Fibre Channel adapter issues on AIX (or VIOS), I often use the fcstat command to assist me in the process. The command may display error numbers similar to the following:

 

$ fcstat fcs0

 

Error opening device: /dev/fscsi0

errno: 00000045

 

The error number (errno) displayed can (in some cases) be used to identify the root cause of a problem. In the example above, the error number is 45. This number is in hex. If we convert this to decimal, the number is 69. Now, if we look for 69 in the /usr/include/errno.h file (on AIX), we discover that this error number relates to a "Network is down" event.

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

 

# echo "ibase=16; 45"|bc

69

 

# grep 69 /usr/include/errno.h

#define ENETDOWN        69      /* Network is down */

 

The AIX error report (errpt or errlog on VIOS) also tells me that there's some type of link error. This helps me focus my investigation toward the most likely problem area. In this case, I suspect either a physical link or cable problem between my FC adapter and the SAN switch.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

7BFEEA1F   1116121816 T H fcs0           LINK ERROR

 

# errpt -aN fcs0 | grep -p Desc

...

Description

LINK ERROR

 

We can use the same process for other errors displayed by the fcstat command. For example:

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000046

 

# echo "ibase=16; 46"|bc

70

 

# grep 70 /usr/include/errno.h

#define ENETUNREACH     70      /* Network is unreachable */

 

This particular error may direct us to the solution outlined in the following tech note:

 

Fibre channel workspace takes over 2 minutes to load but returns no data

http://www-01.ibm.com/support/docview.wss?uid=swg21600064

 

 

fcstat Command (AIX 6.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.2)

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds2/fcstat.htm

 

 

 

There may be better ways to do this, and if there are, please let me know. But lately I've been "hacking around" with cloud-init on AIX and trying to make it behave the way I want it to. There were two problems I faced and solved.

 

My first challenge. The AIX /etc/hosts file isn't updated with the IP address and hostname of the new AIX VM after deployment from PowerVC.

To work-around this niggle*, I added the following short but effective shell script to the Activation Input in the PowerVC GUI. 

 

#!/bin/ksh
myip=`lsattr -El en0 -a netaddr | awk '{print $2}'`
myname=`hostname`
echo Adding $myip $myname to /etc/hosts >> /tmp/mycloud.log
/usr/bin/hostent -a $myip -h $myname
date >> /tmp/mycloud.log

 

 

 

It's ugly but hey, it does the job!

 

And the second niggle. In my customers lab environment, where there's no DNS at all. Everything is /etc/hosts only. Every time they deployed an AIX VM, there was a significant delay for ssh sessions, even with netsvc.conf set to hosts=local4, the dodgy**, default, resolv.conf (search localdomain) forced DNS first....and ssh connections would hang for a minute or so before a login prompt appeared....waiting on name resolution to complete.

 

Some people told me that I could change "manage_resolv_conf" to false in my cloud-init config file and this would prevent the resolv.conf file from being managed (over-written) by cloud-init. But changing that option did nothing. And I really didn't want a resolv.conf file at all anyway!

 

What I really wanted was for cloud-init to deploy the AIX VM and to NOT create an /etc/resolv.conf file. But how? Well, I managed to fudge it***. I made a change to the aix.py python script. With this change, the script now writes out an /etc/resolv.conf.cloud file instead. This works OK.

 

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cat /etc/resolv.conf.cloud
search localdomain

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # mkdir cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cd cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg/
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp ../aix.py .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # vi aix.py

 

Change    resolve_conf_fn = "/etc/resolv.conf"
To        resolve_conf_fn = "/etc/resolv.conf.cloud"

 

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # diff aix.py /tmp/cg/aix.py
35c35
<     resolve_conf_fn = "/etc/resolv.conf.cloud"
---
>     resolve_conf_fn = "/etc/resolv.conf"

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # python -m compileall .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp -p * ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cd ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # rm -r cg

 

Now the customer can connect to new AIX VMs immediately after deployment.

 

# ls -ltr /etc/resolv.conf.cloud
-rw-r--r--    1 root     system           19 Aug 03 12:47 /etc/resolv.conf.cloud

 

It's a hack but it was fun. 

 

And I guess the next time I install the latest release of cloud-init for AIX, I'll need to modify the script again. But I'm OK with this, as I expect the newer release may actually provide me with a fix to each of the problems I faced.

 

*niggle: To cause slight but persistent annoyance, discomfort, or anxiety.

**dodgy: dishonest or unreliable, of low quality.

***fudge it: put together clumsily. Perhaps an alteration of fadge "make suit, fit".

 

My colleague in Lab Services, Ben Swinney, recently configured the Virtual HMC on his Apple Macbook. He was kind enough to write up his experience, so that others could also run the vHMC on their Mac. Me included! 

You can find Ben’s instructions here:

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/vHMC_running_on_Apple_hardware.pdf

Thanks Ben!

 

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

 

"Hello Chris,
 
So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.
 
So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup
 
# lsnim -l netbackup
netbackup:
   class       = resources
   type        = file_res
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /tmp/netbackup
   alloc_count = 0
   server      = master
   dest_dir    = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on. 

Can I actually push files out by themselves? Thanks"

 

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

 

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value     Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value     Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

 

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

 

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm
 
Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really! 

 

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

 

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

First, I'll create a new file_res NIM resource, called soe_conf.

 

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf     resources       file_res

# lsnim -l soe_conf
soe_conf:
   class       = resources
   type        = file_res
   dest_dir    = /usr/local/etc
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /usr/local/etc
   alloc_count = 0
   server      = master

 

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   file_res       = soe_conf
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = not running
   Mstate         = currently running
   cpuid          = 00F94F584C00
   control        = master

 

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

 

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

 

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

 

root@MASTER /# nim -o cust AIXmig

 

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

The file_res resource, is automatically deallocated from the NIM client.

 

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = customization is being performed
   Mstate         = currently running
   cpuid          = 00F94F584C00
   Cstate_result  = success
     
If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable). 
   
  
   

Do you use SSL with nimsh on AIX? No? Well, you might want to consider it. If you regularly use LPM to migrate AIX partitions from one server to another, you may have found that, on occasion, your NIM master has trouble communicating with its NIM clients afterwards. This is by design, as nimsh uses the NIM clients cpuid to authenticate with the NIM master. During an LPM operation, the cpuid of the NIM client changes and its possible the NIM master may reject the client as a result. This problem can occur even when CPU validation is disabled on the NIM master.

 

In the example below, I’ve LPM’ed a NIM client (750lpar1) to another server. Immediately afterwards, I’m able to execute a NIM command against the NIM client, from the NIM master (750lpar4). At this point the NIM client is configured with standard nimsh authentication i.e. no SSL.

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@750lpar1]/var/adm/ras# echo HELLO > /var/adm/ras/nim.installp

 

NIM MASTER:

[root@750lpar4]/ # nim -o change -a validate_cpuid=no master

[root@750lpar4]/ # lsnim -l master | grep -i cpu

   validate_cpuid      = no

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00                << Notice that the cpuid is different.

 

[root@750lpar4]/ # nim -o showlog 750lpar1

HELLO

 

The cpuid is cached by the nimsh daemon, so the previous system id is retained in memory and passed to the NIM master, which allows the operation to complete successfully. But, if I restart the nimsh daemon, on the NIM client, I find that the NIM master is no longer able to communicate with the client.

 

[root@750lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@750lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160610.

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

0042-001 nim: processing error encountered on "master":

   0042-006 m_mkbosi: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar4]/ # nim -o showlog 750lpar1

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied - 750lpar4

 

[root@750lpar1]/var/adm/ras# grep match nimsh.log | grep CD

Wed Apr 27 20:51:12 2016        error: local value passed, '00F603CD4C00', does not match environment value '00F627664C00'

 

One way to work around this problem is to use the procedure outlined at the following link:

 

http://www.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.osdevice/lpm_nim.htm

 

“Migrating a NIM client by using LPM

 

When Live Partition Mobility (LPM) is used to move a machine from one physical server to another and the machine is defined as a Network Installation Management (NIM) client, the NIM administrator must update the cpuid attribute for the NIM client to reflect the new hardware value after the LPM migration completes. To update the cpuid attribute, complete the following steps:

 

On the NIM client, acquire the new cpuid ID by running the following command:

 

uname –a

 

On the NIM master, run the following command:

 

nim -o change -a cpuid=cpuid client”

 

However, there is a better way. Using nimsh, with SSL-enabled authentication, will prevent the checking of cpuid during nimsh service handling.  This is considered the recommended choice of operation (since the client/server can agree upon identity using the certificate information passed during the ssl handshake). Once the certificate is in place, the NIM master will disregard any cpuid validation and instead rely on the success of the SSL-handshake. This configuration will work well with LPM. If using standard nimsh, the limitation of cpuid updating would still apply (because NIM has no way of automatically updating a client once the value has changed).

 

Both the NIM master and client must be configured to support SSL-enabled authentication. To configure SSL-enabled authentication, we can use ‘nimconfig -c’ on the NIM master.

 

[root@750lpar4]/ # oslevel -s

7100-04-01-1543

 

[root@750lpar4]/ # nimconfig -c

0513-029 The tftpd Subsystem is already active.

Multiple instances are not supported.

NIM_MASTER_HOSTNAME=750lpar4

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Target "all" is up to date.

Generating a 1024 bit RSA private key

.................++++++

....................................++++++

writing new private key to '/ssl_nimsh/keys/rootkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

Getting Private key

Generating a 1024 bit RSA private key

...++++++

....++++++

writing new private key to '/ssl_nimsh/keys/clientkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

Generating a 1024 bit RSA private key

.....................++++++

..........++++++

writing new private key to '/ssl_nimsh/keys/serverkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

 

[root@750lpar4]/ # lsnim -l | grep -i ssl

   ssl_support         = yes

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /ssl_nimsh/certs/client.pem

certname= /ssl_nimsh/certs/client.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

certname= /tftpboot/server.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

 

To enable SSL nimsh on the NIM client, we can use the ‘nimclient –c’ command. You can fall-back to non-SSL nimsh, with ‘nimclient –C’.

 

[root@750lpar1]/# oslevel -s

7100-04-01-1543

 

[root@750lpar1]/var/adm/ras# nimclient -c

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Received 2784 Bytes in 0.0 Seconds

0513-044 The nimsh Subsystem was requested to stop.

0513-077 Subsystem has been changed.

0513-059 The nimsh Subsystem has been started. Subsystem PID is 14942422.

 

[root@750lpar1]/var/adm/ras# cat nimsh.log

Fri Apr 29 23:10:45 2016        /usr/sbin/nimsh: NIM Service Handler started from SRC

Fri Apr 29 23:10:45 2016        no environment value for NIM_SECONDARY_PORT

Fri Apr 29 23:10:45 2016        value for hostname is 750lpar1

Fri Apr 29 23:10:45 2016        value for netaddr is: 10.1.50.31

Fri Apr 29 23:10:45 2016        value for route is net,-hopcount,0,,0 and gateway is 10.1.50.1

Fri Apr 29 23:10:45 2016        value for netif is en0

Fri Apr 29 23:10:45 2016        value for netmask is 255.255.255.0

Fri Apr 29 23:10:45 2016        obtained master's hostname: NIM_MASTER_HOSTNAME=750lpar4

 

Fri Apr 29 23:10:45 2016        obtained master's id: NIM_MASTERID=00F603CD4C00

 

Fri Apr 29 23:10:45 2016        value for machine id is 00F603CD4C00

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libssl.so.0.9.8 from /usr/lib/libssl.a ..

Fri Apr 29 23:10:45 2016        Refreshing archive member /usr/lib/libcrypto.so.0.9.8 from /usr/lib/libcrypto.a ..

 

Now I can LPM the NIM client to another server and even if I restart nimsh, the master can still communicate with the client. All thanks to SSL.

 

NIM MASTER:

 

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar1 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

 

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

 

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

 

[root@lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

[root@lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160616.

 

[root@750lpar1]/var/adm/ras# tail -f nimsh.log

Fri Apr 29 23:13:33 2016        passing OpenSSL setting of 1

Fri Apr 29 23:13:33 2016        set symbol table

Fri Apr 29 23:13:33 2016         cert filename discovered: /ssl_nimsh/certs/750lpar4.0

Fri Apr 29 23:13:33 2016        ** OpenSSL FIPS mode enabled successfully

Fri Apr 29 23:13:33 2016        seed_prng

Fri Apr 29 23:13:33 2016        Loading certificates..

Fri Apr 29 23:13:33 2016        Loading private key file..

Fri Apr 29 23:13:33 2016        create BIO

Fri Apr 29 23:13:33 2016        - SSL Connection verified successfully -

Fri Apr 29 23:13:33 2016        sending ack to client

Fri Apr 29 23:13:33 2016        setting descriptors to include 2nd port

Fri Apr 29 23:13:33 2016        command to exec  __  /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_showlog" "-alocation=/var/adm/ras/nim.installp"

 

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

 

+---------------------------------------------------------------------+

                System Backup Image Space Information

              (Sizes are displayed in 1024-byte blocks.)

+---------------------------------------------------------------------+

 

Required = 9457741 (9237 MB)    Available = 51653640 (50444 MB)

 

Creating information file (/image.data) for rootvg.

..etc..

 

 

 

Using NIM to install clients configured with SSL authentication

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_install_ssl.htm

 

Using the certificate viewing file

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/sample_files_certificate.htm%23sample_files_certificate?lang=en

 

 

 

 

 

Is that a versioned WPAR? Use the lswpar command and the –G option to determine the WPAR type. Yeah, I could use clogin and then run oslevel, but this is faster!

 

[root@gibopvc1]/ # lswpar -G p8wpar1

=================================================================

p8wpar1 - Active

=================================================================

Type:                    S

RootVG WPAR:             yes

Owner:                   root

Hostname:                p8wpar1

WPAR-Specific Routing:   no

Virtual IP WPAR:

Directory:               /wpars/p8wpar1

Start/Stop Script:

Auto Start:              no

Private /usr:            yes

Checkpointable:          no

Application:

 

OStype:                  1024

Cross-WPAR IPC:          yes

Architecture:            none

UUID:                    418c1da6-bceb-4b4e-b872-b1c6701da0b6

 

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

 

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

 

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

   0042-061 m_rmpdir: the "liveupdaterte" resource is currently

        allocated for client use

 

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 1

   server      = master

 

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

 

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

 

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

 

# lsnim -Fl liveupdaterte

liveupdaterte:

   id          = 1447111715

   class       = resources

   type        = lpp_source

   comments    = LIVE

   arch        = power

   Rstate      = ready for use

   prev_state  = verification is being performed

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# nim -o remove liveupdaterte

#

 

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

 

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

 

 

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

 

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

 

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. lparaix1nim.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo deallocate -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015).  This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs.  Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

 

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

 

 Major Advantages of using HTTP during NIM Management:

 

• All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
• Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
• Easy to consume by NIM or other products that currently use the client/server model of NFS.
• Able to extend the end-product to support additional protocols (context driven).”

 

 

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources.  The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901.  When active, NIM clients attempt file access and/or scripting customization requests from nimhttp.  If http access fails or is denied, a failover attempt at NFS client access occurs.  Future support will include options to remove NFS client attempts altogether.”

 

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user.  First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

 

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/.  This path location is defined as the document_root and cannot be modified at this time.  Future enhancements will support multiple document_root paths.  The document root path is not limited in depth and may contain many sub-directories.  Client requests are able to traverse the path setting by using the enable_directory_listing option.  If set to “no”, all files being served must reside in the current working directory of document_root.”

 

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable.  To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server.  The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM.  Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

 

# nimconfig –c”

 

I tested this functionality during the AIX 7.2 Early Ship Program.

 

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

 

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

 

NIM MASTER:

 

# lslpp -l | grep nim

  bos.sysmgt.nim.master      7.2.0.0  COMMITTED  Network Install Manager -

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager –

 

NIM CLIENT:

 

# lssrc -s nimsh

Subsystem         Group            PID          Status

 nimsh            nimclient        6554064      active

 

# lslpp -l | grep nim

  bos.sysmgt.nim.client      7.2.0.0  COMMITTED  Network Install Manager -

 

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

 

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

 

# lssrc -s nimhttp

Subsystem         Group            PID          Status

 nimhttp                           6685178      active

 

# ps -ef | grep nimhttp

    root  6685178  4194712   0   Nov 10      -  0:00 /usr/sbin/nimhttpd –v

 

# ls -ltr /httpd.conf

-rw-r--r--    1 root     system         1159 Nov 05 15:31 /httpd.conf

 

 

# cat /httpd.conf

 

#

#---------------------

# http service defines

#---------------------

#

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

#

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

#

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

#

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#

#

#---------------------

# http configuration

#---------------------

#

document_root=/export/nim/

# Designates the directory to serve files from.

#

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

#

enable_proxy=no

# Enable the web service to act as a proxy server.

#

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

#

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

#

 

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdaterte               resources       lpp_source

 

# lsnim -l liveupdaterte

liveu:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /export/nim/cglpp

   alloc_count = 0

   server      = master

 

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r--    1 root     system      2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r--    1 root     system          688 Oct 31 09:33 .toc

 

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

 

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

 

# lslpp -l bos.liveupdate.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

Path: /etc/objrepos

  bos.liveupdate.rte         7.2.0.0  COMMITTED  Live Update Runtime

 

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

 

# tail -f nimhttp.log

 

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015        socks: Copied string

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015        Request Type is

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015        file descriptor is : 6

Mon Nov 16 13:51:23 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015        socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015        socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015        socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:23 2015        nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:23 2015        Request Type is GET

Mon Nov 16 13:51:23 2015        Mime not supported

Mon Nov 16 13:51:23 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015        Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015        Total length sent is 509

Mon Nov 16 13:51:23 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015        Total length sent is 72

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015        Total length sent is 755

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015        socks: Copied string

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015        Request Type is

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015        nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015        nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015        file descriptor is : 6

Mon Nov 16 13:51:24 2015        family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015        source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015        socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015        socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015        socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

 

Mon Nov 16 13:51:24 2015        nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

 

"

Mon Nov 16 13:51:24 2015        Request Type is GET

Mon Nov 16 13:51:24 2015        Mime not supported

Mon Nov 16 13:51:24 2015        Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015        Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015        socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015        Total length sent is 2574336

Mon Nov 16 13:51:24 2015        handle_httpGET: Entering cleanup statement

 

 

“Example:  Commands supporting nimhttp

 

If using push operation (From NIM master)

 

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

 

If using pull operation (From NIM client)

 

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

I enjoy it when I open my email in the morning and find a new message with a subject line of “weird one….”! I immediately prepare myself for whatever challenge awaits. Fortunately I do delight in helping others with their AIX challenges so I usually open these emails first and start to diagnose and troubleshoot the problem!

 

This week I was contacted by someone that was having a little trouble with a mksysb backup on one of their AIX systems.

 

“Hi Chris,

This one has me stumped, any ideas? I’ll have to log a call I think as I’m not sure why this is happening. I run a mksysb and it just backs up 4 files! I also can’t do an alt_disk_copy that also fails.

My /etc/exclude.rootvg is empty.

# cat /etc/exclude.rootvg
# mksysb -i /mksysb/aixlpar1-mksysb

Creating information file (/image.data) for rootvg.

Creating list of files to back up.

Backing up 4 files

4 of 4 files (100%)
0512-038 mksysb: Backup Completed Successfully.

# lsmksysb -f /mksysb/aixlpar1-mksysb
New volume on /mksysb/aixlpar1-mksysb:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Wed Oct 21 22:12:04 EST 2015
Files are backed up by name.
The user is root.
5911 ./bosinst.data
11 ./tmp/vgdata/rootvg/image.info
11837 ./image.data
270567 ./tmp/vgdata/rootvg/backup.data
The total size is 288326 bytes.
The number of archived files is 4.”
 

Yep, that is a weird one! J I replied with the following tips.

 

“Hi,

A couple of things to check and try.....

1. Check the mount command is OK. i.e.

# ls -ltr /usr/sbin/mount
-r-sr-xr-x 3 root system 67040 Aug 18 15:52 /usr/sbin/mount << Not zero size?

# mount

2. Check the find command is OK i.e.

# ls -ltr /usr/bin/find
-r-xr-xr-x 1 bin bin 58608 Aug 18 15:44 /usr/bin/find << Not symlink'ed?

3. Delete the /etc/exclude.rootvg file. Try mksysb again. Does it work?

4. Riun mkszfile and check image.data. Are fs data stanzas listed for each file system?

5. Try with verbose and debug enabled on mksysb/mkszfile commands.

# cd /usr/bin
# cp -p mksysb mksysb.orig
# cp -p mkszfile mkszfile.orig

# vi mksysb

; search for main

#################### main ##########################

#
# set up environment
#
PATH=/usr/bin:/usr/sbin:/sbin:/etc: ; export PATH
export ODMDIR="/etc/objrepos"
NAME=`/usr/bin/basename $0`
PLATFORM=`/usr/sbin/bootinfo -a` # Needed for IA64 (value 4)

Add the following:

#################### main ##########################

set -x
for F in $(typeset +f); do typeset -ft $F; done

#
# set up environment
#

; Do the same (above) for mkszfile.

; Save the files and run mkszfile and mksysb with script (to capture the output).

# script /tmp/mksysb.out
# mkszfile
# mksysb -i -v /mksysb/aixlpar-mksysbimg
# exit”
 

A short time later I received another email stating that the problem had been resolved! Curiously, the find command had been (somehow) destroyed on this system.

 

“LLLLLLLLLLLEEEEEEEEEEEGGGGGGGGGGGGEEEEEEEEEEENNNNNNNNNNNDDDDDDDDDDDDDDDDD!!!!!!!!!!!!!!!!!!!!!!!!

 

/usr/bin/find was a zero size file!!! WTF!

 

I just copied the file from another LPAR and it worked perfect.

 

THANKS!!!!”

 

I thought I’d share this information here, just in case anyone else comes across a similar problem in the future.

In my previous post on AIX Live Updates I discussed how to use the geninstall command to perform a non-disruptive (ifix) update on an AIX system. In this post I wanted to show you how to perform the same task using NIM.

 

NIM can be used to start an AIX Live Update operation on a target machine (NIM client) either from a NIM master or from the NIM client itself (with nimclient).

 

Note: The AIX Live Update operation started by NIM calls the hmcauth command during the cust operation to authenticate to the NIM client with the HMC by using the HMC passwd file. The NIM master is responsible for obtaining password information from the HMC (using ssh).  Without it, NIM clients will not have the password information necessary when running hmcauth as part of the NIM client operation.  So, we must first define an hmc object in NIM and create the password file (used when accessing the console.)  Once this required step has been completed, all clients using NIM live_update have the ability to pass the proper hmc login credentials when configuring 'hmcauth'.

 

First, I need to install the dsm.core fileset and configure SSH keys between the NIM master and the HMC.

 

# lslpp -l dsm.core

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  dsm.core                  7.2.0.0  COMMITTED  Distributed Systems Management

                                               Core

 

# dpasswd -f /etc/ibm/sysmgt/dsm/config/my_password_file -U hscroot

 

# dkeyexch -f /etc/ibm/sysmgt/dsm/config/my_password_file -I hmc -H hsc02

 

Then define a HMC and Managed System NIM object on the NIM master.

 

# nim -o define -t hmc -a if1="find_net hsc02 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/my_password_file hsc02

 

# nim -o define -t cec -a hw_type=8286 -a hw_model=42A -a hw_serial=214F58V -a mgmt_source=hsc02 S824

 

The NIM client must either be defined with or updated to include the Managed System name (Management Source) and LPAR id number.

 

# smit nim_chmac

 

               Change/Show Characteristics of a Machine

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                  [Entry Fields]

  Machine Name                                       [AIXmig]

* Hardware Platform Type                             [chrp]                +

* Kernel to use for Network Boot                     [64]                  +

  Machine Type                                        standalone

  Network Install Machine State                       currently running

  Network Install Control State                       ready for a NIM operation

  Primary Network Install Interface

    Network Name                                      net1

    Host Name                                        [AIXmig]

    Network Adapter Hardware Address                 [0]

    Network Adapter Logical Device Name              [ent]

    Cable Type                                        N/A                  +

    Network Speed Setting                            []                    +

    Network Duplex Setting                           []                    +

  IPL ROM Emulation Device                           []                    +/

  VLAN Tag Priority (0 to 7)                         []                    #

  VLAN Tag Identifier (0 to 4094)                    []                    #

  CPU Id                                             [00F94F584C00]

  Communication Protocol used by client              [nimsh]               +

  NFS Client Reserved Ports                          []                    +

  Comments                                           []

 

  Managing System Information

    LPAR Options

      Identity                                       [88]

      Management Source                              [S824]                                                                                                                                      

 

 

# lsnim -l AIXmig

AIXmig:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = net1 AIXmig 0

   cable_type1    = N/A

   mgmt_profile1  = hsc02 88 S824               <<< LPARD id 88, Mgmt Src S824

   Cstate         = ready for a NIM operation

   prev_state     = ready for a NIM operation

   Mstate         = currently running

   cpuid          = 00F94F584C00

   Cstate_result  = success

 

 

I also need to configure an lpp_source for the ifix location (on the NIM master) and the Live Update data file (on the NIM master). This file can reside on the NIM client if you wish but I’ve chosen to manage all the resources on the NIM master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdatefix       resources       lpp_source

 

# lsnim -l liveupdatefix

liveupdatefix:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/ifix

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/ifix

total 72

-rw-r-----    1 root     system        35625 Oct 15 14:50 dummy.150813.epkg.Z

 

# lsnim -t live_update_data

liveupdate_AIXmig     resources       live_update_data

 

# lsnim -l liveupdate_AIXmig

liveupdate_AIXmig:

   class       = resources

   type        = live_update_data

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/lvupdate.data

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/

total 16

drwxr-xr-x    2 root     system          256 Oct 15 14:54 ifix

-r--r-----    1 root     system         4289 Oct 15 15:04 lvupdate.data

 

# tail -20 /nim/lvup/lvupdate.data

 

#         Users need not provide redundant options such as "-a -U -C and -o"

#         in the trc_option field for trace stanza.

#         Do not add a trace stanza to the lvupdate.data file unless you

#         want the live update commands to be traced.

#

 

general:

        mode = automated

        kext_check = no

 

disks:

        nhdisk  = hdisk0

        mhdisk  = hdisk1

        tohdisk =

        tshdisk =

 

hmc:

        lpar_id  = 88

        management_console = 10.1.50.30

        user = hscroot

 

Now I can perform a preview of the live update operation, from the NIM master. The preview operation will be run on the NIM client called AIXmig.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 595

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffers.

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.

Validating live update input data.

 

With the preview looking good, I can perform the actual update, again starting it from the NIM master.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 644

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffer

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be a

Validating live update input data.

 

Non-interruptable live update operation begins in 10 seconds.

....................................

Initializing live update on original LPAR.

 

Validating original LPAR environment.

 

Beginning live update operation on original LPAR.

 

Requesting resources required for live update.

............

Notifying applications of impending live update.

 

Creating rootvg for boot of surrogate.

................................................................

Starting the surrogate LPAR.

........................................

Creating mirror of original LPAR's rootvg.

................................................

Moving workload to surrogate LPAR.

....

        Blackout Time started.

................................................................................................................................

        Blackout Time end.

 

Workload is running on surrogate LPAR.

....................................

Shutting down the Original LPAR.

................

The live update operation succeeded.

 

If you want, you could initiate the live update from the NIM client using the nimclient command. All the resources reside on the NIM master, but the NIM client starts the operation, not the NIM master.

 

# nimclient -o cust -a live_update=yes -a lpp_source=liveupdatefix -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a filesets=dummy.150813.epkg.Z