Chris's AIX Blog

I created my first ‘versioned’ AIX 5.2 WPAR on AIX 7 today. To get me started, I followed the steps in the following guide from the AIX 7 Open Beta On-line Information Centre:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.cre/creconfig-create-wpar.htm

 

In order to create the WPAR, I needed an AIX 5.2 mksysb file to supply to the mkwpar command.

 

Fortunately, I just happened to have an old AIX 5.2 mksysb image in my archives!

 

I then executed the following command to build the WPAR:

 

# mkwpar -n wpar1 -C -B /home/cgibson/AIX5202_64bit-mksysb

 

The flags to the command are:

 

-n wparname

Specifies the name for the workload partition to be created. You must specify a name, either using the -n flag or in a specification file using the -f flag, unless the -p name or both –w and -o flags are used.

 

-B wparbackupdevice

Specifies a device containing a workload partition backup image. This image is used to populate the workload partition file systems. The wparBackupDevice parameter is a workload partition image that is created with the savewpar, mkcd, or mkdvd command. The -B flag is used by the restwpar command as part of the process of creating a workload partition from a backup image.

 

-C

Creates a versioned workload partition. This option is valid only when additional versioned workload partition software has been installed.

 

 

I was then able to start my new AIX 5.2 WPAR successfully!

 

# startwpar -v wpar1

Starting workload partition wpar1.

Mounting all workload partition file systems.

Mounting /wpars/wpar1

Mounting /wpars/wpar1/home

Mounting /wpars/wpar1/mksysb

Mounting /wpars/wpar1/nre/opt

Mounting /wpars/wpar1/nre/sbin

Mounting /wpars/wpar1/nre/usr

Mounting /wpars/wpar1/opt

Mounting /wpars/wpar1/proc

Mounting /wpars/wpar1/tmp

Mounting /wpars/wpar1/usr

Mounting /wpars/wpar1/usr/local

Mounting /wpars/wpar1/var

Mounting /wpars/wpar1/var/log

Mounting /wpars/wpar1/var/tsm/log

Loading workload partition.

Exporting workload partition devices.

Exporting workload partition kernel extensions.

Starting workload partition subsystem cor_wpar1.

0513-059 The cor_wpar1 Subsystem has been started. Subsystem PID is 8388822.

Verifying workload partition startup.

Return Status = SUCCESS.

 

The WPAR was now in an active state and the associated file systems were mounted (as shown from the Global environment).

 

# lswpar

Name   State  Type  Hostname  Directory     RootVG WPAR

--------------------------------------------------------

wpar1  A      S     wpar1     /wpars/wpar1  no

 

 

# mount | grep wpar

         /dev/lv00        /wpars/wpar1     jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv01        /wpars/wpar1/home jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv02        /wpars/wpar1/mksysb jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /opt             /wpars/wpar1/nre/opt namefs Jul 26 20:13 ro

         /sbin            /wpars/wpar1/nre/sbin namefs Jul 26 20:13 ro

         /usr             /wpars/wpar1/nre/usr namefs Jul 26 20:13 ro

         /dev/lv03        /wpars/wpar1/opt jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /proc            /wpars/wpar1/proc namefs Jul 26 20:13 rw

         /dev/lv04        /wpars/wpar1/tmp jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv05        /wpars/wpar1/usr jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv06        /wpars/wpar1/usr/local jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv07        /wpars/wpar1/var jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv08        /wpars/wpar1/var/log jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv09        /wpars/wpar1/var/tsm/log jfs    Jul 26 20:13 rw,log=/dev/loglv00

 

I was curious what the WPAR environment was going to look like, so I used clogin to access it and run a few commands.

 

From the Global environment I confirmed I was indeed on an AIX 7 system.

 

# uname -W

0

 

# oslevel

V7BETA

 

From within the WPAR, I confirmed that I was indeed running AIX 5.2! Wow!

 

# clogin wpar1

wpar1 : / # oslevel

5.2.0.0

 

And I could see all 8 logical CPUs (4 hardware threads per POWER7 processor i.e. SMT-4).

 

wpar1 : / # sar -P ALL 1 5

 

AIX wpar1 2 5 00F602734C00    07/26/10

 

wpar1 configuration: @lcpu=8  @mem=4096MB  @ent=0.50

 

20:22:20 cpu    %usr    %sys    %wio   %idle   physc   %entc

20:22:21  0        7      78       1      14    0.01     0.0

          1        1      70       0      29    0.01     0.0

          2        0       1       0      99    0.00     0.0

          3        0       0       0     100    0.01     0.0

          4        0      35       0      65    0.00     0.0

          7        0      28       0      72    0.00     0.0

          U        -       -       0      93    0.47    93.9

          -        0       3       0      97    0.03     0.0

 

I noticed an interesting device in the lscfg output.

 

wpar1 : / # lscfg

INSTALLED RESOURCE LIST

 

The following resources are installed on the machine.

+/- = Added or deleted from Resource List.

*   = Diagnostic support not available.

 

  Model Architecture: chrp

  Model Implementation: Multiple Processor, PCI bus

 

+ sys0               System Object

* wio0               WPAR I/O Subsystem

 

Also noticed some new and interesting mount points, for example /nre/opt.

 

wpar1 : / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

Global            131072     99928   24%     1424     5% /

Global            131072    126704    4%       70     1% /home

Global           1048576   1015560    4%       17     1% /mksysb

Global            786432    428904   46%     7331    14% /nre/opt

Global            458752     88400   81%    10020    47% /nre/sbin

Global           4980736     24872  100%    53698    87% /nre/usr

Global            131072     63800   52%     1640    11% /opt

Global                 -         -    -         -     -  /proc

Global            131072    125080    5%       52     1% /tmp

Global           1572864    165744   90%    23183    12% /usr

Global            524288    494464    6%      154     1% /usr/local

Global            131072    111512   15%      493     4% /var

Global            262144    253744    4%       28     1% /var/log

Global            131072    126832    4%       20     1% /var/tsm/log

 

 

I did have one minor problem when I first tried to start my WPAR, but that issue was quickly resolved by the AIX developers on the AIX 7 Open Beta Forum.

 

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=335415&tstart=0

 

Such a fast response to my query demonstrates IBMs commitment to the Open Beta program.

 

I was notified this morning that I’ve been selected to work on an upcoming AIX Redbook in Austin, Texas. I’m really looking forward to it.

 

AIX Differences Guide, PW-0001-R01: http://www.redbooks.ibm.com/residents.nsf/50da6a28780ffa688525701b004a4f21/9032e219ecda947c852576f900691af5?OpenDocument

 

Follow me on Twitter: http://twitter.com/cgibbo

 

I just had to share this entertaining presentation with everyone.

 

http://gibsonnet.net/AIX/ibm/Journey_in_VLAN_world.pps

 

Or the PDF. It's not as interesting or fun, as the animations don't work but you get the idea. I'm sharing the PDF because it appears that you can open this file on Windows fine (in read-only mode) but on a Mac it prompts for a password without an option for opening in read-only. Shame.

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Journey_in_VLAN_world.pdf

 

It was written by Bernard Lemal. He works for IBM GTS Belgium in AIX support.

 

It’s a humorous look at the life of an IP packet and its journey in the world of VLANs, VLAN tagging and Virtual I/O.

 

Plenty of interesting and informative commentary along the way.

 

The part of the IP packet is portrayed by Bernard himself. Take a look at the presentation to see what I mean!

 

I wish I’d seen a presentation like this when I was first introduced to the concept of VLAN tagging, it might have answered several questions early on.

 

Enjoy!

Interested in paid support for LPAR2RRD tool?

 

Let’s all support Pavel and spread the word!

 

From: Pavel Hampl <pavel.hampl@cz.ibm.com> Date: 21 July 2011 3:00:19 AM AEST To: undisclosed-recipients:; Subject: LPAR2RRD professional support announcement Hi Everyone, You are receiving that mail because we have been either in touch in regards of LPA2RRD tool about year ago or you are on LPAR2RRD mailing list. I’ve decided to make professional paid support of LPAR2RRD. Here are my reasons which led me to that thought: - I have not touched the source code for nearly 1.5 year (looks like the last version is quite stable :) ). No time at all and lack of motivation... - Only what I trying to keep is support of the tool, but my response times are not what I am proud of - the tool itself would need big time investment to implement new functionalities, rewrite the web face (would need to hire a web designer for that) etc. Only the solution what I see how to keep continuity and do not let project slowly died is paid support. Basically I feel that as the last chance for any further development or support from my side. I simply cannot manage to do everything I actually do around and still work on LPAR2RRD. If I would get some money from the project then I will be able to prioritize my other activities in favour of LPAR2RRD development and support. From that reason I’ve created new LPAR2RRD home here: http://www.lpar2rrd.com I am about to found a company which would be responsible for support. Based on your feedback I will decide go or no-go. I could even imagine working on LPAR2RRD full time in case of enough support subscribers. I do not intend to force you ordering support. If you are happy with the actual version, you have no issues etc ... that this makes me also happy. Make people happy was the main reason why I spent my free time by developing of the tool since 2006. I do not regret that at all! It was fun. Support details can be found here: http://www.lpar2rrd.com/support.htm Let me know who would be inetrested in that offer. Cheers, Pavel P.S.: For those of you who have installed LPAR2RRD for your customers please distribute this information to them. Thanks.

 

 

OK, it’s official. The AIX 7 Open Beta is now available for testing!

 

An overview of the program is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml

 

Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.

 

I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.

 

The AIX 7 beta code is available for download from here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/download.shtml

 

The user forum, where you can provide feedback and ask questions, is here:

 

https://www.ibm.com/developerworks/forums/forum.jspa?forumID=2255                        

 

I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/library.shtml

 

The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.

 

The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:

 

A.    AIX 5.2 Workload Partitions for AIX 7.

B.    Removal of WPAR local storage device restrictions.

C.    Domain support in Role Based Access Control (RBAC).

D.    Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.

E.    NIM thin server support for IPv6 and NFSv4.

 

I’m interested in testing A, B and D. Here’s why.

 

A.  AIX 5.2 Workload Partitions for AIX 7.

AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.

 

B.  Removal of WPAR local storage device restrictions.

AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?

 

D.  Etherchannel enhancements in 802.3ad mode.

There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I’m interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it’s not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.

 

And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:

 

Installing the AIX Open Beta Driver

The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.

 

Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the ‘mksysb’.

 

1.     Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:

• Type 1 and press Enter to have English during install.

• Type 1 to continue with the install.

• Type 1 to Start Install Now with Default Setting.

 

2.     The system will start installing the AIX 7.0 BETA.

 

3.     Upon completion of the install, the system will reboot. You can then login as “root”, no password is required.

 

Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:

 

·         The Open Beta code is being delivered via an “mksysb” install image. Migration installation is not supported with the open beta driver.

 

·         The open beta driver does not support IBM Systems Director Agent.

 

·         When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.

 

·         The image is known to install without issues on an 8 GB disk.

 

·         oslevel output shows V7BETA.

 

By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17^th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it’s great if you just want to test something on an AIX system.

 

I’ll report back once I’ve got my AIX 7 Open Beta system up and running!

I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.

 

This was the output from his attempt with dgetmacs:

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1.  The routine will continue.

No adapters found for node lpar1.

Could not get mac address for this aix node "lpar1"

 

I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.

 

First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.

 

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54

 

They were surprised to find that the file appeared to be “too small”. They promptly sent me the catalog file from one of their development AIX 7.1 systems.  I replaced the file as follows:

 

# cd  /usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54 IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin         3905 Aug 08 09:54 IBMhsc.netboot.cat.old

-rw-r--r--    1 bin      bin        26374 Dec 23 11:24 IBMhsc.netboot.cat

 

This fixed the problem for me during the residency.

 

So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

 

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

 

lpar1::ent_v::::4A7009608816::U8202.E4B.10582FP-V4-C22-T1::auto::auto::::::n/a::secondary::::

 

lpar1::ent_v::::4A7009608818::U8202.E4B.10582FP-V4-C24-T1::auto::auto::::::n/a::secondary::::

 

This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.

 

# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1

dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1

dnetboot Status: Was successful network booting node lpar1.

 

I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).

 

# oslevel -s

7100-00-02-1041

 

If anyone encounters this problem, feel free to contact me (aix@gibsonnet.net).

 

For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:

 

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

 

This is something that I experience on all new Power/AIX systems that I install:

 

System migrated to AIX 5.3 (or later) might experience double boot

When booting AIX Version 5.3 (or later) on a system that has previously been running an earlier release of AIX, you may notice that the system automatically reboots and restarts the boot process. This is how the firmware processes changed information in the boot image. This reboot also occurs if the process is reversed. A system previously running AIX 5.3 (or later) that is booting a release of AIX prior to 5.3 goes through the same process. This ″double boot″ occurs only once; if the stored value does not change, then the second boot does not occur. If you install AIX 5.3 (or later) and continue to use only that version, this double boot occurs once, and it occurs only if your system was running a pre-AIX 5.3 release before you boot AIX 5.3 (or later). Systems that are preinstalled with AIX 5.3 (or later) and use only that version do not experience the ″double boot.″

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.resources/RELNOTES/610_base_relnotes.pdf

 

Some will panic when this happens, but it’s not an issue. It only happens once, on the first installation of the first AIX or VIOS LPAR you install. After that you will never see it again.

AIX Live Update

Starting with AIX Version 7.2, the AIX operating system provides the AIX Live Update function which eliminates downtime associated with patching the AIX operating system. Previous releases of AIX required systems to be rebooted after an interim fix was applied to a running system. This new feature allows workloads to remain active during a Live Update operation and the operating system can use the interim fix immediately without needing to restart the entire system. In the first release of this feature, AIX Live Update will allow customers to install interim fixes (ifixes) only. Ultimately it may be possible to use this function to install AIX Service Packs (SPs) and Technology Levels (TLs) without a reboot.

 

IBM delivers kernel fixes in the form of ifixes to resolve issues that are reported by customers. If a fix changes the AIX kernel or loaded kernel extensions that cannot be unloaded, the host logical partition (LPAR) must be rebooted. To address this issue, AIX Version 7.1, and earlier, provided concurrent update-enabled ifixes that allowed deployment of some limited kernel fixes to a running LPAR. Unfortunately not all ifixes could be delivered as “concurrent update-enabled”. The AIX Live Update solution is not constrained by the same limitations as in the case of concurrent update enabled ifixes. The AIX 7.2, Live Update feature will allow customers to install ifixes without needing to reboot their AIX systems, avoiding downtime for their mission critical, production workloads.

 

This article (in the link below) will discuss the high-level concepts relating to AIX Live Updates and then provide a real example of how to use the tool to patch a live AIX system. I was fortunate enough to take part in an Early Ship Program (ESP) for AIX 7.2. During the ESP I had the opportunity to test the AIX Live Update feature. I’ll share my experience using this tool in the example that follows.

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/AIXLiveUpdateblog.pdf

 

In my previous post on AIX Live Updates I discussed how to use the geninstall command to perform a non-disruptive (ifix) update on an AIX system. In this post I wanted to show you how to perform the same task using NIM.

 

NIM can be used to start an AIX Live Update operation on a target machine (NIM client) either from a NIM master or from the NIM client itself (with nimclient).

 

Note: The AIX Live Update operation started by NIM calls the hmcauth command during the cust operation to authenticate to the NIM client with the HMC by using the HMC passwd file. The NIM master is responsible for obtaining password information from the HMC (using ssh).  Without it, NIM clients will not have the password information necessary when running hmcauth as part of the NIM client operation.  So, we must first define an hmc object in NIM and create the password file (used when accessing the console.)  Once this required step has been completed, all clients using NIM live_update have the ability to pass the proper hmc login credentials when configuring 'hmcauth'.

 

First, I need to install the dsm.core fileset and configure SSH keys between the NIM master and the HMC.

 

# lslpp -l dsm.core

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  dsm.core                  7.2.0.0  COMMITTED  Distributed Systems Management

                                               Core

 

# dpasswd -f /etc/ibm/sysmgt/dsm/config/my_password_file -U hscroot

 

# dkeyexch -f /etc/ibm/sysmgt/dsm/config/my_password_file -I hmc -H hsc02

 

Then define a HMC and Managed System NIM object on the NIM master.

 

# nim -o define -t hmc -a if1="find_net hsc02 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/my_password_file hsc02

 

# nim -o define -t cec -a hw_type=8286 -a hw_model=42A -a hw_serial=214F58V -a mgmt_source=hsc02 S824

 

The NIM client must either be defined with or updated to include the Managed System name (Management Source) and LPAR id number.

 

# smit nim_chmac

 

               Change/Show Characteristics of a Machine

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                  [Entry Fields]

  Machine Name                                       [AIXmig]

* Hardware Platform Type                             [chrp]                +

* Kernel to use for Network Boot                     [64]                  +

  Machine Type                                        standalone

  Network Install Machine State                       currently running

  Network Install Control State                       ready for a NIM operation

  Primary Network Install Interface

    Network Name                                      net1

    Host Name                                        [AIXmig]

    Network Adapter Hardware Address                 [0]

    Network Adapter Logical Device Name              [ent]

    Cable Type                                        N/A                  +

    Network Speed Setting                            []                    +

    Network Duplex Setting                           []                    +

  IPL ROM Emulation Device                           []                    +/

  VLAN Tag Priority (0 to 7)                         []                    #

  VLAN Tag Identifier (0 to 4094)                    []                    #

  CPU Id                                             [00F94F584C00]

  Communication Protocol used by client              [nimsh]               +

  NFS Client Reserved Ports                          []                    +

  Comments                                           []

 

  Managing System Information

    LPAR Options

      Identity                                       [88]

      Management Source                              [S824]                                                                                                                                      

 

 

# lsnim -l AIXmig

AIXmig:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = net1 AIXmig 0

   cable_type1    = N/A

   mgmt_profile1  = hsc02 88 S824               <<< LPARD id 88, Mgmt Src S824

   Cstate         = ready for a NIM operation

   prev_state     = ready for a NIM operation

   Mstate         = currently running

   cpuid          = 00F94F584C00

   Cstate_result  = success

 

 

I also need to configure an lpp_source for the ifix location (on the NIM master) and the Live Update data file (on the NIM master). This file can reside on the NIM client if you wish but I’ve chosen to manage all the resources on the NIM master.

 

# lsnim -t lpp_source

lpp_sourceaix72     resources       lpp_source

liveupdatefix       resources       lpp_source

 

# lsnim -l liveupdatefix

liveupdatefix:

   class       = resources

   type        = lpp_source

   arch        = power

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/ifix

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/ifix

total 72

-rw-r-----    1 root     system        35625 Oct 15 14:50 dummy.150813.epkg.Z

 

# lsnim -t live_update_data

liveupdate_AIXmig     resources       live_update_data

 

# lsnim -l liveupdate_AIXmig

liveupdate_AIXmig:

   class       = resources

   type        = live_update_data

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /nim/lvup/lvupdate.data

   alloc_count = 0

   server      = master

 

# ls -ltr /nim/lvup/

total 16

drwxr-xr-x    2 root     system          256 Oct 15 14:54 ifix

-r--r-----    1 root     system         4289 Oct 15 15:04 lvupdate.data

 

# tail -20 /nim/lvup/lvupdate.data

 

#         Users need not provide redundant options such as "-a -U -C and -o"

#         in the trc_option field for trace stanza.

#         Do not add a trace stanza to the lvupdate.data file unless you

#         want the live update commands to be traced.

#

 

general:

        mode = automated

        kext_check = no

 

disks:

        nhdisk  = hdisk0

        mhdisk  = hdisk1

        tohdisk =

        tshdisk =

 

hmc:

        lpar_id  = 88

        management_console = 10.1.50.30

        user = hscroot

 

Now I can perform a preview of the live update operation, from the NIM master. The preview operation will be run on the NIM client called AIXmig.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 595

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffers.

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.

Validating live update input data.

 

With the preview looking good, I can perform the actual update, again starting it from the NIM master.

 

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

 

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 644

 

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

 

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

 

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

 

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

 

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffer

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be a

Validating live update input data.

 

Non-interruptable live update operation begins in 10 seconds.

....................................

Initializing live update on original LPAR.

 

Validating original LPAR environment.

 

Beginning live update operation on original LPAR.

 

Requesting resources required for live update.

............

Notifying applications of impending live update.

 

Creating rootvg for boot of surrogate.

................................................................

Starting the surrogate LPAR.

........................................

Creating mirror of original LPAR's rootvg.

................................................

Moving workload to surrogate LPAR.

....

        Blackout Time started.

................................................................................................................................

        Blackout Time end.

 

Workload is running on surrogate LPAR.

....................................

Shutting down the Original LPAR.

................

The live update operation succeeded.

 

If you want, you could initiate the live update from the NIM client using the nimclient command. All the resources reside on the NIM master, but the NIM client starts the operation, not the NIM master.

 

# nimclient -o cust -a live_update=yes -a lpp_source=liveupdatefix -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a filesets=dummy.150813.epkg.Z

 

 

I’ve written about multibos before, here and here. But recently I started experimenting with multibos mksysb migration. A customer asked me how this worked and apart from a high-level view I wasn’t able to provide any real world experience, so I thought I’d give it a try. What follows is just a ‘brain dump’ from my quick test.

 

First of all this isn’t really a migration. It just simply populates a second instance of AIX with a higher-version. It doesn’t really migrate (or merge) your existing configuration into the second instance. So I’m not sure how useful this feature really is right now.

 

Starting with 5.3 TL9 you can add a 6.1 TL2 (or above) instance. This is done with the new –M flag. You must be running with the 64bit kernel.

This isn’t really a migration because it populates the second instance using a mksysb based on the new release.

In 6.1 TL2 a new flag (-M) was added to the mksysb command which allows you to create a mksysb for use with multibos. It creates a backup of BOS (/, /usr, /var, /opt).
bos.alt_disk_install.boot_images must be installed.

To preview the multibos migration
multibos –M <location of mksysb> -spX

To perform the multibos migration
multibos –M <location of mksysb> -sX

It is not advised to run in this environment for an extended period of time. There could be problems if tfactor or maps are used. Be aware that 6.1 specific attributes may not be reflected in the standby instance.

 

So in my lab environment I have two AIX LPARs. One is running AIX 6.1 and the other running AIX 7.1.

 

First I take a mksysb (with the –M flag) of the AIX 7.1 system to a file. This file will be called by multibos to populate the second instance.

 

aix7[/] > mksysb -Mie /data/aix7-mksysb

 

Creating information file (/image.data) for rootvg.

 

Creating list of files to back up.

 

Backing up 71643 files.....

 

71643 of 71643 files (100%)

0512-038 mksysb: Backup Completed Successfully.

 

aix7[/] > ls -ltr /data

total 4276112

drwxr-xr-x    2 root     system          256 Feb 21 20:59 lost+found

-rw-r--r--    1 root     system   2189363200 Feb 21 21:06 aix7-mksysb

 

I copied this file over to my AIX 6.1 system. This was the system that was to be ‘migrated’. The next step was to perform a preview of the multibos operation.

 

root@aix6 /# multibos -sXp -M /data/aix7-mksysb

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Preview

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

Processing preview information ...

 

ACTIVE LV:          hd4

STANDBY LV:         bos_hd4

TYPE:               jfs2

ACTIVE FS:          /

STANDBY FS:         /bos_inst

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd2

STANDBY LV:         bos_hd2

TYPE:               jfs2

ACTIVE FS:          /usr

STANDBY FS:         /bos_inst/usr

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd9var

STANDBY LV:         bos_hd9var

TYPE:               jfs2

ACTIVE FS:          /var

STANDBY FS:         /bos_inst/var

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd10opt

STANDBY LV:         bos_hd10opt

TYPE:               jfs2

ACTIVE FS:          /opt

STANDBY FS:         /bos_inst/opt

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd5

STANDBY LV:         bos_hd5

TYPE:               boot

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              closed

 

ACTIVE LV:          hd8

STANDBY LV:         bos_hd8

TYPE:               jfs2log

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              open

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

With a successful preview out of the way, I initiated the real multibos operation.

 

# multibos -sX -M /data/aix7-mksysb

 

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

The total size is 181596 bytes.

The number of restored files is 2.

 

+-----------------------------------------------------------------------------+

Logical Volumes

+-----------------------------------------------------------------------------+

Creating standby BOS logical volume bos_hd5

Creating standby BOS logical volume bos_hd4

Creating standby BOS logical volume bos_hd2

Creating standby BOS logical volume bos_hd9var

Creating standby BOS logical volume bos_hd10opt

Creating standby BOS logical volume bos_hd8

 

+-----------------------------------------------------------------------------+

File Systems

+-----------------------------------------------------------------------------+

Creating all standby BOS file systems ...

Creating standby BOS file system /bos_inst on logical volume bos_hd4

Creating standby BOS file system /bos_inst/usr on logical volume bos_hd2

Creating standby BOS file system /bos_inst/var on logical volume bos_hd9var

Creating standby BOS file system /bos_inst/opt on logical volume bos_hd10opt

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

BOS Files

+-----------------------------------------------------------------------------+

Including files for file system /

Including files for file system /usr

Including files for file system /var

Including files for file system /opt

 

Copying files using backup/restore utilities ...

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x         6086 ./bosinst.data

x           11 ./tmp/vgdata/rootvg/image.info

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

x            0 ./

x           47 ./.profile

x           63 ./.rhosts

x        13192 ./.sh_history

x            0 ./.ssh

x         1647 ./.ssh/authorized_keys

x            0 ./.topasrecrc

x            0 ./.topasrecrc/topasrec.lock

x       141722 ./.topasrecrc/topasrec.log1

x            0 ./.topasrecrc/topasrec.log2

x          108 ./.vi_history

x            0 ./admin

x            0 ./audit

x            8 ./bin

x            0 ./cdat

x           59 ./cdat/cdat.xml

x            0 ./cg

x            0 ./cgps

x            0 ./dev

x            0 ./dev/.SRC-unix

x            0 ./dev/IPL_rootvg

x            0 ./dev/__pv17.0.10617024

x            0 ./dev/__vg10

x            0 ./dev/audit

.....BLAH.....

x ./dev/zero

Creating standby BOS boot image on boot logical volume bos_hd5

254+0 records in.

254+0 records out.

179+1 records in.

177+1 records out.

saving to '/dev/bos_hd5'

44 CuDv objects to be saved

130 CuAt objects to be saved

21 CuDep objects to be saved

7 CuVPD objects to be saved

351 CuDvDr objects to be saved

2 CuPath objects to be saved

0 CuPathAt objects to be saved

0 CuData objects to be saved

Number of bytes of data to save = 18716

Compressing data

Compressed data size is = 6393

        bi_start     = 0x400

        bi_size      = 0x1677200

        bd_size      = 0x1657200

        ram FS start = 0x8c3570

        ram FS size  = 0xd93abe

        sba_start    = 0x1657600

        sba_size     = 0x20000

        sbd_size     = 0x0

Checking boot image size:

        new save base byte cnt = 0x18fd

Wrote 6397 bytes

Successful completion

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

+-----------------------------------------------------------------------------+

Bootlist Processing

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Setting bootlist to logical volume bos_hd5 on hdisk0.

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:4

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:4

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:2

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:2

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

Now that the second instance had been created, I thought I’d try to start a multibos shell and access the AIX 7.1 environment. I didn’t think this would work but I tried my luck anyway!

 

root@aix6 /# multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

multibos: 0565-110 Cannot start multibos shell due to incompatible levels of active and standby BOS.

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

Log file is /etc/multibos/logs/op.alog

Return Status: FAILURE

 

Upon checking my bootlist output, I noticed (as expected) that the list now contained two extra entries for bos_hd5. These were the boot logical volume entries for the second instance. If I was to boot from this LV I’d be booting into AIX 7.1. Cool.

 

root@aix6 /# bootlist -m normal -o

hdisk0 blv=bos_hd5

hdisk0 blv=bos_hd5

hdisk0 blv=hd5

hdisk0 blv=hd5

 

So at this point, I’d created a second instance of AIX running 7.1. My current version of (running) AIX was AIX 6.1. All I had to do now was reboot the LPAR and let it restart as an AIX 7.1 system.

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

root@aix6 / # shutdown –Fr

 

The LPAR rebooted successfully and I found I was now running AIX 7.1, just as I’d hoped.

 

aix6[/] > oslevel -s

7100-00-01-1037

 

If I wanted to go back to AIX 6.1, I would change my bootlist setting again and restart the LPAR.

 

aix6[/] > bootlist -m normal -o

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

 

aix6[/] > bootlist -m normal hdisk0 blv=hd5 pathid=0 hdisk0 blv=hd5 pathid=1 hdisk0 blv=bos_hd5 pathid=0 hdisk0 blv=bos_hd5 pathid=1

aix6[/] > bootlist -m normal -o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

 

aix6[/] > shutdown –Fr

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

Now that I’ve actually tried this method of migration, I’m not sure I’d actually use it in its current form.

 

Although the migration keeps my hostname and IP address, the file systems are not shared between instances. Most of the target systems configuration is not retained. For example, any user accounts I create on my AIX 6.1 system would also need to be created on the existing AIX.7.1 system which I used to create the AIX 7.1 mksysb image. It reminds me a little of a preservation install.

 

More information on multibos:

 

http://www.aixmind.com/?p=1192

 

IBM made some announcements today relating to their latest POWER7 server offerings. The new line of systems includes new entry level systems and the highly anticipated high-end system, the POWER7 795!  They also officially outlined some of the new features available in AIX 7.1. You can review the details here. I’ve discussed some of these new features here and here. The official AIX 7.1 announcement details are available here.

 

The announcement got me thinking about my recent customer engagements and why some have chosen to deploy AIX into their IBM POWER environments, while others are considering a Linux on POWER solution.

 

I’ve found that it usually comes down to a skills decision more than anything else. Most customers are happy to either continue working with AIX (if they are existing AIX users) or migrate from another UNIX OS to AIX. I’ve seen very few customers actually migrate to Linux on POWER, but I’ve worked with several that have seriously considered it. Those that have chosen to deploy Linux are doing so purely because they have in-house Linux skills. They are concerned that migrating to AIX may be too big a jump for their technical staff. I find this thinking interesting, as most of the customers I’ve dealt with who run other UNIX OS’s like Tru64, Solaris or HP-UX are more than happy to migrate to AIX. They believe the move is relatively minor and doesn’t require massive re-training of their UNIX admins. I tend to agree.

 

For me, AIX is my preferred “Enterprise class” UNIX Operating System. Notice I’m prefacing this with the words Enterprise class. Don’t get me wrong, I have worked with Linux systems in both small and large customer environments. It is a great OS. But I’ve found that it really only fits into environments that have a relatively small number of users and where significant downtime can be tolerated for things like operating system maintenance. This doesn’t fit the Enterprise class of UNIX server OS’s that I’m thinking of here. When I contemplate the word Enterprise, I think of servers and operating systems that can respond to business demands in terms of performance, reliability, stability and availability. An Enterprise UNIX can provide all of these things without compromise. Linux can offer performance and reliability (in my opinion). However, from what I’ve seen, it lacks features & functions in the areas of stability and availability. AIX on the other hand ticks all the boxes. Again, this is just my opinion based on my experiences with both AIX and Linux in the Enterprise landscape. Others will no doubt have their own experiences that may or may not match my own.

 

So when I’m designing an Enterprise UNIX server environment for a customer, I always start with an AIX on POWER base. If the customer wants Linux, sure I can look at that too, but I strongly recommend AIX as my preferred choice for large systems. Most of my customers are running relatively large SAP/Oracle systems. AIX on POWER is a great combination for large Enterprise systems. If you need to deploy large database systems that must service tens of thousands of users (like a big SAP system), then I believe AIX is the perfect OS on which to provide a platform for these large scale systems.

 

AIX is a very mature and powerful UNIX OS. It has been a major player in the UNIX server market for over 20 years (as shown below). Some people are just not aware of how mature, robust and stable the AIX OS has become over the years. There are many impressive aspects of the OS in the areas of performance, scalability, reliability, management and administration.

 

 

Just looking at some of the administration capabilities built into AIX are enough for me to always recommend AIX over Linux (or any other UNIX OS), when it comes to large Enterprise servers.

 

For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler. This is an interactive tool that is part of the AIX OS. Almost all tasks that an AIX administrator may need to perform can be executed using this tool. It is a text-based tool (there is also an X interface but I recommend sticking with the text-based menus). Everything it does, it does through standard AIX commands and Korn shell functions. This feature is especially useful when you need to automate a repetitive task; you can have SMIT create the proper command-line sequence, and you can then use those commands in your own script. My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.

 

The AIX Logical Volume Manager (LVM) is built into the OS, for free. AIX LVM helps UNIX system administrators manage their storage in a very flexible manner. LVM allows logical volumes to span multiple physical volumes. Data on logical volumes appears to be contiguous to the user, but might not be contiguous on the physical volume. This allows file systems, paging space, and other logical volumes to be resized or relocated, span multiple physical volumes, and have their contents replicated for greater flexibility and availability. It provides capabilities for mirroring data across disks, migrating data across disks & storage subsystems, expand/shrink filesystems and more.......all of which can be performed dynamically.....no downtime required. The concept, implementation and interface to the AIX LVM is one of a kind. All of its features support the ‘continuous availability’ philosophy.

 

One of the biggest reasons that I love AIX over Linux is the mksysb. It’s built into the OS and allows you to create a bootable image of your AIX system. This image can be used to restore a broken AIX system or for cloning other systems. The cloning feature is truly amazing. You can take an image created on a low-end system and deploy it on any POWER system, all the way up to the high-end POWER boxes. This simplifies the installation and cloning processing when you need to install and manage many AIX LPARs. By using an SOE mksysb image you can deploy consistent AIX images across your Enterprise POWER server environment.

 

This brings me to another wonderful feature of AIX, the Network Installation Manager (NIM). NIM is powerful network installation tool (comparable to Linux Kickstart). Using NIM you can backup/restore, update and upgrade one or more AIX systems either individually or simultaneously. This can all be achieved over a network connection, removing the need for handling physical installation media forever.

 

Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates. Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows.

 

When it comes to upgrading the OS to a new release of AIX, the nimadm utility can assist the administrator greatly in this task. The nimadm utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a NIM client's rootvg and migrate the disk to a newer version or release of AIX. All of this can be done without disruption to the client (there is no outage required to perform the migration). After the migration is finished, the only downtime required will be a single scheduled reboot of the system.

 

AIX 6.1 introduced new capability that most UNIX operating systems are still working on. Concurrent Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS that can provide continuous availability, even if it needs to be patched. AIX now has the ability to update certain kernel components and kernel extensions in place, without needing a system reboot. In addition, concurrent updates can be removed from the system without needing a reboot. Can you do this with other UNIX OSs?

 

And that’s just some of the features that make AIX the only UNIX OS that I recommend for Enterprise systems. There are many more tools and features that I couldn’t live without (like alt_disk_install, savevg, installp, WPARs, etc, the list goes on). If you are new to AIX and you are considering what your next UNIX OS should be, then I recommend you take a very close look at AIX.

 

And finally, the support provided by IBM, is first class. Whenever I’ve needed assistance with an AIX issue or query, I have always received timely, professional and useful advice. On the rare occasions where I’ve uncovered a new bug, IBM AIX support have always been quick to provide me with an interim fix to resolve or workaround a problem. That’s the sort of support you’d expect for an Enterprise UNIX OS, isn’t it? What’s the support like from your current UNIX (or Linux) OS vendor?

 

Linux is still a viable UNIX operating system. However, I think it’s more suited to certain workloads like small to medium size mail, web and other utility servers and services. AIX, however, would be my platform of choice for my 10TB Oracle database running SAP ERP, not just for performance reasons, but primarily because of the system administration features of AIX that allow me to support and manage the system without impacting my customers or enforcing reboots/outages whenever I need to change something on the system.

 

All IBM need to do is create their own Linux distribution (Blue Linux perhaps?) that has all the features of AIX built in and then I’m sold! But why would they? We already have AIX.

 

Follow me on Twitter: http://twitter.com/cgibbo

 

AIX has a new “critical volume group” capability which will monitor for the loss or failure of a volume group. You can apply this to any volume group, including rootvg. If applied to rootvg, then you can monitor for the loss of the root volume group.

 

This feature may be useful if your AIX LPAR experiences a loss of SAN connectivity e.g. total loss of access to SAN storage and/or all SAN switches. Typically, when this happens, AIX will continue to run, in memory for a period of time and will not immediately crash. Often you can still log on to the AIX system but if you attempt to write a file you’ll see in I/O error. But even then the system may (potentially) remain up. When the SAN issue is resolved the AIX system may continue running, with file systems in read-only mode (or not, it depends) but to really resolve the issue you would still need to reboot the AIX LPAR in order for it regain access to its disks. This can result in the need to run fsck against file systems. Note that the behaviour you encounter will be impacted by a variety of factors, such as length and type of outage. As always, your mileage may vary!

 

You can encounter this behaviour with both VSCSI and NPIV SAN booted LPARs. This new AIX VG option, which caters for the scenario described above, is not enabled by default. From the chvg man page:

 

“-r y | n                      Changes the critical volume group (VG) option of the volume group.

 

n                                  Disables the critical VG option of the volume group.

 

y                                  Enables the critical VG option of the volume group. If the volume group is set

to the critical VG, any I/O request failure starts the Logical Volume Manager

(LVM) metadata write operation to check the state of the disk before

returning the I/O failure. If the critical VG option is set to rootvg and if the

volume group losses access to quorum set of disks (or all disks if quorum

disabled), instead of moving the VG to an offline state, the node is crashed

and a message is displayed on the console.”

 

PowerHA also caters for and supports this now....and should already be enabled by default. You want this feature enabled for your HA clusters so that they respond appropriately to loss of the root volume group and initiate a failover.

 

smitty sysmirror -> Custom Cluster Configuration -> Events -> System Events -> Change/Show Event Response (smitty cm_change_show_sys_event)

 

                     Change/Show Event Response

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                        [Entry Fields]

* Event Name            ROOTVG                 +

 

* Response              Log event and reboot   +

* Active                Yes                    +

 

"Exploitation of LVM rootvg failure monitoring

 

AIX LVM has recently added the capability to change a volume group to be a known as critical volume group. Though PowerHA has allowed critical volume groups in the past, that

only applied to non-operating system/data volume groups. PowerHA v7.2 now also takes advantage of this functionality specifically for rootvg. If the volume group is set to the critical VG, any I/O request failure starts the Logical Volume Manager (LVM) metadata write operation to check the state of the disk before returning the I/O failure. If the critical VG option is set to rootvg and if the volume group losses access to quorum set of disks (or all disks if quorum is disabled), instead of moving the VG to an offline state, the node is crashed and a message is displayed on the console. You can set and validate rootvg as a critical volume group by executing the commands shown below. The command has to run once since we are using the CAA distributed command clcmd.

 

# clcmd chvg -r y rootvg

# clcmd lsvg rootvg |grep CRIT

DISK BLOCK SIZE: 512 CRITICAL VG: yes

DISK BLOCK SIZE: 512 CRITICAL VG: yes"

 

To test this new feature in my lab, I simulated a disk "failure" or accidental unmapping/removal of a rootvg disk from an LPAR.

 

On the AIX LPAR, prior to disk failure simulation, I turn on the “CRITICAL VG” option for rootvg.

 

# oslevel -s

7200-00-00-0000

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    no

 

# chvg -r y rootvg

 

# lsvg rootvg | grep CRIT

DISK BLOCK SIZE:    512                      CRITICAL VG:    yes

 

On the VIOS, I unmap the rootvg disk from the corresponding vhost adapter:

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

$ lu -list | grep volume-A | grep Lab

volume-AIX71LabImage-b~ 10240       7790        acc7e518f3f561d4719fcecfff88b0f1

volume-AIX71LabImage-b~ 1024        1021        81823bdb90537250b95b3b2f35bb55f9

 

$ lu -unmap -luudid acc7e518f3f561d4719fcecfff88b0f1

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

On the AIX LPAR, I attempt to create (write) a file in /tmp (which resides in rootvg):

 

# touch /tmp/mynewfile

 

The LPAR stops responding immediately. I can no longer connect or login to it.

 

I then remap the disk and restart the LPAR. The AIX error report shows that the system halted due a critical VG going offline.

 

$ lu -map -luudid acc7e518f3f561d4719fcecfff88b0f1 -vadapter vhost30

 

$ lsmap -vadapter vhost30

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost30         U8286.42A.214F58V-V2-C38                     0x0000001d

 

VTD                   vtscsi59

Status                Available

LUN                   0x8100000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-boot--26bf124d-40b3.acc7e518f3f561d4719fcecfff88b0f1

Physloc

Mirrored              N/A

 

VTD                   vtscsi60

Status                Available

LUN                   0x8200000000000000

Backing device        volume-AIX71LabImage-b8588f28-00000033-data--1830bd26-ba42.81823bdb90537250b95b3b2f35bb55f9

Physloc

Mirrored              N/A

 

# errpt -a

...

LABEL:          KERNEL_PANIC

IDENTIFIER:     225E3B63

 

Date/Time:       Wed Mar  1 14:27:51 AEDT 2017

Sequence Number: 215

Machine Id:      00F94F584C00

Node Id:         aix72lab

Class:           S

Type:            TEMP

WPAR:            Global

Resource Name:   PANIC

 

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

 

        Recommended Actions

        PERFORM PROBLEM DETERMINATION PROCEDURES

 

Detail Data

ASSERT STRING

 

PANIC STRING

Critical VG Force off, halting.

 

This feature is available with AIX 6.1 or later.

 

IV52743: ADD CRITICAL VG SUPPORT FOR ROOTVG APPLIES TO AIX 7100-03

http://www-01.ibm.com/support/docview.wss?uid=isg1IV52743

 

You can specify the –Z flag with the AIX snap command to prevent it from collecting system dump data. The flag is not shown in the man page for snap. However, it is listed in the usage information.

 

# oslevel -s

7100-03-03-1415

 

# sysdumpdev -L

 

 

Device name:         /dev/lg_dumplv

Major device number: 10

Minor device number: 11

Size:                174708224 bytes

Uncompressed Size:   1131385360 bytes

Date/Time:           Tue Nov 11 12:56:15 2014

Dump status:         0

Type of dump:        fw-assisted

dump completed successfully

 

...

 

# man snap

 

snap Command

 

Purpose

 

       Gathers system configuration information.

 

Syntax

 

       snap [-@] [ -a ] [-z "product_name=prd_name,..." | "class=myclass,.." | ALL] [-M Timeout][ -A ] [ -b ] [ -B ] [ -c ] [ -C ] [ -D ] [ -f ] [ -F] [ -g ] [ -G ] [ -i ] [ -k ] [ -l ] [ -L ][

       -n ] [ -N ] [ -p ] [ -r ] [ -R  ] [ -s ] [ -S ] [ -t ] [ -T  Filename ] [ -w  ] [ -X ] [ -Y ][ -o OutputDevice ] [ -d Dir ] [ -v Component ] [ -O FileSplitSize ] [ -P Files ] [ script1

       script2 ... | All | file:filepath ]

 

       snap -e [ -m Nodelist ] [ -d Dir ]

 

       snap -z ADD ["product_name=prod_name" "class=myclass" "command_path=/tmp/myprod_myscript -a"]

 

       snap -z DELETE ["product_name=prod_name" ]

 

Description

 

       The snap command gathers system configuration information and compresses the information into a pax file. The file may then be written to a device such as tape or DVD, or transmitted to a

       remote system. The information gathered with the snap command might be required to identify and resolve system problems.

 

...

 

# snap

Usage: snap [-@gFGStnfkDpsAliIbBNLRwhHCZXYM] [ -z "product_name=productname, ..." | "class=classname, ..." | ALL ] [-o outputdevice] [-d dir] [All |

file:filename | scriptname1 [scriptname2] ...]

       snap [-aZ] [-o outputdevice] [-d dir]

       snap [-gGStnfkDpsAliIbBNLRwhHCZY] [-c] [-O splitsize] [-d dir] [All |

file:filename | scriptname1 [scriptname2] ...]

       snap [-o outputdevice] [-d dir]

       snap [-v component]

       snap [-c] [-O splitsize] [-d dir]

       snap [-r] [-d dir]

       snap [-gGS ]

       snap [-gG ]

       snap [-gh ]

       snap [-gS ]

       snap [ -HC | -P files ]

       snap [-g] [-T trcfile]

       snap [-T trcfile]

       snap [All]

       snap [file:filename]

       snap [scriptname1 [scriptname2] ...]

       snap [-e [-m nodenamelist]]

 

                 snap [ -z ADD "product_name=productname" "class=classname" "command_path=Absolutepathofyourdebugcommand" ]

                 snap [ -z DELETE "product_name=yourproductname" ]

                 snap [ -z "product_name=productname, ..." | "class=classname, ..." | ALL ]

                 snap [ -M snap_time_out_value ]

        -@ Gather workload partition information

        -a Gather all information.

        -g Gather general information.

        -T Captures a specific trace (specify full path name of the trcfile).

        -G Include Pd* ODM files in general information.

        -t Gather tcpip information.

        -n Gather nfs information.

        -f Gather file system information.

        -k Gather kernel information.

        -D Gather dump and /unix along with livedump information.

        -D -x Perform Automated Dump Analysis.

        -p Gather printer information.

        -s Gather sna information.

        -S Include security files in general information.

        -A Gather async (tty) information.

        -X Gather X.25 LPP information.

        -Y Gather IB Debug information.

        -i Gather Install information.

        -I Gather PCI-X-SCSI and SISSAS information.

        -l Gather programming language information.

        -b Gather SSA Adapter/Disk information.

        -B Skip SSA adapter dump collection when -b is specified.

        -R Gather SCSI RAID information.

        -L Gather logical volume manager (LVM) information.

        -N Suppress free space checking

        -w Gather workload manager (wlm) information.

        -H Gather general information without platform/scanout data.

        -C Gather just platform/scanout files.

        -P Gather just the specified platform/scanout files.

        -h Gather Hardware information.

 

        -e Gather HACMP information.

        -m node name list [separated by ,] to gather HACMP information.

 

        -c Create snap.pax.Z file.

        -o Send information to removable output device (/dev/rfd0).

        -d Directory to put information (/tmp/ibmsupt).

        -r Remove directory (/tmp/ibmsupt).

 

        -F Gather flash adapter related information.

 

        -v Output component snap file to stdout

            Current component choices are:

                 'async dump dump/autoload filesys general kernel lang printer XS25 sna tcpip install ssa scraid lvm wlm wpars hacmp pcixscsi sissas ctsupt products flash IB'

 

        -z To run third party debug commands

        -M To set the timeout for third party debug commands

           If, used with optoins other than -a/-z its value is ignored

        -Z Prevent dump collection.

 

If you run snap whilst a system dump has been recorded on the dump device (logical volume), then it will be collected and included in the snap data file. If there is a valid dump to collect you’ll see the message highlighted below in the snap output.

 

# snap -ac

Checking space requirement for general information.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. done.

Checking space requirement for tcpip information....................................................... done.

Checking space requirement for kernel information............... done.

Checking space requirement for printer information.... done.

Checking space requirement for dump information........

        Attention:  The dump is compressed, and we were not able to verify it is

        consistent with /unix.  Processing continues.

. done.

Checking space requirement for sna information.../var/sna not found

 done.

Checking space requirement for filesys information.................................. done.

Checking space requirement for async information................ done.

Checking space requirement for lang information.......... done.

...etc...

Gathering caa data

Gathering pcixscsi system information....... done.

Gathering sissas system information...... done.

Gathering IB system information..............................

Creating compressed pax file...

Starting pax/compress process... Please wait... done.

 

-rw-------    1 0        0         238702810 Nov 11 13:20 snap.pax.Z

 

The dump data is saved to /tmp/ibmsupt/dump, as shown below.

 

# cd /tmp/ibmsupt/dump

# ls -ltr

total 374408

-r-xr-xr-x    1 bin      bin         7975536 May 16 22:01 kdb_64

-rw-------    1 root     system         2118 Nov 11 12:57 minidump_last

-rw-------    1 root     system      8977113 Nov 11 13:19 unix.Z

-rw-------    1 root     system        13954 Nov 11 13:19 mdmprpt.out

lrwxrwxrwx    1 root     system            6 Nov 11 13:19 trcdead -> kdb_64

lrwxrwxrwx    1 root     system            6 Nov 11 13:19 livedumpdead -> kdb_64

lrwxrwxrwx    1 root     system            6 Nov 11 13:19 kdb -> kdb_64

lrwxrwxrwx    1 root     system            6 Nov 11 13:19 errdead -> kdb_64

-rw-------    1 root     system          784 Nov 11 13:19 dump.snap

-rw-------    1 root     system    174708224 Nov 11 13:19 dump.BZ

drwx------    2 root     system         4096 Nov 11 13:19 autoload

 

Specifying the –Z flag to snap prevents this data from being collected by snap.

 

# snap -aZc

Checking space requirement for general information............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. done.

Checking space requirement for tcpip information....................................................... done.

Checking space requirement for kernel information............... done.

Checking space requirement for printer information.... done.

Checking space requirement for dump information........ done.

Checking space requirement for sna information.../var/sna not found

 done.

Checking space requirement for filesys information.................................. done.

Checking space requirement for async information................ done.

Checking space requirement for lang information.......... done.

...etc...

Gathering caa data

Gathering pcixscsi system information....... done.

Gathering sissas system information...... done.

Gathering IB system information..............................

Creating compressed pax file...

Starting pax/compress process... Please wait... done.

 

-rw-------    1 0        0          42606819 Nov 11 13:29 snap.pax.Z

 

The snap file is much smaller without the system dump collection.

 

-rw-------    1 0        0   238702810 Nov 11 13:20 snap.pax.Z – includes dump info

-rw-------    1 0        0    42606819 Nov 11 13:29 snap.pax.Z – no dump info

 

If you are on a version of AIX that doesn’t have the snap –Z flag (such as AIX 5.3) then the alternative way to capture snap data without including the dump info is to run snap –a, when that is finished, remove the /tmp/ibmsupt/dump directory and run snap –c to create the snap pax file.

 

More information on snap and AIX system dumps:

 

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds5/snap.htm?lang=en

https://www.ibm.com/developerworks/community/blogs/AIXDownUnder/entry/a_nice_clean_dump2?lang=en

 

 

Here’s some handy tips and tricks for managing WPARs in your AIX environment.

 

1. How to share a global file system with a WPAR.

 

- In the Global.

 

root@750lpar11]/ # uname -W

0

 

- Global File System that you want to share with the WPAR (750wpar2).

 

[root@750lpar11]/ # df -g /cg

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/cglv          1.98      0.30   85%       64     1% /cg

 

[root@750lpar11]/ # lswpar

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar1  A      S     750wpar1  /wpars/750wpar1  yes

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

- Create "namefs" file system from the Global. /cg file system will be mounted under /stuff in the WPAR.

 

[root@750lpar11]/ # crfs -v namefs -A yes -d /cg -m /wpars/750wpar2/stuff -u 750wpar2

 

[root@750lpar11]/ # lsfs /wpars/750wpar2/stuff

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/cg             --         /wpars/750wpar2/stuff  namefs --      --         no   no

 

root@750lpar11]/ # grep -p /wpars/750wpar2/stuff /etc/filesystems

/wpars/750wpar2/stuff:

        dev             = /cg

        vfs             = namefs

        mount           = true

        type            = 750wpar2

        account         = false

 

- Mount the file system, in the Global.

 

[root@750lpar11]/ # mount /wpars/750wpar2/stuff

 

root@750lpar11]/ # mount | grep stuff

         /cg              /wpars/750wpar2/stuff namefs Jun 23 10:27 rw

 

- In the WPAR.

 

[root@750wpar2]/stuff # uname -W

8

[root@750wpar2]/stuff # df -g /stuff

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

Global             1.98      0.30   85%       64     1% /stuff

 

2. Don't remove the ipv6 address from the global LPAR or else you may not be able to start your WPARs.

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6=''

lo0 changed

 

[root@gibopvc1]/ # stopwpar -Fv p8wpar1

Stopping workload partition 'p8wpar1'.

Stopping workload partition subsystem 'cor_p8wpar1'.

0513-044 The cor_p8wpar1 Subsystem was requested to stop.

Shutting down all workload partition processes.

WPAR='p8wpar1' CID=1

sysV mq ID=2097153 key=0x4107001c uid=0 gid=9

sysV sem ID=5242893 key=0x62023457 uid=0 gid=0

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1'.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

startwpar: 0960-231 ATTENTION: '/usr/lib/wpars/loadwpar' failed with return code 13.

startwpar: 0960-244 Error loading workload partition.

Unmounting all workload partition file systems.

Umounting '/wpars/p8wpar1/usr'.

Umounting '/wpars/p8wpar1/opt'.

Umounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Umounting '/wpars/p8wpar1'.

Return Status = FAILURE.

[root@gibopvc1]/ #

 

[root@gibopvc1]/ # chdev -l lo0 -a netaddr6='::1'

lo0 changed

[root@gibopvc1]/ #

[root@gibopvc1]/ #

[root@gibopvc1]/ # startwpar -v p8wpar1

Starting workload partition 'p8wpar1'.

Mounting all workload partition file systems.

Mounting '/wpars/p8wpar1'.

Mounting '/wpars/p8wpar1/etc/objrepos/wboot'.

Mounting '/wpars/p8wpar1/opt'.

Mounting '/wpars/p8wpar1/usr'.

Loading workload partition.

en1 net default: gateway 10.1.1.10

Exporting workload partition devices.

Exporting workload partition kernel extensions.

Starting workload partition subsystem 'cor_p8wpar1'.

0513-059 The cor_p8wpar1 Subsystem has been started. Subsystem PID is 9896106.

Verifying workload partition startup.

Return Status = SUCCESS.

[root@gibopvc1]/ #

 

3. Follow this procedure if you are unable to clogin to a WPAR because root logins are not allowed.

 

[root@750lpar11]/ GLOBAL # uname -W

0

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

Remote logins are not allowed for this account.

 

[root@750lpar11]/ GLOBAL # chroot /wpars/750wpar1 /usr/bin/ksh

750lpar11 : / # chuser rlogin=true root

750lpar11 : / # exit

 

[root@750lpar11]/ GLOBAL # clogin 750wpar1

750wpar1 : / # oslevel -s

5300-12-09-1341

750wpar1 : / # uname -W

4

 

4. How back up your WPAR to a NIM master.

 

Backing up WPAR to NIM master over NFS mounted file system.

--------------------------------------------------------------------------------------

 

[root@750lpar11]/ GLOBAL # mount nimmast:/export/images/wpars /mnt

[root@750lpar11]/ GLOBAL # savewpar -if /mnt/750wpar1-savewpar-image

Creating information file (/image.data) for 750wpar1-savewpar-image.

Creating list of files to back up

Backing up 92766 files......................

92766 of 92766 files backed up (100%)

0512-038 savewpar: Backup Completed Successfully.

 

############################################################################

 

Backing up WPAR with NIM.

--------------------------------------

On the NIM master, check the Global LPAR is already configured as a NIM client.

 

# nim -o showlog 750lpar11

BEGIN:Thu Jan 30 17:25:47 2014:013017254714

Command line is:

/usr/sbin/installp -acNgXY -e /var/adm/ras/nim.installp -f \

/tmp/.workdir.12583118.12255434_1/.genlib.installp.list.12255434-d \

/tmp/_nim_dir_11862198/mnt0

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

openssh.man.en_US           6.0.0.6100      USR         APPLY       SUCCESS

 

END:Thu Jan 30 17:25:51 2014:013017255114

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

Check that the WPAR hostname/IP can be resolved from the NIM master.

 

# host 750wpar2

750wpar2 is 10.1.50.44

 

# host 10.1.50.44

750wpar2 is 10.1.50.44

 

Define the WPAR in the NIM database. Where 750lpar11 is the Global partition NIM client name and 750wpar2 is the WPAR name.

 

# nim -o define -t wpar -a mgmt_profile1="750lpar11 750wpar2" -a if1="find_net 750wpar2 0" 750wpar2

 

Verify that the NIM master can now query the status of the WPAR.

 

# nim -o lswpar 750wpar2

Name      State  Type  Hostname  Directory        RootVG WPAR

--------------------------------------------------------------

750wpar2  A      S     750wpar2  /wpars/750wpar2  yes

 

# lsnim -l 750lpar11

750lpar11:

   class          = machines

   type           = standalone

   connect        = nimsh

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar11 FAFAC00EF002

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

   manages        = 750wpar2

 

# lsnim -l 750wpar2

750wpar2:

   class         = machines

   type          = wpar

   connect       = shell

   platform      = chrp

   if1           = 10_1_50 750wpar2 0

   mgmt_profile1 = 750lpar11 750wpar2

   Cstate        = ready for a NIM operation

   prev_state    = managed system defined but not yet created

   Mstate        = currently running

 

Backup the WPAR from the NIM master.

 

# nim -o define -t savewpar -a server=master -a location=/export/mksysb/cg/750wpar2-savewpar-image -a source=750wpar2 -a mk_image=yes 750wpar2-savewpar-image

 

Creating list of files to back up.

.

Backing up 64560 files..........

 

64560 of 64560 files (100%)

0512-038 savewpar: Backup Completed Successfully.

  

 

5. Moving a Rootvg Versioned Workload Partition (VWPAR) to Another Disk

 

"I tested in the lab, for a versioned wpar you can use it more like a regular LPAR - extend a disk into rootvg, mirror on it, unmirror from the old one, reduce out and rmdev.  No bosboot or bootlist needed.  There is no bootset in a versioned wpar for some reason.”

Example -  5.3 versioned wpar, rootvg with rootvg on hdisk11 from global env.

global environment:
 

add a new disk (hdisk10) to the vwpar:

# chwpar -D devname=hdisk10 rootvg=yes 53rvg

chwpar: 0960-721 The workload partition's rootvg must be extended with the new disk(s). Please run extendvg inside the workload partition prior to stopping or rebooting it.

In the versioned wpar:

# clogin 53rvg
# cfgmgr
# extendvg rootvg hdisk1
# mirrorvg rootvg hdisk1
# unmirrorvg rootvg hdisk0
# reducevg rootvg hdisk0
# rmdev -dl hdisk0
# exit (go back to global env)

Now in the global deallocate the old rootvg disk.

# chwpar -K -D devname=hdisk11 53rvg

Now you can shut down and start the wpar, it will boot from the other disk.

# stopwpar 53rvg
# startwpar 53rvg

Apparently bootset commands don't work with a versioned rootvg wpar:

# lswpar -Br 53rvg
Name   Device Name      Virtual Device  RootVG  Bootset
--------------------------------------------------------

# chwpar -B devname=hdisk10 rootvg=yes 53rvg
chwpar: 0960-808 Cannot add bootsets to versioned workload partitions.
 

 

6.Moving a Rootvg Workload Partition (WPAR) to Another Disk

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024257

 

Technote (FAQ)

 

Question

I have a rootvg WPAR that is on one disk, is there a method to move it to a new disk?

 

Answer

There may be an occasion where you have created a rootvg WPAR on a specific disk and you want to move the entire WPAR to another disk. One example might be that the original disk is from an older storage enclosure, and you wish to move the WPAR to newly purchased storage, connected to the system.

 

You can do this by means of an alternate bootset. Similar to how using the alt_disk_copy command in a global LPAR will create a copy of rootvg on another disk, an alternate bootset is a copy of a WPAR's rootvg on another disk.

 

The example in this technote will use a rootvg wpar that is on a single disk (hdisk11), and has private /opt and /usr filesystems (AKA a "detached" WPAR). This was initially created using these options:

# mkwpar -D devname=hdisk11 rootvg=yes -l -n rootvgwpar
# startwpar rootvgwpar 

1. List the bootsets for the rootvg WPAR (-Br) we are interested in:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk11      hdisk0      yes   0 

So we can see that hdisk11 is being used for the rootvg in the WPAR, as internal hdisk0.

2a. Now we allocate a new unused disk to the WPAR, and set a bootset on it in the same action:

# chwpar -B devname=hdisk9 rootvgwpar
Creating a bootset for WPAR rootvgwpar. Please wait... 

After this is finished we can see that a new bootset has been created on hdisk9:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk1      no   1
rootvgwpar  hdisk11      hdisk0      yes   0 

2b. If we log in to the WPAR we'll see a rootvg and an alternate rootvg:

# clogin rootvgwpar

# lspv
hdisk0     c7a96733e1ebfc      rootvg      active
hdisk1     c7a9673433f0a7      altvg.1 

3. From the global environment again, we can set the bootlist for the WPAR to boot from the bootset on the new disk:

# chwpar -b bootlist=1 rootvgwpar 
check that it was set properly:

# lswpar -b rootvgwpar
Name     Bootlist
---------------------
rootvgwpar ف 

4. Reboot the wpar on the new bootset.

# rebootwpar rootvgwpar

Stopping workload partition rootvgwpar.
Stopping workload partition subsystem cor_rootvgwpar.
0513-044 The cor_rootvgwpar Subsystem was requested to stop.
stopwpar: 0960-261 Waiting up to 600 seconds for workload partition to halt.
Shutting down all workload partition processes.
Unmounting all workload partition file systems.

Starting workload partition rootvgwpar.
Mounting all workload partition file systems.
Loading workload partition.
Exporting workload partition devices.
Exporting workload partition kernel extensions.
Starting workload partition subsystem cor_rootvgwpar.
0513-059 The cor_rootvgwpar Subsystem has been started. Subsystem PID is 8323226.
Verifying workload partition startup.

If we look at the bootsets we'll see that the 2nd disk now has rootvg officially on it:

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes  1
rootvgwpar  hdisk11      hdisk1      no   ـ 

That bootset also believes hdisk9 is "hdisk0" for it, and the other disk is hdisk1. Notice the bootset ID has not changed, bootset 0 is still on (global) disk hdisk11 and bootset 1 on (global) disk hdisk9.

If we log into the WPAR we can see this:

# lspv
hdisk0     c7a9673433f0a7      rootvg      active
hdisk1     c7a96733e1ebfc      altvg.0 

The volume group "altvg.0" is our original rootvg.

5. At this point if everything is looking good, we can remove the original bootset from the global environment:

# chwpar -K -B bootset=0 rootvgwpar

# lswpar -Br rootvgwpar
Name     Device Name    Virtual Device  RootVG  Bootset
-------------------------------------------------------------
rootvgwpar  hdisk9      hdisk0      yes   1

# lswpar -D rootvgwpar | grep disk
rootvgwpar  disk   hdisk9      yes   EXPORTED
rootvgwpar  disk   hdisk11      no    EXPORTED

6. Now we can remove the original rootvg disk:

# chwpar -K -D devname=hdisk11 rootvgwpar 

7. Remember to run cfgmgr to put the disk in an Available state again so it can be used by another WPAR or volume group:

# lsdev -xc disk -l hdisk11
hdisk11 Defined 00-01-02 MPIO IBM 2076 FC Disk

# cfgmgr

# lsdev -xc disk -l hdisk11
hdisk11 Available 00-01-02 MPIO IBM 2076 FC Disk

 

 

 

 

I LPM'ed an AIX LPAR from Frame A to Frame B. The LPAR id before the LPM was 99 and after the LPM it's still 99. And the system id has changed, as expected. Good.

 

Before migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,017268B60 Hardware system identifier False

 

After migration.

 

root@lpar1 / # uname -L

99 lpar1

 

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,0157F5B16 Hardware system identifier False

 

Great, that's exactly what I wanted.

 

Hang on a minute! What kind of sorcery is this?!

 

root@lpar1 / # errpt | grep sys

60AFC9E5   0405132517 I O sys0           Partition ID changed since last boot.

 

root@lpar1 / # errpt -aN sys0

---------------------------------------------------------------------------

LABEL:          PARTID_CHANGE

IDENTIFIER:     60AFC9E5

 

Date/Time:       Wed Apr  5 13:25:53 EET 2017

Sequence Number: 81372

Machine Id:      000157F5B16

Node Id:         lpar1

Class:           O

Type:            INFO

WPAR:            Global

Resource Name:   sys0

 

Description

Partition ID changed since last boot.

 

Whaddaya mean my partiton ID changed since last boot?! It hasn't! I know this for a fact! Look at the uname output above!!

 

Well, I'm 100% correct, the partition id HAS NOT CHANGED! But the system id has....and this is the cause of the "informational" message in the AIX error report.

 

AIX uses the output from 'uname -f' to detect if the system AND/OR partition id have changed since last boot.

 

So, in my case, the partition id (99 dec/63 hex) did not change but (as you'd expect with LPM) the system id (shown in hex below) has changed, hence the message "Partition ID changed since last boot".

 

Before LPM.

 

root@lpar1 / # uname -f

80000BA02DC00063

 

After LPM.

 

root@lpar1 / # uname -f

800016B466F00063

 

From the uname man page:

 

-F Displays a system identification string comprised of hexadecimal characters. This identification string is the same for all partitions on a particular system.

                                                     

-f Similar to the F flag, except that the partition number is also used in the calculation of this string. The resulting identification string is unique for each partition on a particular system.

 

So, no problem here. Nothing to see, nothing to do. Move along! 

 

In a previous post I discussed how you can identify some of the different types of a PowerVM Capacity on Demand (CoD) activation keys from IBM.

 

Recently I had to Activate Memory Expansion (AME) on a couple of POWER7 systems. I discovered that all of the keys contained a similar string. It appears that if a CoD key contains the string CA1F0000000800 then it is safe to assume it will activate AME for a particular system. e.g.  

 

9741EF3AE6969F17CA1F0000000800419D 

937A1240F00F5B05CA1F0000000800413D

 

And while I’m talking about AME, I thought I’d share this tip as well.

 

I was performing a demo of AME for my team and wanted to change the AME expansion factor using DLPAR during the demo. I did not want to use the HMC GUI but rather the HMC command line (as it’s faster).

 

To change the expansion factor for an LPAR (that’s enabled for AME), you can use the chhwres command from the HMC CLI.

 

During the demo I highlighted the current (running) expansion factor for the LPAR (using the lshwres command).

 

hscroot@hmc1:~> lshwres -m 795-1 -r mem --level lpar --filter "lpar_ids=189"

lpar_name=aixlpar1,lpar_id=89,curr_min_mem=1024,curr_mem=4096,curr_max_mem=32768,pend_min_mem=1024,pend_mem=4096,pend_max_mem=32768,run_min_mem=1024,run_mem=4096,curr_min_num_huge_pages=0,curr_num_huge_pages=0,curr_max_num_huge_pages=0,pend_min_num_huge_pages=0,pend_num_huge_pages=0,pend_max_num_huge_pages=0,run_num_huge_pages=0,mem_mode=ded,curr_mem_expansion=2.0,pend_mem_expansion=2.0,curr_hpt_ratio=1:64,curr_bsr_arrays=0

 

Then I increased the expansion factor from 2.0 to 3.0 (using DLPAR).

 

hscroot@hmc1:~> chhwres -r mem -m 795-1 -o s -p aixlpar1 -a "mem_expansion=3"

 

Using the lshwres command I then confirmed that the expansion factor changed (from the HMC command line).

 

hscroot@hmc1:~> lshwres -m 795-1 -r mem --level lpar --filter "lpar_ids=189"

lpar_name=aixlpar1,lpar_id=89,curr_min_mem=1024,curr_mem=4096,curr_max_mem=32768,pend_min_mem=1024,pend_mem=4096,pend_max_mem=32768,run_min_mem=1024,run_mem=4096,curr_min_num_huge_pages=0,curr_num_huge_pages=0,curr_max_num_huge_pages=0,pend_min_num_huge_pages=0,pend_num_huge_pages=0,pend_max_num_huge_pages=0,run_num_huge_pages=0,mem_mode=ded,curr_mem_expansion=3.0,pend_mem_expansion=3.0,curr_hpt_ratio=1:64,curr_bsr_arrays=0

 

I also ran the lparstat command on the LPAR (before and after the DLPAR operation) to demonstrate the changed expansion factor value and the increase in expanded memory size.

 

aixlpar1 : / # lparstat -i | grep Exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 2.00

Target Memory Expansion Size               : 8192 MB

 

aixlpar1 : /# lparstat -i | grep Exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 3.00

Target Memory Expansion Size               : 12288 MB

 

The latest release of the VIOS provides a way to automatically backup your virtual device configuration whenever it changes. From the IBM Knowledge Center for POWER8:

 

https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_viosbr.htm

 

"The viosbr command automatically creates a backup, whenever there are any configuration changes. This functionality is known as the autoviosbr backup. It is triggered every hour, and checks if there are any configuration changes, or any other changes. If it detects any changes, a backup is created. Otherwise, no action is taken. The backup files resulting from the autoviosbr backup are located under the default path /home/padmin/cfgbackups with the names autoviosbr_SSP.<cluster_name>.tar.gz for cluster level and autoviosbr_<hostname>.tar.gz for node level. The cluster-level backup file is present only in the default path of the database node.

 

The -autobackup flag is provided for the autoviosbr backup functionality. By default, autoviosbr backup is enabled on the system. To disable the autoviosbr backup, use the stop parameter and to enable it you can use the start parameter. When the autoviosbr backupis disabled, no autoviosbr related tar.gz file is generated.

 

To check if the autoviosbr backup file, present in the default path is up to date, you can use the status parameter. To access the cluster-level backup file on any node of the cluster, use the save parameter. This action is necessary as the cluster-level backup file is present in the default path of the database node only.

 

If the node is a part of cluster, you can use the -type flag to specify the parameter. The parameter can be either cluster or node, depending on if it is a cluster-level or a node-level backup.

Flags""

 

On your VIOS, under oem_setup_env (as root), you'll find the following new entry in root's crontab:

 

# crontab -l | grep autoviosbr

0 * * * * /usr/ios/sbin/autoviosbr -start 1>/dev/null 2>/dev/null

 

This entry will check for any configuration changes and generate a new backup if necessary. Here's an example from my VIOS, running 2.2.5.10.

 

$ ioslevel

2.2.5.10

 

I create a new virtual optical device, which should trigger a new backup the next time the autoviosbr script runs (once per hour). Prior to creating the device, viosbr shows the autobackup status as Complete (no changes). I ensure that autobackup is configured by stopping and starting it with the viosbr command.

 

$ viosbr -autobackup stop -type node

Autobackup stopped successfully.

 

$ viosbr -autobackup start -type node

Autobackup started successfully.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

$ mkvdev -fbo -vadapter vhost34

vtopt13 Available

 

Immediately after the vtopt device is created, the autobackup status displays as Pending (something has changed but has not yet been backed up).

 

$ viosbr -autobackup status -type node

Node configuration changes:Pending.

 

The autoviosbr file is created in /home/padmin/cfgbackups.

 

$ r oem

oem_setup_env

 

# cd /home/padmin/cfgbackups

# ls -tlr

total 72

-rw-r--r--    1 root     staff         12189 May 10 14:00 autoviosbr_s824vio2.tar.gz

 

On the hour, the autoviosbr script runs, notices that the configuration has changed and generates a new viosbr backup file. The viosbr autobackup status changes to Complete.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

# ls -ltr

total 88

-rw-r--r--    1 root     staff         12218 May 10 15:00 autoviosbr_s824vio2.tar.gz

 

What's inside the tar file?

 

# cp /home/padmin/cfgbackups/autoviosbr_s824vio2.tar.gz /tmp/cg/

# cd /tmp/cg

# gzip -d autoviosbr_s824vio2.tar.gz

# tar -xvf autoviosbr_s824vio2.tar

x ArtexRules_MTM8286-42A02214F58VP2.xml, 20549 bytes, 41 media blocks.

x autoviosbr_s824vio2, 198554 bytes, 388 media blocks.

# ls -ltr

total 888

-rw-r--r--   1 root staff  198554 May 10 15:00 autoviosbr_s824vio2

-rw-r--r--   1 root staff  20549 May 10 15:00 ArtexRules_MTM8286-42A02214F58VP2.xml

-rw-r--r--   1 root staff  225280 May 10 15:00 autoviosbr_s824vio2.tar

 

The ArtexRules file contains device specific configuration attributes.

 

# head ArtexRules_MTM8286-42A02214F58VP2.xml

<?xml version="1.0" encoding="UTF-8"?>

<Profile origin="get" version="3.0.0" date="2017-05-10T04:10:54Z">

 <Catalog id="devParam.disk.fcp.mpioosdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioosdisk"/>

  </Parameter>

 </Catalog>

 <Catalog id="devParam.disk.fcp.mpioapdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioapdisk"/>

#

 

The autoviosbr file contains virtual device configuration information. In this file I can confirm that the configuration of my new virtual optical device (vtopt13) has been backed up.

 

# grep vtopt13 autoviosbr_s824vio2

                <name>vtopt13</name>

                        <vtdName>vtopt13</vtdName>

                        <value>vtopt13</value>

                                          

# vi autoviosbr_s824vio2

...

        <virtDevOPT>

                <name>vtopt13</name>

                <state>AVAILABLE</state>

                <locCode>U8286.42A.214F58V-V2-C101-L2</locCode>

                <unique_type>virtual_target/vtdev/fbopt</unique_type>

                <type>BDOPFBOPT</type>

                <fbOpt>

                </fbOpt>

                <CuAtcount>2</CuAtcount>

                <CuAt name="LogicalUnitAddr" value="0x8200000000000000" action="FALSE" />

                <CuAt name="udid" value="19cc3cb76ad94674c3" action="TRUE" />

        </virtDevOPT>

...

 

If you want to force a backup to run, before the next hour, manually run the following as root (oem_setup_env):

 

$ oem_setup_env

# /usr/ios/sbin/autoviosbr -start

 

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

 

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

        0514-018 The values specified for the following attributes

                 are not valid:

     max_transfer     Maximum TRANSFER Size

 

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth     256                              Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

 

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

 

1. Remove the cd0 device from the LPAR i.e.

   # rmdev –dl cd0

 

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

   $ rmvdev –vtd vtopt0

 

3. Shutdown the LPAR.

 

4. Reboot the VIOS.

 

5. Restart the LPAR.

 

6. Change the max_transfer value for the hdisk on the LPAR.

 

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

 

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 0                                Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x100000                         Maximum TRANSFER Size      True

pvid            00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth     3                                Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

The following explanation was given by IBM support:

 

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

Have you ever encountered this error message on AIX before?

 

# man -k loopmount
man: 0703-310 file man not found.
Create the whatis database using the <catman -w> command

Of course you have! If you’ve ever installed AIX from scratch then you have undoubtedly seen this message.

 

And of course, you dutifully executed the ‘catman –w’ command to rectify the situation. Didn’t you?

But what if  ‘catman –w’  doesn’t work as expected?

 

I recently came across this minor issue. Running catman produced the following error message:

 

# catman -w    

catman: 0703-102 /usr/lbin/mkwhatis: The error number is 1.

 

The first place to check was the MANPATH environment variable (which didn’t look right).

 

# echo $MANPATH

:/opt/csm/man

 

So I unset it and ran catman again. This time, with ‘great success’.

 

# unset MANPATH

# catman -w  

 

Now I could search for keywords against the /usr/share/man/whatis database that contain the string “loopmount”.

 

# man -k loopmount

loopmount(1)    - Associate an image file to a loopback device.

 

Another interesting issue I’ve also come across is when catman runs with apparent success but I still can’t find keyword entries when I run  ‘man –k’.

 

# catman -w

# man -k loopmount

loopmount: Cannot find matching entry.

 

I found that manually setting MANPATH and re-running catman rectifies this situation.

 

# export MANPATH=/usr/share/man

# catman -w

# man -k lsvg

loopmount(1)    - Associate an image file to a loopback device.

 

Of course the whatis command can also perform a similar function. Only difference is that you need to know the command beforehand.

 

# whatis loopmount

loopmount(1)    - Associate an image file to a loopback device.

The catman command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds1/catman.htm

 The whatis command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds6/whatis.htm

Locating a command by keyword:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/HT_locating_command_keyword.htm

The /usr/share directory:

 http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/usr_share_dir.htm

 

 

 

This entry is similar in theme to one of my previous posts about verifying your hdisk queue_depth settings with kdb. This time we want to check if an attribute for a Virtual FC (VFC) adapter has been modified and whether or not AIX has been restarted since the change. The attribute I’m interested in is num_cmd_elems. This value is often changed from its default settings, in AIX environments, to improve I/O performance on SAN attached storage.

 

From kdb you can identify the VFC adapters configured on an AIX system using the vfcs subcommand. Not only does this tell you what adapters you have, but it also identifies the VIOS each adapter is connected to and the corresponding vfchost adapter. Nice!

 

(0)> vfcs

NAME      ADDRESS             STATE   HOST    HOST_ADAP  OPENED NUM_ACTIVE

fcs0      0xF1000A00103D4000  0x0008  vio1    vfchost10  0x01   0x0000

fcs1      0xF1000A00103D6000  0x0008  vio2    vfchost10  0x01   0x0000

 

You can view the current (running) configuration of a VFC adapter using the kdb vfcs subcommand and the name of the VFC adapter, for example fcs1:

 

(0)> vfcs fcs1

adapter: fcs0   partition_num: 0x14     partition_name: aixlpar1

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

state: 0x8  opened: 0x1  flags: 0x1

host_name: vio1       host_device: vfchost10

maxCmds: FA     maxDMALength: 100000    SCSIid: 6F1E9A

portName: C050760399100168      nodeName: C050760399100168      nport_id:

6F1E9A

cmd_q.base_addr: 0xF1000A00102D0000     asyncq.base_addr: 0xF1000A0019761000

crq[0]: 0x8001000000000000      crq[1]: 0x8013960

ctl_head: 0x0   ctl_tail: 0x0

cmd_head: 0x0   cmd_tail: 0x0

pool_head: 0xF1000A001975BD08   pool_tail: 0xF1000A0019759D88

head_active: 0x0        tail_active: 0x0

cancel_head: 0xF1000A0019758090 cancel_tail: 0xF1000A0019758E58

num_cmds_allowed: 0xFA  num_cmds_active: 0x0    num_cmds_pending: 0x0

head_flush_q: 0x0       tail_flush_q: 0x0       cmd_flush_q: 0x0

no_elem: 0x0  no_dma: 0x0  no_sglist: 0x0  bad_mad: 0x0  qfull: 0x0

h_dropped: 0x0  link_down_cnt: 0x0  num_frames: 0x0  tx_lock: 0x0

rcv_lock: 0x0   ctl_lock: 0x0   lock: 0xFFFFFFFFFFFFFFFF

ctl_event: 0xFFFFFFFFFFFFFFFF   open_event: 0xFFFFFFFFFFFFFFFF

reset_cmd: 0x0  log_cmd: 0x0

link_cmd: 0x0   ctl_cmd->cmd_state: 0x0

inp_reqs: 0x12594       out_reqs: 0x1A7FD4      ctrl_reqs: 0x44

inp_bytes: 0x14973F90  out_bytes: 0xEC14EC18  npiv_logout_sent: 0x0

npiv_logout_rcvd: 0x0  act_hi_water: 0x10  pend_hi_water: 0x1

 

Using the output from this command we can determine the current (running) value for a number of VFC attributes, including num_cmd_elems.

 

So I start with an adapter with a num_cmd_elems value of 200. Both the lsattr command and kdb report 200 (C8 in hex) for num_cmd_elems.

 

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 200 Maximum Number of COMMAND Elements True

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

 

I change num_cmd_elems to 400 with chdev –P (remember, the –P flag only updates the AIX ODM, and not the running configuration of the device in the AIX kernel. You must either reboot for this change to take effect or offline & online the device).

 

# chdev -l fcs1 -a num_cmd_elems=400 -P

fcs1 changed

 

Now the lsattr command reports num_cmd_elems is set to 400 in the ODM.

 

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

 

But kdb still has num_cmd_elems set to C8 (200).

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

 

Using this technique I now have a way of checking if an AIX system has been restarted since an attribute was changed on a VFC adapter.

 

At this point I could suggest a reboot of the system. Or I could take the adapter offline and then online for the changes to take effect. For example:

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

 

# rmpath -l hdisk0 -p fscsi1

# rmpath -l hdisk1 -p fscsi1

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Defined hdisk0 fscsi1

Defined hdisk1 fscsi1

 

# rmdev -Rl fcs1

fscsi1 Defined

fcs1 Defined

 

# cfgmgr

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

 

# lsattr -El fcs1 –a num_cmd_elems

num_cmd_elems 400      Maximum Number of COMMAND Elements True

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0x190    location_code: U9119.FHA.87654A1-V20-C10-T1

 

;; 0x190 hex = 400 decimal

 

Please note: Be careful when using the kdb command. If used incorrectly, you can crash an AIX system!

Just say you change the queue_depth on a hdisk with chdev –P. This updates the devices ODM information only, not its running configuration. The new value will take effect next time I reboot the system. So now I have a different queue_depth in the ODM compared to the devices current running config (in the kernel).

 

What if I forget that I’ve made this change to the ODM and forget to reboot the system for many months? Someone complains of an I/O performance issue....I check the queue_depths and find they appear to be set appropriately but I still see disk queue full conditions on my hdisks. But have I rebooted since changing the values?

 

How do I know if the ODM matches the devices running configuration?

 

For example, I start with a queue_depth of 3, which is confirmed by looking at lsattr (ODM) and kdb (running config) output:

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 3 Queue DEPTH True

 

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;                                                   < In Hex.

 

Now I change the queue_depth using chdev –P i.e. only updating the ODM.

 

# chdev -l hdisk6 -a queue_depth=256 -P

hdisk6 changed

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

         

kdb reports that the disks running configuration still has a queue_depth of 3.

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;

 

Now if I varyoff the VG and change the disk queue_depth, both lsattr (ODM) and kdb (the running config) show the same value:

 

 

# umount /test

# varyoffvg testvg

 

# chdev -l hdisk6 -a queue_depth=256

hdisk6 changed

 

# varyonvg testvg

# mount /test

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x100;                                    < In Hex = Dec 256.

 

# echo "ibase=16 ; 100" | bc

256

 

This is one way of checking you’ve rebooted since you changed your queue_depth attributes. I’ve tried this on AIX 6.1 and 7.1 only.

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.kdb/doc/kdb/kdb_pdf.pdf

 

Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:

 

·         Clusterwide event management.

·         Communication and storage events:

o   Node UP and node DOWN

o   Network adapter UP and DOWN

o   Network address change

o   Point-of-contact UP and DOWN

o   Disk UP and DOWN

o   Predefined and user-defined events

·         Clusterwide storage naming service

·         Clusterwide command distribution

·         Clusterwide communication making use of networking and storage connectivity

 

All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.

 

CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.

 

As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.

 

 

So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.

 

What follows are some of the most important “snippets” from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).

 

Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.

 

The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.

 

The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.

 

The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.

 

A multicast address is used for cluster communications between the nodes in the cluster.

These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.

 

Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.

 

When you first configure CAA, the following actions are performed:

 

•             The cluster is created using the mkcluster command.

•             The cluster configuration is written to the raw section of the cluster repository disk.

•             Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.

•             Special volume groups and logical volumes are created on the cluster repository disk.

•             Cluster file systems are created on the special volume group.

•             The cluster repository database is created on both primary and secondary nodes.

•             The cluster repository database is started.

•             Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

•             Storage framework register lists are created on the cluster repository disk.

•             A global device namespace is created and interaction with LVM starts for handling associated volume group events.

•             A clusterwide multicast address is established.

•             The node discovers all of the available communication interfaces.

•             The cluster interface monitoring starts.

•             The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.

•             The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

 

In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.

 

# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02

mkcluster: Cluster shared disks are automatically renamed to names such as

        cldisk1, [cldisk2, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, they will be renamed

        later by the clconfd daemon, when the node is available and the disks

        are not busy.

 

You’ll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I’d run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!

 

This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.

 

7502lp01:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2f73e46e2                    rootvg          active

 

7502lp02:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2895e4cbe                    rootvg          active

 

Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!

 

# lscluster -m

Calling node query for all nodes

Node query number of nodes examined: 2

 

 

        Node name: 7502lp01

        Cluster shorthand id for node: 1

        uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304

        State of node:  UP

        Smoothed rtt to node: 7

        Mean Deviation in network rtt to node: 3

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 1

        Point-of-contact interface & contact state

         en0  UP

 

------------------------------

 

        Node name: 7502lp02

        Cluster shorthand id for node: 2

        uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814

        State of node:  UP  NODE_LOCAL

        Smoothed rtt to node: 0

        Mean Deviation in network rtt to node: 0

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 0

        Point-of-contact interface & contact state

         n/a

 

CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.

 

# lscluster –i –n

Node 7502lp01

Node uuid = 110b2422-7efc-11df-aed7-1612a0003002

Number of interfaces discovered = 1

        Interface number 1 en0

                ifnet type = 6 ndd type = 7

                Mac address length = 6

                Mac address =  16.12.a0.0.30.2

                Smoothed rrt across interface = 7

                Mean Deviation in network rrt across interface = 3

                Probe interval for interface = 100 ms

                ifnet flags for interface = 0x1e080863

                ndd flags for interface = 0x21081b

                Interface state  UP

                Number of regular addresses configured on interface = 1

                IPV4 ADDRESS: 9.3.28.136  broadcast 9.3.28.159  netmask 255.255.255.224

                Number of cluster multicast addresses configured on interface = 1

                IPV4 MULTICAST ADDRESS: 228.8.8.8  broadcast 0.0.0.0  netmask 0.0.0.0

 

CAA tells you what disks are in the cluster plus information on those disks, including the state and type.

 

# lscluster –d

 

Storage Interface Query

Cluster Name:  mycluster

Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002

Number of nodes reporting = 2

Number of nodes expected = 2

Node oscar-dev8.austin.ibm.com

Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593

Number of disk discovered = 2

        cldisk1

          state : UP

          uDid  : 533E3E213600A0B8000475C200000E7114BC825930F1818      FAStT03IB

Mfcp05VDASD03AIXvscsi

             type  : CLUSDISK

          uUid  : 9807465b-3194-f1e5-8a04-1f044bc82593

        hdisk1

          state : UP

          uDid  :

          uUid  : 600a0b80-0047-5d0a-0000-e6094bc82690

          type  : REPDISK

 

You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.

 

# clcmd ps -ef

 

 

-------------------------------

NODE 7502lp01

-------------------------------

     UID     PID         PPID      C    STIME    TTY  TIME CMD

    root       1           0       0   Aug 25      -  0:00 /etc/init

    root  655526           1       0   Aug 25      -  0:18 [cimserve]

    root 1310840           1       0   Aug 25      -  0:00 /usr/ccs/bin/shlap64

    root 2490482           1       0   Aug 25      -  0:42 /usr/sbin/srcmstr

    root 3145874         3801262   0 11:17:13      -  0:00 telnetd -a

    root 3276994         2490482   0   Aug 25      -  0:10 /usr/sbin/snmpd

 

-------------------------------

NODE 7502lp02

-------------------------------

     UID     PID         PPID   C    STIME    TTY  TIME CMD

    root       1           0   1   Aug 28      -  0:00 /etc/init

    root 1441934           1   0   Aug 28      -  0:39 /usr/sbin/syncd 60

    root 1638496           1   0   Aug 28      -  0:00 /usr/ccs/bin/shlap64

    root 2228252     2556054   0   Aug 28      -  0:00 /usr/sbin/hostmibd

   

I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I’m eagerly awaiting the first draft of this book so that I can learn how these two components will “play” together.

 

Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.clusteraware/claware_main.htm

 

 

Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/aix_ev.htm