Chris's AIX Blog

with Tags: nim

Migrating you NIM Master to AIX 7.2 using the VIOS VML

cggibbo Tags: master using sddpcm 7.2 vml chris the nimadm to you migrating gibson nim vios aix 1,801 Views

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

Modified by cggibbo

NIM Master, NIMSH and SSL on AIX 7.1 TL4 SP3

cggibbo Tags: nimsh sp3 nim hang lslpp chris 7.1 ssl tl4 7100-04-03-1642 aix gibson iv93152 -o libssl.so 3,367 Views

Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.

[root@750lpar4]/ # nim -o lslpp 750lpar9

The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:

…

Thu Jan 12 14:31:49 2017 Loading certificates..

Thu Jan 12 14:31:49 2017 Loading private key file..

Thu Jan 12 14:31:49 2017 create BIO

…

NIM master: 750lpar4

7100-04-03-1543

NIM client: 750lpar9

7100-04-02-1614

[root@750lpar4]/ # lsnim -l 750lpar9

750lpar9:

class = machines

type = standalone

connect = nimsh (secure)

platform = chrp

netboot_kernel = 64

if1 = 10_1_50 750lpar9 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F603CD4C00

Cstate_result = success

The root cause of the problem become apparent when we ran truss against the nim –o command.

[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9

[root@750lpar4]/ # cat truss.lsnim.out

…

13959372: C o u l d n o t l o a d m o d u l e / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m e r r o r : N o s u c h

13959372: f i l e o r d i r e c t o r y

…

19267612: C o u l d n o t l o a d m o d u l e / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m e r r o r : N o s u

19267612: c h f i l e o r d i r e c t o r y

The required shared library object files were missing on the NIM master.

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found

We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.

[root@750lpar4]/usr/lib # slibclean

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x 1 root system 724913 Jan 18 09:08 libssl.so

-rwxr-xr-x 1 root system 3031337 Jan 18 09:08 libcrypto.so

After that, the nim –o commands started working again.

[root@750lpar4]/usr/lib # nim -o showlog 750lpar9

HELLO

So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run. But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.

I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.

Here’s one good reason to setup NIMSH over SSL.

NIMSH, SSL and LPM

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en

The following link is a great reference guide for configuring NIMSH over SSL.

NIMSH over SSL

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024615

Modified by cggibbo

Using NIM to distribute files to your AIX servers.

cggibbo Tags: resource files server copy master sync chris blog client aix gibson nim 3 Comments 4,462 Views

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

"Hello Chris,

So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.

So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup

# lsnim -l netbackup
netbackup:
class = resources
type = file_res
Rstate = ready for use
prev_state = unavailable for use
location = /tmp/netbackup
alloc_count = 0
server = master
dest_dir = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on.

Can I actually push files out by themselves? Thanks"

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm

Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really!

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r-- 1 root system 542 Aug 22 09:53 logger.conf
-rw-r--r-- 1 root system 3219 Aug 22 10:41 fugazi.cf

First, I'll create a new file_res NIM resource, called soe_conf.

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf resources file_res

# lsnim -l soe_conf
soe_conf:
class = resources
type = file_res
dest_dir = /usr/local/etc
Rstate = ready for use
prev_state = unavailable for use
location = /usr/local/etc
alloc_count = 0
server = master

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
class = machines
type = standalone
connect = nimsh
file_res = soe_conf
platform = chrp
netboot_kernel = 64
if1 = VLAN50 AIXmig 0
cable_type1 = N/A
Cstate = ready for a NIM operation
prev_state = not running
Mstate = currently running
cpuid = 00F94F584C00
control = master

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

root@MASTER /# nim -o cust AIXmig

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r-- 1 root system 542 Aug 22 09:53 logger.conf
-rw-r--r-- 1 root system 3219 Aug 22 10:41 fugazi.cf

The file_res resource, is automatically deallocated from the NIM client.

root@MASTER / # lsnim -l AIXmig
AIXmig:
class = machines
type = standalone
connect = nimsh
platform = chrp
netboot_kernel = 64
if1 = VLAN50 AIXmig 0
cable_type1 = N/A
Cstate = ready for a NIM operation
prev_state = customization is being performed
Mstate = currently running
cpuid = 00F94F584C00
Cstate_result = success

If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable).

Modified by cggibbo

NIMSH, SSL and LPM.

cggibbo Tags: processing error aix blog encountered on ssl lpm chris "master": nim: gibson nimsh nim 0042-001 4 Comments 8,626 Views

Do you use SSL with nimsh on AIX? No? Well, you might want to consider it. If you regularly use LPM to migrate AIX partitions from one server to another, you may have found that, on occasion, your NIM master has trouble communicating with its NIM clients afterwards. This is by design, as nimsh uses the NIM clients cpuid to authenticate with the NIM master. During an LPM operation, the cpuid of the NIM client changes and its possible the NIM master may reject the client as a result. This problem can occur even when CPU validation is disabled on the NIM master.

In the example below, I’ve LPM’ed a NIM client (750lpar1) to another server. Immediately afterwards, I’m able to execute a NIM command against the NIM client, from the NIM master (750lpar4). At this point the NIM client is configured with standard nimsh authentication i.e. no SSL.

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

[root@750lpar1]/var/adm/ras# echo HELLO > /var/adm/ras/nim.installp

NIM MASTER:

[root@750lpar4]/ # nim -o change -a validate_cpuid=no master

[root@750lpar4]/ # lsnim -l master | grep -i cpu

validate_cpuid = no

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

class = machines

type = standalone

connect = nimsh

platform = chrp

netboot_kernel = 64

if1 = 10_1_50 750lpar1 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F603CD4C00 << Notice that the cpuid is different.

[root@750lpar4]/ # nim -o showlog 750lpar1

HELLO

The cpuid is cached by the nimsh daemon, so the previous system id is retained in memory and passed to the NIM master, which allows the operation to complete successfully. But, if I restart the nimsh daemon, on the NIM client, I find that the NIM master is no longer able to communicate with the client.

[root@750lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

[root@750lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160610.

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

0042-001 nim: processing error encountered on "master":

0042-006 m_mkbosi: (From_Master) connect Error 0

0042-008 nimsh: Request denied - 750lpar4

[root@750lpar4]/ # nim -o showlog 750lpar1

0042-001 nim: processing error encountered on "master":

0042-006 m_showlog: (From_Master) connect Error 0

0042-008 nimsh: Request denied - 750lpar4

[root@750lpar1]/var/adm/ras# grep match nimsh.log | grep CD

Wed Apr 27 20:51:12 2016 error: local value passed, '00F603CD4C00', does not match environment value '00F627664C00'

One way to work around this problem is to use the procedure outlined at the following link:

http://www.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.osdevice/lpm_nim.htm

“Migrating a NIM client by using LPM

When Live Partition Mobility (LPM) is used to move a machine from one physical server to another and the machine is defined as a Network Installation Management (NIM) client, the NIM administrator must update the cpuid attribute for the NIM client to reflect the new hardware value after the LPM migration completes. To update the cpuid attribute, complete the following steps:

On the NIM client, acquire the new cpuid ID by running the following command:

uname –a

On the NIM master, run the following command:

nim -o change -a cpuid=cpuid client”

However, there is a better way. Using nimsh, with SSL-enabled authentication, will prevent the checking of cpuid during nimsh service handling. This is considered the recommended choice of operation (since the client/server can agree upon identity using the certificate information passed during the ssl handshake). Once the certificate is in place, the NIM master will disregard any cpuid validation and instead rely on the success of the SSL-handshake. This configuration will work well with LPM. If using standard nimsh, the limitation of cpuid updating would still apply (because NIM has no way of automatically updating a client once the value has changed).

Both the NIM master and client must be configured to support SSL-enabled authentication. To configure SSL-enabled authentication, we can use ‘nimconfig -c’ on the NIM master.

[root@750lpar4]/ # oslevel -s

7100-04-01-1543

[root@750lpar4]/ # nimconfig -c

0513-029 The tftpd Subsystem is already active.

Multiple instances are not supported.

NIM_MASTER_HOSTNAME=750lpar4

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Target "all" is up to date.

Generating a 1024 bit RSA private key

.................++++++

....................................++++++

writing new private key to '/ssl_nimsh/keys/rootkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

Getting Private key

Generating a 1024 bit RSA private key

...++++++

....++++++

writing new private key to '/ssl_nimsh/keys/clientkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

Generating a 1024 bit RSA private key

.....................++++++

..........++++++

writing new private key to '/ssl_nimsh/keys/serverkey.pem'

-----

Signature ok

subject=/C=US/ST=Texas/L=Austin/O=ibm.com

Getting CA Private Key

[root@750lpar4]/ # lsnim -l | grep -i ssl

ssl_support = yes

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /ssl_nimsh/certs/client.pem

certname= /ssl_nimsh/certs/client.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

[root@750lpar4]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

certname= /tftpboot/server.pem

subject= /C=US/ST=Texas/L=Austin/O=ibm.com

issuer= /C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA

notAfter=Apr 28 10:25:17 2017 GMT

To enable SSL nimsh on the NIM client, we can use the ‘nimclient –c’ command. You can fall-back to non-SSL nimsh, with ‘nimclient –C’.

[root@750lpar1]/# oslevel -s

7100-04-01-1543

[root@750lpar1]/var/adm/ras# nimclient -c

x - /usr/lib/libssl.so.0.9.8

x - /usr/lib/libcrypto.so.0.9.8

Received 2784 Bytes in 0.0 Seconds

0513-044 The nimsh Subsystem was requested to stop.

0513-077 Subsystem has been changed.

0513-059 The nimsh Subsystem has been started. Subsystem PID is 14942422.

[root@750lpar1]/var/adm/ras# cat nimsh.log

Fri Apr 29 23:10:45 2016 /usr/sbin/nimsh: NIM Service Handler started from SRC

Fri Apr 29 23:10:45 2016 no environment value for NIM_SECONDARY_PORT

Fri Apr 29 23:10:45 2016 value for hostname is 750lpar1

Fri Apr 29 23:10:45 2016 value for netaddr is: 10.1.50.31

Fri Apr 29 23:10:45 2016 value for route is net,-hopcount,0,,0 and gateway is 10.1.50.1

Fri Apr 29 23:10:45 2016 value for netif is en0

Fri Apr 29 23:10:45 2016 value for netmask is 255.255.255.0

Fri Apr 29 23:10:45 2016 obtained master's hostname: NIM_MASTER_HOSTNAME=750lpar4

Fri Apr 29 23:10:45 2016 obtained master's id: NIM_MASTERID=00F603CD4C00

Fri Apr 29 23:10:45 2016 value for machine id is 00F603CD4C00

Fri Apr 29 23:10:45 2016 Refreshing archive member /usr/lib/libssl.so.0.9.8 from /usr/lib/libssl.a ..

Fri Apr 29 23:10:45 2016 Refreshing archive member /usr/lib/libcrypto.so.0.9.8 from /usr/lib/libcrypto.a ..

Now I can LPM the NIM client to another server and even if I restart nimsh, the master can still communicate with the client. All thanks to SSL.

NIM MASTER:

[root@750lpar4]/ # lsnim -l 750lpar1

750lpar1:

class = machines

type = standalone

connect = nimsh (secure)

platform = chrp

netboot_kernel = 64

if1 = 10_1_50 750lpar1 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F603CD4C00

NIM CLIENT (AFTER LPM):

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F603CD4C00

[root@750lpar1]/ # uname -a

AIX 750lpar1 1 7 00F627664C00

[root@lpar1]/ # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

[root@lpar1]/ # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 6160616.

[root@750lpar1]/var/adm/ras# tail -f nimsh.log

Fri Apr 29 23:13:33 2016 passing OpenSSL setting of 1

Fri Apr 29 23:13:33 2016 set symbol table

Fri Apr 29 23:13:33 2016 cert filename discovered: /ssl_nimsh/certs/750lpar4.0

Fri Apr 29 23:13:33 2016 ** OpenSSL FIPS mode enabled successfully

Fri Apr 29 23:13:33 2016 seed_prng

Fri Apr 29 23:13:33 2016 Loading certificates..

Fri Apr 29 23:13:33 2016 Loading private key file..

Fri Apr 29 23:13:33 2016 create BIO

Fri Apr 29 23:13:33 2016 - SSL Connection verified successfully -

Fri Apr 29 23:13:33 2016 sending ack to client

Fri Apr 29 23:13:33 2016 setting descriptors to include 2nd port

Fri Apr 29 23:13:33 2016 command to exec __ /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_showlog" "-alocation=/var/adm/ras/nim.installp"

[root@750lpar4]/ nim -o define -t mksysb -a source=750lpar1 -a mksysb_flags=-e -a mk_image=yes -a server=master -a location=/export/mksysb/cg/750lpar1-mksysb 750lpar1-mksysb

+---------------------------------------------------------------------+

System Backup Image Space Information

(Sizes are displayed in 1024-byte blocks.)

+---------------------------------------------------------------------+

Required = 9457741 (9237 MB) Available = 51653640 (50444 MB)

Creating information file (/image.data) for rootvg.

..etc..

Using NIM to install clients configured with SSL authentication

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_install_ssl.htm

Using the certificate viewing file

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/sample_files_certificate.htm%23sample_files_certificate?lang=en

Modified by cggibbo

NIM - "...the resource is currently allocated for client use" - er, NO it's not!

cggibbo Tags: blog currently nim client is 0042-001 /usr/lpp/bos.sysmgt/nim/m... gibson aix for 0042-061 chris use allocated m_chattr 1 Comment 8,228 Views

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

0042-061 m_rmpdir: the "liveupdaterte" resource is currently

allocated for client use

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

# lsnim -Fl liveupdaterte

liveupdaterte:

id = 1447111715

class = resources

type = lpp_source

comments = LIVE

arch = power

Rstate = ready for use

prev_state = verification is being performed

location = /export/nim/cglpp

alloc_count = 1

server = master

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

# lsnim -Fl liveupdaterte

liveupdaterte:

id = 1447111715

class = resources

type = lpp_source

comments = LIVE

arch = power

Rstate = ready for use

prev_state = verification is being performed

location = /export/nim/cglpp

alloc_count = 0

server = master

# nim -o remove liveupdaterte

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

#!/usr/bin/ksh

# Reset a NIM client.

if [[ "$1" = "" ]] ; then

echo Please specify a NIM client to reset e.g. lparaix1nim.

else

if lsnim -l $1 > /dev/null 2>&1 ; then

nim -o reset -F $1

nim -Fo deallocate -a subclass=all $1

nim -Fo change -a cpuid= $1

else

echo Not a valid NIM client!?

Modified by cggibbo

NIM support for installing AIX updates and new packages over HTTP

cggibbo Tags: chris gibson nim blog http aix 7,201 Views

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015). This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs. Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

Major Advantages of using HTTP during NIM Management:

All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
Easy to consume by NIM or other products that currently use the client/server model of NFS.
Able to extend the end-product to support additional protocols (context driven).”

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources. The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901. When active, NIM clients attempt file access and/or scripting customization requests from nimhttp. If http access fails or is denied, a failover attempt at NFS client access occurs. Future support will include options to remove NFS client attempts altogether.”

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user. First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/. This path location is defined as the document_root and cannot be modified at this time. Future enhancements will support multiple document_root paths. The document root path is not limited in depth and may contain many sub-directories. Client requests are able to traverse the path setting by using the enable_directory_listing option. If set to “no”, all files being served must reside in the current working directory of document_root.”

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable. To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server. The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM. Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

# nimconfig –c”

I tested this functionality during the AIX 7.2 Early Ship Program.

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

NIM MASTER:

# lslpp -l | grep nim

bos.sysmgt.nim.master 7.2.0.0 COMMITTED Network Install Manager -

bos.sysmgt.nim.client 7.2.0.0 COMMITTED Network Install Manager –

NIM CLIENT:

# lssrc -s nimsh

Subsystem Group PID Status

nimsh nimclient 6554064 active

# lslpp -l | grep nim

bos.sysmgt.nim.client 7.2.0.0 COMMITTED Network Install Manager -

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

# lssrc -s nimhttp

Subsystem Group PID Status

nimhttp 6685178 active

# ps -ef | grep nimhttp

root 6685178 4194712 0 Nov 10 - 0:00 /usr/sbin/nimhttpd –v

# ls -ltr /httpd.conf

-rw-r--r-- 1 root system 1159 Nov 05 15:31 /httpd.conf

# cat /httpd.conf

#---------------------

# http service defines

#---------------------

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#---------------------

# http configuration

#---------------------

document_root=/export/nim/

# Designates the directory to serve files from.

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

enable_proxy=no

# Enable the web service to act as a proxy server.

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

# lsnim -t lpp_source

lpp_sourceaix72 resources lpp_source

liveupdaterte resources lpp_source

# lsnim -l liveupdaterte

liveu:

class = resources

type = lpp_source

arch = power

Rstate = ready for use

prev_state = unavailable for use

location = /export/nim/cglpp

alloc_count = 0

server = master

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r-- 1 root system 2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r-- 1 root system 688 Oct 31 09:33 .toc

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

# lslpp -l bos.liveupdate.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

bos.liveupdate.rte 7.2.0.0 COMMITTED Live Update Runtime

Path: /etc/objrepos

bos.liveupdate.rte 7.2.0.0 COMMITTED Live Update Runtime

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

# tail -f nimhttp.log

Mon Nov 16 13:51:23 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015 file descriptor is : 6

Mon Nov 16 13:51:23 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015 socks: Request string

Mon Nov 16 13:51:23 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015 socks: Copied string

Mon Nov 16 13:51:23 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015 Request Type is

Mon Nov 16 13:51:23 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015 file descriptor is : 6

Mon Nov 16 13:51:23 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015 socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015 socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015 socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 Request Type is GET

Mon Nov 16 13:51:23 2015 Mime not supported

Mon Nov 16 13:51:23 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015 Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015 Total length sent is 509

Mon Nov 16 13:51:23 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015 Total length sent is 72

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015 Total length sent is 755

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 Total length sent is 2574336

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

“Example: Commands supporting nimhttp

If using push operation (From NIM master)

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

If using pull operation (From NIM client)

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

Modified by cggibbo

AIX Live Update using NIM

cggibbo Tags: lku aix blog chris nimclient nim 7.2 live update gibson 2 Comments 6,126 Views

In my previous post on AIX Live Updates I discussed how to use the geninstall command to perform a non-disruptive (ifix) update on an AIX system. In this post I wanted to show you how to perform the same task using NIM.

NIM can be used to start an AIX Live Update operation on a target machine (NIM client) either from a NIM master or from the NIM client itself (with nimclient).

Note: The AIX Live Update operation started by NIM calls the hmcauth command during the cust operation to authenticate to the NIM client with the HMC by using the HMC passwd file. The NIM master is responsible for obtaining password information from the HMC (using ssh). Without it, NIM clients will not have the password information necessary when running hmcauth as part of the NIM client operation. So, we must first define an hmc object in NIM and create the password file (used when accessing the console.) Once this required step has been completed, all clients using NIM live_update have the ability to pass the proper hmc login credentials when configuring 'hmcauth'.

First, I need to install the dsm.core fileset and configure SSH keys between the NIM master and the HMC.

# lslpp -l dsm.core

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

dsm.core 7.2.0.0 COMMITTED Distributed Systems Management

Core

# dpasswd -f /etc/ibm/sysmgt/dsm/config/my_password_file -U hscroot

# dkeyexch -f /etc/ibm/sysmgt/dsm/config/my_password_file -I hmc -H hsc02

Then define a HMC and Managed System NIM object on the NIM master.

# nim -o define -t hmc -a if1="find_net hsc02 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/my_password_file hsc02

# nim -o define -t cec -a hw_type=8286 -a hw_model=42A -a hw_serial=214F58V -a mgmt_source=hsc02 S824

The NIM client must either be defined with or updated to include the Managed System name (Management Source) and LPAR id number.

# smit nim_chmac

Change/Show Characteristics of a Machine

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

Machine Name [AIXmig]

* Hardware Platform Type [chrp] +

* Kernel to use for Network Boot [64] +

Machine Type standalone

Network Install Machine State currently running

Network Install Control State ready for a NIM operation

Primary Network Install Interface

Network Name net1

Host Name [AIXmig]

Network Adapter Hardware Address [0]

Network Adapter Logical Device Name [ent]

Cable Type N/A +

Network Speed Setting [] +

Network Duplex Setting [] +

IPL ROM Emulation Device [] +/

VLAN Tag Priority (0 to 7) [] #

VLAN Tag Identifier (0 to 4094) [] #

CPU Id [00F94F584C00]

Communication Protocol used by client [nimsh] +

NFS Client Reserved Ports [] +

Comments []

Managing System Information

LPAR Options

Identity [88]

Management Source [S824]

# lsnim -l AIXmig

AIXmig:

class = machines

type = standalone

connect = nimsh

platform = chrp

netboot_kernel = 64

if1 = net1 AIXmig 0

cable_type1 = N/A

mgmt_profile1 = hsc02 88 S824 <<< LPARD id 88, Mgmt Src S824

Cstate = ready for a NIM operation

prev_state = ready for a NIM operation

Mstate = currently running

cpuid = 00F94F584C00

Cstate_result = success

I also need to configure an lpp_source for the ifix location (on the NIM master) and the Live Update data file (on the NIM master). This file can reside on the NIM client if you wish but I’ve chosen to manage all the resources on the NIM master.

# lsnim -t lpp_source

lpp_sourceaix72 resources lpp_source

liveupdatefix resources lpp_source

# lsnim -l liveupdatefix

liveupdatefix:

class = resources

type = lpp_source

arch = power

Rstate = ready for use

prev_state = unavailable for use

location = /nim/lvup/ifix

alloc_count = 0

server = master

# ls -ltr /nim/lvup/ifix

total 72

-rw-r----- 1 root system 35625 Oct 15 14:50 dummy.150813.epkg.Z

# lsnim -t live_update_data

liveupdate_AIXmig resources live_update_data

# lsnim -l liveupdate_AIXmig

liveupdate_AIXmig:

class = resources

type = live_update_data

Rstate = ready for use

prev_state = unavailable for use

location = /nim/lvup/lvupdate.data

alloc_count = 0

server = master

# ls -ltr /nim/lvup/

total 16

drwxr-xr-x 2 root system 256 Oct 15 14:54 ifix

-r--r----- 1 root system 4289 Oct 15 15:04 lvupdate.data

# tail -20 /nim/lvup/lvupdate.data

# Users need not provide redundant options such as "-a -U -C and -o"

# in the trc_option field for trace stanza.

# Do not add a trace stanza to the lvupdate.data file unless you

# want the live update commands to be traced.

general:

mode = automated

kext_check = no

disks:

nhdisk = hdisk0

mhdisk = hdisk1

tohdisk =

tshdisk =

hmc:

lpar_id = 88

management_console = 10.1.50.30

user = hscroot

Now I can perform a preview of the live update operation, from the NIM master. The preview operation will be run on the NIM client called AIXmig.

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 595

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffers.

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.

Validating live update input data.

With the preview looking good, I can perform the actual update, again starting it from the NIM master.

# nim -o cust -a live_update=yes -a live_update_data=liveupdate_AIXmig -a lpp_source=liveupdatefix -a filesets=dummy.150813.epkg.Z AIXmig

Computing the estimated time for the live update operation:

-------------------------------------------------------

LPAR: AIXmig

Mode: F

Blackout_time(s): 72

Global_time(s): 644

Checking mirror vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

Checking new root vg device size:

------------------------------------------

Required device size: 7504 MB

Given device size: 10239 MB

PASSED: device size is sufficient.

Checking temporary paging space device size:

------------------------------------------

Required device size: 512 MB

Checking temporary dump device size:

------------------------------------------

Required device size: 100 MB

Validating the adapters and their paths:

------------------------------------------

PASSED: adapters can be divided into two sets so that each has paths to all disks.

Checking other requirements:

------------------------------------------

PASSED: sufficient space available in /var.

PASSED: sufficient space available in /.

PASSED: no existing altinst_rootvg.

PASSED: rootvg is not part of a snapshot.

PASSED: pkcs11 is not installed.

PASSED: rootvg is not part of a snapshot.

PASSED: The trustchk Trusted Execution Policy is not on.

PASSED: The trustchk Trusted Library Policy is not on.

PASSED: The trustchk TSD_FILES_LOCK policy is not on.

PASSED: the boot disk is set to the current rootvg.

PASSED: the mirrorvg name is available.

PASSED: the rootvg is uniformly mirrored.

PASSED: the rootvg does not have the maximum number of mirror copies.

PASSED: the rootvg does not have stale logical volumes.

PASSED: all of the mounted file systems are of a supported type.

PASSED: this AIX instance is not diskless.

PASSED: no Kerberos configured for NFS mounts.

PASSED: multibos environment not present.

PASSED: Trusted Computing Base not defined.

PASSED: no local tape devices found.

PASSED: live update not executed from console.

PASSED: the execution environment is valid.

PASSED: enough available space for /var to dump Component Trace buffers.

PASSED: enough available space for /var to dump Light weight memory Trace buffer

PASSED: all devices are virtual devices.

PASSED: No active workload partition found.

PASSED: nfs configuration supported.

PASSED: HMC token is present.

PASSED: HMC token is valid.

PASSED: HMC requests successful.

PASSED: Provided LPAR ID is available.

PASSED: A virtual slot is available.

PASSED: RSCT daemons are active.

PASSED: no Kerberos configuration.

PASSED: lpar is not remote restart capable.

PASSED: no virtual log device configured.

PASSED: lpar is using dedicated memory.

PASSED: the disk configuration is supported.

PASSED: no Generic Routing Encapsulation (GRE) tunnel configured.

PASSED: Firmware level is supported.

PASSED: vNIC resources available.

PASSED: Consolidated system trace buffers size is within the limit of 64 MB.

INFO: Any system dumps present in the current dump logical volumes will not be a

Validating live update input data.

Non-interruptable live update operation begins in 10 seconds.

....................................

Initializing live update on original LPAR.

Validating original LPAR environment.

Beginning live update operation on original LPAR.

Requesting resources required for live update.

............

Notifying applications of impending live update.

Creating rootvg for boot of surrogate.

................................................................

Starting the surrogate LPAR.

........................................

Creating mirror of original LPAR's rootvg.

................................................

Moving workload to surrogate LPAR.

....

Blackout Time started.

................................................................................................................................

Blackout Time end.

Workload is running on surrogate LPAR.

....................................

Shutting down the Original LPAR.

................

The live update operation succeeded.

If you want, you could initiate the live update from the NIM client using the nimclient command. All the resources reside on the NIM master, but the NIM client starts the operation, not the NIM master.

# nimclient -o cust -a live_update=yes -a lpp_source=liveupdatefix -a live_update_data=liveupdate_AIXmig -a installp_flags="-p" -a filesets=dummy.150813.epkg.Z

Modified by cggibbo

Restoring a Versioned WPAR with NIM

cggibbo Tags: vwpar gibson chris iv68299 blog wpar mkwpar aix restore volume wparvg group rootvg nim 2 Comments 8,063 Views

I was contacted recently by a customer who was attempting to restore an AIX 5.3 Versioned WPAR (VWPAR) from backup using NIM. The restore worked OK but the data was restored to the wrong volume group!

When the VWPAR was created, the –g option was specified with mkwpar to force the creation of the VWPAR file systems in a separate volume group (named wparvg) rather than the default location of the Global root volume group (rootvg).

# mkwpar -g wparvg -n p8vw2 -B /cg/53gibbo.mksysb -C -O

Running lsvg against wparvg confirmed the file systems were in the right location after creation.

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

fslv02 jfs2 4 4 1 open/syncd /wpars/p8vw2

fslv03 jfs2 2 2 1 open/syncd /wpars/p8vw2/home

fslv04 jfs2 12 12 1 open/syncd /wpars/p8vw2/opt

fslv05 jfs2 6 6 1 open/syncd /wpars/p8vw2/tmp

fslv06 jfs2 56 56 1 open/syncd /wpars/p8vw2/usr

fslv07 jfs2 12 12 1 open/syncd /wpars/p8vw2/var

Before handing the VWPAR over for production use, the customer wanted to ensure they could successfully backup and recover the VWPAR using NIM. First they took a backup of the VWPAR using NIM. From the NIM master, they created a “savewpar backup image”, as shown below.

Note: The WPAR was already defined as a NIM client. Refer to section “12.2 Adding a System WPAR to NIM” in the “Exploiting IBM AIX Workload Partitions (SG24-7955-00)” Redbook for information on how to perform this task.

# smit nim_mkres

savewpar = a workload partition backup image

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [p8vw2-backup]

* Resource Type savewpar

* Server of Resource [master] +

* Location of Resource [/export/vwpar/p8vw2-backup] /

NFS Client Security Method [] +

NFS Version Access [] +

Comments []

Source for Replication [] +

-OR-

System Backup Image Creation Options:

CREATE system backup image? yes +

WPAR to backup [p8vw2] +

....

# lsnim -l p8vw2

p8vw2:

class = machines

type = wpar

connect = shell

platform = chrp

if1 = 10_1_50 p8vw2 0

mgmt_profile1 = gibopvc1 p8vw2

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F94F584C00

Cstate_result = success

# lsnim -t savewpar

p8vw2-backup resources savewpar

In the Global environment, we then stopped and removed the VWPAR (p8vw2).

# stopwpar -F p8vw2

# rmwpar -F p8vw2

Back on the NIM master, we attempted to restore the VWPAR from the recently created backup image (p8vw2-backup).

# smit nim_wpar_create

p8vw2

Create a Managed Workload Partition

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Target Name [p8vw2]

Remain NIM client after install? [yes] +

Specification Resource [] +

WPAR Options

WPAR Name p8vw2

Resource for WPAR Backup Image [p8vw2-backup] +

Resource for System Backup Image [] +

Alternate DEVEXPORTS for installation [] +

Alternate SECATTRS for installation [] +

The restore completed successfully but to our surprise, the VWPAR file systems were in the Global rootvg not wparvg.

# lsvg –l rootvg

rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

hd5 boot 1 1 1 closed/syncd N/A

hd6 paging 8 8 1 open/syncd N/A

hd8 jfs2log 1 1 1 open/syncd N/A

hd4 jfs2 6 6 1 open/syncd /

hd2 jfs2 35 35 1 open/syncd /usr

hd9var jfs2 7 7 1 open/syncd /var

hd3 jfs2 2 2 1 open/syncd /tmp

hd1 jfs2 1 1 1 open/syncd /home

hd10opt jfs2 5 5 1 open/syncd /opt

hd11admin jfs2 2 2 1 open/syncd /admin

lg_dumplv sysdump 16 16 1 closed/syncd N/A

livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump

cglv jfs2 100 100 1 open/syncd /cg

fslv02 jfs2 2 2 1 open/syncd /wpars/p8vw2

fslv03 jfs2 1 1 1 open/syncd /wpars/p8vw2/home

fslv04 jfs2 6 6 1 open/syncd /wpars/p8vw2/opt

fslv05 jfs2 3 3 1 open/syncd /wpars/p8vw2/tmp

fslv06 jfs2 28 28 1 open/syncd /wpars/p8vw2/usr

fslv07 jfs2 6 6 1 open/syncd /wpars/p8vw2/var

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 closed/syncd N/A

We attempted the restore again but this time we explicitly included a WPAR “Specification Resource”. We did this to ensure that the restwpar process was using the correct specification file.

# smit nim_wpar_create

p8vw2

Create a Managed Workload Partition

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Target Name [p8vw2]

Remain NIM client after install? [yes] +

Specification Resource [p8vw2-spec] +

WPAR Options

WPAR Name p8vw2

Resource for WPAR Backup Image [p8vw2-backup] +

Resource for System Backup Image [] +

Alternate DEVEXPORTS for installation [] +

Alternate SECATTRS for installation [] +

We created a WPAR Specification NIM resource. The file was created in the Global environment, using the mkwpar command to write out the VWPAR specification details to a text file. This file was then copied to the NIM master to be used to create the NIM resource.

# mkwpar -e p8vw2 -w -o /tmp/cg/p8vw2_cg.cf

# lsnim -l p8vw2-spec

p8vw2-spec:

class = resources

type = wpar_spec

Rstate = ready for use

prev_state = unavailable for use

location = /tmp/cg/p8vw2_cg.cf

alloc_count = 1

server = master

# lsnim -t wpar_spec

p8vw2-spec resources wpar_spec

The specification file contained the volume group name (wparvg) where each of the VWPAR file systems where located.

# grep vg /tmp/cg/p8vw2_cg.cf

rootvgwpar = "no"

vg = "wparvg"

# grep -p vg /tmp/cg/p8vw2_cg.cf

general:

version = "1"

name = "p8vw2"

hostname = "p8vw2"

checkpointable = "no"

directory = "/wpars/p8vw2"

privateusr = "yes"

uuid = "3e7a2bfb-6060-4770-ad7e-4d6b2a84f657"

devices = "/etc/wpars/devexports"

architecture = "none"

ostype = "1024"

xwparipc = "no"

auto = "no"

rootvgwpar = "no"

preserve = "no"

routing = "no"

mount:

logname = "/dev/loglv00"

directory = "/home"

vfs = "jfs2"

vg = "wparvg"

size = "131072"

mount:

logname = "/dev/loglv00"

mountopts = "rw"

directory = "/opt"

vfs = "jfs2"

vg = "wparvg"

size = "786432"

mount:

logname = "/dev/loglv00"

directory = "/var"

vfs = "jfs2"

vg = "wparvg"

size = "786432"

mount:

logname = "/dev/loglv00"

directory = "/tmp"

vfs = "jfs2"

vg = "wparvg"

size = "393216"

mount:

logname = "/dev/loglv00"

directory = "/"

vfs = "jfs2"

vg = "wparvg"

size = "262144"

mount:

logname = "/dev/loglv00"

mountopts = "rw"

directory = "/usr"

vfs = "jfs2"

vg = "wparvg"

size = "3670016"

However, even with the specification file in place, the result was the same and the VWPAR file systems were created in rootvg rather than wparvg.

We eventually discovered that this behaviour was a bug.

IV68299: RESTWPAR OF VWPAR IS CREATED IN ROOTVG

http://www-01.ibm.com/support/docview.wss?uid=isg1IV68299

Error description

When restoring a backup of a VWPAR that resided in a

non-root volume group, the WPAR is created in rootvg,

even though the image.data file in the backup indicates the

desired volume group.

Local fix

Specify an alternate default volume group using the -M

flag of the restvg command. For example:

# restwpar -f backup_location -M "-g wparvg" -n wpar_name

Problem summary

When restoring a backup of a VWPAR that resided in a

non-root volume group, the WPAR is created in rootvg,

even though the image.data file in the backup indicates the

desired volume group.

Problem conclusion

Added flag in restwpar to call mkwpar with ignore_lvs=no by

default.

Note: Both the Global environment and the NIM master were running AIX 7100-03-04-1441.

We were able to request in ifix from AIX support. Once we installed the ifix, in the Global environment, the restore process via NIM worked as expected and the VWPAR file systems were recovered in wparvg. We did not need to use the WPAR specification NIM resource.

# emgr -l

ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ======================================

1 S IV68299s4a 04/20/15 01:08:20 RESTWPAR VWPAR CREATED IN ROOTVG

STATE codes:

S = STABLE

M = MOUNTED

U = UNMOUNTED

Q = REBOOT REQUIRED

B = BROKEN

I = INSTALLING

R = REMOVING

T = TESTED

P = PATCHED

N = NOT PATCHED

SP = STABLE + PATCHED

SN = STABLE + NOT PATCHED

QP = BOOT IMAGE MODIFIED + PATCHED

QN = BOOT IMAGE MODIFIED + NOT PATCHED

RQ = REMOVING + REBOOT REQUIRED

# diff /usr/emgrdata/efixdata/IV68299s4a/save/EFSAVE1 /usr/sbin/restwpar

459a460

> IGNORE_LVS="ignore_lvs=no"

660a662

> IGNORE_LVS=

664c666

< IMAGE_DATA_SETUP="$IMAGE_DATA_SETUP $SHRINK $IGNORE_MAPS"

---

> IMAGE_DATA_SETUP="$IMAGE_DATA_SETUP $SHRINK $IGNORE_MAPS $IGNORE_LVS"

# lsvg -l rootvg

rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

hd5 boot 1 1 1 closed/syncd N/A

hd6 paging 8 8 1 open/syncd N/A

hd8 jfs2log 1 1 1 open/syncd N/A

hd4 jfs2 6 6 1 open/syncd /

hd2 jfs2 35 35 1 open/syncd /usr

hd9var jfs2 7 7 1 open/syncd /var

hd3 jfs2 2 2 1 open/syncd /tmp

hd1 jfs2 1 1 1 open/syncd /home

hd10opt jfs2 5 5 1 open/syncd /opt

hd11admin jfs2 2 2 1 open/syncd /admin

lg_dumplv sysdump 16 16 1 closed/syncd N/A

livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump

cglv jfs2 100 100 1 open/syncd /cg

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

fslv02 jfs2 4 4 1 open/syncd /wpars/p8wpar2

fslv03 jfs2 2 2 1 open/syncd /wpars/p8wpar2/home

fslv04 jfs2 12 12 1 open/syncd /wpars/p8wpar2/opt

fslv05 jfs2 6 6 1 open/syncd /wpars/p8wpar2/tmp

fslv06 jfs2 56 56 1 open/syncd /wpars/p8wpar2/usr

fslv07 jfs2 12 12 1 open/syncd /wpars/p8wpar2/var

# uname -W

# lswpar

Name State Type Hostname Directory RootVG WPAR

------------------------------------------------------------

p8vw2 A S p8wpar2 /wpars/p8wpar2 no

# clogin p8vw2

*******************************************************************************

* *

* Welcome to AIX Version 5.3! *

* *

* Please see the README file in /usr/lpp/bos for information pertinent to *

* this release of the AIX Operating System. *

* *

*******************************************************************************

# oslevel -s

5300-12-09-1341

Tips and tricks with dsh on AIX

cggibbo Tags: dsh chris gibson there 0042-053 22 no different aix lsnim object dsh_remote_opts dsh_context named port nim is 8 Comments 15,904 Views

I received the following errors whilst running dsh on a NIM master recently.

root@nim1 : / # dsh -waixlpar1 date

0042-053 lsnim: there is no NIM object named "aixlpar1"

The node aixlpar1 is not defined in NIM database.

aixlpar1: Mon Aug 4 14:01:57 EET 2014

I had to set the following environment variable, shown below. By setting DSH_CONTEXT to DSH this prevented the dsh command from referring to the NIM database and instead forced it to query a user-defined node list.

root@nim1 : / # export DSH_CONTEXT=DSH

root@nim1 : / # dsh -waixlpar1 date

aixlpar1: Mon Aug 4 14:02:22 EET 2014

root@nim1 : / # env | grep -i dsh

DSH_CONTEXT=DSH

DSH_NODE_RSH=/usr/bin/ssh

root@nim1 : / # dsh -q

DSH:DCP_DEVICE_OPTS=

DSH:DCP_DEVICE_RCP=

DSH:DCP_NODE_OPTS=-q

DSH:DCP_NODE_RCP=/usr/bin/scp

DSH:DSH_CONTEXT=DSH

DSH:DSH_DEVICE_LIST=

DSH:DSH_DEVICE_OPTS=

DSH:DSH_DEVICE_RCP=

DSH:DSH_DEVICE_RSH=

DSH:DSH_ENVIRONMENT=

DSH:DSH_FANOUT=

DSH:DSH_LOG=

DSH:DSH_NODEGROUP_PATH=

DSH:DSH_NODE_LIST=/usr/local/etc/csmnodes.list

DSH:DSH_NODE_OPTS=

DSH:DSH_NODE_RCP=

DSH:DSH_NODE_RSH=/usr/bin/ssh

DSH:DSH_OUTPUT=

DSH:DSH_PATH=

DSH:DSH_REPORT=

DSH:DSH_SYNTAX=

DSH:DSH_TIMEOUT=

DSH:RSYNC_RSH=

Here’s another dsh tip I picked up. By default dsh will use the default port for ssh connections to nodes. For example, by default sshd listens on port 22 on an AIX node. I recently came across a customer environment where they had configured sshd to listen on port 6666 (not the real port number!). They wanted to use dsh from a NIM master which would connect to all the defined nodes in their custom list. When they ran it they got the following error message:

# dsh date

aixlpar1: ssh: connect to host aixlpar1 port 22: Connection refused

dsh: 2617-009 aixlpar1 remote shell had exit code 255

On the AIX node, we could see that sshd was listening on port 6666:

# netstat -a | grep 6666 | grep LIST

tcp6 0 0 *.6666 *.* LISTEN

tcp4 0 0 *.6666 *.* LISTEN

We needed to find a way to force dsh to use a different port number when starting the ssh connection. This was accomplished by setting the DSH_REMOTE_OPTS variable, as shown below.

[root@nim1]/ # export DSH_REMOTE_OPTS=-p6666

[root@nim1]/ # dsh date

aixlpar1: Tue Aug 5 17:37:16 2014

[root@nim1]/ # env | grep DSH

DSH_REMOTE_CMD=/usr/bin/ssh

DSH_NODE_LIST=/etc/ibm/sysmgt/dsm/nodelist

DSH_REMOTE_OPTS=-p6666

DSH_NODE_RSH=/usr/bin/ssh

DSH CONTEXT

The DSH CONTEXT is the in-built context for all the DSH Utilities commands. It permits a user-defined node group database contained in the local file system. The DSH_NODEGROUP_PATH environment variable specifies the path to the node group database. Each file in this directory represents a node group, and contains one host name or TCP/IP address for each node that is a group member. Blank lines and comment lines beginning with a # symbol are ignored. If all nodes are requested for the DSH CONTEXT, a full node list is built from all groups in the DSH_NODEGROUP_PATH directory, and cached in /var/ibm/sysmgt/dsm/dsh/$DSH_NODEGROUP_PATH/AllNodes. This file is recreated each time a group file is modified or added to the DSH_NODEGROUP_PATH directory. Device targets are not supported in the DSH context.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/dsh.htm?lang=en

DSH_REMOTE_OPTS

Includes the options specified in the remote command when the command is forwarded to the remote nodes.

http://ps-2.kev009.com/wisclibrary/aix52/usr/share/man/info/en_US/a_doc_lib/cmds/aixcmds3/mgmtsvr.htm

Updating your NIM master to AIX 7.1 TL2? Take note!

cggibbo Tags: not nim working tl2 issue chris_gibson 7.1 aix resources de-allocate 7 Comments 20,606 Views

After updating my NIM master to AIX 7.1 TL2 SP1 (7100-02-01-1245), I noticed a problem. Whenever I installed a new AIX partition using NIM, the resources allocated to the NIM client were not being de-allocated, even though the installation was completing successfully. Also, if I tried to run my usual ‘NIM client reset’ script (below), the resources were still allocated.

#!/usr/bin/ksh

# Reset a NIM client.

if [[ "$1" = "" ]] ; then

echo Please specify a NIM client to reset e.g. aixlpar1.

else

if lsnim -l $1 > /dev/null 2>&1 ; then

nim -o reset -F $1

nim -Fo de-allocate -a subclass=all $1

nim -Fo change -a cpuid= $1

else

echo Not a valid NIM client!?

For example, here’s my NIM client with the lpp_source, mksysb and SPOT resources assigned to it (even though the AIX install completed OK).

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

class = machines

type = standalone

connect = shell

platform = chrp

netboot_kernel = 64

if1 = network1 aixlpar1 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

boot = boot

lpp_source = lpp_sourceaix710105

mksysb = aixlpar1-71

nim_script = nim_script

spot = spotaix710105

cpuid = 00C453C75C00

control = master

Cstate_result = success

installed_image = aixlpar1-71

My workaround was to use 'smit nim_mac_res' to manually de-allocate resources from the client:

====

De-allocate Network Install Resources

aixlpar1 machines standalone

>lpp_sourceaix710105 lpp_source

>spotaix710105 spot

>aixlpar1-71 mksysb

====

It appears that others were also experiencing this problem. I found the following thread on the IBM developerWorks AIX user forum:

NIM de-allocate with subclass=all doesn't work:

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=464710

Someone even provided their own version of a ‘NIM client reset’ script that offered another workaround to the problem:

nim -Fo reset $1

for i in bosinst_data fb_script image_data installp_bundle lpp_source mksysb nim_script script spot ; do

ressources_to_clean=$(lsnim -l $1 | grep $i | awk '{print $NF}')

for j in ${ressources_to_clean} ; do

nim -Fo de-allocate -a $i=$j $1

done

I reported the issue to IBM AIX support and discovered that this is a known issue with 7.1 TL2.

“This is a known issue which recently has been documented in 2 open

APARs: IV32570 and IV32670: UPDATE NIM BINARIES FOR UPDATES.

Error description

One known symptom (of this issue) is that "nim -Fo

de-allocate -a subclass=all <client>" returns successfully but the

resource will still be allocated.

The problem occured because the "update" pack that brings you from

7.1.1.x to 7.1.2.0 missed a piece that effected the bos.sysmgt.7.1.2.0

fileset.

Currently there is no fix available and no projected timeframe when this

issue will be fixed (i assume it will be with the next SP for AIX

7100-02)

However, a workaround is allready documented in our incidents

database - you simply need to force reinstall the bos.sysmgt.nim.master

7.1.2.0 fileset using your AIX 7.1 TL02 media.

This is not a backleveling of the fileset, you are simply force

overwriting the full base level at the same version that you are

currently at. It will still be at 7100-02.

The workaround is a force overwrite of the bos.sysmgt.nim.master 7.1.2.0

from media."

The AIX support team suggested that I download the necessary install fileset from the following location:

ftp://testcase.software.ibm.com/fromibm/aix/bos.sysmgt.7.1.2.0.I

I backed up my NIM database before re-installing the fileset, just in case.

---------------------------------------------------------

Starting NIM Database Backup on nim1 at Wed Dec 12 12:39:43 EETDT 2012

a ./etc/objrepos/nim_attr 32 blocks.

a ./etc/objrepos/nim_attr.vc 112 blocks.

a ./etc/objrepos/nim_object 8 blocks.

a ./etc/objrepos/nim_object.vc 24 blocks.

a ./etc/NIM.level 1 blocks.

a ./etc/niminfo 1 blocks.

a ./etc/NIM.primary.cpuid 1 blocks.

Finished NIM Database Backup on nim1 at Wed Dec 12 12:39:44 EETDT 2012

---------------------------------------------------------

I placed the fileset into a separate directory on the NIM master and ran inutoc.

root@nim1 : /home/cgibson/NIMPMR # ls -ltr

total 36170

-rw-r--r-- 1 cgibson staff 18512896 Dec 12 12:21 bos.sysmgt.7.1.2.0.I

-rw-r--r-- 1 root system 4778 Dec 12 12:35 .toc

Then I re-installed the fileset (force OVERWRITE) on the NIM master:

Install Software

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* INPUT device / directory for software .

* SOFTWARE to install [@ 7.1.2.0 Network Install Manager - Master Tools @@I:bos.sysmgt.nim.master 7>

PREVIEW only? (install operation will NOT occur) no

COMMIT software updates? yes

SAVE replaced files? no

AUTOMATICALLY install requisite software? no

EXTEND file systems if space needed? yes

OVERWRITE same or newer versions? yes

VERIFY install and check file sizes? no

Include corresponding LANGUAGE filesets? yes

DETAILED output? yes

Process multiple volumes? yes

ACCEPT new license agreements? yes

Preview new LICENSE agreements? no

The fileset re-installed without a problem.

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

bos.sysmgt.nim.master 7.1.2.0 USR APPLY SUCCESS

root@nim1 : /home/cgibson/NIMPMR # lslpp -l bos.sysmgt.nim.master

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

bos.sysmgt.nim.master 7.1.2.0 COMMITTED Network Install Manager -

Master Tools

I re-tried installing AIX on a NIM client.

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

class = machines

type = standalone

connect = shell

platform = chrp

netboot_kernel = 64

if1 = network1 aixlpar1 0

cable_type1 = N/A

Cstate = Base Operating System installation is being performed

prev_state = BOS installation has been enabled

Mstate = in the process of booting

info = verifying_data_files

boot = boot

lpp_source = lpp_sourceaix710105

mksysb = aixlpar1-71

nim_script = nim_script

spot = spotaix710105

cpuid = 00C453C75C00

control = master

Cstate_result = success

installed_image = aixlpar1-71

Once the install completed I found that all allocated NIM resources (lpp_source, mksysb and SPOT) were de-allocated. Problem solved.

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

class = machines

type = standalone

connect = shell

platform = chrp

netboot_kernel = 64

if1 = network1 aixlpar1 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = in the process of booting

Mstate = not running

cpuid = 00C453C75C00

Cstate_result = success

installed_image = aixlpar1-71

So if you are planning on updating your NIM master to 7.1 TL2, please take note and make sure you implement the workaround.

SSH Host keys and AIX 7.1 Migration.

cggibbo Tags: has warning: host aix remote identification nim changed lppsource migration ssh 6 Comments 35,668 Views

Recent releases of AIX installation media (for 7.1 and 6.1) now contain the OpenSSH base installation filesets. This is very handy; we no longer need to download or locate the software from other sources.

One thing to consider is what this means for future AIX migrations.

If you are migrating a system (that already has a version of SSH installed) to AIX 7.1 then you may notice that the first time you attempt to connect to the server (after the 7.1 migration) the following ssh message appears:

root@nim1 : / # ssh aixlpar1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

59:68:05:71:60:b5:d1:96:87:df:f6:9c:ca:9a:14:3e.

Please contact your system administrator.

Add correct host key in /.ssh/known_hosts to get rid of this message.

Offending RSA key in /.ssh/known_hosts:17

RSA host key for aixlpar1 has changed and you have requested strict checking.

Host key verification failed.

In the output above I’m attempting to SSH from another system to the newly migrated AIX 7.1 LPAR. This is essentially informing us that the SSH host keys on the AIX 7.1 server don’t match the host key stored in the local systems /.ssh/known_hosts file. Something has changed.

Now of course I could simply accept this change and update my known_hosts files, like so:

root@nim1 : / # ssh-keygen -R aixlpar1

/.ssh/known_hosts updated.

Original contents retained as /.ssh/known_hosts.old

With known_hosts updated, I’m able to SSH to the AIX 7.1 system successfully.

cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date

Mon Aug 20 19:44:20 EET 2012

But that’s just for my SSH known_hosts file only. What about all the users that connect to this system via SSH/SFTP/SCP? Do I really expect all of them to update their known_hosts file with the new host key information?

This could create problems for automated tasks, like file transfers. If these transfers stop working then their could be “hell to pay”. So the question I’m often asked is what can I do to prevent this from happening in the first place? Luckily there is a way.

In this example, we are using nimadm to migrate from AIX 5.3 to 7.1. The AIX 7.1 lpp_source resource was created using the AIX 7.1 installation media DVDs. All filesets were copied from the DVDs, verbatim, to the new 7.1 lpp_source resource on the NIM master.

First we verify that the openssh* filesets are in fact in the AIX 7.1 lpp_source on the NIM master.

root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh

openssh.base.client 5.4.0.6100 I N usr,root

openssh.base.client 5.8.0.6101 I N usr,root

openssh.base.server 5.4.0.6100 I N usr,root

openssh.base.server 5.8.0.6101 I N usr,root

openssh.man.en_US 5.4.0.6100 I N usr

openssh.man.en_US 5.8.0.6101 I N usr

openssh.msg.EN_US 5.8.0.6101 I N usr

openssh.msg.en_US 5.4.0.6100 I N usr

openssh.msg.en_US 5.8.0.6101 I N usr

On the NIM client (running AIX 5.3), we verify there is an older version of SSH already installed. The migration will remove these filesets (and the associated /etc/ssh_host_* files). The newer version of SSH will be installed and new ssh_host_key* files will be generated (hence the problem with the remote SSH clients known_hosts files no longer holding the correct host keys).

aixlpar1 : / # oslevel -s

5300-12-04-1119

aixlpar1 : / # lslpp -l openssh*

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

openssh.base.client 4.3.0.5301 COMMITTED Open Secure Shell Commands

openssh.base.server 4.3.0.5301 COMMITTED Open Secure Shell Server

openssh.license 4.3.0.5301 COMMITTED Open Secure Shell License

openssh.man.en_US 4.3.0.5301 COMMITTED Open Secure Shell

Documentation - U.S. English

The following SSH files are regenerated during the new fileset install:

aixlpar1 : /home/cgibson/ssh_orig # ls -ltr /etc/ssh/ssh_host*

-rw-r--r-- 1 root system 210 Feb 03 2006 /etc/ssh/ssh_host_rsa_key.pub

-rw------- 1 root system 887 Feb 03 2006 /etc/ssh/ssh_host_rsa_key

-rw-r--r-- 1 root system 319 Feb 03 2006 /etc/ssh/ssh_host_key.pub

-rw------- 1 root system 515 Feb 03 2006 /etc/ssh/ssh_host_key

-rw-r--r-- 1 root system 590 Feb 03 2006 /etc/ssh/ssh_host_dsa_key.pub

-rw------- 1 root system 668 Feb 03 2006 /etc/ssh/ssh_host_dsa_key

You can confirm the filesets are being installed during the nimadm operation by reviewing your clients nimadm log file in /var/adm/ras/alt_mig:

root@nim1 : /var/adm/ras/alt_mig # grep –i openssh aixlpar1_alt_mig.log

openssh.base.client 5.8.0.6101 USR APPLY SUCCESS

openssh.base.server 5.8.0.6101 USR APPLY SUCCESS

openssh.base.client 5.8.0.6101 ROOT APPLY SUCCESS

openssh.base.server 5.8.0.6101 ROOT APPLY SUCCESS

openssh.man.en_US 5.8.0.6101 USR APPLY SUCCESS

openssh.msg.en_US 5.8.0.6101 USR APPLY SUCCESS

aixlpar1 : / # ls -ltr /etc/ssh

total 352

-rw-r--r-- 1 root system 210 Feb 03 2006 ssh_host_rsa_key.pub

-rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key

-rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub

-rw------- 1 root system 515 Feb 03 2006 ssh_host_key

-rw-r--r-- 1 root system 590 Feb 03 2006 ssh_host_dsa_key.pub

-rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key

-rw-r--r-- 1 root system 2341 Apr 26 2007 ssh_prng_cmds

-rw------- 1 root system 132839 Apr 26 2007 moduli

-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config

-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner

-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config

-rw-r----- 1 root system 7 Aug 20 16:10 sshd.pid

To work around this issue, we can move the openssh* filesets out of the lpp_source directory (installp/ppc), like so:

root@nim1 : / # lsnim -l lpp_sourceaix710101

lpp_sourceaix710101:

class = resources

type = lpp_source

arch = power

Rstate = ready for use

prev_state = ready for use

location = /export/lpp_source/lpp_sourceaix710101

simages = yes

alloc_count = 0

server = master

root@nim1 : / # cd /export/lpp_source/lpp_sourceaix710101

root@nim1 : /export/lpp_source/lpp_sourceaix710101 # find . | grep -i openssh

./installp/ppc/openssh.base

./installp/ppc/openssh.man.en_US

./installp/ppc/openssh.msg.en_US

./installp/ppc/openssh.msg.en_US.5.8.0.6101.I

./installp/ppc/openssh.msg.EN_US.5.8.0.6101.I

./installp/ppc/openssh.man.en_US.5.8.0.6101.I

./installp/ppc/openssh.base.5.8.0.6101.I

root@nim1 : /export/lpp_source/lpp_sourceaix710101 # cd ./installp/ppc/

root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # ls -ltr openssh*

-rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US

-rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US

-rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base

-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.en_US.5.8.0.6101.I

-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.EN_US.5.8.0.6101.I

-rw-r----- 1 root system 153600 Nov 16 2011 openssh.man.en_US.5.8.0.6101.I

-rw-r----- 1 root system 6298624 Nov 16 2011 openssh.base.5.8.0.6101.I

I’ve chosen to move the filesets to a different directory (notice I’ve created a .toc file in the new location). I also need to copy (not move) the openssl* filesets to the new location as well.

root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # mkdir /export/lpp_source/lpp_sourceaix710101/ssh

root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # mv openssh* /export/lpp_source/lpp_sourceaix710101/ssh

root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # cp -p openssl* /export/lpp_source/lpp_sourceaix710101/ssh

root@nim1 : / # cd /export/lpp_source/lpp_sourceaix710101/ssh

root@nim1 : /export/lpp_source/lpp_sourceaix710101/ssh # inutoc .

root@nim1 : /export/lpp_source/lpp_sourceaix710101/ssh # ls -ltr

total 58648

-rw-r----- 1 root system 6298624 Nov 16 2011 openssl.base

-rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US

-rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US

-rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base

-rw-r----- 1 root system 2350080 Nov 16 2011 openssl.man.en_US

-rw-r----- 1 root system 28672 Nov 16 2011 openssl.license

-rw-r----- 1 root system 2350080 Nov 16 2011 openssl.man.en_US.0.9.8.1800.I

-rw-r----- 1 root system 3072 Nov 16 2011 openssl.license.0.9.8.1800.I

-rw-r----- 1 root system 6307840 Nov 16 2011 openssl.base.0.9.8.1800.I

-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.en_US.5.8.0.6101.I

-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.EN_US.5.8.0.6101.I

-rw-r----- 1 root system 153600 Nov 16 2011 openssh.man.en_US.5.8.0.6101.I

-rw-r----- 1 root system 6298624 Nov 16 2011 openssh.base.5.8.0.6101.I

-rw-r--r-- 1 root system 4966 Aug 20 20:41 .toc

Now that openssh is no longer in the lppsource, I check the AIX 7.1 lpp_source and update its TOC:

root@nim1 : / # nim -o check lpp_sourceaix710101

The openssh filesets are no longer listed as part of the lppsource:

root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh

root@nim1 : / #

At this point we perform a standard nimadm operation to migrate from AIX 5.3 to 7.1. Openssh is not installed during the migration:

root@nim1: /var/adm/ras/alt_mig # grep -i openssh aixlpar1_alt_mig.log

root@nim1: /var/adm/ras/alt_mig #

After the migration, ssh connections to the server function just as they did before the migration.

cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date

Mon Aug 20 20:36:31 EETDT 2012

Now we can update the openssh filesets to the newer level. This will install the newer filesets but preserve the existing SSH host keys.

aixlpar1 : / # mount nim1:/export/lpp_source/lpp_sourceaix710101/ssh /mnt

aixlpar1 : / # installp -Y -a -d /mnt ALL

...etc...

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

openssl.man.en_US 0.9.8.1800 USR APPLY SUCCESS

openssl.license 0.9.8.1800 USR APPLY SUCCESS

openssl.base 0.9.8.1800 USR APPLY SUCCESS

openssl.base 0.9.8.1800 ROOT APPLY SUCCESS

openssh.base.client 5.8.0.6101 USR APPLY SUCCESS

openssh.base.server 5.8.0.6101 USR APPLY SUCCESS

openssh.base.client 5.8.0.6101 ROOT APPLY SUCCESS

openssh.base.server 5.8.0.6101 ROOT APPLY SUCCESS

openssh.msg.en_US 5.8.0.6101 USR APPLY SUCCESS

openssh.msg.EN_US 5.8.0.6101 USR APPLY SUCCESS

openssh.man.en_US 5.8.0.6101 USR APPLY SUCCESS

Rather than update these filesets manually after the migration, you can include this step as a post migration task with nimadm.

An alternative way to work around this problem (after the fact) would be to restore the original ssh_host_key* files from a backup. For example, I copied the original ssh_host_key* files to my home directory before starting the AIX migration.

aixlpar1 : / # cd /etc

aixlpar1 : /etc # cp -pr ssh /home/cgibson/ssh_orig/

In the output below, I discover that my ssh_host_key* files have all been recreated during the migration.

aixlpar1 : /etc/ssh # ls -ltr

total 352

-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config

-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner

-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config

-rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid

-rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds

-rw------- 1 root system 132839 Aug 20 21:19 moduli

-rw-r----- 1 root system 382 Aug 20 21:45 ssh_host_rsa_key.pub

-rw------- 1 root system 1679 Aug 20 21:45 ssh_host_rsa_key

-rw-r----- 1 root system 630 Aug 20 21:45 ssh_host_key.pub

-rw------- 1 root system 965 Aug 20 21:45 ssh_host_key

-rw-r----- 1 root system 590 Aug 20 21:45 ssh_host_dsa_key.pub

-rw------- 1 root system 668 Aug 20 21:45 ssh_host_dsa_key

I copy the original files back to the /etc/ssh directory. The sshd subsystem is also restarted to pick up the updated ssh_host* files.

aixlpar1 : /etc/ssh # cp -p /home/cgibson/ssh_orig/ssh_host_* .

aixlpar1 : /etc/ssh # ls -ltr

total 352

-rw-r--r-- 1 root system 210 Feb 03 2006 ssh_host_rsa_key.pub

-rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key

-rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub

-rw------- 1 root system 515 Feb 03 2006 ssh_host_key

-rw-r--r-- 1 root system 590 Feb 03 2006 ssh_host_dsa_key.pub

-rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key

-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config

-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner

-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config

-rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid

-rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds

-rw------- 1 root system 132839 Aug 20 21:19 moduli

aixlpar1 : /etc/ssh # stopsrc -s sshd

0513-044 The sshd Subsystem was requested to stop.

aixlpar1 : /etc/ssh # startsrc -s sshd

0513-059 The sshd Subsystem has been started. Subsystem PID is 3997822.

Using a post-migration script with nimadm

cggibbo Tags: script aix nim post-migration chris_gibson nimadm 3 Comments 20,687 Views

I’ve received a couple of requests for an example of using a post migration script with nimadm. What follows is a simple example of using such a resource with NIM. If you are not familiar with the nimadm tool then perhaps you’d like to start first by reading my article on using nimadm to migrate to AIX 6.1.

The nimadm utility can perform both pre and post migration tasks. This is accomplished by running NIM scripts either before or after a migration. The tool accepts the following flags for pre and post migration script resources:

-a PreMigrationScript Specifies the pre-migration NIM script resource.

-z PostMigrationScript Specifies the post-migration NIM script resource.

pre-migration

This script resource that is run on the NIM master, but in the environment of the client's alt_inst file system that is mounted on the master (this is done by using the chroot command). This script is run before the migration begins.

post-migration

This script resource is similar to the pre-migration script, but it is executed after the migration is complete.

We are going to focus on post-migration only, although the configuration is the same for both.

In this example I need to uninstall and install a 3^rd party device fileset for a storage device. I need to perform this task as part of the migration process. To protect the innocent, I have not named the storage vendor in this post. But I will say that it was not IBM storage we are dealing with in this case.

Before we start, first we collect all the necessary device filesets that provide support for this type of storage on AIX. We place them into a local directory on my NIM master. Along with the software, I also place a copy of my NIM script in the same directory on the NIM master. The script name is XYZpost.ksh.

root@nim1 : /usr/local/XYZ # ls -ltr

total 544

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1001I

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1002U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1003U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1004U

-r-xr-xr-x 1 root system 51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x 1 root system 715 May 24 16:57 XYZpost.ksh

-rw-r--r-- 1 root system 2310 May 25 14:57 .toc

The contents of my script are simple. This script will de-install the old device fileset and then immediately install the latest version of the VendorXYZ’s device fileset. The script will then change the attributes for the vendor’s storage to more appropriate default values.

#!/usr/bin/ksh

echo “Uninstalling XYZ fileset: devices.fcp.disk.VendorXYZ.mpio.rte.”

installp -u devices.fcp.disk.VendorXYZ.mpio.rte

echo Return code is: $?

echo echo “Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte.”

cd /usr/local/XYZ/

installp -aXd . devices.fcp.disk.VendorXYZ.mpio.rte

echo Return code is: $?

echo “Changing default ODM (PdAt) attributes for VendorXYZ storage.”

set -x

odmget -q uniquetype="disk/fcp/xyzmpio and attribute=queue_depth" PdAt |sed s/deflt\ =\ \"2\"/deflt\ =\ \"32\"/g >/usr/local/XYZ/chg_pdat_qd

odmchange -o PdAt -q uniquetype="disk/fcp/xyzmpio and attribute=queue_depth" /usr/local/XYZ/chg_pdat_qd

odmget -q uniquetype="disk/fcp/xyzmpio and attribute=reserve_policy" PdAt |sed s/deflt\ =\ \"single_path\"/deflt\ =\ \"no_reserve\"/g >/usr/local/XYZ/chg_pdat_rp

odmchange -o PdAt -q uniquetype="disk/fcp/xyzmpio and attribute=reserve_policy" /usr/local/XYZ/chg_pdat_rp

odmget -q uniquetype="PCM/friend/VendorXYZ and attribute=algorithm" PdAt |sed s/deflt\ =\ \"fail_over\"/deflt\ =\ \"round_robin\"/g >/usr/local/XYZ/chg_pdat_al

odmchange -o PdAt -q uniquetype="PCM/friend/VendorXYZ and attribute=algorithm" /usr/local/XYZ/chg_pdat_al

set +x

cd /

At this point I copy the same directory and all of its contents to the NIM client.

root@nim1 : /usr/local # scp –pr XYZ lparaix01:/usr/local/

…etc…

lparaix01 : /usr/local/XYZ # ls -ltr

total 0

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1001I

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1002U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1003U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1004U

-r-xr-xr-x 1 root system 51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x 1 root system 715 May 24 16:57 XYZpost.ksh

-rw-r--r-- 1 root system 2310 May 25 14:57 .toc

Make sure that any scripts you write for use with nimadm start with an appropriate ‘hashbang’ to announce it is a shell script and the shell that must be used to execute it e.g. #!/usr/bin/ksh. If you forget to do this nimadm will fail to execute your script and will report an error message similar to the following:

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

/lparaix01_alt/alt_inst/tmp/.alt_mig_chroot_script.11731036: Cannot run a file that does not have a valid format.

The next step is to define the script as a NIM resource so that nimadm can call the resource during the migration process. I’ve decided to call this new NIM resource, XYZPOST.

This is easily achieved using smit nim_mkres:

root@nim1 : / # smit nim_mkres

| script = an executable file which is executed on a client |

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [XYZPOST]

* Resource Type script

* Server of Resource [master] +

* Location of Resource [/usr/local/XYZ/XYZpost.ksh] /

We can confirm that the NIM script resource is now available using the lsnim command.

root@nim1 : / # lsnim -t script

XYZPOST resources script

root@nim1 : / # lsnim -l XYZPOST

XYZPOST:

class = resources

type = script

Rstate = ready for use

prev_state = unavailable for use

location = /usr/local/XYZ/XYZpost.ksh

alloc_count = 0

server = master

Now that the script is in place, and defined to NIM, we are ready to test it. We will migrate the system from AIX 5.3 to AIX 6.1 using nimadm. Once the migration phase is complete (phases 1 to 6), the post-migration script will be executed in the NIM clients nimadm (chroot) environment on the NIM master. Once this is finished the NIM clients data is synced back to the NIM clients alternate disk and the boot image is created. The migration process is then complete.

We add the –z flag to our nimadm command line options to specify the post migration resource.

In normal operation we would simply let nimadm run all phases in sequence with the following command.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -Y

In our case we would like to test the post-migration script phase as a separate step.

We will start by running only the first 6 phases. We will stop at the completion of phase 6 and run phase 7 (post-migration script) manually, before moving on to phases 8 through 12.

You’ll observe that we have specified the –z option with the name of the NIM script resource and the –P flag with phases 1 to 6 only.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P1,2,3,4,5,6 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 1.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P1 -d "hdisk2"

Calling mkszfile to create new /image.data file.

...etc...

Filesets processed: 1 of 2 (Total time: 3 secs).

installp: COMMITTING software for:

X11.compat.lib.X11R6_motif 6.1.6.15

Finished processing all filesets. (Total time: 4 secs).

...etc...

install_all_updates: Checking for recommended maintenance level 6100-06.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-06

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

Known Recommended Maintenance Levels

------------------------------------

Restoring device ODM database.

root@nim1 : / #

Phase 6 has completed successfully. The NIM clients rootvg data has been migrated from AIX 5.3 to 6.1 on the NIM master. The data has not yet been synced back to the NIM client.

At this stage we can now run phase 7 separately and ensure that it performs the required task. We expect it will de-install the device fileset, install the latest version and change the ODM default attributes for the device type. Again, you’ll notice that we specify the –P flag for phase 7 only.

Let’s test it.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P7 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

Uninstalling XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

Pre-deinstall Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

FILESET STATISTICS

------------------

1 Selected to be deinstalled, of which:

1 Passed pre-deinstall verification

----

1 Total to be deinstalled

SUCCESSES

---------

Filesets listed in this section passed pre-deinstall verification

and will be removed.

Selected Filesets

-----------------

devices.fcp.disk.VendorXYZ.mpio.rte 1.2.0.0 # AIX MPIO Support for VendorXYZ Disk

<< End of Success Section >>

+-----------------------------------------------------------------------------+

Deinstalling Software...

+-----------------------------------------------------------------------------+

installp: DEINSTALLING software for:

devices.fcp.disk.VendorXYZ.mpio.rte 1.2.0.0

...etc...

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.2.0.0 USR DEINSTALL SUCCESS

Return code is: 0

Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

FILESET STATISTICS

------------------

1 Selected to be installed, of which:

1 Passed pre-installation verification

----

1 Total to be installed

SUCCESSES

---------

Filesets listed in this section passed pre-installation verification

and will be installed.

Selected Filesets

-----------------

devices.fcp.disk.VendorXYZ.mpio.rte 1.4.0.0 # AIX MPIO Support for VendorXYZ Disk

<< End of Success Section >>

+-----------------------------------------------------------------------------+

Installing Software...

+-----------------------------------------------------------------------------+

installp: APPLYING software for:

devices.fcp.disk.VendorXYZ.mpio.rte 1.4.0.0

. . . . . << Copyright notice for devices.fcp.disk.VendorXYZ.mpio >> . . . . . . .

...etc...

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.4.0.0 USR APPLY SUCCESS

Return code is: 0

Changing default ODM (PdAt) attributes for VendorXYZ storage

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth PdAt

+ sed s/deflt = "2"/deflt = "32"/g

+ 1> /usr/local/XYZ/chg_pdat_qd

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth /usr/local/XYZ/chg_pdat_qd

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy PdAt

+ sed s/deflt = "single_path"/deflt = "no_reserve"/g

+ 1> /usr/local/XYZ/chg_pdat_rp

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy /usr/local/XYZ/chg_pdat_rp

+ odmget -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm PdAt

+ sed s/deflt = "fail_over"/deflt = "round_robin"/g

+ 1> /usr/local/XYZ/chg_pdat_al

+ odmchange -o PdAt -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm /usr/local/XYZ/chg_pdat_al

root@nim1 : / #

Great news! Our script has worked as expected. The old fileset was de-installed, the new fileset was installed and the PdAt default attributes were changed successfully.

Note: You can also review the post migration script output at a later date if you wish. All nimadm activities are logged, on the NIM master, to /var/adm/ras/alt_mig/NIMclientname_alt_mig.log (where NIMclientname is the name of the NIM client being migrated by nimadm).

With regard to nimadm log files, please be aware that if you choose to run nimadm in phases (as I’ve shown in this example) that each run will generate a new log file. So in my case, when I ran phases 1 to 6, this created a log file named lparaix01_alt_mig.log. When I ran phase 7, the original log file was moved to lparaix01_alt_mig.log.prev. A new log file was created and used for phase 7. Then when I ran phases 8 to 12, the phase 7 log file was moved to lparaix01_alt_mig.log.prev and a new log file was used for phases 8-12. For this reason you may want to backup each log file to a unique file name as you execute each phase group, so that you do not lose any of the information logged to the .log or .log.prev files.

Now we can complete the rest of the migration and execute the remaining phases, 8 through 12.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P8,9,10,11,12 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 46989 512 byte blocks.

Writing boot image to client's alternate boot disk hdisk2.

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Expanding /alt_inst/usr client filesystem.

Filesystem size changed to 4325376

Adjusting size for /usr/local

Adjusting size for /usr/local/chksys

Adjusting size for /var

Adjusting size for /var/log

Adjusting size for /var/log/audit

Adjusting size for /var/tsm/log

Syncing cache data to client ...

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /lparaix01_alt/alt_inst/var/log/audit

forced unmount of /lparaix01_alt/alt_inst/var/log

forced unmount of /lparaix01_alt/alt_inst/var

forced unmount of /lparaix01_alt/alt_inst/usr/local

forced unmount of /lparaix01_alt/alt_inst/usr

forced unmount of /lparaix01_alt/alt_inst/tmp

forced unmount of /lparaix01_alt/alt_inst/opt

forced unmount of /lparaix01_alt/alt_inst/home

forced unmount of /lparaix01_alt/alt_inst/admin

forced unmount of /lparaix01_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /lparaix01_alt/alt_inst

Removing cache file system /lparaix01_alt/alt_inst/admin

Removing cache file system /lparaix01_alt/alt_inst/home

Removing cache file system /lparaix01_alt/alt_inst/opt

Removing cache file system /lparaix01_alt/alt_inst/tmp

Removing cache file system /lparaix01_alt/alt_inst/usr

Removing cache file system /lparaix01_alt/alt_inst/usr/local

Removing cache file system /lparaix01_alt/alt_inst/var

Removing cache file system /lparaix01_alt/alt_inst/var/log

Removing cache file system /lparaix01_alt/alt_inst/var/log/audit

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 3.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P3 -d "hdisk2"

## Phase 3 ###################

Verifying altinst_rootvg...

Modifying ODM on cloned disk.

forced unmount of /alt_inst/var/log/audit

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk2 blv=hd5

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

Cleaning up alt_disk_migration on client lparaix01.

root@nim1 : / #

Our nimadm operation is finished. The NIM client is rebooted. We verify it is now running AIX 6.1. And the correct version of the vendor’s device fileset is installed.

lparaix01 : / # oslevel -s

6100-06-05-1115

lparaix01 : / # lslpp -l devices.fcp.disk.VendorXYZ.mpio.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

devices.fcp.disk.VendorXYZ.mpio.rte

1.4.0.0 COMMITTED AIX MPIO Support for VendorXYZ Disk

lparaix01 : / # lsdev -Cc disk

hdisk1 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk2 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk3 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk4 Available 20-T1-01 VendorXYZ MPIO Disk

And there you have it, an example of using a post migration script with nimadm.

nimadm Command

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

More NIM troubleshooting....some basics...

cggibbo Tags: resource to filesystem m_mk_lpp_source: oslevel space failed all a check "master" 0042-185 on already nim copy due not exists aix instfix c_ch_nfsexp: rest or permissions link filesets nfs 0042-081 found. for /etc/.tmp.nim.exports export were and 1 Comment 29,401 Views

It’s no secret that I like NIM. I’ve written about NIM on several occasions, in Redbooks, technical articles and on my blog. It’s a great tool for installing and maintaining AIX. Occasionally though, it can throw you an unusual error or two when things go wrong. For the uninitiated resolving these errors can be challenging.

Here’s a couple of NIM related questions I received recently. Both are troubleshooting queries.

“I'm getting the following error when trying to create a NIM lpp_source for AIX 6.1”:

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [6100-06_lpp]

* Resource Type lpp_source

* Server of Resource [master] +

* Location of Resource [/export/eznim] /

NFS Client Security Method [] +

NFS Version Access [] +

Architecture of Resource [] +

Source of Install Images [/dev/cd0] +/

Names of Option Packages []

Show Progress [yes] +

Comments []

...

Preparing to copy install images (this will take several minutes)...

0042-001 nim: processing error encountered on "master":

0042-081 m_mk_lpp_source: a resource already exists on "master" at

location "/export/eznim/bid_ow"; due to NFS export restrictions, the

new location "/export/eznim" may not be used

...

I could see what could be causing the issue but I first asked to see the contents of the /etc/exports file. Just to make sure that /export/eznim was not explicitly exported. It wasn’t.

# grep eznim /etc/exports

/export/eznim/vios/mksysb_image -ro,root=vio1,access=vio1

/export/eznim/bid_ow -ro,root=nim1,access=nim1

I suggested they re-try the operation again but this time specify the full path to the lpp_source directory, like so:

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [6100-06_lpp]

* Resource Type lpp_source

* Server of Resource [master] +

* Location of Resource [/export/eznim/6100-06_lpp] /

NFS Client Security Method [] +

NFS Version Access [] +

Architecture of Resource [] +

Source of Install Images [/dev/cd0] +/

Names of Option Packages []

Show Progress [yes] +

Comments []

This worked fine. And the lpp_source was created successfully.

# lsnim -l 6100-06_lpp

6100-06_lpp:

class = resources

type = lpp_source

locked = 26738726

Rstate = unavailable for use

prev_state =

location = /export/eznim/6100-06_lpp

alloc_count = 0

server = master

The second issue related to down-level filesets after an AIX 6.1 migration with nimadm. There was also an issue with disk space that generated an error during NIM operations.

“Hi Chris,

Thanks for replying back to my post. I am a newbie to AIX/pSeries world.

I was following your document on upgrading AIX 5.3 to 6.1 using nimadm, it seemed to have worked partially for me, as I got to AIX 6100-00-00. The mksysb and the SPOT I created were for AIX 6100-06-05.

Before I re-attempt the upgrade, is there anything I need to be clean up on AIX 5.3 side that could have perhaps caused issues with the upgrade.

Thanks”

I recommended that the following command be run on the NIM client, to determine which filesets were down-level after the migration.

# oslevel -s

# instfix -i | grep AIX

# instfix -i | grep SP

If the output contained messages like "Not all filesets for 6100-06_AIX_ML were found", or something similar, then they should run the following command against each line (there could be more than one).

# instfix -icqk 6100-06_AIX_ML | grep ":-:"

This will tell you which filesets were not updated.

The output from ‘oslevel –s’and ‘instfix –i | grep AIX’ looked OK.

# oslevel -s

6100-06-04-1112

# instfix -i | grep AIX

All filesets for 6100-00_AIX_ML were found.

All filesets for 6100-01_AIX_ML were found.

All filesets for 6100-02_AIX_ML were found.

All filesets for 6100-03_AIX_ML were found.

All filesets for 6100-04_AIX_ML were found.

All filesets for 6100-05_AIX_ML were found.

All filesets for 6100-06_AIX_ML were found.

But the output from ‘instfix –i | grep SP’ indicated an issue with 61-06-051115_SP.

# instfix -i | grep SP

All filesets for 61-00-010748_SP were found.

All filesets for 61-00-020750_SP were found.

All filesets for 61-00-030808_SP were found.

All filesets for 61-00-040815_SP were found.

All filesets for 61-01-010823_SP were found.

All filesets for 61-00-050822_SP were found.

All filesets for 61-00-060834_SP were found.

All filesets for 61-01-020834_SP were found.

All filesets for 61-00-070846_SP were found.

All filesets for 61-01-030846_SP were found.

All filesets for 61-02-010847_SP were found.

All filesets for 61-02-020849_SP were found.

All filesets for 61-00-080909_SP were found.

All filesets for 61-01-040909_SP were found.

All filesets for 61-02-030909_SP were found.

All filesets for 61-03-010921_SP were found.

All filesets for 61-00-090920_SP were found.

All filesets for 61-01-050920_SP were found.

All filesets for 61-02-040920_SP were found.

All filesets for 61-00-100939_SP were found.

All filesets for 61-01-060939_SP were found.

All filesets for 61-02-050939_SP were found.

All filesets for 61-03-020939_SP were found.

All filesets for 61-04-010944_SP were found.

All filesets for 61-04-021007_SP were found.

All filesets for 61-04-031009_SP were found.

All filesets for 61-00-110943_SP were found.

All filesets for 61-01-081014_SP were found.

All filesets for 61-02-060943_SP were found.

All filesets for 61-03-030943_SP were found.

All filesets for 61-03-041014_SP were found.

All filesets for 61-04-041014_SP were found.

All filesets for 61-05-011016_SP were found.

All filesets for 61-01-091015_SP were found.

All filesets for 61-02-081015_SP were found.

All filesets for 61-03-051015_SP were found.

All filesets for 61-04-051015_SP were found.

All filesets for 61-01-070943_SP were found.

All filesets for 61-02-071014_SP were found.

All filesets for 61-02-091034_SP were found.

All filesets for 61-03-061034_SP were found.

All filesets for 61-04-061034_SP were found.

All filesets for 61-05-021034_SP were found.

All filesets for 61-06-011043_SP were found.

All filesets for 61-02-101036_SP were found.

All filesets for 61-03-071036_SP were found.

All filesets for 61-04-071036_SP were found.

All filesets for 61-05-031036_SP were found.

All filesets for 61-06-021044_SP were found.

All filesets for 61-06-031048_SP were found.

All filesets for 61-05-041048_SP were found.

All filesets for 61-04-081048_SP were found.

All filesets for 61-03-081048_SP were found.

All filesets for 61-03-091112_SP were found.

All filesets for 61-04-091112_SP were found.

All filesets for 61-05-051112_SP were found.

All filesets for 61-06-041112_SP were found.

Not all filesets for 61-06-051115_SP were found.

The culprit was the csm.hc_utils fileset from TL6 SP5. It was still at level 1.7.1.0 instead of the expected 1.7.1.1.

# instfix -ciqk "61-06-051115_SP" |grep ":-:"

61-06-051115_SP:csm.hc_utils:1.7.1.1:1.7.1.0:-:AIX 6100-06-05 Service Pack

I recommended that the csm.hc_utils 1.7.1.1 fileset be added to the AIX 6.1 lpp_source. The migration was attempted again and the client migrated to AIX 6.1 TLP6 SP5 successfully. I also suggested they determine if this fileset was really used and/or needed (CSM is no longer supported/available starting with AIX 7.1 anyway).

And just when I thought all his problems were resolved, he replied with: “however upon trying to install these from the NIM server, I get errors on NIM”.

*********************************************************

errors:

0042-001 nim: processing error encountered on "master": |

| 0042-001 m_showres: processing error encountered on "master": |

| 0042-185 c_ch_nfsexp: Failed to link or copy /etc/.tmp.nim.exports. |

| 12451894 to /etc/exports |

| Check permissions and filesystem space

*********************************************************

Fortunately a quick look at the output from the df command highlighted the problem immediately.

He was out of space in the root file system. The very place that /etc/exports, and the NIM database (/etc/objrepos), reside.

# df -g

Filesystem GB blocks Free %Used Iused %Iused Mounted on

/dev/hd4 2.50 0.00 100% 3388 1% /

/dev/hd2 4.50 1.74 62% 42150 4% /usr

/dev/hd9var 1.00 0.52 48% 3262 2% /var

/dev/hd3 4.00 3.84 5% 74 1% /tmp

/dev/hd1 0.50 0.37 27% 479 1% /home

/proc - - - - - /proc

/dev/hd10opt 0.75 0.21 73% 5698 3% /opt

/dev/livedump 0.25 0.24 4% 17 1% /var/adm/ras/livedump

/dev/exportlv 136.25 124.03 9% 20128 1% /export

After cleaning up unwanted files in / and making sure there was plenty of free space, the error no longer appeared in future NIM operations.

Defining a new Network Install Interface on a NIM master.

cggibbo Tags: new gibson network install nim installation chris interface manager aix 21,987 Views

OK, so I had a NIM master with two network interfaces, en0 and en1. The en0 interface was connected to the 192.168.10.0 network and the en1 interface was connected to the 10.1.1.0 network. When the master was initially configured, we chose en0 as the primary install interface. The master could now install NIM clients on the 192.168.10.0 network, without any additional NIM network configuration.

root@nim1 / # ifconfig –a

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255

tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

inet 10.1.1.10 netmask 0xffffff00 broadcast 10.1.1.255

tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>

inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255

inet6 ::1%1/0

tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

There was a single NIM network definition for the 192.168.10.0 network (net_192_168_10) in the NIM database.

root@nim1 / # lsnim -c networks

net_192_168_10 networks ent

Sure enough, the time came when we needed to install NIM clients on the 10.1.1.0 network. Rather than adding NIM routes, we chose to configure the NIM master with an additional install interface on the 10.1.1.0 network. Fortunately, the NIM master was already “directly” connected to the 10.1.1.0 network on its en1 interface. So all we had to do was update the NIM configuration.

First we added a new entry in the /etc/hosts file for the NIM master’s 10.1.1.0 network address. The hostname for this interface was nim1i.

root@nim1 / # grep nim1 /etc/hosts

192.168.10.10 nim1

10.1.1.10 nim1i

root@nim1 / # host nim1i

nim1i is 10.1.1.10

Next we defined a new network install interface for the NIM master via the smit fastpath, ‘smit nim_mac_if’.

root@nim1 / # smit nim_mac_if

Define a Network Install Interface

+--------------------------------------------------------------------------+

| Machine Name |

| |

| Move cursor to desired item and press Enter. |

| |

| master machines master |

| lpar14 machines standalone |

| lpar15 machines standalone |

| lpar16 machines standalone |

| |

| F1=Help F2=Refresh F3=Cancel |

| F8=Image F10=Exit Enter=Do |

F1=Help F2=Refresh | /=Find n=Find Next | F8=Image

F9=Shell F10=Exit +--------------------------------------------------------------------------+

We entered the hostname of the NIM master on the 10.1.1.0 network i.e. nim1i.

Define a Network Install Interface

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Host Name of Network Install Interface [nim1i]

We named the new NIM network, net_10_1_1 and entered the appropriate subnet mask for this network. We did not enter a default gateway for the client or the NIM master (it is unnecessary, as the master is “directly” connected to this network).

Define a Network Install Interface

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Machine Name master

Network Install Interface

* Cable Type N/A +

Network Speed Setting [] +

Network Duplex Setting [] +

* NIM Network [net_10_1_1]

* Network Type ent

* Ethernet Type Standard +

* Subnetmask [255.255.255.0]

Default Gateway Used by Machine []

Default Gateway Used by Master []

* Host Name nim1i

Network Adapter Hardware Address [0]

Network Adapter Logical Device Name []

There were now two network definitions in our NIM database. The Nstate attribute was set to ‘ready for use’, so our definition was OK to use.

root@nim1 / # lsnim -c networks

net_192_!68_10 networks ent

net_10_1_1 networks ent

root@nim1 / # lsnim -l net_10_1_1

net_10_1_1:

class = networks

type = ent

comments = Generated during definition of nim1i

Nstate = ready for use

prev_state = information is missing from this object's definition

net_addr = 10.1.1.0

snm = 255.255.255.0

The NIM master has two network interfaces defined.

root@nim1 / # lsnim -l master | grep if

if_defined = chrp.64.ent

if1 = net_192_168_10 nim1 0

if2 = net_10_1_1 nim1i 0

Now we could install new NIM clients using this network. We verified this by adding a new NIM client to the NIM database.

# smit nim_mkmac

Define a Machine

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Host Name of Machine [lpar22i]

(Primary Network Install Interface)

After pressing enter on the previous smit panel, we were immediately presented with a panel to modify the attributes of the NIM client and (more importantly) the correct network and hostname were automatically selected as part of the clients definition.

Define a Machine

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* NIM Machine Name [lpar22]

* Machine Type [standalone] +

* Hardware Platform Type [chrp] +

Kernel to use for Network Boot [64] +

Communication Protocol used by client [nimsh] +

Primary Network Install Interface

* Cable Type N/A +

Network Speed Setting [] +

Network Duplex Setting [] +

* NIM Network net_10_1_1

* Host Name lpar22i

Network Adapter Hardware Address [0]

Network Adapter Logical Device Name []

IPL ROM Emulation Device [] +/

CPU Id []

Machine Group [] +

Managing System Information

WPAR Options

Managing System []

-OR-

LPAR Options

Identity []

Management Source [] +

Comments []

If we had configured NIM incorrectly i.e. it was not aware of the 10.1.1.0 network, then we would have been prompted for network information before we could configure the NIM client (as shown below).

Define a Machine

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Host Name of Machine [lpar22i]

(Primary Network Install Interface)

+--------------------------------------------------------------------------+

| Type of Network Attached to Primary Network Install Interface |

| |

| Move cursor to desired item and press Enter. |

| |

| tok = token ring network |

| ent = ethernet network |

| fddi = FDDI network |

| generic = generic network (no network boot capability) |

| atm = ATM network |

| ent6 = IPv6 ethernet network |

| hfi = HFI network |

| |

| F1=Help F2=Refresh F3=Cancel |

nimsh and SAP

cggibbo Tags: aix nim chris sap 3901 gibson nimsh 8 Comments 34,272 Views

Here’s a problem I’ve encountered more than once.

Someone installs SAP onto an AIX system and decides to use TCP port 3901 as an SAP service port. This is the same port used by nimsh. In some rare cases, nimsh may not be active on the LPAR, which makes it easy for the SAP installation to hijack port 3901. If nimsh is active, the person installing SAP may consciously stop nimsh and use port 3901 for SAP anyway. Hopefully that doesn’t happen. Hopefully, they will talk to the AIX administrator and discuss the best way forward. Hopefully...

In either case, if the port is taken by SAP, nimsh will no longer work. If you love using NIM as much as I do, this is a real problem! We could revert back to using rsh but no-one will do this anymore because of concerns around security. And rightfully so!

The ports used by nimsh (3901 and 3902) are registered to Internet Assigned Number Authority (IANA). These port numbers appear in the /etc/services file.

nimsh 3901/tcp # NIM Service Handler

nimsh 3901/udp # NIM Service Handler

nimaux 3902/tcp # NIMsh Auxiliary Port

nimaux 3902/udp # NIMsh Auxiliary Port

Considering these port numbers are registered with IANA, we can usually persuade our SAP colleagues to change their SAP installation to use a different port number. However, depending on the skills/experience of the SAP resource, one of two things usually happens 1) They take an outage, re-install SAP and choose a different port number or 2) The more experienced/confident SAP basis resource will take an outage and modify the instance to use a different port: without reinstalling SAP.

Perhaps SAP need to include a warning in their install notes, advising customers not to use port 3901 on AIX systems (i.e. best practice)?

Now, if you must change nimsh to use a different port number, it is possible. But not recommended.

To do this, you must change the /etc/services file on the NIM master and the NIM client to reflect the same port numbers for nimsh. This will work until the NIM master or the NIM client have their services file overwritten by way of install or fileset updates. After which, the default values for nimsh will be reinstated.

You would also need to change the services file on all of your NIM clients. Every time you performed a NIM fileset update, you would need to remember to change the /etc/services file again. This is painful and bound to catch someone out eventually!

In the following example I’ll demonstrate how to change the port number used by nimsh.

We start with a typical nimsh configuration using port 3901. On the NIM client, nimsh is listening on port 3901.

nimaix / # lsof -i tcp:3901

Value of I :77 np:0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

nimsh 2883740 root 4u IPv6 0xf1000e0006f843b0 0t0 TCP *:nimsh (LISTEN)

On the NIM master we can connect to the NIM client on port 3901.

nimmast / # nim -o lslpp nimaix | head -10

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

Firefox.base.rte 2.0.0.20 COMMITTED Firefox Web Browser

ICU4C.rte 6.1.4.0 COMMITTED International Components for

Unicode

6.1.4.1 COMMITTED International Components for

Unicode

6.1.6.0 COMMITTED International Components for

Unicode

...

The /etc/services file on both the NIM client and the master have the same port number for nimsh.

nimmast / # grep nimsh /etc/services

nimsh 3901/tcp # NIM Service Handler

nimsh 3901/udp # NIM Service Handler

nimaix / # grep nimsh /etc/services

nimsh 3901/tcp # NIM Service Handler

nimsh 3901/udp # NIM Service Handler

To change the port number for nimsh, first we select an unused port number, in this case 39011.

nimaix / # lsof -i tcp:39011

Value of I :77 np:0

We update the /etc/services file on the master and the client to reflect the new port number.

nimaix / # grep nimsh /etc/services

#nimsh 3901/tcp # NIM Service Handler

#nimsh 3901/udp # NIM Service Handler

nimsh 39011/tcp # NIM Service Handler

nimsh 39011/udp # NIM Service Handler

nimmast / # grep nimsh /etc/services

#nimsh 3901/tcp # NIM Service Handler

#nimsh 3901/udp # NIM Service Handler

nimsh 39011/tcp # NIM Service Handler

nimsh 39011/udp # NIM Service Handler

We stop and start nimsh on the client and confirm that nimsh is now listening on the new port number, 39011.

nimaix / # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

nimaix / # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 2883742.

nimaix / # lsof -i tcp:39011

Value of I :77 np:0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

nimsh 2883742 root 4u IPv6 0xf1000e0006f85bb0 0t0 TCP *:nimsh (LISTEN)

From the NIM master we connect to the NIM client using the nim command.

nimmast / # nim -o lslpp nimaix | head -10

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

Firefox.base.rte 2.0.0.20 COMMITTED Firefox Web Browser

ICU4C.rte 6.1.4.0 COMMITTED International Components for

Unicode

6.1.4.1 COMMITTED International Components for

Unicode

6.1.6.0 COMMITTED International Components for

Unicode

....

We can confirm that we have connected to the NIM client on port 39011 by looking at the output from lsof and netstat. There is a TCP session established between the master and the client on port 39011.

nimmast / # lsof -i tcp:39011

Value of I :93 np:0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

m_showlog 430080 root 8u IPv4 0xf1000700047693b0 0t5 TCP nimmast:1023->nimaix:nimsh (ESTABLISHED)

nimmast / # netstat -na | grep 39011

tcp4 0 0 172.29.144.167.1023 172.29.152.241.39011 ESTABLISHED

nimaix / # lsof -itcp:39011

Value of I :75 np:0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

nimsh 2293836 root 0u IPv6 0xf1000e0004d313b0 0t5 TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh 2293836 root 1u IPv6 0xf1000e0004d313b0 0t5 TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh 2293836 root 2u IPv6 0xf1000e0004d313b0 0t5 TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh 2293836 root 4u IPv6 0xf1000e0006f85bb0 0t0 TCP *:nimsh (LISTEN)

nimsh 2293836 root 5u IPv6 0xf1000e0004d313b0 0t5 TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh 2883742 root 4u IPv6 0xf1000e0006f85bb0 0t0 TCP *:nimsh (LISTEN)

nimsh 2883742 root 5u IPv6 0xf1000e0004d313b0 0t5 TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

If the /etc/services file on either the NIM client or master is incorrect we would see error messages similar to the following:

0042-006 m_lslpp: (From_Master) connect A remote host refused an attempted connect operation.

nconn: connect() failed, errno is 79nimaix: A remote host refused an attempted connect operation.

Would I recommend this approach? No. This is not a good, permanent solution to the problem.

Do I think IBM would support this configuration? No.

Why am I writing about it then? Because it might help someone caught in a bind. You could use this method temporarily until you’ve sorted out your SAP port problem.

Feed for Blog Entries ▼

Blogs

Chris's AIX Blog

▼ Tags

▼ Similar Blogs

AIX Down Unde...

Application P...

Asset Managem...

Application I...

AIXpert Blog

▼ Similar Ideation Blogs

IBM PureData-...

IBM Forms Exp...

WebSphere App...

Rational Soft...

IBM Configura...

▼ Archive

▼ Blog Authors

Chris's AIX Blog

Migrating you NIM Master to AIX 7.2 using the VIOS VML

NIM Master, NIMSH and SSL on AIX 7.1 TL4 SP3

Using NIM to distribute files to your AIX servers.

NIMSH, SSL and LPM.

NIM - "...the resource is currently allocated for client use" - er, NO it's not!

NIM support for installing AIX updates and new packages over HTTP

AIX Live Update using NIM

Restoring a Versioned WPAR with NIM

Tips and tricks with dsh on AIX

Updating your NIM master to AIX 7.1 TL2? Take note!

SSH Host keys and AIX 7.1 Migration.

Using a post-migration script with nimadm

More NIM troubleshooting....some basics...

Defining a new Network Install Interface on a NIM master.

nimsh and SAP