Chris's AIX Blog

with Tags: chris

Is that a versioned WPAR I see before me?

cggibbo Tags: versioned aix gibson type wpar chris 3,422 Views

Is that a versioned WPAR? Use the lswpar command and the –G option to determine the WPAR type. Yeah, I could use clogin and then run oslevel, but this is faster!

[root@gibopvc1]/ # lswpar -G p8wpar1

=================================================================

p8wpar1 - Active

=================================================================

Type: S

RootVG WPAR: yes

Owner: root

Hostname: p8wpar1

WPAR-Specific Routing: no

Virtual IP WPAR:

Directory: /wpars/p8wpar1

Start/Stop Script:

Auto Start: no

Private /usr: yes

Checkpointable: no

Application:

OStype: 1024

Cross-WPAR IPC: yes

Architecture: none

UUID: 418c1da6-bceb-4b4e-b872-b1c6701da0b6

NIM - "...the resource is currently allocated for client use" - er, NO it's not!

cggibbo Tags: blog currently nim client is 0042-001 /usr/lpp/bos.sysmgt/nim/m... gibson aix for 0042-061 use chris allocated m_chattr 7,397 Views

On the odd occasion, NIM may report that a resource is allocated to a NIM client, when, in fact, it is not. Typically, you’d check that the resource was, in fact, not allocated for use to any NIM client and if it was, you would reset the client; and this would resolve the issue. But if that doesn’t work, you may need to take an additional action to resolve the problem. This doesn’t happen very often but it can frustrate you when it does.

Here’s an example of the problem. I try to remove an lpp_source resource but I’m told that it’s still allocated to a client. But it isn’t, I tell you!

# nim -o remove liveupdaterte

0042-001 nim: processing error encountered on "master":

0042-061 m_rmpdir: the "liveupdaterte" resource is currently

allocated for client use

Even lsnim is telling me that the resource is still allocated, somewhere, because alloc_count is set to 1.

# lsnim -Fl liveupdaterte

liveupdaterte:

id = 1447111715

class = resources

type = lpp_source

comments = LIVE

arch = power

Rstate = ready for use

prev_state = verification is being performed

location = /export/nim/cglpp

alloc_count = 1

server = master

After trying to de-allocate the resources, by resetting my NIM clients (see my script at the bottom of the page), and still receiving the same error, I’m left with little choice but to manually reset the alloc_count value to 0, using the (almost undocumented) /usr/lpp/bos.sysmgt/nim/methods/m_chattr NIM utility.

# /usr/lpp/bos.sysmgt/nim/methods/m_chattr -a alloc_count=0 liveupdaterte

Now lsnim shows that the resource is no longer allocated to a client and I can remove the lpp_source resource from the NIM database.

# lsnim -Fl liveupdaterte

liveupdaterte:

id = 1447111715

class = resources

type = lpp_source

comments = LIVE

arch = power

Rstate = ready for use

prev_state = verification is being performed

location = /export/nim/cglpp

alloc_count = 0

server = master

# nim -o remove liveupdaterte

I say “almost undocumented” because you can find one reference to the m_chattr command under the NIM error and warning messages page in the AIX 7.1 knowledge center.

https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.install/nim_error_messages.htm

Message	0042-228
Explanation	Invalid release level.
User Action	The release level of the resource is incomplete, or incorrectly specified. The level of the resource can be obtained by running the lsnim -l ResourceName command and viewing the version, release, and mod attributes. To correct the problem, either recreate the resource, or modify the NIM database to contain the correct level using the command on the NIM master:/usr/lpp/bos.sysmgt/nim/methods/m_chattr -a Attribute = Value ResourceName, where Attribute is version, release, or mod; Value is the correct value; and ResourceName is the name of the resource with the incorrect level specification.

One question that comes to mind is how did the NIM resource end up in this state? Most likely it was the result of a failed NIM operation on the lpp_source and NIM client to which it was to be allocated. This can be tricky to pick up and almost always, it’s the next person who tries to use the resource that finds the problem and has no idea what events led up to this point.

As always, use caution when experimenting with this tool. If in doubt, take a backup of your NIM database before you start messing with the attributes, just in case you need it in the future.

Here’s my NIM client reset script. It resets the client and de-allocates any resources assigned to it. It also resets the NIM client cpuid (this is not always required) but I often use the same NIM client to install multiple AIX partitions across several Power servers, so it’s useful to me only (probably)! You can remove that line if need be.

#!/usr/bin/ksh

# Reset a NIM client.

if [[ "$1" = "" ]] ; then

echo Please specify a NIM client to reset e.g. lparaix1nim.

else

if lsnim -l $1 > /dev/null 2>&1 ; then

nim -o reset -F $1

nim -Fo deallocate -a subclass=all $1

nim -Fo change -a cpuid= $1

else

echo Not a valid NIM client!?

Modified by cggibbo

NIM support for installing AIX updates and new packages over HTTP

cggibbo Tags: chris gibson nim blog aix http 6,535 Views

There’s a new NIM HTTP service handler included with AIX 7.2 (due for release next month, December 2015). This new service is designed “…….to help Clients better conform to emerging data center policies restricting the use of NFS, NIM will now have support to apply updates to AIX or install new packages over HTTPs. Initial AIX installs will still require the use of NFS version 3 or the more secure NFS version 4 protocol.

In addition to fileset installs, NIM customization activities such as script execution and file_res copying also support access over HTTPs.

Major Advantages of using HTTP during NIM Management:

All communication occurs over a single http port, so the authorization through a firewall is quite easy to manage.
Actions are driven from the client's end (the install target), so remote access isn't necessary for pushing the commands.
Easy to consume by NIM or other products that currently use the client/server model of NFS.
Able to extend the end-product to support additional protocols (context driven).”

“How Does it Work?

AIX ships a new service handler (in 7.2.0) that provides http access to NIM resources. The service name (defined in /etc/services) is nimhttp and it listens for requests over port 4901. When active, NIM clients attempt file access and/or scripting customization requests from nimhttp. If http access fails or is denied, a failover attempt at NFS client access occurs. Future support will include options to remove NFS client attempts altogether.”

“On startup, the nimhttp service attempts to read the httpd.conf configuration file -‐-‐ located in the default home directory of the user. First time users will notice that starting the service without a configuration file will result in one being created and populated with default service values.”

“document_root

….for now, the key detail to point out is that NIM expects all http accessible files to exist under the path of /export/nim/. This path location is defined as the document_root and cannot be modified at this time. Future enhancements will support multiple document_root paths. The document root path is not limited in depth and may contain many sub-directories. Client requests are able to traverse the path setting by using the enable_directory_listing option. If set to “no”, all files being served must reside in the current working directory of document_root.”

“The default authentication used in nimhttp for client access is a basic protocol handshake and is probably considered by some (if not all) as undesirable. To enable the more secure Digest Authentication method, users must provide valid paths for certificate authority and root certificate files for the server. The certificate authority and root PEM files used in nimhttp are easily created using the existing SSL management option in NIM. Run the following command on the NIM master to create the ssl.cert_authority and ssl.pemfiles used by the nimhttp service:

# nimconfig –c”

I tested this functionality during the AIX 7.2 Early Ship Program.

Warning: The information shown here was collected from testing conducted with beta level code. Some details may change in the final release.

Configuring the service was easy. For the sake of simplicity I chose not to use SSL with the authentication mechanism. With my NIM master already configured, all I need to do is confirm that the NIM client fileset is installed on the master and any client I wish to manage with the HTTP service.

NIM MASTER:

# lslpp -l | grep nim

bos.sysmgt.nim.master 7.2.0.0 COMMITTED Network Install Manager -

bos.sysmgt.nim.client 7.2.0.0 COMMITTED Network Install Manager –

NIM CLIENT:

# lssrc -s nimsh

Subsystem Group PID Status

nimsh nimclient 6554064 active

# lslpp -l | grep nim

bos.sysmgt.nim.client 7.2.0.0 COMMITTED Network Install Manager -

Start the NIMHTTP service on the NIM master. This starts the nimhttpd daemon (on the master only) and creates the default httpd.conf file (in root’s home directory, /).

# startsrc -s nimhttp

0513-059 The nimhttp Subsystem has been started. Subsystem PID is 6685178.

# lssrc -s nimhttp

Subsystem Group PID Status

nimhttp 6685178 active

# ps -ef | grep nimhttp

root 6685178 4194712 0 Nov 10 - 0:00 /usr/sbin/nimhttpd –v

# ls -ltr /httpd.conf

-rw-r--r-- 1 root system 1159 Nov 05 15:31 /httpd.conf

# cat /httpd.conf

#---------------------

# http service defines

#---------------------

service.name=nimhttp

# Designates the service name used when discovering the listening port for requests (i.e., nimhttp)

service.log=/var/adm/ras/nimhttp.log

# Log of access attempts and equivalent responses. Also useful for debug purposes.

# service.proxy_port=

# Designates the service port number used when configured as a proxy.

# service.access_list=

# White-list of IP (host) addresses which have access to our http file service. All others are denied.

#---------------------

# http configuration

#---------------------

document_root=/export/nim/

# Designates the directory to serve files from.

enable_directory_listing=yes

# Allow requests for listing served files/directories under the document root.

enable_proxy=no

# Enable the web service to act as a proxy server.

ssl.cert_authority=/ssl_nimsh/certs/root.pem

# Designates the file location of the certificate authority used for digital certificate signing.

ssl.pemfile=/ssl_nimsh/certs/server.pem

# Designates the file location of the PEM format file which contains both a certificate and private key.

I configured a new lpp_source resource (liveupdaterte) on the NIM master. I ensured that all the files for the lpp_source were in the correct location (i.e. /export/nim) . This restriction will be lifted in the future, but during my testing the service required all files to be served from /export/nim, on the master.

# lsnim -t lpp_source

lpp_sourceaix72 resources lpp_source

liveupdaterte resources lpp_source

# lsnim -l liveupdaterte

liveu:

class = resources

type = lpp_source

arch = power

Rstate = ready for use

prev_state = unavailable for use

location = /export/nim/cglpp

alloc_count = 0

server = master

# ls –ltr /export/nim/cglpp

total 4864

-rw-r--r-- 1 root system 2483200 Oct 16 04:23 bos.liveupdate.7.2.0.0.I

-rw-r--r-- 1 root system 688 Oct 31 09:33 .toc

From the client I initiated a nimclient pull request, to install the bos.liveupdate.rte fileset.

# nimclient -o cust -a lpp_source=liveupdaterte -a filesets=bos.liveupdate.rte

# lslpp -l bos.liveupdate.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

bos.liveupdate.rte 7.2.0.0 COMMITTED Live Update Runtime

Path: /etc/objrepos

bos.liveupdate.rte 7.2.0.0 COMMITTED Live Update Runtime

The bos.liveupdate.rte fileset installed OK, from the lpp_source and I noticed nimhttp service activity was logged to the /var/adm/ras/nimhttp.log file, on the NIM master.

# tail -f nimhttp.log

Mon Nov 16 13:51:23 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015 file descriptor is : 6

Mon Nov 16 13:51:23 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:23 2015 socks: Request string

Mon Nov 16 13:51:23 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:23 2015 socks: Copied string

Mon Nov 16 13:51:23 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:23 2015 Request Type is

Mon Nov 16 13:51:23 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:23 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:23 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:23 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:23 2015 file descriptor is : 6

Mon Nov 16 13:51:23 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:23 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:23 2015 socks: 20004238 128 <- 70 bytes (PLAIN)

Mon Nov 16 13:51:23 2015 socks: Request string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 nim_buffer: (append) len is 70, buffer length is 70

Mon Nov 16 13:51:23 2015 socks: Copied string GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 nim_http: data string passed to get_http_request: "GET /export/nim/scripts/AIXmig.script HTTP/1.1

Connection: close

Mon Nov 16 13:51:23 2015 Request Type is GET

Mon Nov 16 13:51:23 2015 Mime not supported

Mon Nov 16 13:51:23 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:23 2015 Sending file over socket 6. Expected length is 509

Mon Nov 16 13:51:23 2015 Total length sent is 509

Mon Nov 16 13:51:23 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:23 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 54 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 54, buffer length is 54

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 Sending Response Header "302 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 72

Mon Nov 16 13:51:24 2015 Total length sent is 72

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 59 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 59, buffer length is 59

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/.toc HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 755

Mon Nov 16 13:51:24 2015 Total length sent is 755

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 203 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 203, buffer length is 203

Mon Nov 16 13:51:24 2015 socks: Copied string

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: ""

Mon Nov 16 13:51:24 2015 Request Type is

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 nim_http: queue socket create product (memory *)20004238

Mon Nov 16 13:51:24 2015 nim_http: file descriptor is 6

Mon Nov 16 13:51:24 2015 nim_buffer: (resize) buffer size is 0

Mon Nov 16 13:51:24 2015 file descriptor is : 6

Mon Nov 16 13:51:24 2015 family is : 2 (AF_INET)

Mon Nov 16 13:51:24 2015 source address is : 10.1.50.252

Mon Nov 16 13:51:24 2015 socks: 20004238 128 <- 79 bytes (PLAIN)

Mon Nov 16 13:51:24 2015 socks: Request string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_buffer: (append) len is 79, buffer length is 79

Mon Nov 16 13:51:24 2015 socks: Copied string GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 nim_http: data string passed to get_http_request: "GET /export/nim/cglpp/bos.liveupdate.7.2.0.0.I HTTP/1.1

Connection: close

Mon Nov 16 13:51:24 2015 Request Type is GET

Mon Nov 16 13:51:24 2015 Mime not supported

Mon Nov 16 13:51:24 2015 Sending Response Header "200 OK"

Mon Nov 16 13:51:24 2015 Sending file over socket 6. Expected length is 2574336

Mon Nov 16 13:51:24 2015 socks: Removing socksObject 2ff1eca0

Mon Nov 16 13:51:24 2015 Total length sent is 2574336

Mon Nov 16 13:51:24 2015 handle_httpGET: Entering cleanup statement

“Example: Commands supporting nimhttp

If using push operation (From NIM master)

# nim –o cust –a file_res=<obj_name> <client_obj_name>

# nim –o cust –a script=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a filesets=<fileset names to install> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name> <client_obj_name>

# nim –o cust –a lpp_source=<obj_name> ‐a fixes=update_all <client_obj_name>

If using pull operation (From NIM client)

# nimclient -o cust -a lpp_source=<obj_name> -a filesets=<fileset names to install>

# nimclient –o cust –a file_res=<obj_name>

# nimclient –o cust –a script=<obj_name> nimclient –o cust –a lpp_source=<obj_name> -‐a filesets=<fileset names to install>

# nimclient –o cust –a lpp_source=<obj_name> -a installp_bundle=<obj_name>

# nimclient –o cust –a lpp_source=<obj_name> -a fixes=update”

Modified by cggibbo

Queue depth setting in a Versioned WPAR on AIX

cggibbo Tags: disk depth wpar performance aix versioned io queue vwpar chris gibson 5,957 Views

One of my customers was configuring a new AIX 5.3 Versioned WPAR when they came across a very interesting issue. I thought I’d share the experience here, just in case anyone else comes across the problem. We configured the VWPAR to host an old application. The setup was relatively straight forward, restore the AIX 5.3 mksysb into the VWPAR and export the data disk from the Global into the VWPAR, import the volume group and mount the file systems. Job done! However, we noticed some fairly poor performance during application load tests. After some investigation we discovered that disk I/O performance was worse in the VWPAR than on the source LPAR. The question was, why?

We initially suspected the customers SAN and/or the storage subsystem, but both of these came back clean with no errors or configuration issues. In the end, the problem was related to a lack of ODM attributes in the PdAt object class, which prevented the VWPAR disk from using the correct queue depth setting.

Let me explain by demonstrating the problem and the workaround.

First, let’s add a new disk to a VWPAR. This will be used for a data volume group and file system. The disk in question is hdisk3.

# uname -W

# lsdev -Cc disk

hdisk0 Available Virtual SCSI Disk Drive

hdisk1 Available Virtual SCSI Disk Drive

hdisk2 Defined Virtual SCSI Disk Drive

hdisk3 Available Virtual SCSI Disk Drive <<<<<<

We set the disk queue depth to an appropriate number, in this case 256.

Note: This value will differ depending on the storage subsystem type, so check with your storage team and/or vendor for the best setting for your environment.

# chdev -l hdisk3 -a queue_depth=256

hdisk3 changed

Using the lsattr command, we verify that the queue depth attribute is set correctly in both the ODM and the AIX kernel.

# lsattr -El hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

We can also use kdb to verify the setting in the kernel. Remember at this stage, we are concentrating on hdisk3, which is referenced with a specific kernel device address in kdb.

# echo scsidisk | kdb

START END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME ADDRESS STATE CMDS_OUT CURRBUF LOW

hdisk0 0xF1000A01505DC000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk0 0xF1000A01C0148000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk1 0xF1000A01505D4000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk3 0xF1000A01C014C000 0x00000001 0x0000 0x0000000000000000 0x0 <<<<<<<

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

ushort queue_depth = 0x100;

From the output above, we can see that the queue depth is correctly i.e. set to 0x100 in Hex (256 in decimal).

Next, we export hdisk3 to the VWPAR using the chwpar command. The disk, as expected, enters a Defined state in the Global environment. It is known as hdisk1 in the VWPAR.

# chwpar -D devname=hdisk3 p8wpar1

# lswpar -D p8wpar1 | head -2 ; lswpar -D p8wpar1 | grep hdisk

Name Device Name Type Virtual Device RootVG Status

-------------------------------------------------------------------

p8wpar1 hdisk3 disk hdisk1 no EXPORTED <<<<<<

p8wpar1 hdisk2 disk hdisk0 yes EXPORTED

[root@gibopvc1]/ # lsdev -Cc disk

hdisk0 Available Virtual SCSI Disk Drive

hdisk1 Available Virtual SCSI Disk Drive

hdisk2 Defined Virtual SCSI Disk Drive

hdisk3 Defined Virtual SCSI Disk Drive

In the VWPAR, we run cfgmgr to discover the disk. We create a new data volume group (datavg) and file system (datafs) for application use (note: the steps to create the VG and FS are not shown below). This is for demonstration purposes only; the customer imported the data volume groups on their system.

# clogin p8wpar1

*******************************************************************************

* *

* Welcome to AIX Version 5.3! *

* *

* Please see the README file in /usr/lpp/bos for information pertinent to *

* this release of the AIX Operating System. *

* *

*******************************************************************************

Last login: Sun Apr 26 21:27:03 2015 on /dev/Global from

# uname -W

# lspv

hdisk0 00f94f58a0b98ca2 rootvg active

# cfgmgr ; lspv

hdisk0 00f94f58a0b98ca2 rootvg active

hdisk1 none None

# lsvg

rootvg

datavg

# lsvg -l datavg

datavg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

datalv jfs2 1024 1024 1 open/syncd /datafs

loglv00 jfs2log 1 1 1 open/syncd N/A

We perform a very simple I/O test in the /datafs file system. We write/create a 1GB file and time the execution. We noticed immediately that the task took longer than expected.

# cd /datafs

# time lmktemp Afile 1024M

Afile

real 0m7.22s <<<<<<<<<<<<<<< SLOW?

user 0m0.04s

sys 0m1.36s

We ran the iostat command, from the Global environment, and noticed that “serv qfull” was constantly non-zero (very large numbers) for hdisk3. Essentially the hdisk queue was full all the time. This was bad and unexpected, given the queue depth setting of 256!

# iostat -DTRl 1

System configuration: lcpu=8 drives=4 paths=8 vdisks=2

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:53

hdisk0 0.0 4.1K 1.0 0.0 4.1K 0.0 0.0 0.0 0.0 0 0 1.0 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:53

hdisk2 1.0 12.3K 3.0 0.0 12.3K 0.0 0.0 0.0 0.0 0 0 3.0 0.5 0.3 0.6 0 0 0.4 0.0 0.9 0.0 0.0 3.0 03:11:53

hdisk3 100.0 140.5M 1072.0 0.0 140.5M 0.0 0.0 0.0 0.0 0 0 1072.0 0.9 0.8 24.0 0 0 176.7 107.5 279.8 152.0 0.0 1072.0 03:11:53

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:54

hdisk0 0.0 4.1K 1.0 4.1K 0.0 1.0 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:54

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:54

hdisk3 100.0 154.0M 1175.0 0.0 154.0M 0.0 0.0 0.0 0.0 0 0 1175.0 0.8 0.8 1.8 0 0 161.9 108.0 217.7 303.0 1.0 1175.0 03:11:54

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:55

hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:55

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:55

hdisk3 100.0 150.3M 1147.0 0.0 150.3M 0.0 0.0 0.0 0.0 0 0 1147.0 0.9 0.8 10.6 0 0 165.9 108.5 239.4 304.0 1.0 1147.0 03:11:55

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:56

hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:56

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:56

hdisk3 100.0 155.5M 1186.0 0.0 155.5M 0.0 0.0 0.0 0.0 0 0 1186.0 0.8 0.8 1.8 0 0 161.6 106.8 217.8 307.0 1.0 1186.0 03:11:56

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:57

hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:57

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:11:57

hdisk3 67.0 102.2M 780.0 0.0 102.2M 0.0 0.0 0.0 0.0 0 0 780.0 0.9 0.8 11.8 0 0 166.1 106.3 232.8 53.0 0.0 779.0 03:11:57

Now comes the interesting part. With a little help from our friends in IBM support, using kdb we found that the queue depth was reported as being set to 1 in the kernel and not 256! You’ll also notice here that the hdisk name has changed from hdisk3 to hdisk1. This happened as a result of exporting hdisk3 to the VWPAR. The disk is known as hdisk1 in the VWPAR (not hdisk3) but the kernel address is the same.

# echo scsidisk | kdb

START END <name>

0000000000001000 0000000005840000 start+000FD8

F00000002FF47600 F00000002FFDF9C8 __ublock+000000

000000002FF22FF4 000000002FF22FF8 environ+000000

000000002FF22FF8 000000002FF22FFC errno+000000

F1000F0A00000000 F1000F0A10000000 pvproc+000000

F1000F0A10000000 F1000F0A18000000 pvthread+000000

read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01E4E20

(0)> scsidisk

"scsidisk_list" address...[0x0]

NAME ADDRESS STATE CMDS_OUT CURRBUF LOW

hdisk0 0xF1000A01505DC000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk0 0xF1000A01C0148000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk1 0xF1000A01505D4000 0x00000002 0x0000 0x0000000000000000 0x0

hdisk1 0xF1000A01C014C000 0x00000001 0x0000 0x0000000000000000 0x0 <<<<

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

ushort queue_depth = 0x1; <<<< WRONG QUEUE DEPTH!

# lsattr -Pl hdisk3 -a queue_depth

queue_depth 256 Queue DEPTH True

The kdb output above proved that the queue depth was set to 1 in the VWPAR. Even though the ODM still had the attribute set to 256 (in both the Global and VWPAR environments).

We discovered that this behaviour was a bug.

IV63665: CAN'T SET THE QUEUE_DEPTH IN A VERSION WPAR.

http://www-01.ibm.com/support/docview.wss?uid=isg1IV63665

APAR status

Closed as fixed if next.

Error description

When a customer run the following in a version

WPAR such as 5.2 or 5.3. The disk in the kernel

will not have the queue depth set in kernel.

1. chdev -P -l hdisk5 -a queue_depth=8

2. odmget CuAt | grep -p hdisk5 | less

CuAt:

name = "hdisk5"

attribute = "queue_depth"

value = "8"

type = "R"

generic = "UD"

rep = "nr"

nls_index = 30

3. stopwpar -N <wpar_name>

The customer then will see that the performance

is still the same. When a perfpmr is sent in

from the global lpar you will see that the

queue_depth is set to 1 for all of the disk

exported to the wpar.

Local fix

N/A

Problem summary

When a customer run the following in a version

WPAR such as 5.2 or 5.3. The disk in the kernel

will not have the queue depth set in kernel.

Fortunately, IBM support was able to provide us with a workaround. The first step was to add the missing vparent PdAt entry to the ODM in the Global environment.

# cat addodm_pdat_for_vparent.txt

PdAt:

uniquetype = "wio/common/vparent"

attribute = "naca_1_spt"

deflt = "1"

values = "1"

width = ""

type = "R"

generic = ""

rep = "n"

nls_index = 0

# odmadd addodm_pdat_for_vparent.txt

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

uniquetype = "wio/common/vparent"

attribute = "naca_1_spt"

deflt = "1"

values = "1"

width = ""

type = "R"

generic = ""

rep = "n"

nls_index = 0

We did the same in the VWPAR.

# clogin p8wpar1

# uname -W

# odmget PdAt | grep -p "wio/common/vparent"

# odmadd addodm_pdat_for_vparent.txt

# odmget PdAt | grep -p "wio/common/vparent"

PdAt:

uniquetype = "wio/common/vparent"

attribute = "naca_1_spt"

deflt = "1"

values = "1"

width = ""

type = "R"

generic = ""

rep = "n"

nls_index = 0

In the VWPAR, we removed the hdisk and then discovered it again, ensuring that the queue depth attribute was set to 256 in the ODM.

# uname –W

# rmdev -dl hdisk1

hdisk1 deleted

# cfgmgr

# lspv

hdisk0 00f94f58a0b98ca2 rootvg active

hdisk1 none None

# lsattr -El hdisk1 –a queue_depth

queue_depth 256 Queue DEPTH True

# odmget CuAt | grep -p queue

CuAt:

name = "hdisk1"

attribute = "queue_depth"

value = "256"

type = "R"

generic = "UD"

rep = "nr"

nls_index = 12

Back in the Global environment we checked that the queue depth was set correctly in the kernel. And it was!

# uname -W

# echo scsidisk 0xF1000A01C014C000 | kdb | grep queue_depth

ushort queue_depth = 0x100;

We re-ran the simple I/O test and immediately found that the test ran faster and the hdisk queue (for hdisk3, as shown by iostat from the Global environment) was no longer full. Subsequent application load tests showed much better performance.

# time lmktemp Afile 1024M

Afile

real 0m3.15s <<<< BETTER!

user 0m0.03s

sys 0m1.60s

# iostat -DTRl 1

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:14

hdisk0 0.0 4.1K 1.0 4.1K 0.0 1.0 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:14

hdisk2 0.0 20.5K 5.0 20.5K 0.0 5.0 0.2 0.2 0.4 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 5.0 03:20:14

hdisk3 86.0 280.6M 2144.0 0.0 280.6M 0.0 0.0 0.0 0.0 0 0 2144.0 1.2 0.3 2.9 0 0 0.0 0.0 0.0 0.0 4.0 0.0 03:20:14

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:15

hdisk0 0.0 8.2K 2.0 0.0 8.2K 0.0 0.0 0.0 0.0 0 0 2.0 1.2 0.6 1.9 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:15

hdisk2 0.0 4.1K 1.0 0.0 4.1K 0.0 0.0 0.0 0.0 0 0 1.0 0.6 0.6 0.6 0 0 0.0 0.0 0.0 0.0 0.0 1.0 03:20:15

hdisk3 100.0 327.0M 2495.0 0.0 327.0M 0.0 0.0 0.0 0.0 0 0 2495.0 1.3 0.8 10.0 0 0 0.0 0.0 0.0 0.0 2.0 0.0 03:20:15

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:16

hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:16

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:16

hdisk3 100.0 354.3M 2703.0 0.0 354.3M 0.0 0.0 0.0 0.0 0 0 2703.0 2.1 0.9 16.1 0 0 0.0 0.0 0.0 0.0 5.0 0.0 03:20:16

Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:17

hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:17

hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 03:20:17

hdisk3 32.0 111.7M 852.0 0.0 111.7M 0.0 0.0 0.0 0.0 0 0 852.0 1.3 0.9 2.8 0 0 0.0 0.0 0.0 0.0 1.0 0.0 03:20:17

Please Note: kdb will not work inside a WPAR. If you attempt to run it, you’ll receive the following error message. Run kdb from the Global environment instead.

# clogin p8wpar1

# kdb

The specified kernel file is a 64-bit kernel

open: A file or directory in the path name does not exist.

cannot open /dev/pmem

Modified by cggibbo

Troubleshooting AIX hdisk configuration errors

cggibbo Tags: device specified liblpp.a does configuration odm hds the gibson customized not error 0514-023 vsp method database. exist /usr/lib/methods/cfgscsid... in pdpathat chris aix 1 Comment 13,304 Views

Last year I was working for a customer that was upgrading several AIX 5.3 systems to AIX 6.1. The migrations were successful for the most part, but we did encounter one issue that took a little time to resolve.

The customer was using nimadm to migrate. This process worked fine, however, on a couple of systems a strange error was encountered after the migration. The LPAR was booted into AIX 6.1 and everything came up fine. The applications were started and users started accessing the system.

It was several days later, when the AIX administrator attempted to configure new storage on the AIX system, when the first sign of trouble appeared. He had asked his Storage guy to assign a couple of new disks to his LPAR (via NPIV/VFC). As soon as the Storage admin had completed the assignment, the AIX admin ran cfgmgr to detect and configure the new hdisks. Immediately, cfgmgr reported the following error:

Method error (/usr/lib/methods/cfgscsidisk):

0514-023 The specified device does not exist in the

customized device configuration database.

Initially, the AIX team suspected there was some fault with either the storage device or the zoning of the disk. Both of these items were checked and doubled-checked and were found to be OK. Our next step was to run cfgmgr again, but this time we wanted a greater level of detail captured. To do this we used the following environment variable to force cfgmgr to be ‘more verbose’.

# export CFGLOG="cmd,meth,lib,verbosity:9"

We ran cfgmgr and went to the /var/adm/ras/cfglog file to view the results with the alog command. However, we noticed that the cfglog file had a size of zero (0) and contained no data.

# cd /var/adm/ras

# ls –l cfglog

-rw-r----- 1 root system 0 May 16 13:22 cfglog

We decided to recreate the cfglog alog file and run mkdev again to reproduce the disk configuration error.

# rm cfglog

# echo "Create cfglog `date`"|alog -t cfg

# mkdev -l hdisk0

Method error (/usr/lib/methods/cfgscsidisk):

0514-023 The specified device does not exist in the

customized device configuration database.

This time we found some useful data in the cfglog file.

# alog -t cfg -o

MS 31981804 28835876 /usr/lib/methods/cfgscsidisk -l hdisk39

M4 31981804 Parallel mode = 0

M4 31981804 Get CuDv for hdisk39

M4 31981804 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 31981804 Get parent CuDv, name=fscsi0

M4 31981804 ..is_mpio_capable()

M4 31981804 Device is MPIO

M4 31981804 ..get_paths()

M4 31981804 Getting CuPaths for name='hdisk39'

M4 31981804 Found 1 paths

M0 31981804 cfgcommon.c 225 mpio_init error, rc=23

MS 28835892 31981568 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 28835892 Parallel mode = 0

M4 28835892 Get CuDv for hdisk0

M4 28835892 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 28835892 Get parent CuDv, name=fscsi0

M4 28835892 ..is_mpio_capable()

M4 28835892 Device is MPIO

M4 28835892 ..get_paths()

M4 28835892 Getting CuPaths for name='hdisk0'

M4 28835892 Found 2 paths

M0 28835892 cfgcommon.c 225 mpio_init error, rc=23

MS 25690326 27328608 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 25690326 Parallel mode = 0

M4 25690326 Get CuDv for hdisk0

M4 25690326 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 25690326 Get parent CuDv, name=fscsi0

M4 25690326 ..is_mpio_capable()

M4 25690326 Device is MPIO

M4 25690326 ..get_paths()

M4 25690326 Getting CuPaths for name='hdisk0'

M4 25690326 Found 2 paths

M0 25690326 cfgcommon.c 225 mpio_init error, rc=23

The configuration method was attempting to configure a disk device type of htcvspmpio (which was correct) but it was unable to configure the device paths (mpio_init error rc=23). We suspected that the system was missing some sort device driver support for the type of storage in use.

Cutting a very long story short, we determined, with the help of the IBM AIX support team, that the issue stemmed from “old” AIX installation media used to create the AIX 6.1 TL6 SP5 SPOT and lppsource on the NIM master. Old AIX 6.1 media was originally used (several years ago) to create the NIM resources and was gradually updated over time, all the way up to TL6 SP5.

IBM support identified that the older install media contained a liblpp.a file that was missing the necessary PdPathAt ODM files. Newer install media contained a fix to add the appropriate entries to the bos.rte.cfgfiles. e.g.

SPOT created using OLD install media.

======================================

# ar xv /export/spot/spotaix610605_OLD/usr/lpp/bos/liblpp.a bos.rte.cfgfiles

x - bos.rte.cfgfiles

# grep PdPathAt bos.rte.cfgfiles

SPOT created using NEW install media.

======================================

# ar xv /export/spot/spotaix610605_NEW/usr/lpp/bos/liblpp.a bos.rte.cfgfiles

x - bos.rte.cfgfiles

# grep PdPathAt bos.rte.cfgfiles

/usr/lib/objrepos/PdPathAt v4preserve

/usr/lib/objrepos/PdPathAt.vc v4preserve

We recreated the SPOT and lppsource resources using newer media on the NIM master. We were then able to migrate the AIX 5.3 LPAR to 6.1 without encountering the issues faced previously.

Modified by cggibbo

Replicated rootvg, ghostdev and NPIV

cggibbo Tags: chris hds recovery led rootvg fc support ghostdev replication lpar hung gibson 554 boot replicated chris_gibson vfc vsp aix san channel virtual fibre dr disaster npiv hang 13 Comments 34,751 Views

My team and I have recently been trying to stream line our AIX disaster recovery process. We’ve been looking for ways to reduce our overall recovery time. Several ideas were tossed around such as a) using a standby DR LPAR with AIX already installed and using rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy (with the –O flag for a device reset) to clone rootvg to an alternate disk which is then replicated to DR. These methods may work but are cumbersome to administer and (in the case of alt_disk_copy) require additional (permanent) resources on every production system. With over 120 production instances of AIX, the disk space requirements start to add up.

So far we’ve concluded that the best way to achieve our goal is by using SAN replicated rootvg volumes at our DR site.

Our current DR process relies on recovery of AIX systems from mksysb images from a NIM master. All our data (non-rootvg) LUNs are already replicated to our DR site. The aim was to change the process and ‘recover’ our AIX images using replicated rootvg LUNs. This will reduce our overall recovery time at DR (which is crucial if we are to meet the proposed recovery time objectives set by our business). Based on current IBM documentation we were relatively comfortable with the proposed approach. The following IBM developerWorks article (originally published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported. More easily switch AIX environments from one system to another and help achieve higher availability and reduced down time. These scenarios also allow for fast deployment of new systems using cloning.”

The document focuses on fully virtualised environments that utilise shared processors and VIO servers. One area where this document is currently lacking in information is the use of NPIV and virtual fibre channel adapters in a DR scenario. We reached out to our contacts in the AIX development space and asked the following question:

“Hoping you can help us find some statements regarding support for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?

The following IBM developerWorks article discusses VSCSI and we are looking for something similar for NPIV.

http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html

My guess is that restrictions similar to those for physical FC adapters will apply here? But I'm hoping that given the adapters are virtual the limitations may be relaxed.

Are you aware of any statement regarding support (or not) for booting from another system using a disk subsystem image of rootvg replicated to another disk subsystem when using NPIV? And what, if any, additional requirements/restrictions may apply when using NPIV?”

We received the following responses:

“There are some additional considerations when using NPIV for booting from a replicated rootvg. With NPIV the client partitions has virtual Fibre Channel adapter ports, but has physical access to the actual (physical) disk devices. There may be an increased chance of needing to update the boot list via the Open Firmware SMS menu. Since the clients have access to the actual disks, you have the possibility of running multipathing software besides AIX MPIO. If you are using a multipathing software to manage the NPIV attached disks besides AIX MPIO, then you should contact the vendor that provided the software to check their support statement,

Since one or more of the physical devices will change when booting from an NPIV replicated rootvg, it is recommend to set the ghostdev attribute. The ghostdev attribute will trigger when it detects the AIX image is booting from either a different partition or server. Ghostdev attribute should not trigger during LPM operations (Live Partition Mobility). Once triggered, ghostdev will clear the customized ODM database. This will cause detected devices to be discovered as new devices (with default settings), and avoid the issue with missing/stale device entries in ODM. Since ghostdev does clear the entire customized ODM database, this will require you import your data (non-rootvg) volume groups again, and perform any (device) attribute customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1". Ghostdev must be set before the rootvg is replicated.

Similar with virtual devices, the client partition is booting an existing rootvg where the hardware may be different. It's possible some applications have dependency on tracking the actual physical devices (instead of data on the disks). For example PowerHA, may keep track of a disk for a cluster health checks. If you do have applications that have a dependency on tracking physical devices, then additional setup (of those applications) may be required after the first boot from the replicated rootvg.

We do have multiple customers using NPIV for such scenarios. I believe most of them worked with IBM Lab Based Services to assist with implementing such a configuration, and some of the customers required some custom scripts to further customize their system after booting from the replicated rootvg. Those customers set the ghostdev attribute, and had custom scripts to import their data (non-rootvg) volume groups, and update PowerHA to point to the new health check disk.

You should get support for such an NPIV setup with IBM as long as you follow the considerations listed in the WhitePaper.”

“Development has approved using NPIV to do this for one customer. Below are more detailed requirements for this DR strategy using NPIV. If a Disaster Recovery (DR) environment not using PowerHA Enterprise Edition is used, then we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and as well as the limitations of such a DR deployment. Deployments as detailed in the white paper are supported by IBM. However note that such a deployment has many manually instituted responsibilities on the customer to setup and maintain such an environment. IBM expects that the customer carefully manage these manual steps without any mistakes. The White paper does not currently cover using NPIV in such a DR scenario.

We have the following guidelines regarding the configuration, which includes NPIV as an option: All of the system configuration should be virtualized, with the possible exception of disk devices when using NPIV. If NPIV is used then AIX MPIO must be used as the multi-pathing solution. If multi-pathing software is used besides AIX MPIO, then the vendor of that software must be contacted regarding a support statement. Install AIX (at least minimum required TLs/SPs for desired AIX version) and software stack (Middleware and applications) on primary systems, which is compatible with systems at both sites. Primary and secondary sites should be using systems with similar hardware, same microcode levels, and same VIOS levels.

Many manual steps are needed to setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then discover the unique identification on the primary site and map the disks to the corresponding replication disks. Map the same appropriately on VIOS on secondary site. Level of VIOS should support attribute to open the secondary devices passively. This setting needs to be setup correctly on the VIOS on secondary site. Operating environment should not have subnet dependencies. Manage the replication relationships accurately. Manually may need to switch the secondary disks to primary node. Raw disk usage may cause problems. Some middle ware products may bypass Operating system and use the disk directly. They might have their own restrictions for this environment. (e.g. anything that is device location code or storage LUN unique ID dependant may have issues when the cloned image is restarted on the secondary system with replicated storage).

Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system. If the "ghostdev" attribute is not set, then the booting from the alternate site will result in devices in ODM showing up in "Defined" or "Missing" state.

After a failover to secondary site, may need to reset the boot device list for each LPAR before the boot of the LPAR using SMS menus of the firmware Note that this is not an exhaustive list of issues. Refer to the white paper and study the same as it applies to the environment. So as long as they are using MPIO, you're OK. If not using MPIO and some OEM storage, then the storage vendor must also support it.”

While these responses indicate that this form of recovery is supported by IBM, we were still looking to IBM for clarity on the support position. It has been noted that other IBM customers have had mixed responses when contacting AIX support for feedback and assistance with this type of DR procedure. And it’s not hard to see why when you read statements like this from the “Supported Methods of Duplicating an AIX System” document:

“Unsupported Methods

1. Using a bitwise copy of a rootvg disk to another disk.

This bitwise copy can be a one-time snapshot copy such as flashcopy, from one disk to another, or a continuously-updating copy method, such as Metro Mirror.

While these methods will give you an exact duplicate of the installed AIX operating system, the copy of the OS may not be bootable. A typical scenario where this is tried is when one system is a production host and there is a desire to create a duplicate system at a disaster recovery site in a remote location.

2. Removing the rootvg disks from one system and inserting into another.

This also applies to re-zoning SAN disks that contain the rootvg so another host can see them and attempt to boot from them.

Why don't these methods work?

The reason for this is there are many objects in an AIX system that are unique to it; Hardware location codes, World-Wide Port Names, partition identifiers, and Vital Product Data (VPD) to name a few. Most of these objects or identifiers are stored in the ODM and used by AIX commands.

If a disk containing the AIX rootvg in one system is copied bit-for-bit (or removed), then inserted in another system, the firmware in the second system will describe an entirely different device tree than the AIX ODM expects to find, because it is operating on different hardware. Devices that were previously seen will show missing or removed, and usually the system will typically fail to boot with LED 554 (unknown boot disk).”

So, as a secondary objective, we have been working closely with our local IBM representatives to obtain some surety from IBM that our proposed DR strategy for AIX is fully supported by both the AIX development and support teams.

With that in mind I’ll provide an overview of our new DR approach and hope that it offers others insight to alternative method for recovery and also to assist IBM in further understanding what some of the “larger” AIX customers are looking for in terms of simplified AIX disaster recovery.

What follows is a detailed description of our IBM AIX, PowerVM/Power Systems environment, the proposed recovery steps and other items for consideration.

Our Environment:

· - AIX 5.3*, 6.1 & 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.

· - VIOS 2.2.1.3 – both production and DR.

· - VIOS physical FC adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735) – Firmware level: 2.00X7.

· - All client LPARs utilise Virtual FC adapters (NPIV) for disk storage.

· - Production: POWER7 795 (9119-FHB). Firmware level: AH730_078.

· - DR: POWER6 595 (9119-FHA). Firmware level: EH350_120.

· - HDS VSP storage – at both production and DR sites.

· - AIX MPIO only – HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is not installed.

· - Production: Dual VIOS.

· - DR: Single VIOS.

· - Production: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.

· - DR: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a single VIOS. i.e. single path only.

· - Please refer to the following table for a summary of the environment details.

Our Recovery Procedure:

1. - Change the sys0 ghostdev attribute value to 1 on the source production AIX system. Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system.

2. - Take note of the source systems rootvg hdisk PVID.

3. - Select the source production rootvg LUN for replication on the HDS VSP.

4. - Replicate the LUN from the production site to the DR HDS VSP.

5. - In a DR test, suspend HDS replication from production to DR.

6. - Assign the replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the WWPN of the virtual FC adapter on the DR LPAR.

7. - Attempt to boot the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to update the boot list i.e. select the correct boot disk, check for the same PVID as the source host.

8. - Once the LPAR has successfully booted, the AIX administrator would configure the necessary devices i.e. import data volume groups, configure network interfaces, etc. This may also be scripted for execution during the first boot process.

9. - Please refer to the following diagrams for a visual representation of the proposed process.

Some Notes/Caveats:

The following is a list of items that we understand are possible limitations and issues with our new DR process.

· - Booting from replicated rootvg disks may fail for several reasons, such as, a) there is unexpected corruption in the replicated LUN image due to rootvg not being quiesced during replication or b) there is a unidentified issue with the AIX system that is only apparent the next time the system is booted; this could be mis-configuration by the administrator or some other unforseen problem.

· - In the event that an LPAR fails to boot via a replicated rootvg LUN, a backup method is available for recovery. Switching back to manual NIM mksysb restore provides a sufficient backup should the replicated rootvg be unusable.

· - If the "ghostdev" attribute is not set, then booting from the DR site will result in devices in the ODM showing up in a "Defined" or "Missing" state.

· - Once a DR test is completed, the DR LPAR should be de-activated immediately so that SAN disk replication can be restarted between production and DR. Failure to perform this step may result in the DR LPAR failing as a result of file system corruption.

· - At present we are using AIX MPIO only. There is discussion of using HDLM in the future. We will contact HDS for a support statement regarding booting from replicated rootvg LUNs with HDLM installed.

· - The ghostdev attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.

So far all of our testing has been successful. We verified that we could replicate an SOE rootvg image of AIX 6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk. Based on these tests there doesn’t appear to be anything stopping us from using this method for DR purposes. The following table outlines the different versions of AIX we tested and the results.

Once the system was booted we needed to perform some post boot configuration tasks. These tasks were handled by two scripts that were called from /etc/inittab. On the source system we installed the new scripts (in /etc) and added new entries to the /etc/inittab file. These scripts only run if the systemid matches that of the DR systemid. Note: Only partial contents of each script are shown below…but you get the idea.

# mkitab -i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# lsitab -a | grep DR

AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIX_DRconfig.ksh:

#!/bin/ksh

#set -xv

#################################################################

# This script will configure the AIX environment for DR purposes.

# This script will only run if the systemid matches the systemid of the DR system.

# aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'

# 0211D11C1 = DR POWER System

#################################################################

MYNAME=$(basename $0)

DR_SYSTEMID=0211D11C1

LSATTR_SYSTEMID_DR=`lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`

if [ "$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID" ]

then

#Set the DR hostname, IP address, netmask and gateway.

#mktcpip -h <hostname> -a <IP to use in DR> -m <network mask> -i en0 -g <gateway IP>

#en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0 10.2.2.10,255.255.255.0

dr_hostname=`cat /usr/local/dr/dr_hostname.txt`

dr_defgw=`cat /usr/local/dr/dr_defgw.txt`

dr_intf=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $1}'`

dr_ip=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $2}'`

dr_netm=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $3}'`

echo "$MYNAME: Setting DR hostname and IP address."

mktcpip -h $dr_hostname -a $dr_ip -m $dr_netm -i $dr_intf -g $dr_defgw

#Configure IP aliases

#chdev -l en0 -a alias4=10.1.1.10,255.255.255.0

#Remove IP aliases

#chdev -l en0 -a delalias4=10.1.1.10,255.255.255.0

for i in `cat /usr/local/dr/dr_en0.txt | awk -F: '{print $4}'`

chdev -l $dr_intf -a alias4=$i

done

for inet in `cat dr_inet.txt | grep -v en0`

for i in `cat /usr/local/dr/dr_$inet.txt | awk -F: '{print $4}'`

chdev -l $inet -a alias4=$i

done

#Configure the AIX environment for DR.

echo "$MYNAME: Configure the AIX environment for DR."

echo

#Delete the MDC DNS entries

echo "$MYNAME: Configuring DNS for DR."

namerslv -d -i 10.1.6.21

namerslv -d -i 10.1.4.21

namerslv -d -i 10.1.5.21

#Add the DR DNS entry

namerslv -a -i 10.1.7.38

echo

echo "$MYNAME: Configuring bootlist for DR."

bootlist -m normal -o

bootlist -m normal hdisk0

bootlist -m normal -o

echo

echo "$MYNAME: Configuring /etc/inittab for DR."

lsitab -a | grep nim

chitab "nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console 2>&1 # inform nim we're running"

lsitab -a | grep nim

echo

else

MYNAME=$(basename $0)

echo "$MYNAME: The systemid $LSATTR_SYSTEMID_DR does not match the expected DR systemid of $DR_SYSTEMID."

echo "$MYNAME: This script should only be executed at DR."

echo "$MYNAME: If you are not booting the system at the DR site, then you can ignore this message."

echo "$MYNAME: No changes have been perfomed. Script is exiting."

The ghostdev attribute essentially provides us with a clean ODM and allows the system to discover new devices and build the ODM from scratch. If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554) because of a new device tree and/or missing devices. You might be able to recover from this situation (without restoring from mksysb) by performing the steps outlined on pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for the presentation).

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf

How much memory or CPU will that CoD code activate anyway?

cggibbo Tags: cod power gibson mod aix activation demand pod capacity chris code on vet powervm 11 Comments 29,547 Views

This little tip was passed on to me by a friendly IBM hardware engineer many years ago.

When entering a capacity on demand (CoD) code into a Power system, you can tell how many processors and how much memory will be activated, just by looking at the code you’ve given by IBM.

For example, the following codes, when entered for the appropriate Power system, will enable 4 processors (POD) and 64GB of memory (MOD). I can also tell* that once the VET code is entered, this system will be licensed for PowerVM Enterprise Edition (2C28).

System Type: 9117 Serial Number: 06-B719P
Type	Activation Code	Posted Date (MM/DD/YYYY)
VET	F2FB60ERC4FF3DB50CA1F00002C28004177	04/30/2010
POD	FA94E589F2067F0C54680000000400419B	04/30/2010
MOD	B154E7097C231A85821200000064004102	04/30/2010

Activation type definitions
POD:	CUoD Processor Activation Code
MOD:	CUoD Memory Activation Code
TCOD:	On/Off CoD Enablement Code
TMOD:	On/Off CoD Memory Enablement Code
VET:	Virtualization Technology Code
	(PowerVM, Enterprise Enablement, WWPN, Active Memory Expansion)
STDP:	Standard Trial CoD Processor Activation Code
STDM:	Standard Trial CoD Memory Activation Code
STME:	Standard Trial Active Memory Expansion Code
EXCP:	Exception Trial CoD Processor Activation Code
EXCM:	Exception Trial CoD Memory Activation Code
USTA:	Utility CoD Enablement Code
USTO:	Utility CoD Termination Code
URPT:	Utility CoD Reporting Code
PAID:	Utility CoD PrePaid Code

0000 = Reset to base defaults (No PowerVM)
1xxx = Express
2c00 = Standard
2c20 = Enterprise

*Refer to the following link for more information on “How to tell which PowerVM Editions feature was ordered”: https://www-304.ibm.com/support/docview.wss?uid=isg3T1010860

You can obtain your CoD codes from the following website: http://www-912.ibm.com/pod/pod

======

UPDATE 12/9/2011: Script from Chris-R that collects CoD information for a system from the IBM CoD Website. Thanks Chris-R, nice job!

Done!

$ ./activations.pl 9117 10 06ebp
02 processors activated on 05/23/2011
024 GB memory activated on 05/23/2011
02 processors activated on 03/11/2011
016 GB memory activated on 03/11/2011
055 GB memory activated on 03/02/2011
06 processors activated on 02/22/2011
096 GB memory activated on 02/22/2011
01 processors activated on 01/12/2011
016 GB memory activated on 01/12/2011
06 processors activated on 03/17/2010
072 GB memory activated on 03/17/2010
19 processors activated on 03/16/2010
228 GB memory activated on 03/16/2010
================
TOTAL CPU=36 RAM=519168MB (507GB)

$cat activations.pl
#!/usr/bin/perl
#
# PoC Script to go out on the net to the IBM POD site and tally up activations for a given i or p Series machine
# Chris-R rutherfc@gmail.com
# v1.0 9/9/2011
#
# todo - add logic for processor deactivation - who uses that? IBM dont even publish the code on the pod site ;)
#
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("mozilla 8.0");
# $ua->proxy(['http'], 'http://proxy:8080/');
$ua->timeout(10);
use HTTP::Request::Common qw(POST);
if ($#ARGV != 2) {
print "usage: $0 MODEL XX XXXXX e.g $0 9119 83 9f6bf\n";
exit;
}
($model, $serial1, $serial2) = @ARGV;

##### main #####
get('http://www-912.ibm.com/pod/pod',"$serial2.htm");
html2txt("$serial2.htm","$serial2.txt");
total("$serial2.txt");
exit;

sub get # fakes a mozilla browser, fills in the CGI form and snags the returned page to a local html file
{
my $req = (POST $_[0],
["system_type" => $model,
"system_serial_number1" => $serial1,
"system_serial_number2" => $serial2 ]);
$request = $ua->request($req);
$activations = $request->content;
open(POD,">$_[1]");
print POD $activations;
close(POD);
}

sub html2txt # strips out the crap and converts the table to a local txt file to parse
{
open(HTML,"<$_[0]");
open(TXT,">$_[1]");
while (<HTML>) {
if (/<\/table>/) {$f = 0;};
if (/<th>Posted Date \(MM/) {$f = 1;};
if ($f == 1) {
# poor mans HTML::TableExtract - excuse my sed like perl....
s/<tr align="center">/:/g;
s/<[^>][^>]*>//g;
s/ //g;
s/\n//g;
s/:/\n/g;
print TXT $_;
};
};
close(TXT);
close(HTML);
}
sub total # totals up the de/activations to get totals
{
open(TXT,"<$_[0]");
$[ = 1; $\ = "\n";# set array base & output record separator
while (<TXT>) {
($code,$hex,$date) = split(' ', $_, -1);
if (/POD/) {
$p = substr($hex, 27, 2);
print $p . ' processors activated on ' . $date;
$pt = $pt + $p;
};
if (/MOD/) {
$r = substr($hex, 26, 3);
print $r . ' GB memory activated on ' . $date;
$rt = $rt + $r;
};
if (/RMEM/) {
$r = substr($hex, 27, 2);
print $r . ' GB memory activated on ' . $date;
$rt = $rt - $r;
};
};
print '================';
print 'TOTAL CPU=' . $pt . ' RAM=' . $rt*1024 . 'MB (' . $rt . 'GB)';
close(TXT);
}
</TXT></tr></th></HTML>

JFS2 logging disabled during mksysb restore.

cggibbo Tags: mksysb log=null disabled logging jfs2 chris aix gibson 1 Comment 10,435 Views

While attending the IBM Power Systems Symposium this week, I learnt that starting with AIX 7.1 (and AIX 6.1 TL6) JFS2 logging is disabled during a mksysb restore. You may be familiar with disabling JFS2 logs, if not, take a look at this IBM technical note:

https://www-304.ibm.com/support/docview.wss?uid=isg3T1012643

I’ve been unable to find any official documentation from IBM that mentions this new enhancement to the mksysb restore process. However, when I checked my own AIX 7.1 system I found the following statement/code in the /usr/lpp/bosinst/bi_main script:

# Disable jfs2 logs for temp mounts -- (def#751604 performance)

if [[ "$2" = "TRUE" ]]; then

mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

else

mount -v $LVTYPE -o log=/dev/$LOGDEV,nomanager /dev/$LV $MROOT$FS || $ERROR

If I compare this against my AIX 5.3 system, I don’t find any reference to this enhancement.

AIX 7.1:

[/usr/lpp/bosinst] > oslevel -s

7100-00-03-1115

[/usr/lpp/bosinst] # grep log=NULL bi_main

bi_main: mount -v $LVTYPE -o log=NULL,nomanager /dev/$LV $MROOT$FS || $ERROR

AIX 5.3:

[/usr/lpp/bosinst] > oslevel -s

5300-11-03-1013

[/usr/lpp/bosinst] # grep log=NULL bi_main

[/usr/lpp/bosinst] #

This change is clever. I mean why would we ever need JFS2 logging during a restore? By disabling logging, the restore process will be faster. Simple, really.

AIX or Linux on POWER?

cggibbo Tags: chris linux gibson ibm power aix 8 Comments 23,427 Views

IBM made some announcements today relating to their latest POWER7 server offerings. The new line of systems includes new entry level systems and the highly anticipated high-end system, the POWER7 795! They also officially outlined some of the new features available in AIX 7.1. You can review the details here. I’ve discussed some of these new features here and here. The official AIX 7.1 announcement details are available here.

The announcement got me thinking about my recent customer engagements and why some have chosen to deploy AIX into their IBM POWER environments, while others are considering a Linux on POWER solution.

I’ve found that it usually comes down to a skills decision more than anything else. Most customers are happy to either continue working with AIX (if they are existing AIX users) or migrate from another UNIX OS to AIX. I’ve seen very few customers actually migrate to Linux on POWER, but I’ve worked with several that have seriously considered it. Those that have chosen to deploy Linux are doing so purely because they have in-house Linux skills. They are concerned that migrating to AIX may be too big a jump for their technical staff. I find this thinking interesting, as most of the customers I’ve dealt with who run other UNIX OS’s like Tru64, Solaris or HP-UX are more than happy to migrate to AIX. They believe the move is relatively minor and doesn’t require massive re-training of their UNIX admins. I tend to agree.

For me, AIX is my preferred “Enterprise class” UNIX Operating System. Notice I’m prefacing this with the words Enterprise class. Don’t get me wrong, I have worked with Linux systems in both small and large customer environments. It is a great OS. But I’ve found that it really only fits into environments that have a relatively small number of users and where significant downtime can be tolerated for things like operating system maintenance. This doesn’t fit the Enterprise class of UNIX server OS’s that I’m thinking of here. When I contemplate the word Enterprise, I think of servers and operating systems that can respond to business demands in terms of performance, reliability, stability and availability. An Enterprise UNIX can provide all of these things without compromise. Linux can offer performance and reliability (in my opinion). However, from what I’ve seen, it lacks features & functions in the areas of stability and availability. AIX on the other hand ticks all the boxes. Again, this is just my opinion based on my experiences with both AIX and Linux in the Enterprise landscape. Others will no doubt have their own experiences that may or may not match my own.

So when I’m designing an Enterprise UNIX server environment for a customer, I always start with an AIX on POWER base. If the customer wants Linux, sure I can look at that too, but I strongly recommend AIX as my preferred choice for large systems. Most of my customers are running relatively large SAP/Oracle systems. AIX on POWER is a great combination for large Enterprise systems. If you need to deploy large database systems that must service tens of thousands of users (like a big SAP system), then I believe AIX is the perfect OS on which to provide a platform for these large scale systems.

AIX is a very mature and powerful UNIX OS. It has been a major player in the UNIX server market for over 20 years (as shown below). Some people are just not aware of how mature, robust and stable the AIX OS has become over the years. There are many impressive aspects of the OS in the areas of performance, scalability, reliability, management and administration.

Just looking at some of the administration capabilities built into AIX are enough for me to always recommend AIX over Linux (or any other UNIX OS), when it comes to large Enterprise servers.

For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler. This is an interactive tool that is part of the AIX OS. Almost all tasks that an AIX administrator may need to perform can be executed using this tool. It is a text-based tool (there is also an X interface but I recommend sticking with the text-based menus). Everything it does, it does through standard AIX commands and Korn shell functions. This feature is especially useful when you need to automate a repetitive task; you can have SMIT create the proper command-line sequence, and you can then use those commands in your own script. My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.

The AIX Logical Volume Manager (LVM) is built into the OS, for free. AIX LVM helps UNIX system administrators manage their storage in a very flexible manner. LVM allows logical volumes to span multiple physical volumes. Data on logical volumes appears to be contiguous to the user, but might not be contiguous on the physical volume. This allows file systems, paging space, and other logical volumes to be resized or relocated, span multiple physical volumes, and have their contents replicated for greater flexibility and availability. It provides capabilities for mirroring data across disks, migrating data across disks & storage subsystems, expand/shrink filesystems and more.......all of which can be performed dynamically.....no downtime required. The concept, implementation and interface to the AIX LVM is one of a kind. All of its features support the ‘continuous availability’ philosophy.

One of the biggest reasons that I love AIX over Linux is the mksysb. It’s built into the OS and allows you to create a bootable image of your AIX system. This image can be used to restore a broken AIX system or for cloning other systems. The cloning feature is truly amazing. You can take an image created on a low-end system and deploy it on any POWER system, all the way up to the high-end POWER boxes. This simplifies the installation and cloning processing when you need to install and manage many AIX LPARs. By using an SOE mksysb image you can deploy consistent AIX images across your Enterprise POWER server environment.

This brings me to another wonderful feature of AIX, the Network Installation Manager (NIM). NIM is powerful network installation tool (comparable to Linux Kickstart). Using NIM you can backup/restore, update and upgrade one or more AIX systems either individually or simultaneously. This can all be achieved over a network connection, removing the need for handling physical installation media forever.

Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates. Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows.

When it comes to upgrading the OS to a new release of AIX, the nimadm utility can assist the administrator greatly in this task. The nimadm utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a NIM client's rootvg and migrate the disk to a newer version or release of AIX. All of this can be done without disruption to the client (there is no outage required to perform the migration). After the migration is finished, the only downtime required will be a single scheduled reboot of the system.

AIX 6.1 introduced new capability that most UNIX operating systems are still working on. Concurrent Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS that can provide continuous availability, even if it needs to be patched. AIX now has the ability to update certain kernel components and kernel extensions in place, without needing a system reboot. In addition, concurrent updates can be removed from the system without needing a reboot. Can you do this with other UNIX OSs?

And that’s just some of the features that make AIX the only UNIX OS that I recommend for Enterprise systems. There are many more tools and features that I couldn’t live without (like alt_disk_install, savevg, installp, WPARs, etc, the list goes on). If you are new to AIX and you are considering what your next UNIX OS should be, then I recommend you take a very close look at AIX.

And finally, the support provided by IBM, is first class. Whenever I’ve needed assistance with an AIX issue or query, I have always received timely, professional and useful advice. On the rare occasions where I’ve uncovered a new bug, IBM AIX support have always been quick to provide me with an interim fix to resolve or workaround a problem. That’s the sort of support you’d expect for an Enterprise UNIX OS, isn’t it? What’s the support like from your current UNIX (or Linux) OS vendor?

Linux is still a viable UNIX operating system. However, I think it’s more suited to certain workloads like small to medium size mail, web and other utility servers and services. AIX, however, would be my platform of choice for my 10TB Oracle database running SAP ERP, not just for performance reasons, but primarily because of the system administration features of AIX that allow me to support and manage the system without impacting my customers or enforcing reboots/outages whenever I need to change something on the system.

All IBM need to do is create their own Linux distribution (Blue Linux perhaps?) that has all the features of AIX built in and then I’m sold! But why would they? We already have AIX.

Follow me on Twitter: http://twitter.com/cgibbo

rmsock on AIX.

cggibbo Tags: rmsock netstat aix gibson lpar wpar lsof chris 29,760 Views

I like lsof.

Whenever I’m building a new AIX system I always make sure to install it. I really like the fact that I can quickly list processes that are connected to TCP and UDP ports on my system. For example, to check for the current SSH connections on my system I can run lsof and check port 22 (SSH). Immediately I have a good idea of the existing SSH sessions/connections. I can also check to see if the SSH server (sshd daemon) is running and listening (LISTEN) on my AIX partition.

# lsof -i tcp:22

Value of I :77 np:0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

sshd 340040 root 3u IPv4 0xf100020002e4a3b0 0t277394 TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd 385184 u0008904 3u IPv4 0xf100020002e4a3b0 0t277394 TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd 409808 root 3u IPv4 0xf100020001c48bb0 0t0 TCP *:ssh (LISTEN)

But sometimes I work on systems that don’t have lsof installed. It may not be practical or appropriate for me to install it either. So I have to find another tool (or tools) that will do something similar.

Of course, I could use netstat to check that a server daemon was listening on a particular TCP port and view any established connections. But this doesn’t give me the associated process id’s.

$ netstat -a | grep -i ssh

tcp4 0 0 *.ssh *.* LISTEN

tcp4 0 48 aix01.ssh 172.29.131.16.50284 ESTABLISHED

Fortunately, the rmsock command can provide that information. So if I wanted to find the process id for the sshd daemon that is listening on my system I’d do the following. First I need to find the socket id using netstat*.

# netstat -@aA | grep -i ssh | grep LIST | grep Global

Global f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN

Then I can use rmsock to discover the process id associated with the sockect. In this case it’s PID 282700.

$ rmsock f1000200003e9bb0 tcpcb

The socket 0x3e9808 is being held by proccess 282700 (sshd).

Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket. Note that the second argument of rmsock is the protocol. It's tcpcb in this example to indicate that the protocol is TCP. The results of the command are also logged to /var/adm/ras/rmsock.log.

# tail /var/adm/ras/rmsock.log

socket 0xf100020001c45008 held by process 434420 (writesrv) can't be removed.

socket 0xf100020000663008 held by process 418040 (java) can't be removed.

socket 0xf1000200012ad008 held by process 418040 (java) can't be removed.

socket 0xf100020000dec008 held by process 163840 (inetd) can't be removed.

socket 0xf100020000deb008 held by process 163840 (inetd) can't be removed.

socket 0xf10002000016f808 held by process 192554 (snmpdv3ne) can't be removed.

socket 0xf100020001c51808 held by process 442596 (dtlogin) can't be removed.

socket 0xf1000200012a4008 held by process 418040 (java) can't be removed.

socket 0xf100020000666008 held by process 315640 (java) can't be removed.

socket 0xf100020000deb808 held by process 163840 (inetd) can't be removed.

The following links have more information on the rmsock command:

http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg21264632

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/rmsock.htm

http://www.ibmsystemsmag.com/print/print.aspx?print_page=%2Faix%2Ftipstechniques%2F6666printp1.aspx&string_referer=/aix/tipstechniques/6666p1.aspx

http://aixblogs.blogspot.com/2009/03/using-netstat-and-rmsock-to-identify.html

*Note: In my example I specified the @ symbol with the netstat command. I also grep’ed for the string Global. You may have to do the same if you have WPARs running on your system. In my case I have two active WPARs who both have their own sshd process. My Global environment also has an sshd process. So in total there are three sshd daemons that I can view from the Global environment. By specifiying the @ symbol with netstat, I can quickly determine which process belongs to the Global environment and those that exist within each WPAR.

# netstat -aA | grep -i ssh | grep LISTEN

f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN

f100070000198bb0 tcp4 0 0 *.ssh *.* LISTEN

f100070000194bb0 tcp4 0 0 *.ssh *.* LISTEN

# netstat -@aA | grep -i ssh | grep LISTEN

Global f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN

wpar1 f100070000198bb0 tcp4 0 0 *.ssh *.* LISTEN

wpar2 f100070000194bb0 tcp4 0 0 *.ssh *.* LISTEN

AIX 7 Open Beta is live - it's time to test!

cggibbo Tags: chris 7 beta gibson aix open 2 Comments 10,419 Views

OK, it’s official. The AIX 7 Open Beta is now available for testing!

An overview of the program is available here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml

Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.

I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.

The AIX 7 beta code is available for download from here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/download.shtml

The user forum, where you can provide feedback and ask questions, is here:

https://www.ibm.com/developerworks/forums/forum.jspa?forumID=2255

I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/library.shtml

The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.

The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:

A. AIX 5.2 Workload Partitions for AIX 7.

B. Removal of WPAR local storage device restrictions.

C. Domain support in Role Based Access Control (RBAC).

D. Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.

E. NIM thin server support for IPv6 and NFSv4.

I’m interested in testing A, B and D. Here’s why.

A. AIX 5.2 Workload Partitions for AIX 7.

AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.

B. Removal of WPAR local storage device restrictions.

AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?

D. Etherchannel enhancements in 802.3ad mode.

There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I’m interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it’s not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.

And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:

Installing the AIX Open Beta Driver

The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.

Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the ‘mksysb’.

1. Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:

• Type 1 and press Enter to have English during install.

• Type 1 to continue with the install.

• Type 1 to Start Install Now with Default Setting.

2. The system will start installing the AIX 7.0 BETA.

3. Upon completion of the install, the system will reboot. You can then login as “root”, no password is required.

Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:

· The Open Beta code is being delivered via an “mksysb” install image. Migration installation is not supported with the open beta driver.

· The open beta driver does not support IBM Systems Director Agent.

· When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.

· The image is known to install without issues on an 8 GB disk.

· oslevel output shows V7BETA.

By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17^th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it’s great if you just want to test something on an AIX system.

I’ll report back once I’ve got my AIX 7 Open Beta system up and running!

saposcol and shared memory on AIX.

cggibbo Tags: aix ipcs ipcrm memory gibson chris shared saposcol 1 Comment 25,058 Views

If you are a system administrator that is responsible for managing AIX systems that run SAP, then you’ve probably had an experience similar to the following?

OK, so one day my SAP Basis administrator contacts me and says “I can’t start saposcol......can you please reboot the system?” I quickly reply, “Was there an error message when trying to restart saposcol?”. He replies, “No”. Again I return very quickly, “OK, have you checked to see if there are any shared memory segments left for saposcol?”. Just as quick, he replies “How do I do that?”

Together we try starting saposcol and what we find is that it thinks it’s already running (as shown below, PID 327924). But there is no such process!

$ saposcol -l

-l

09:37:24 22.01.2009 LOG: Effective User Id is root

***********************************************************************

* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

* Usage: saposcol -l: Start OS Collector

* saposcol -k: Stop OS Collector

* saposcol -d: OS Collector Dialog Mode

* saposcol -s: OS Collector Status

* The OS Collector (PID 327924) is already running .....

************************************************************************

$ ps -fp 327924

UID PID PPID C STIME TTY TIME CMD

So we try to stop saposcol. This of course fails, as there is no PID 327924 running!

$ saposcol -k

Setting Stop Flag :

09:19:29 22.01.2009 LOG: ==== Stop Flag was set by saposcol (kill_collector()).

09:19:29 22.01.2009 LOG: ==== The collection process will stop as soon as possible

********************

can't kill process 327924.

kill: No such process

ERROR:No reaction from collecting process 327924.

Please kill collecting process.

My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?

Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.

http://www.saptechies.com/os-collector-saposcol/

7. Question: How do I remove the shared memory key of saposcol?

Answer: Sometimes it may be necessary to remove the shared memory key of saposcol (see point 6). Caution: Please be very careful! This procedure should be performed only after checking that saposcol is really not running (see point 4) and only in cases when other options (see point 6) really do not work! For this, execute command “ipcs -ma” and note the line that contains saposcol key “4dbe”. You need the shared memory ID. After that, execute command ipcrm -m ID. Now the command “saposcol -s” should show that saposcol is not running and that the shared memory is not attached. The shared memory key will be created automatically by the saposcol when the collector is next started: “saposcol -l”.

So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.

There are several process id’s ‘attached’ to this segment. However, only one of them actually exists (PID 2293794).

# ipcs -ma | grep 4dbe

m 2097156 0x00004dbe --rw-rw-rw- root sapsys root sapsys 0 1870188 467164 2293794 9:39:43 9:39:43 8:12:43

# ps -fp 1870188

UID PID PPID C STIME TTY TIME CMD

# ps -fp 467164

UID PID PPID C STIME TTY TIME CMD

# ps -fp 2293794

UID PID PPID C STIME TTY TIME CMD

sapadm 2293794 1 0 19:26:41 - 1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01

I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.

# ipcrm -m 2097156

# ipcs -ma | grep 4dbe

We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.

# ipcs -ma | grep 4dbe

m 3145732 0x00004dbe --rw-rw-rw- root sapsys root sapsys 1 1870188 2293818 2207892 10:14:25 10:14:25 10:12:47

# ps -ef | grep oscol

sapadm 2064616 1 0 10:12:53 - 0:00 saposcol -l

# /usr/sap/SAP/SYS/exe/run/saposcol -s

**************************************************************

Collector Versions :

running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

dialog : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

Shared Memory : attached

Number of records : 17640

Active Flag : active (01)

Operating System : AIX aix01 3 5 00C01C704C00

Collector PID : 2064616 (001F80E8)

Collector : running

Start time coll. : Thu Jan 22 10:12:53 2009

Current Time : Thu Jan 22 10:17:46 2009

Last write access : Thu Jan 22 10:17:38 2009

Last Read Access : Thu Jan 22 10:16:16 2009

Collection Interval : 10 sec (next delay).

Collection Interval : 10 sec (last ).

Status : free

Collect Details : required

Refresh : required

Header Extention Structure

Number of x-header Records : 1

Number of Communication Records : 60

Number of free Com. Records : 60

Resulting offset to 1.data rec. : 61

Trace level : 2

Collector in IDLE - mode ? : NO

become idle after 300 sec without read access.

Length of Idle Interval : 60 sec

Length of norm.Interval : 10 sec

**************************************************************

Problem solved and no reboot required.....this is AIX after all! :)

Migrating you NIM Master to AIX 7.2 using the VIOS VML

cggibbo Tags: using master 7.2 sddpcm vml chris the nimadm you to migrating gibson nim vios aix 1,107 Views

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

Modified by cggibbo

AIX, LPM & Partition ID sorcery! Why does AIX report that my partition ID has changed after LPM when it hasn't?

cggibbo Tags: last uname lpm chris since aix id partition changed boot gibson 1,747 Views

I LPM'ed an AIX LPAR from Frame A to Frame B. The LPAR id before the LPM was 99 and after the LPM it's still 99. And the system id has changed, as expected. Good.

Before migration.

root@lpar1 / # uname -L

99 lpar1

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,017268B60 Hardware system identifier False

After migration.

root@lpar1 / # uname -L

99 lpar1

root@lpar1 / # lsattr -El sys0 -a systemid

systemid IBM,0157F5B16 Hardware system identifier False

Great, that's exactly what I wanted.

Hang on a minute! What kind of sorcery is this?!

root@lpar1 / # errpt | grep sys

60AFC9E5 0405132517 I O sys0 Partition ID changed since last boot.

root@lpar1 / # errpt -aN sys0

---------------------------------------------------------------------------

LABEL: PARTID_CHANGE

IDENTIFIER: 60AFC9E5

Date/Time: Wed Apr 5 13:25:53 EET 2017

Sequence Number: 81372

Machine Id: 000157F5B16

Node Id: lpar1

Class: O

Type: INFO

WPAR: Global

Resource Name: sys0

Description

Partition ID changed since last boot.

Whaddaya mean my partiton ID changed since last boot?! It hasn't! I know this for a fact! Look at the uname output above!!

Well, I'm 100% correct, the partition id HAS NOT CHANGED! But the system id has....and this is the cause of the "informational" message in the AIX error report.

AIX uses the output from 'uname -f' to detect if the system AND/OR partition id have changed since last boot.

So, in my case, the partition id (99 dec/63 hex) did not change but (as you'd expect with LPM) the system id (shown in hex below) has changed, hence the message "Partition ID changed since last boot".

Before LPM.

root@lpar1 / # uname -f

80000BA02DC00063

After LPM.

root@lpar1 / # uname -f

800016B466F00063

From the uname man page:

-F Displays a system identification string comprised of hexadecimal characters. This identification string is the same for all partitions on a particular system.

-f Similar to the F flag, except that the partition number is also used in the calculation of this string. The resulting identification string is unique for each partition on a particular system.

So, no problem here. Nothing to see, nothing to do. Move along!

Modified by cggibbo

NIM Master, NIMSH and SSL on AIX 7.1 TL4 SP3

cggibbo Tags: nimsh nim sp3 hang lslpp chris ssl 7.1 aix 7100-04-03-1642 tl4 gibson -o iv93152 libssl.so 2,667 Views

Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.

[root@750lpar4]/ # nim -o lslpp 750lpar9

The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:

…

Thu Jan 12 14:31:49 2017 Loading certificates..

Thu Jan 12 14:31:49 2017 Loading private key file..

Thu Jan 12 14:31:49 2017 create BIO

…

NIM master: 750lpar4

7100-04-03-1543

NIM client: 750lpar9

7100-04-02-1614

[root@750lpar4]/ # lsnim -l 750lpar9

750lpar9:

class = machines

type = standalone

connect = nimsh (secure)

platform = chrp

netboot_kernel = 64

if1 = 10_1_50 750lpar9 0

cable_type1 = N/A

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F603CD4C00

Cstate_result = success

The root cause of the problem become apparent when we ran truss against the nim –o command.

[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9

[root@750lpar4]/ # cat truss.lsnim.out

…

13959372: C o u l d n o t l o a d m o d u l e / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m e r r o r : N o s u c h

13959372: f i l e o r d i r e c t o r y

…

19267612: C o u l d n o t l o a d m o d u l e / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m e r r o r : N o s u

19267612: c h f i l e o r d i r e c t o r y

The required shared library object files were missing on the NIM master.

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found

We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.

[root@750lpar4]/usr/lib # slibclean

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x 1 root system 724913 Jan 18 09:08 libssl.so

-rwxr-xr-x 1 root system 3031337 Jan 18 09:08 libcrypto.so

After that, the nim –o commands started working again.

[root@750lpar4]/usr/lib # nim -o showlog 750lpar9

HELLO

So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run. But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.

I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.

Here’s one good reason to setup NIMSH over SSL.

NIMSH, SSL and LPM

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en

The following link is a great reference guide for configuring NIMSH over SSL.

NIMSH over SSL

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024615

Modified by cggibbo

Feed for Blog Entries ▼

Blogs

Chris's AIX Blog

▼ Tags

▼ Similar Blogs

AIX Down Unde...

Application P...

Asset Managem...

Application I...

AIXpert Blog

▼ Similar Ideation Blogs

IBM PureData-...

IBM Forms Exp...

WebSphere App...

IBM Platform ...

Rational Soft...

▼ Archive

▼ Blog Authors

Chris's AIX Blog

Is that a versioned WPAR I see before me?

NIM - "...the resource is currently allocated for client use" - er, NO it's not!

NIM support for installing AIX updates and new packages over HTTP

Queue depth setting in a Versioned WPAR on AIX

Troubleshooting AIX hdisk configuration errors

Replicated rootvg, ghostdev and NPIV

How much memory or CPU will that CoD code activate anyway?

JFS2 logging disabled during mksysb restore.

AIX or Linux on POWER?

rmsock on AIX.

AIX 7 Open Beta is live - it's time to test!

saposcol and shared memory on AIX.

Migrating you NIM Master to AIX 7.2 using the VIOS VML

AIX, LPM & Partition ID sorcery! Why does AIX report that my partition ID has changed after LPM when it hasn't?

NIM Master, NIMSH and SSL on AIX 7.1 TL4 SP3