Network Tracing Techniques for RMC Connection Issues

Answer

When trying to debug the RMC communication channel between a management console such as the HMC and a logical partition its important to keep in mind the terminology IBM Power Systems developers use to reference Private and Public connections. The private connection is the communication channel between the HMC and the managed server's Service Processor. Its intended to be on a private network separated from production related traffic. The public connection is for remote communications to the management console from users and for the management console to communicate with logical partitions (LPARs) or other external systems. The figure below shows a private connection (blue and red lines) from a pair of redundant HMCs to a managed server as well as the public communications channel (in green) to LPARs. The HMC to LPAR communications is called the RMC connection. As the green cloud symbol suggests below, there can be many varied networks that RMC communications traffic might have to flow across. Sometimes the RMC network traffic flow over such a complex network route that debugging the communications related to it can require networking tracing tools.

The application traces available for RMC are enabled by default and gathered by the ctsnap data collection script. The application traces do not trace network activity, but can indicate if there is some network issue preventing RMC from working. If additional network debugging is required for complex configurations then OS tools such as tcpdump (Linux or AIX) and iptrace (AIX) might need to be used. If you are asked by IBM Support to collect tcpdump and iptraces for an LPAR that is having RMC connection problems then you will want to do so in a matter that captures sufficient data to debug the connection properly. This typically involves using the commands as described below.

1. Start tracing on both LPAR and HMC

a. Start capturing AIX or VIOS iptrace data for an RMC issue (as root) by running following command.

startsrc -s iptrace -a "-a -b -p 657 /tmp/<PMR>-iptrace.cap"

b. Start tracing on HMC (as root.) using tcpdump. Make sure you have pesh passwords for the HMC and can access the root shell prior to starting.

tcpdump -i any -w /tmp/<PMR>-tcpdump.cap -s 0 port 657

NOTE: this will run in the active shell till you terminate it.

2. Create traffic flow over the RMC channel by stopping and starting the subsystem as root on the LPAR.

/usr/sbin/rsct/bin/rmcctrl -z

/usr/sbin/rsct/bin/rmcctrl -A

/usr/sbin/rsct/bin/rmcctrl -p

3. Wait two to three minutes to allow the RMC communication attempts between HMC and LPAR to pass then stop the tracing as follows.

a. On AIX run following command

stopsrc -s iptrace

b. On HMC use the "Ctrl-C" key sequence within the trace shell to terminate tcpdump

4. Send the data to IBM along with other diagnostic files they might have requested (typically pedbg from HMC and snap and ctsnap from AIX or VIOS).

a. On AIX or VIOS the collection of ctsnap is as follows (as root)

/usr/sbin/rsct/bin/ctsnap -x runrpttr

(collects archive in /tmp/ctsupt)

b. On HMC collecting a pedbg is done as user hscpe

pedbg -c -q 4

(collects archive in /dump)

5. Make sure your trace files are marked with the PMR, branch and country code as a file prefix so the data can be automatically processed in IBM's tool systems.

Example: If PMR is 12345, branch 678, and country code 000 (12345.678.000 then the file names should begin with 12345.678.000. So your iptrace should be named 12345.678.000-iptrace.cap and your tcpdump 12345.678.000-tcpdump.cap.

Where you send the data depends on whether or not you can follow normal data transmission procedures and send them to IBM's testcase server or ECuRep. If you have to follow special handling procedures such as those specified for IBM's Blue Diamond project then send appropriately.

Normal delivery of Electronic Data is to use ECuRep tools

Healthcare Customers than need to follow special handling of data should use the Blue Diamond support portals.