MustGather: AIX PowerHA SystemMirror problems

The following procedures are for data collection, not for problem diagnosis. Gathering this information before calling IBM support can aid in problem determination and save time resolving Problem Management Records (PMRs).

Run # zsnap --HACMP

This zsnap command gathers PowerHA data and creates the testcase file in one step. If you already have a PMR number, see the example below.

• Data

  The zsnap command for PowerHA SystemMirror gathers the same information as snap at this time. The data include:

  • Data from both nodes
  • CAA data (PowerHA 7.1 and up)
  • RSCT information (PowerHA 6.1 and lower)
  • AIX information: bootinfo, lslpp, emgr, lsdev disk data, lspv lsvg, lsfs, mount, df, lscfg, lsattr on fibre channel adapter, process table, env data
  • Network information: netstat -in, netstat -rn, netstat -v, netstat -m, lsdev adapter and interface data, tty, lsattr on network adapters ODM data for both PowerHA and AIX
  • error report
  • Configuration files: clhosts, clinfo.rc, harc.net, netmon.cf, rhosts, clip_config,environment, inetd.conf, limits, profile, resolv.conf, snmpd.log, snmpdv3.log, filesystems, inittab, netprobe.log, rc.net, services, snmpd.peers, syslog.conf, clvg_config, hosts, ipHarvest.log, netsvc.conf, rc.nfs, snmpd.conf and snmpdv3.conf, ifrestrict
  • AHAFS data
  • PowerHA logs: autoclstrcfgmonitor.out, autoverify.log, cell temp log, clverify, clavan.log, cluster.log, clcomd.log, clcomddiag.log,clconfigassist.log, hacmp.out clstrmgr.debug, clstrmgr.debug.long clevents, clevmgrdevents, clinfo.log, clutils.log, clver_CA_daemon_invoke_client.log, clver_debug.log, cspoc.log, dhcpsa.log, dnssa.log, domino_server.log, emuhacmp.out, hacmprd_run_rcovcmd.debug, application monitor logs, smart assistant logs, smit.log, migration.log
  • PowerHA data: hostname information, cllsif information, cluster state data, cluster daemon data, resource group information, cluster topology information
• Example

  See zsnap usage for all available options.

  # zsnap --HACMP --pmr 12345,123,123

  The example gathers the appropriate data and creates a testcase file with the IBM standard naming convention for quicker processing. You will be prompted to send the file to IBM using the FTP protocol. If you don't have a PMR number, omit the --pmr flag to build the testcase file.

  You can also run the zsnap command from the AIX SMIT menus.

  For information on sending testcase information, read Sending Testcases Electronically.

The snap command is the standard AIX tool that gathers data and stores that information in /tmp/ibmsupt/. The snap command does not gather the following additional PowerHA related information.

• Data

  See zsnap Data section above for the data collected by the snap command.

• Sample snap procedure for PowerHA

  See snap usage for all available options

  Follow these steps to gather the PowerHA data.

  1. Run the snap -r command to remove all previously gathered data on all of the nodes in the cluster.
  2. Gather the additional information and put it in /tmp/ibmsupt/testcase. You may need to recreate the testcase directory.
  3. Run # snap -e on just one node.
  4. Rename the testcase file to adhere to IBM testcase file naming conventions, and then send the file to IBM.
  5. For information on sending testcase information, read Sending Testcases Electronically.

The following procedures are for data collection, not for problem diagnosis. Gathering this information before calling IBM support can aid in problem determination and save time resolving Problem Management Records (PMRs).

This clsnap command is the command on Linux that gathers data and stores that information in /tmp/ibmsupt/.

• Sample snap procedure for PowerHA on Linux

  Follow these steps to gather the PowerHA on Linux data.

  1. To gather logs from all cluster nodes, run on just one node:
     
     # clsnap
  2. A compressed tar file is created under the /tmp/imbsupt/hacmp directory.
  3. Rename the testcase file to adhere to IBM testcase file naming conventions, and then send the file to IBM.
  4. For information on sending testcase information, read Sending Testcases Electronically.

The following procedures are for data collection, not for problem diagnosis. Gathering this information before calling IBM support can aid in problem determination and save time resolving Problem Management Records (PMRs).

The snap command is the standard AIX tool that gathers data and stores that information in /tmp/ibmsupt/.

• Sample snap procedure for GDR and VMR

  Follow these steps to gather the GDR and VMR data.

  1. On the KSYS node, run the snap -r command to remove all previously gathered data
  2. On the KSYS node run:
     
     # snap vmsnap
  3. Up to VMR 130 Service Pack 1, the above command creates a testcase file named /tmp/ibmsupt/ksys.pax.Z .
     Starting from VMR 130 Service Pack 2 and later, the above command creates a testcase file named /tmp/ibmsupt/vmsnap/ksys.pax.Z .
  4. Rename the testcase file to adhere to IBM testcase file naming conventions, and then send the file to IBM.
  5. For information on sending testcase information, read Sending Testcases Electronically.