Gathering logs for a VMware environment

The generate_postmortem.sh script gathers all logs for troubleshooting and diagnostics.

About this task

When you contact IBM Support, logs and VMware environment data are normally required to help diagnosis. Run the generate_postmortem.sh script on each API Connect VM to gather all the required logs and data. The data that is gathered by the script includes OS-level information from each VM, so if you have a three replica deployment, it is recommended to run the generate_postmortem.sh script on each VM in the cluster.

The logs that are gathered by the generate_postmortem.sh script include the logs of the currently running API Connect processes. The logs have a maximum size, and older log messages are not retained when the maximum size is reached. To ensure that the logs gathered cover the time when the problem occurred, it is recommended to reproduce the problem and then immediately gather the logs. Offboarding the logs to an external server is the recommended best practice for long-term storage of log data. See Configuring remote logging for a VMware deployment.

Procedure

  1. Connect to the target appliance with SSH:
    ssh apicadm@<ova appliance hostname>
  2. Switch to the root user:
    sudo -i
  3. Download the generate_postmortem.sh script. Enter the following command:
    curl -s -o generate_postmortem.sh https://raw.githubusercontent.com/ibm-apiconnect/v10-postmortem/master/generate_postmortem.sh
    Note: If your API Connect VMs do not have external internet access, download the generate_postmortem.sh script to another computer that has internet access, and copy it to your API Connect VMs.
  4. Add execute permissions to the generate_postmortem.sh script. Enter the following command:
    chmod +x generate_postmortem.sh
  5. Run the postmortem tool by using the following command:
    ./generate_postmortem.sh --ova --pull-appliance-logs
  6. Optional: Specify either management, portal, or analytics for more detailed logs from that subsystem. Specify --diagnostic-all for detailed logs of every subsystem:
    • Portal:
      ./generate_postmortem.sh --ova --diagnostic-portal --pull-appliance-logs
    • Management:
      1. Download an additional file that gathers management database information: crunchy_gather.py:
        wget https://github.com/ibm-apiconnect/v10-postmortem/blob/master/crunchy_gather.py
      2. Run generate_postmortem.sh with --diagnostic-manager:
        ./generate_postmortem.sh --ova --diagnostic-manager --pull-appliance-logs
    • Analytics:
      ./generate_postmortem.sh --ova --diagnostic-analytics --pull-appliance-logs
    Note: The diagnostic level logs generated with the --diagnostic-<subsystem name> argument produce a much larger output file.
  7. If you see errors when you run the generate_postmortem.sh script, run again with --debug:
    ./generate_postmortem.sh --ova --debug &>debug.log

    For more documentation on the script, see https://github.com/ibm-apiconnect/v10-postmortem.