Gathering logs in a VMware environment

How to gather the API Connect system logs and configuration information for use in troubleshooting and diagnostics.

When you open a support case for API Connect, system logs and environment data are normally required to help diagnosis. Several tools are used for gathering this data:

  • The apic-mustgather tool tool collects all the logs and system information from the management, portal, and analytics VMs.
  • The apicops tool gathers extra debug information about the synchronization of the management subsystem with the gateways.
  • DataPower error reports and debug logs provide the information that is required from the gateway appliance.

Installing the apic-mustgather tool

Download the latest version of the apic-mustgather tool from https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather, copy it to your API Connect VMs, and give it execute permissions. For example:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
scp apic-mustgather apicadm@<API Connect VM hostname>:/home/apicadm
ssh apicadm@<API Connect VM hostname>
sudo chmod +x apic-mustgather
Important: If you downloaded this tool previously, check that you still have the latest version:
sudo ./apic-mustgather --version

The apic-mustgather tool gathers OS-level information from each VM, so if you have a three replica API Connect deployment, it is recommended to install and run the apic-mustgather tool on each VM in the cluster.

The apic-mustgather tool gathers the logs of the currently running processes. The logs have a maximum size and older log messages are not retained when the maximum size is reached. To ensure that the logs gathered cover the time when the problem occurred, reproduce the problem and then gather the logs without delay. Offboarding the logs to an external server is the best practice for long-term storage of log data. See Configuring remote logging for a VMware deployment.

Gathering the management subsystem data on VMware

  1. Install the apic-mustgather tool and apicops on all your management VMs.
  2. Re-create the problem that you want to diagnose.
  3. On each API Connect VM, run the following commands:
    sudo ./apic-mustgather
    sudo apic status > apic_status.out
    sudo apic version > apic_version.out
  4. Copy off the generated files from each VM:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
  5. Create an archive of your project directory:
    sudo tar -cf projectdir.tar <project directory path>
  6. Upload the files to the support case:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
    • projectdir.tar

Gathering the portal subsystem data on VMware

  1. Install the apic-mustgather tool and apicops on all your management VMs.
  2. Re-create the problem that you want to diagnose.
  3. On each API Connect VM, run the following commands:
    sudo ./apic-mustgather
    sudo apic status > apic_status.out
    sudo apic version > apic_version.out
  4. Copy off the generated files from each VM:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
  5. Create an archive of your project directory:
    sudo tar -cf projectdir.tar <project directory path>
  6. Upload the files to the support case:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
    • projectdir.tar

Gathering the analytics subsystem data on VMware

  1. Install the apic-mustgather tool and apicops on all your management VMs.
  2. Re-create the problem that you want to diagnose.
  3. On each API Connect VM, run the following commands:
    sudo ./apic-mustgather
    sudo apic status > apic_status.out
    sudo apic version > apic_version.out
  4. Copy off the generated files from each VM:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
  5. Create an archive of your project directory:
    sudo tar -cf projectdir.tar <project directory path>
  6. Upload the files to the support case:
    • /tmp/apic-mustgather-*.tar.gz (apic-mustgather tool output)
    • apic_status.out
    • apic_version.out
    • projectdir.tar

Gathering the DataPower appliance subsystem data

Before you reproduce the problem, configure DataPower and management subsystem logging as described in the following steps:

  1. Use SSH to log in to each gateway and access the DataPower CLI. For more information about accessing the DataPower CLI, see Command-line access in the DataPower documentation.

  2. Configure the log target in the API Connect application domain. Repeat these steps for each gateway server in the cluster:

    sw <apiconnect domain>
configure terminal
logging target gwd-log
type file
format text
timestamp zulu
size 50000
local-file logtemp:///gwd-log.log
event apic-gw-service debug
exit
apic-gw-service;admin-state disabled;exit
apic-gw-service;admin-state enabled;exit
write mem
exit
  3. OPTIONAL: Enable debug of gateway-peering on each gateway. 
    sw <apiconnect domain>
diagnostics
gateway-peering-debug <gw peering object name>
exit
    Replace <gw peering object name> with the correct name of the peering object. To identify the peering object name, run the following command within the API Connect domain: 
    show gateway-peering-status
  4. Install the apic-mustgather tool and apicops on all of your management subsystem VMs.

Reproduce the problem, taking note of the exact time of the reproduction, and then follow these steps to gather the logs:

  1. Gather the management subsystem logs, following the steps in Gathering the management subsystem data on VMware.
  2. On one of the management VMs, run the apicops command and collect the output:
    ./apicops debug:info
  3. Generate an error-report on each gateway in your gateway cluster:
    sw default
conf; save error-repor
  4. If you enabled gateway peering debug, then get the log file and disable debug logging. Run the following command for each gateway in the cluster. Replace <gw peering object name> with the correct peering object name.
    sw <apiconnect domain>
diagnostics
gateway-peering-dump <gw peering object name>
no gateway-peering-debug <gw peering object name>
exit

    To identify the <gw peering object name>, run the following command within the API Connect domain:

    show gateway-peering-status

Upload the following to the support case:

  • Management subsystem apic-mustgather and apicops output.
  • Gateway service logs: Upload the logs at logtemp://gwd-log.log within the API Connect domain.
  • Gateway error reports: Include the error report file named <error report filename>.txt.gz.
  • If you enabled gateway peering debug, then upload gatewaypeering.log and gatewaypeeringmonitor.log from temporary:///<name of gateway peering object in API Connect application domain>.
  • Optional: Other data for troubleshooting specific APIs:
    • Temporary files: Include all relevant files that are located under temporary://.
    • Configuration and YAML Files: Upload YAML files related to the failing API and the DataPower configuration for the application domain.
    • Transaction Probes: Provide data from the probe of the failing transaction as outlined in the API probe configuration instructions.
  • Optional: Special considerations for v5-compatible gateways:
    • YAML Files: Include related YAML files.
    • Transaction probes: Upload data from the probe of the failing transaction.
    • Document cache: Provide an export of the document cache for webapi and webapi-internal.
Important: Ensure to update the support case with the date and time of when the problem occurred or was reproduced.