Z APM Connect Distributed Gateway possible error scenarios

Edit online

Learn how to correct or avoid some of the possible errors that can occur when you install and configure Z APM Connect DG, and enable security connection between Z APM Connect DG and z/OS®.

 Possible error scenarios: installing and configuring 

 Possible error scenarios: enabling security connection 

 Submitting problems to IBM Software Support 

If you cannot determine the cause of the errors, you can gather diagnostic information about the problem and contact IBM Software Support for a solution. For more information, see Submitting problems to IBM Software Support.

Z APM Connect DG is not running

You can check the status of Z APM Connect DG on the Z APM Connect DG machine by running the following command:
Installation Format Command
OpenShift 
oc get pods
Kubernetes 
kubectl get pods
Docker standalone 
sudo docker ps | grep zapm

You will see a STATUS column containing the amount of time the container has been up as well as the health.

For example: 
Up 3 hours (healthy)
If any container is unhealthy, you can find out more by running:
Installation Format Command
OpenShift 
oc describe [pod name]
Kubernetes 
kubectl describe pods [pod name]
Docker standalone 
docker inspect [container]

Z APM Connect DG is not started

After issuing the proper command to start the Z APM Connect DG, if it does not start or stay up, you can look for any errors in the logs.

The most common errors are as follows:
  • (Docker standalone only) The listening port was already in use by another process. Ensure that port 30455 is available on the VM.
  • The Z APM Connect DG is improperly configured to communicate with Instana. Ensure that credentials are not invalid: access key, username/password, OAuth token. Check the logs for errors.
  • The Z APM Connect DG is set to listen for connections from z/OS using TLS but a truststore or keystore are missing or the passwords that are set are incorrect.

Z APM Connect DG is not restarted automatically or is shut down in an unsafe manner

Installation Format Command
OpenShift If Z APM Connect DG starts successfully, OpenShift handles crashes and restarts pods as needed. If pods are crashing consistently, it is likely due to resources limits and allocating more nodes to the cluster will fix it. If it crashes immediately upon startup, the problem is likely with Instana credential configuration.
Kubernetes If Z APM Connect DG starts successfully, Kubernetes handles crashes and restarts pods as needed. If pods are crashing consistently, it is likely due to resources limits and allocating more nodes to the cluster will fix it. If it crashes immediately upon startup, the problem is likely with Instana credential configuration.
Docker standalone Docker attempts to restart a crashing container three times. If a container crashes three times, Z APM Connect DG will need to be restarted manually. Upon crashing three times, the output of docker inspect [container] will contain the necessary logs or reason for the failure.

Z APM Connect DG fails to communicate with the Instana backend

Z APM Connect DG is not configured properly to communicate with the Instana backend. The Instana agent endpoint URL, agent key, or both are incorrect, or are changed.

You can change or update the Instana server configuration information.
Installation Format Command
OpenShift Edit the values.yaml, ensure the credentials are correct and issue the following command.
helm upgrade --namespace <desired project> -f <config yaml> <name> <chart location>

z/OS Connect EE is unable to connect to Z APM Connect DG

Edit agmconfig.properties which is located under the AGMMonitor/config/... directory. The bootstrap.servers property needs to be set to the hostname and port of the Kafka server.
Note: This file is in ASCII format. Ensure the value for bootstrap.servers matches exactly the value specified on the distributed side.
This varies based on installation options:
  • Standalone: hostname of the Z APM Connect DG
  • Non-helm: value for KAFKA_CFG_ADVERTISED_LISTENERS in kafka.yaml
  • Helm: value for advertisedHostname in the kafka section of values.yaml . If the two values do not match then DNS mapping will be required. For more information, see Configuring the local host table (optional).

SSL secure communication is not established between IBM Z APM Connect Base and Z APM Connect DG

The following errors are likely:
  • The security settings of Z APM Connect DG or the Z APM Connect Base are not configured properly during the installation.
  • The security settings are changed after your configuration.
You can solve the problem by checking:
  • Whether your z/OS machine is configured to use a secure (TLS/SSL) port to connect to the Z APM Connect DG.
  • Whether the Z APM Connect DG is configured similarly.
  • The security settings for connecting to the Instana server and port.

Imported certificate to RACF already exists

If the imported certificate already exists, browse the certificates to find the label name. Use the following command to display the information about the CA certificates owned by the command issuer:
RACDCERT LIST CERTAUTH
If the certificate is not found, remove CERTAUTH and issue the RACDCERT LIST command to display the information about the CA and site certificate owned by the command issuer:
RACDCERT LIST
If you are looking for a certificate created under a particular ID, use ID(<userid>) instead of CERTAUTH:
RACDCERT LIST ID(userid)

Procs are under different user IDs but send to the same Linux server

Take the following actions to solve this problem:
  • Ensure that each users ID has access to the root certificate.
  • With a RACF® key ring, confirm that the key ring contains an internal CA, a personal certificate per user ID, and the imported CA certificate from the Linux machine.

The date range is incorrect

In RACF, use the RACDCERT LIST command to see the end dates of the certificates.

The certificates might need to be renewed. Check with your local process for monitoring certificate expiry and renewal to avoid outages. For more information about RACF certificate renewal, see Renewing an expiring certificate.