Troubleshooting
Problem
This document describes the general information and diagnostic data needed to start troubleshooting issues related to IBM Automation Decision Services.
The document applies for the Standalone edition and the Cloud Pak for Automation edition.
When you open a case for problems related to Automation Decision Services, include the diagnostics gathered following this document to expedite its resolution.
Resolving The Problem
Overview of Automation Decision Services diagnostic information
Collect diagnostic information - must provide for every case
Component specific diagnostic configuration - provide for the specific component you are having an issue with
Everything needed to collect Automation Decision Services diagnostic information is detailed here.
- Provided a detailed description of your issue. Include screen captures and steps to recreate the issue if possible.
- Is the issue intermittent or recreatable? Has there always been an issue or did it start only after a change occurred?
- What is the business impact? Is the issue impacting any deadlines?
- Provide a reference to the documentation used when the issue happens.
- Which platform are you using: OpenShift, managed OpenShift, other Kubernetes platform?
- Provide version (and vendor as needed) for the following products:
- Automation Decision Services deployment version
- Database vendor and version.
- LDAP vendor and version.
- If this installation is an IBM Cloud Pak for Automation deployment, what version are you using?
- For deployment issues, were images downloaded from Entitled Registry or Passport Advantage?
To collect information automatically follow this steps:
- Download the Automation Decision Services repository
It provides shell scripts to collect data automatically. - Unzip the repository
- Move to the must-gather directory.
- Follow the instructions in the readme file to gather diagnostics automatically
If you prefer gathering information manually follow these steps.
Note:
- Run diagnostic commands from an empty directory to make it easy to package the files collected.
- Run the commands from the project or namespace containing Automation Decision Services or use the -n <namespace> flag with all kubectl commands.
Steps:
- For installation on OpenShift provide the output of this command:
oc version > version.txt
- Provide the output of this command:
kubectl version >> version.txt
Note:
- Running the two previous commands in order produces a single versions.txt file containing both versions
- For installation on other Kubernetes technology provide version and architecture
- Provide the Custom Resource (CR) .yaml file used by the operator to set up the environment
For Cloud Pak installation:kubectl get icp4acluster -o yaml > config.yaml
kubectl get AutomationDecisionService-o yaml > config.yaml
-
kubectl get nodes -o wide > nodes.txt
- Collect information about the pod statuses
kubectl get pods > pods.txt
- Collect information about the pod containers
kubectl get pods -o jsonpath="{..image}" > containerInfo.txt
- On OpenShift gather route configuration
oc get route > routes.txt
- For installation on other technologies, get the ingresses
kubectl get ingresses > ingresses.txt
Collect the defined secrets -
kubectl get secrets > secrets.txt
- Collect the defined persistent volume claims
kubectl get pvc > pvcs.txt
- Collect the description and log of any pod that you are having issues with:
kubectl describe pod <pod-name> > describe-<podname>.txt kubectl logs <podname> > log-<podname>.log
- On Openshift gather the operator log
namespace=$(oc project -q) icp4a_instance=$(oc get icp4acluster -o=jsonpath='{.items[0].metadata.name}') operator_pod=$(oc get pod | grep operator | awk '{print $1}') oc cp ${operator_pod}:/tmp/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/${namespace}/${icp4a_instance}/artifacts/latest/stdout operator.log
- Tar the collection directory and send it to the support team.
Use the undermentioned trace strings and follow steps in the section How to set trace strings and collect logs? to generate the data.
The name of the pod on which the generated logs are available is provided in each section.
The name of the pod on which the generated logs are available is provided in each section.
- REST API
CR format Configuration Trace location standalone spec.rest_api.log_trace_specification In the ads-rest-api pod Cloud Pak spec.ads_configuration.rest_api.log_trace_specification In the icp4adeploy-ads-rest-api pod Trace String Description com.ibm.odm.expresso.web=finer Show debug level logs about the rest api operations com.ibm.odm.foundation.adapters=finer Show debug level logs about operations on decision artifacts org.apache.httpcomponents=finer Log all http traffic going through the httpclient with debug level - Git Service
CR format Configuration Trace location standalone spec.git_service.log_trace_specification In the ads-git-service pod Cloud Pak spec.ads_configuration.git_service.log_trace_specification In the icp4adeploy-ads-git-service pod Trace String Description com.ibm.odm.expresso=finer Show debug level logs about git operations com.ibm.odm.gitservice=finer Show debug level logs about the service global operation and mongo db related logs For enterprise GitHub, confirm that the Git id and Git Access Token are used for making the connection. Do not use your email address that is used for SSO.- Parsing Service
CR format Configuration Trace location standalone spec.parsing_service.log_trace_specification In the ads-parsing-service pod Cloud Pak spec.ads_configuration.parsing_service.log_trace_specification In the icp4adeploy-ads-parsing-service pod Trace String Description com.ibm.odm.parsing=finer Show debug level logs about the parsing service global operation com.ibm.odm.foundation.adapters=finer Show debug level logs about build and parsing of decisions
- Parsing Service
- Run Service
CR format
Configuration
Trace location
standalone
spec.run_service.log_trace_specification
In the ads-run-service pod
Cloud Pak
spec.ads_configuration.run_service.log_trace_specification
In the icp4adeploy-ads-run-service pod
Trace String Description com.ibm.odm.runservice=finer Show debug level logs about the run service global operation com.ibm.odm.foundation.adapters=finer Show debug level logs about build and running of decisions
- If the deploy link is unavailable in the Deploy tab, confirm that:
- the project is connected to a Git repository,
- changes are shared,
- a version is created.
- If a failure is seen while building or deploying a decision service, check the errors by clicking the View logs icon next to the deployment status.
Copy and save this log to a file deployment.log in the collection directory.
To include the exception stack trace in the response payload set a property to true in the CR
CR format | Configuration | Trace location |
---|---|---|
standalone | spec.: true | In the response payload |
Cloud Pak | spec.ads_configuration.: true | In the response payload |
Follow these steps to enable tracing in the Automation Decision Services containers.
- Edit the CR .yaml file used by the operator to create the Automation Decision Services pods. You can use the file gathered earlier in this document. Modify the log_trace_specification property in the Automation Decision Services logs section of the yaml file. Follow the documentation syntax defined for the traceSpecification attribute .
spec:
...
ads_configuration:
component_as_informed_by_support_team:
log_trace_specification: '*=info:component_specific_trace_string_as_requested_by_support_team' - Update the operator with the new configuration.
kubectl apply -f my_cr.yaml --overwrite=true
- For Cloud Pak installation, the operator can take up to 30 minutes to recognize the change and update the configuration.
- You can grep the log file for traceSpecification to see when the trace settings change.
- By default, the logs are saved in the standard output of the respective pods in JSON format. Collect the logs, FFDC, and trace by tarring the files on the respective pod.
- Disable the trace by setting log_trace_specification back to "*=info" and applying the CR changes again.
Document Location
Worldwide
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS2JQC","label":"IBM Cloud Pak for Automation"},"ARM Category":[{"code":"a8m3p000000GnuZAAS","label":"Automation Decision Service-\u003EADS App Development"},{"code":"a8m3p000000GnuaAAC","label":"Automation Decision Service-\u003EADS App Management"},{"code":"a8m3p000000GnuVAAS","label":"Automation Decision Service-\u003EADS Install\\Upgrade\\Setup"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD1E4E","label":"IBM Automation Decision Services"},"ARM Category":[{"code":"a8m3p000000hCCwAAM","label":"ADS standalone"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
01 June 2023
UID
ibm16405888