MustGather: Collecting data to diagnose issues with Business Automation Insights (BAI)

Troubleshooting

Problem

This document describes the general information and diagnostic data needed to start troubleshooting issues related to Business Automation Insights (BAI), which is included in IBM Cloud Pak for Automation. Include the diagnostics retrieved from using this document when opening a case for Business Automation Insights (BAI) or Cloud Pak for Automation.

Resolving The Problem

Overview of Business Automation Insights diagnostic information

As needed diagnostic information

BAI for a Server
Emitters
Operator Logs
Downloadable MustGather scripts

Detailed diagnostic collection steps

These steps are the detailed steps to gather different types of data for Business Automation Insights (BAI). When running diagnostic commands, run them from an empty collection directory to make it easy to package the files. Run the commands from the project or namespace containing BAI or use the -n <namespace> flag with all kubectl commands.

1: Provide a detailed description of the problem and your environment

Provided a detailed description of your issue. Include screen captures and re-create steps if possible.
Is it an intermittent or recreatable issue? Has this always been an issue or an issue that started only after a change occurred?
What is the business impact? Should we be aware of any deadlines impacted by the issue?
Provide a reference to the documentation being followed for the failing operation
Which platform are you using (OpenShift, managed OpenShift, other Kubernetes platform)?
What database is used by UMS [Derby, Db2, Oracle]?

2: Gather the following configuration information

If using OpenShift, provide the output of this command:
```
oc version > version.txt
```
Provide the output of this command:
```
kubectl version >> version.txt
```
Note: Running the two commands above in order will produce a single versions.txt file containing both versions
Provide the Custom Resource(CR) .yaml file used by the operator to set up the environment
```
kubectl get icp4acluster -o yaml > config.yaml
```
Collection information about the nodes.
```
kubectl get nodes -o wide > nodes.txt
```
Collect information about the pod statuses
```
kubectl get pods > pods.txt
```

Collect information about the pod containers

kubectl get pods -o jsonpath="{..image}" > containerInfo.txt

On OpenShift gather route configuration
```
kubectl get route > routes.txt
```
Note: If needed, more detailed route config information can be gotten with -o yaml option
Collect the defined secrets
```
kubectl get secrets > secrets.txt
```
Collect the defined persistent volume claims
```
kubectl get pvc > pvcs.txt
```

Collect the config map

kubectl get configmap <release-name>-bai-env -o yaml

Collect the event logs

kubectl get events --sort-by=.metadata.creationTimestamp

Collect the description and log of any pod you are having issues with (If your UMS is connected to DB2 collect log from the DB2 pod as well):
```
kubectl describe pod <pod-name> > describe-<podname>.txt
kubectl logs <podname> > log-<podname>.log
```
As of version 21.x, Business Automation Insights relies on IBM Automation Foundation for many components such as Elasticsearch and Flink. If experiencing issues with these components, please collect data from IBM Automation Foundation MustGather as well.

3: BAI for a server

Provide the log for each service using

<BAI_Install_Root>/bin/bai-logs <service>

Provide the main configuration file
```
<BAI_Install_Root>/.env
```

Provide the custom configuration located in

tar cvf config.tar <BAI_Install_Root>/config

Provide the output of the following commands

docker-compose -f <BAI_Install_Root>/data/bai.yml images

docker-compose -f <BAI_Install_Root>/data/bai.yml ps

4: Emitters

BPM Emitter
- Provide the SystemOut.log for the BPM/BAW server
- BAW On-prem tracing
  - Enable the trace in the WebSphere Console
    1. In the WebSphere Integrated Solutions console, navigate to Troubleshooting > Logs and Trace
    2. For each support cluster member, select the server then Change log detail levels
    3. Switch to the Runtime tab and add the following trace string This change is dynamic so the servers do not need to be restarted
      - com.ibm.bpm.mon.oi.*=all
      - For playing back events from Performance Data Warehouse (19.0.1+), append: com.ibm.bpm.bui.emitter.*=all
    4. Reproduce the issue and provide the resulting trace.log files for the servers
- For BAW on Containers, set the trace string using the instructions here: BAW on Containers MustGather
- Check that DEF is enabled
  - The defconfig.xml file must be present in the Business Automation Workflow Dmgr profile
    <BAW_Install_Root>\profiles\DmgrProfile\config\cells\<CellName>
    and should be replicated (identical) on all Business Automation Workflow nodes configuration
    <BAW_Install_Root>\profiles\Node1Profile\config\cells\<CellName>
  - Provide the defconfig.xml file
- Collect DEF details
- Provide all the BPM event emitter BPMEventEmitter.yml configuration files
  Typical locations are:
  - <BAW_Install_Root>\profiles\DmgrProfile\config\cells\<CellName>\clusters\<ClusterName>\analytics\config\BPMEventEmitter.yml
  - <BAW_Install_Root>\profiles\DmgrProfile\config\cells\<CellName>\nodes\<NodeName>\servers\<ClusterMemberName>\analystics\config\BPMEventEmitter.yml
  - <BAW_Install_Root>\profiles\NodeProfile\config\cells\<CellName>\nodes\<NodeName>\servers\<ClusterMemberName>\analytics\config\BPMEventEmitter.yml
Case Emitter
- Provide the SystemOut.log for the Business Process Manager/Business Automation Workflow server
- Business Automation Workflow On-prem tracing
  - Enable the trace in the WebSphere Console
    1. In the WebSphere Integrated Solutions console, navigate to Troubleshooting > Logs and Trace
    2. For each application cluster member, select the server then Change log detail levels
    3. Switch to the Runtime tab and add the following trace string
      com.ibm.dba.oi.*=finest:com.ibm.oi.icm.*=finest
      This change is dynamic so the servers do not need to be restarted
    4. Reproduce the issue and provide the resulting trace.log files for the servers
- For Business Automation Workflow on Containers, set the trace string using the instructions here: BAW on Containers MustGather
- Provide a screen capture of the ContentEngine's Audit Processing Bookmark
  - Be sure to include the bookmark entry with the display name of "Emitter_[logicalUniqueId-configured]_*"
  - Navigate to ACCE > Object Store > [Your Target Object Store] > Administrative > Audit Disposition > Audit Processing Bookmarks
    Note the "Last processed" value from the bookmark entry
- CaseEventEmitter.json file found inside the deployed CaseEventEmitter application within the WEB_INF/classes directory
- Query Content Engine Target Object store database to get the export of the events. Get top 500 events from "Event" table with "audit_sequence" greater than the "Last processed" value noted from the bookmark entry.
  Sample Query:
  SELECT TOP 500 * FROM Event WHERE audit_sequence > {Last processed value from bookmark} ORDER BY audit_sequence asc
Content Event Emitter
- Log in to your Kafka broker server
- Go to your KAFKA_HOME/config directory
- Edit the log4j.properties file and set log4j.logger.kafka=DEBUG,kafkaAppender
- Restart the Kafka broker server
- Collect the server.log in the KAFKA_HOME/logs directory
ODM Event Emitter
- See Troubleshooting the ODM Event Emitter

5: Collect Operator Logs

If you are having issues during the deployment by the operator then collect the operator logs described in the installation troubleshooting page.

6: Downloadable MustGather scripts

Each of the following downloadable .zip files provides a shell script together with a readme file have been for you to collect data automatically. Please download the one you need and extract it into a location of your choice. Read the readme file contained first for further instructions.

For standard BAI v20.0.2, v20.0.3, and V21.0.1 installations use: BAI Cluster MustGather Script .zip
For BAI for a Server v20.0.2, v20.0.3, and V21.0.1 installations use: BAI for a Server MustGather Script .zip

What to do next

Review the log files and traces at the time of the problem to try to determine the source of the problem.
Check these locations for known issues:
- Search in the Cloud Pak for Automation Support Page.
- Review the Business Automation Insights documentation.
Once you completed gathering all the needed information and diagnostics, you can add them to your case. Alternatively, you can upload files to ECURep. For more information, see Enhanced Customer Data Repository (ECuRep) - Overview.

Related Information

IBM Automation Foundation MustGather

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"Component":"Business Automation Insights","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Product Synonym

Business Automation Insights; BAI

Was this topic helpful?

Document Information

Modified date:
08 March 2022

UID

ibm10791875

Tips