Logging considerations

By default, logging of the capabilities is enabled in your cluster and the logs are stored in a dedicated persistent data store. You can also collect and forward the standard output stdout logs from the containers to help you troubleshoot issues and improve their health and performance.

The following information must be considered before you install FileNet® Content Manager.

Logs and log persistence
Java dumps

Logs and log persistence

All containerized applications write to the standard output and standard error streams, which can be viewed on a pod level by using oc logs on the command line. To view them in the Red Hat® OpenShift® Container Platform (OCP) web console, click Workload > Pod > Pod Details > Logs.

However, if a container crashes, a pod is evicted, or a node dies, you probably still want to access the application logs. Therefore, logs need a separate storage location and a lifecycle that is independent of nodes, pods, or containers. OCP provides a logging solution that is based on the EFK stack: Elasticsearch, Fluentd, and Kibana. Fluentd collects all the node and container logs and stores them in dedicated project indexes. Kibana is the centralized user interface where you can create visualizations and dashboards with the aggregated data.

Most components can be configured to write logs to a persistent volume (PV). Logs can be stored under a /logs/application directory on the PV.

The following table shows which containers produce and persist logs.

Table 1. Logging information for capability components
Component	Does it produce logs	Can it persist logs
FileNet Content Manager
cpe	Yes	Yes (dedicated log PVC cpe-logstore-pvc). For more information, see Must gather .
cmis	Yes	Yes (dedicated log PVC cmis-logstore-pvc). For more information, see Must gather .
graphql	Yes	Yes (dedicated log PVC graphql-logstore-pvc). For more information, see Must gather .
Task Manager (tm)	Yes	Yes (dedicated log PVC tm-logstore-pvc). For more information, see Must gather .
ier	Yes	Yes (dedicated log PVC ier-pvc). For more information, see Must gather: Collecting data to diagnose issues with Enterprise Records .
css	Yes	Yes (dedicated log PVC css-logstore-pvc). For more information, see Must gather .
iccsap	Yes	Yes (dedicated log PVC iccsap-logstore-pvc). For more information, see Must gather .
Navigator
icn	Yes	For more information, see Must gather .
Business Automation Insights
BPC	Yes	Not persisted. For more information, see Must gather .
Flink job/task managers	Yes	Not persisted. For more information, see Must gather .
Other components	Yes	Not persisted. For more information, see Must gather .

FileNet Content Manager can also use the following logging configuration parameters to further customize how the logs are stored.

  # # Logging setting
  # logging_configuration:
  #   mon_log_parse: false
  #   mon_log_service_endpoint: localhost:5044
  #   private_logging_enabled: false
  #   logging_type: default
  #   mon_log_path: /path_to_extra_log

For more information, see Logging parameters.

You can customize the logs for all of the other capabilities that are written to stdout by referring to the configuration parameters for the capability. For more information, see Certified Kubernetes configuration parameters.

Java dumps

Remember: Java™ dumps cannot be generated for Node.js based applications.

For more information about collecting data in Cloud Pak for Business Automation, see the Must gather External link opens a new window or tab .

Get a shell to a running pod where you want to generate a dump:
```
oc exec -it <running pod> bash
```
Dumps are created in the "/config/dumps" directory or the /opt/ibm/wlp/output/defaultServer/ directory.
Change directory to the ../wlp/bin directory:
```
cd /opt/ibm/wlp/bin
```
For components that have no PVC to store dumps, copy the dumps locally by using the following command:
```
oc cp <pod name>:/config/dumps/<javacore or heapdump file name> <javacore or heapdump file name>
```
Determine whether the dump storage is attached to the pod by running the following command:
```
df | grep /opt/ibm/wlp/output/defaultServer/dump
```
Note: If a BAW and PFS Liberty server JVM crashes, then you also see dumps generated.
Get the dumps by tarring the files in the dump store persistent volume (baw/pfs-dumpstore-pvc). You can find the path by reviewing the get pvc output. All server pods send dumps to the same persistent volume. A sub-directory for each pod is created by using the pod name.
Determine the log volume where logs are being persisted for the current pod and make sure you can write a test file:
```
ls /opt/ibm/wlp/usr/servers/defaultServer/logs/<pod name>
```

Run the touch command:

touch /opt/ibm/wlp/usr/servers/defaultServer/logs/<pod name>

Execute dump command and specify the log path for the server dump zip file:

server dump defaultServer --archive=/opt/ibm/wlp/usr/servers/defaultServer/logs/<pod name>/package_file_name.dump.zip --include=heap

Expect the following output:

Dumping server defaultServer.
Server defaultServer dump complete in /opt/ibm/wlp/usr/servers/defaultServer/logs/<pod_name>/package_file_name.dump.zip.

For Flink job/task managers, you can retrieve the PID of the Java process by running the following command:

oc exec -it <CR_NAME>-bai-event-xxx-eve-yyy-ep-job/taskmanager-xxx -- bash -c "ps -ef | grep java"

Create the thread dump:

oc exec -it <CR_NAME>-bai-event-xxx-eve-yyy-ep-job/taskmanager-xxx -- bash -c "kill -3 <PID>"

Retrieve the file name of the dump:

oc exec -it <CR_NAME>-bai-event-xxx-eve-yyy-ep-job/taskmanager-xxx -- bash -c "ls -ltr"

Copy the file locally:

oc cp <pod name>:<filename> <file name>