Troubleshooting WebSphere Liberty operators

You might encounter an issue when you install, configure, or manage operators. You can run oc or kubectl commands to check the status of pods, operators, and custom resources and to investigate problems. Run the MustGather tool to collect information about clusters and send the information to IBM® Support.

To run oc commands, you need the Red Hat® OpenShift® command-line interface (CLI). To run kubectl commands if Red Hat OpenShift is not installed, you need the Kubernetes command line tool.

Tip: The documentation shows oc commands. To run kubectl commands, replace oc with kubectl in the commands.

Troubleshooting operators

Run the following oc commands to investigate problems with pods and operators.

Check the WebSphere® Liberty operator.

oc get pods -l name=websphere-liberty-operator

Output from the get pods command shows the pod name and status.

NAME                                          READY   STATUS    RESTARTS   AGE
websphere-liberty-operator-5c4548d98f-xgqtg   1/1     Running   0          2m29s

Check the operator events. In the describe pod command, replace pod_name with a pod name from the get pods output.
```
oc describe pod pod_name
```
The following example command uses the WebSphere Liberty operator pod name.
```
oc describe pod websphere-liberty-operator-5c4548d98f-xgqtg
```
Check the operator logs. In the logs command, replace pod_name with a pod name from the get pods output.
```
oc logs pod_name
```

Troubleshooting custom resources

If the operator is running as wanted, check the status of the WebSphereLibertyApplication CR instance.

The following commands use wlapp, which is the short name for WebSphereLibertyApplication.

Check the CR status. In the get wlapp command, replace app_name with the name of your CR instance.

oc get wlapp app_name -o wide

The following example shows the command with my-app for app_name and the output.

oc get wlapp my-app -o wide

NAME             IMAGE                         EXPOSED   RECONCILED   REASON    MESSAGE   AGE
my-app   quay.io/my-repo/my-app:1.0    false     True                             1h

Check the CR effective fields. In the get wlapp command, replace app_name with the name of your CR instance.

oc get wlapp app_name -o yaml

Ensure that the effective CR values in the output are what you want. If the CR successfully reconciled, the output has Reconciled in the status section.

$ oc get wlapp my-app -o yaml

  apiVersion: liberty.websphere.ibm.com/v1
  kind: WebSphereLibertyApplication
  ...
  status:
    conditions:
    - lastUpdateTime: "2020-01-08T22:06:50Z"
      status: "True"
      type: Reconciled

Check the CR events. In the describe wlap command, replace app_name with the name of your CR instance.
```
oc describe wlapp app_name
```

Lost connections when using OpenJ9 version openj9-0.33.1

If you are running a WebSphereLibertyApplication custom resource that uses JITServer with OpenJ9 version openj9-0.33.1, the application might get lost connections to the JITServer with errors in the application and JIT Server containers. To prevent this problem, upgrade OpenJ9 to version 0.35.0.

The following code shows an example error in an application pod.

139745560807168:error:140940F4:SSL routines:ssl3_read_bytes:unexpected message:ssl/record/rec_layer_s3.c:1477:
#JITServer: t=865045 Lost connection to the server (serverUID=2529813496315317418)

The following shows an example error in a JIT server pod

139825751693056:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:355:

The following code shows how to get the version information for OpenJ9.

sh-4.4$ java -version
openjdk version "17.0.4.1" 2022-08-12
IBM Semeru Runtime Open Edition 17.0.4.1 (build 17.0.4.1+1)
Eclipse OpenJ9 VM 17.0.4.1 (build openj9-0.33.1, JRE 17 Linux amd64-64-Bit Compressed References 20220812_266 (JIT enabled, AOT enabled)
OpenJ9   - 1d9d16830
OMR      - b58aa2708
JCL      - 1f4d354e654 based on jdk-17.0.4.1+1)

The following code sample shows details of the Liberty version.

********************************************************************************
product = Open Liberty 22.0.0.10 (wlp-1.0.69.cl221020220912-1100)
wlp.install.dir = /opt/ol/wlp/
server.output.dir = /opt/ol/wlp/output/defaultServer/
java.home = /opt/java/openjdk
java.version = 17.0.4.1
java.runtime = IBM Semeru Runtime Open Edition (17.0.4.1+1)
os = Linux (4.18.0-372.19.1.el8_6.x86_64; amd64) (en_US)
process = 1@daytrader7-7f795bd46b-8vtkl
Classpath = /opt/ol/wlp/bin/tools/ws-server.jar:/opt/ol/wlp/bin/tools/ws-javaagent.jar
Java Library path = /opt/java/openjdk/lib/default:/opt/java/openjdk/lib:/usr/lib64:/usr/lib
********************************************************************************

The following code sample shows that OpenJ9 version 0.35.0 is installed.

sh-4.4$ java -version
openjdk version "17.0.5" 2022-10-18
IBM Semeru Runtime Open Edition 17.0.5.0 (build 17.0.5+8)
Eclipse OpenJ9 VM 17.0.5.0 (build openj9-0.35.0, JRE 17 Linux amd64-64-Bit Compressed References 20221018_325 (JIT enabled, AOT enabled)
OpenJ9   - e04a7f6c1
OMR      - 85a21674f
JCL      - 32d2c409a33 based on jdk-17.0.5+8)

The following code sample shows details of the Liberty version.

********************************************************************************
product = Open Liberty 22.0.0.12 (wlp-1.0.71.cl221220221107-1900)
wlp.install.dir = /opt/ol/wlp/
server.output.dir = /opt/ol/wlp/output/defaultServer/
java.home = /opt/java/openjdk
java.version = 17.0.5
java.runtime = IBM Semeru Runtime Open Edition (17.0.5+8)
os = Linux (4.18.0-305.57.1.el8_4.x86_64; amd64) (en_US)
process = 1@daytrader7-0
Classpath = /opt/ol/wlp/bin/tools/ws-server.jar:/opt/ol/wlp/bin/tools/ws-javaage
nt.jar
Java Library path = /opt/java/openjdk/lib/default:/opt/java/openjdk/lib:/usr/lib
64:/usr/lib
********************************************************************************

Gathering information about clusters with MustGather

You can use the MustGather tool to collect information about your cluster. IBM Support uses the collected information to help you troubleshoot and fix problems.

Run the Red Hat OpenShift must-gather and describe commands before you contact IBM Support to save time.

Open a command-line prompt to the directory where you want to store the MustGather data.
Log in to your Red Hat OpenShift cluster. The username must have cluster-admin permissions.
```
oc login https://your_cluster_hostname -u username -p password
```
Run the Red Hat OpenShift must-gather command.
Replace instance_namespace and operator_namespace with the namespace values of your installation.
```
oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.4 -- gather -n instance_namespace,operator_namespace -m failure,overview,cloudpak
```
The command compresses the collected data into a .tgz output file and stores it in ./must-gather.local.random_number.
Run Red Hat OpenShift describe commands to gather data about the state of your WebSphereLibertyApplication, WebSphereLibertyDump, and WebSphereLibertyTrace instances.
The MustGather tool does not gather data about custom resources. Use the describe command to gather data about instances. Replace the instance_namespace variables with the namespace values for your instances.
```
oc describe wlapp -n WebSphereLibertyApplication_instance_namespace > wlapp.txt
oc describe wldump -n WebSphereLibertyDump_instance_namespace > wldump.txt
oc describe wltrace -n WebSphereLibertyTrace_instance_namespace > wltrace.txt
```
Open a Support Ticket with IBM Support and attach the MustGather .tgz output file.

Getting help from IBM Support

Open a Support Ticket with IBM Support and add information that can help IBM Support troubleshoot and fix the problem.

Click Open a case on the WebSphere Application Server support or Let's troubleshoot page.
Add information that can help IBM Support determine the cause of the error.
In the ticket, describe the error. If the error is difficult to describe, then provide a screen capture of the error. Also, provide pertinent information, such as a description of your cluster configuration and the component that is failing or having issues.
If you used the MustGather tool to collect information about your cluster, attach the MustGather .tgz output file and the .txt files that have data about your WebSphereLibertyApplication, WebSphereLibertyDump, and WebSphereLibertyTrace instances.