Troubleshooting
Problem
Resolving The Problem
Overview of User Management Service diagnostic information
General diagnostic information
Detailed diagnostic collection steps
Where to find logs in User Management Service (UMS)
1: Provide a detailed description of the problem and your environment
- Provided a detailed description of your issue. Include screen captures and re-create steps if possible.
- Is it an intermittent or re-creatable issue? Has this always been an issue or an issue that started only after a change occurred?
- What is the business impact? Should we be aware of any deadlines impacted by the issue?
- Provide a reference to the documentation being followed for the failing operation
- Which platform are you using (OpenShift, managed OpenShift, other Kubernetes platform)?
- What database is used by UMS [Derby, Db2, Oracle]?
2: Gather the following configuration information
- If using OpenShift, provide the output of this command:
oc version > version.txt
- Provide the output of this command:
kubectl version >> version.txt
- Provide the Custom Resource(CR) .yaml file used by the operator to set up the environment
kubectl get icp4acluster -o yaml > config.yaml
- Collection information about the nodes.
kubectl get nodes -o wide > nodes.txt
- Collect information about the pod statuses
kubectl get pods > pods.txt
-
Collect information about the pod containers
kubectl get pods -o jsonpath="{..image}" > containerInfo.txt
- On OpenShift gather route configuration
kubectl get route > routes.txt
- Collect the defined secrets
kubectl get secrets > secrets.txt
- Collect the defined persistent volume claims
kubectl get pvc > pvc.txt
- Collect the description and log of any pod you are having issues with (If your UMS is connected to DB2 collect log from the DB2 pod as well).
- For 21.0.x:
pod = <podname> kubectl describe pod $pod > $pod-description.txt kubectl get pod $pod -o yaml > $pod-configuration.txt kubectl cp $pod:/logs/UMS/$pod/ $pod-logs
- For 20.0.x:
pod = <podname> kubectl describe pod $pod > $pod-description.txt kubectl get pod $pod -o yaml > $pod-configuration.txt Depending on the pod (sso, scim, teams or allinone) execute: kubectl cp $pod:/logs/application/UMS/sso/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/teams/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/scim/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/allinone/$pod/ $pod-logs
- For 21.0.x:
- Collect the configuration mounted to the containers
kubectl get cm icp4adeploy-ums-configmap -o yaml > UMSConfigMap.yaml
- Collect the status of problematic pods. From within the terminal of the pod itself:
curl -sk https://localhost:9443/umshealth/rest/health/ready | jq
3: Log and Tracing data for WebSphere Liberty
-
Edit your Custom Resource
-
Configure tracing in section ums_configuration.logs.trace_specification. If you are using IBM Cloud Pak for Automation 20.0.3 and you have dedicated_pods enabled, then configure tracing for each of the individual capabilities in the corresponding configuration section.
-
To enable security tracing specify the following trace string
-
If dedicated_pods option is disabled
ums_configuration: ... logs: ... trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all"
-
If dedicated_pods option is enabled
ums_configuration: ... <pod> logs: ... trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all"
-
-
To enable UMS tracing specify the following trace string-
If dedicated_pods option is disabled
ums_configuration: ... logs: ... trace_specification: "com.ibm.dba.ums.*=all"
-
If dedicated_pods option is enabled
ums_configuration: ... <pod> logs: ... trace_specification: "com.ibm.dba.ums.*=all"
-
- To collect both, security and UMS data, combine the above trace strings
-
If dedicated_pods option is disabled
ums_configuration: ... logs: ... trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all:com.ibm.dba.ums.*=all"
-
If dedicated_pods option is enabled
ums_configuration: ... <pod> logs: ... trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all:com.ibm.dba.ums.*=all"
See reference MustGather: Web Single Sign-on problems with WebSphere Application Server for more details on collecting data for WebSphere Liberty.
-
-
Save the Custom Resource & apply your changes
kubectl apply -f my-cr.yaml
-
Reproduce the issue.
-
For every UMS pod collect log and traces by running:
pod = <podname> kubectl logs $pod > $pod.log
- Collect the trace. In addition to using the `kubectl logs` command, if persistent storage has been setup (available starting in version 20.0.3), you may obtain the logs and trace from the persistent volume specified in the `ums_configuration.existing_claim_name` property of your configuration, under the UMS directory. See UMS parameters for more details. More details can be found at Where to find logs in User Management Service (UMS)
The trace may be collected by copying using the oc command as well as long as the pod is available (where $pod refers to any pod with persistent storage enabled):pod = <podname> kubectl describe pod $pod > $pod-description.txt kubectl get pod $pod -o yaml > $pod-configuration.txt kubectl cp $pod:/logs/UMS/$pod/ $pod-logs
For 21.0.x:
- For 20.0.x:
pod = <podname> kubectl describe pod $pod > $pod-description.txt kubectl get pod $pod -o yaml > $pod-configuration.txt Depending on the pod (sso, scim, teams or allinone) execute: kubectl cp $pod:/logs/application/UMS/sso/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/teams/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/scim/$pod/ $pod-logs kubectl cp $pod:/logs/application/UMS/allinone/$pod/ $pod-logs
4: Collect the following information in case of OpenID Connect issues
- Get a specific client:
curl -k -v -u <umsadmin>:<umspassword> -s https://<ums-host>/oidc/endpoint/ums/registration/yourclient
- Get all connected clients:
curl -k -v -u <umsadmin>:<umspassword> -s https://<ums-host>/oidc/endpoint/ums/registration
5: User Management Service as an on-prem component
As an additional specification, setup UMS tracing:
com.ibm.dba.ums.*=all
6: Collect Operator Logs
7. Downloadable MustGather script
What to do next
- Review the log files and traces at the time of the problem to try to determine the source of the problem.
-
Check these locations for known issues:
-
Search in the Cloud Pak for Automation Support Page.
-
Review the User Management Service documentation.
-
- Once you completed gathering all the needed info and diagnostics, you can add them to your case. Alternatively, you can upload files to ECURep. For more information, see Enhanced Customer Data Repository (ECuRep) - Overview.
Related Information
Document Location
Worldwide
Product Synonym
CP4A;UMS
Was this topic helpful?
Document Information
Modified date:
06 October 2023
UID
ibm11076031