IBM Support

How to generate sosreport on a OpenShift Container Platform (OCP) cluster node on IBM Power Systems

How To


Summary

This document describes how to collect diagnostic data from the Red Hat OpenShift Container Platform (OCP) that might be required for problem determination. Additionally, this document provides information on how to send the gathered data to IBM.

Objective

  • Learn how to collect diagnostic data from the Red Hat OpenShift Container Platform (OCP) on IBM Power Systems required for troubleshooting Hardware I/O errors.
  • Learn how to send the gathered data to IBM.
     

Environment

Red Hat OpenShift Container Platform (OCP)
Red Hat Enterprise Linux CoreOS (RHCOS)
Red Hat Enterprise Linux (RHEL)
IBM Power Systems

Steps

For Red Hat Enterprise Linux CoreOS (RHCOS) partition containing physical hardware, follow the steps to generate a sosreport file used for troubleshooting Hardware I/O errors:

1. Download and install OpenShift command-line interface (CLI). The OpenShift CLI (oc) is used to interact with Red Hat OpenShift Container Platform from a command-line interface and binary is supported on following operating systems - Linux, Windows, or macOS.

2. Logon as user with the cluster-admin role.

3. Generate API token from Red Hat OpenShift web console (Profile -> Copy Login Command):
image-20220308161619-1
4. Log in with the generated API token, for example:
oc login --token=sha256~vm0HWVsaqPdJaiDOondCF2nH2DNhNXyHMDy6hEJCaYs --server=https://api.openshiftcluster-9.com:6443
5. Run the following command to list cluster nodes:
oc get nodes -o wideCopied to clipboard.
Example:
oc get nodes
Note: Identify the target node by using the 'Node Name' and 'Internal IP'. 'Kernel Version' can be used to identify the node architecture in our case ppc64le is for IBM Power Systems, s390x for Mainframe or x86 for x86 System x (MVS).
Example:
ocp get nodes s390x
Note: 'Kernel Version' can be used to identify the node architecture in our case s390x for Mainframe.
6. To enter debug session on the target node, run following command:
oc debug node/my-cluster-node-nameCopied to clipboard.
7. To run binary files contained in the host’s executable paths, change the root directory to "/host":
chroot /hostCopied to clipboard.
8. 'Sos' plug-in is part of toolbox container, run following command to access the plug-in:
toolboxCopied to clipboard.
Note:
If a message, "Container 'toolbox-' exist. Trying to start…", is displayed on the terminal after running the 'toolbox' command, type 'exit' and remove the toolbox container with command 'sudo podman rm toolbox-' and rerun the 'toolbox' command to avoid issues with sosreport plug-ins.
sudo podman rm toolbox-Copied to clipboard.
9. Generate a new sosreport archive by running command:
sos reportCopied to clipboard.
Press ENTER to continue, or CTRL+C to quit.
When instructed to “enter the case ID that you are generating this report for []”, type: <IBM or Red Hat case number >
A file called "sosreport-nodename-currentdate.tar.xz" is created in “/host/var/tmp/sosreport-nodename-currentdate.tar.xz“

10. Copy the sosreport archive to the target system by using 'oc':
For example, if the file is called sosreport-master-0-master-0-2022-02-11-mqgqger.tar.xz, run the following command to copy the archive to the target system:
oc debug node/master-0 -- bash -c 'cat /host/var/tmp/sosreport-master-0-master-0-2022-02-11-mqgqger.tar.xz' > /tmp/sosreport-master-0-master-0-2022-02-11-mqgqger.tar.xzCopied to clipboard.
11.  If required, transfer the file to a system with internet access. When using FTP as a protocol, make sure to use the binary transfer method.
12. Secure upload the file to existing IBM case via Ecurep URL:

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI000BK","label":"Power System E1080 Server (9080-HEX)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI0007E","label":"Power System E950 Server (9040-MR9)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI000B2","label":"IBM Power System S914 (9009-41G)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI0005H","label":"Power System L922 Server (9008-22L)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGDMKK","label":"Power System AC922 Server (8335-GTH)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGDMMD","label":"Power System AC922 Server (8335-GTC)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI0007I","label":"Power System E980 Server (9080-M9S)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHKN6","label":"IBM Cloud Platform Common Services"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNJN0","label":"Power System LC922 Server (9006-22C)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSDSQ3D","label":"Power System S824 Server (8286-42A)"},"ARM Category":[],"ARM Case Number":[],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"HW1A1","label":"IBM Power Systems"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
08 June 2022

UID

ibm16556490