MustGather for IBM Sterling Intelligent Promising Containers

Note: If you have cluster permissions, run the diagnostic script mentioned in Section C and attach the diagnostics. If not, proceed and provide the diagnostics for the issue you are facing.

• Must Gather Information for Operator and Catalog Source Issues

• Version and Pod Status

  Objective: To check the operational status and versions of Catalog Sources and associated Pods.

  Command:

  oc get catalogsource,pods -n <namespace>

  Note: Replace <namespace> with the specific namespace where the catalog source is deployed. For OpenShift, this typically is openshift-marketplace. In vanilla Kubernetes, the namespace might vary based on your setup.

• Catalog Source Details

  Objective: To collect detailed YAML configuration data of the Catalog Sources for in-depth troubleshooting.

  Commands:

  1. General Catalog Source Information:

     oc get catalogsource -n <namespace> -o yaml > operator_troubleshooting_csinfo.yaml

  2. Specific Catalog Source Configuration:

     oc get catalogsource <catalogsource-name> -n <namespace> -o yaml > specific_catalogsource_details.yaml

     This command allows for retrieving YAML configurations of specific catalog sources, such as ibm-sip-catalog or ibm-oms-gateway-catalog, providing a granular view necessary for deeper troubleshooting.

  Notes on Namespace Specification: Ensure <namespace> correctly reflects where the catalog source is deployed. The default in OpenShift is often openshift-marketplace. It's important to adjust this based on your Kubernetes or OpenShift configuration settings.

• Operator Group Details

  Objective: Retrieve details about the Operator Group in the specified namespace to ensure correct configuration and alignment with the deployed operators.

  Command:

  oc get OperatorGroup -n <namespace>

  Note: Replace <namespace> with the specific namespace where the subscription or operator is deployed. This is crucial as the Operator Group details need to match the namespace of the operational environment to provide accurate diagnostics.

• Subscription Details

  Retrieve Subscription Names

  Objective: Gather the names of all subscriptions in the specified namespace to check their current status and configuration.

  Setup: Set the namespace variable to ensure all commands are consistently applied to the correct namespace:

  ns=<namespace> # Replace <namespace> with the actual namespace

  Command:

  subs=$(oc get sub -n $ns --no-headers -o custom-columns=:metadata.name)

  This command saves the names of all subscriptions in the specified namespace into a variable, making it easier to use in subsequent commands.

  Subscription Details

  Objective: After obtaining the names of the subscriptions, fetch detailed information about each subscription.

  Command:

  oc get sub $subs -n $ns -o yaml > operator_troubleshooting_subinfo.yaml

  Note: Ensure that the $ns variable is set to the namespace where the subscription or operator is deployed. This approach not only streamlines the command execution but also minimizes the risk of applying commands to the wrong namespace, ensuring accurate and relevant data collection.

• Logs Collection

  IBM-OMS-gateway-operator Logs

  Objective: List and save logs for IBM-OMS-gateway-operator pods.

  Setup: Set the namespace variable to ensure all commands are consistently applied to the correct namespace:

  ns=<namespace> # Replace <namespace> with the actual namespace

  Commands:

  1. Get the IBM-OMS-gateway-operator Pods:

     pods_OMS_gateway=$(oc get pods -n $ns -l app.kubernetes.io/name=ibm-oms-gateway-operator -o custom-columns=NAME:.metadata.name --no-headers)

  2. Export logs if any pods are found:

     if [ -z "$pods_OMS_gateway" ]; then echo "No IBM-OMS-gateway-operator pods found in namespace $ns."; else for pod in $pods_OMS_gateway; do oc logs $pod -n $ns > ${pod}_logs.txt echo "Logs for pod $pod have been saved to ${pod}_logs.txt" done fi

  IBM-sip-operator Logs

  Objective: List and save logs for IBM-sip-operator pods.

  Commands:

  1. Get the IBM-sip-operator Pods:

     pods_sip_operator=$(oc get pods -n $ns -l app.kubernetes.io/name=ibm-sip-operator -o custom-columns=NAME:.metadata.name --no-headers)

  2. Export logs if any pods are found:

     if [ -z "$pods_sip_operator" ]; then echo "No IBM-sip-operator pods found in namespace $ns."; else for pod in $pods_sip_operator; do oc logs $pod -n $ns > ${pod}_logs.txt echo "Logs for pod $pod have been saved to ${pod}_logs.txt" done fi

  Notes:

  Ensure that the $ns variable is set to the namespace where the IBM-OMS-gateway-operator and IBM-sip-operator are deployed. This method helps maintain consistency and accuracy while collecting logs.

• Deployment and Service Group Status

  SIP Environment Deployment Status

  Objective: Gather the status of the SIP Environment deployment.

  Command:

  oc describe sipenvironments.apps.sip.ibm.com -n <namespace> > operator_troubleshooting_deploymentstatusinfo.txt

  Note: Replace <namespace> with the actual namespace where the SIP environment is deployed. This command provides a detailed description of the SIP environment deployment status, which is saved to a file for troubleshooting.

  Service Group Status

  Objective: Retrieve the status of various service groups in YAML format.

  Setup: Set the namespace variable to ensure all commands are consistently applied to the correct environment:

  ns=<namespace> # Replace <namespace> with the actual namespace

  1. List All Instances:

     IV Service Group:

     kubectl get ivservicegroups.apps.sip.ibm.com -n $ns

     Promising Service Group:

     kubectl get promisingservicegroups.apps.sip.ibm.com -n $ns

     Utility Service Group:

     kubectl get utilityservicegroups.apps.sip.ibm.com -n $ns

  2. Set the Instance Variable:

     instance=<instance> # Replace <instance> with the actual instance name (e.g., dev, prod, etc.)

  3. Retrieve Specific Instance Details:

     IV Service Group:

     oc get IVServiceGroup $instance -n $ns -o yaml > ${instance}_IVServiceGroup.yaml

     Promising Service Group:

     oc get PromisingServiceGroup $instance -n $ns -o yaml > ${instance}_PromisingServiceGroup.yaml

     Utility Service Group:

     oc get UtilityServiceGroup $instance -n $ns -o yaml > ${instance}_UtilityServiceGroup.yaml

  Notes:

  Namespace and Instance Reference: Ensure that the $ns variable is set to the namespace where the SIP environment and service groups are deployed, and the $instance variable is set to the specific instance name.

• Stateful Sets and CRDs

  Check the Status of Stateful Sets and Custom Resource Definitions (CRDs)

  Objective: Check the status of Stateful Sets and Custom Resource Definitions (CRDs) associated with SIP.

  Setup: Set the namespace variable to ensure all commands are consistently applied to the correct environment:

  ns=<namespace> # Replace <namespace> with the actual namespace

  Commands:

  oc get statefulset,crd -n $ns | grep sip

• If Catalog Source Pods Are Not Being Created During SIP Deployment Due to PodSecurityStandards (PSS) Enforcement

  Adjusting Pod Security Standards

  Objective: Change the namespace to a less restrictive PSS profile to allow the creation of Catalog Source pods.

  Setup: Set the namespace variable to ensure all commands are consistently applied to the correct environment:

  ns=<namespace> # Replace <namespace> with the actual namespace where the catalog source is deployed

  Commands:

  1. Adjust PSS Profile:

     kubectl label --overwrite ns $ns pod-security.kubernetes.io/enforce=baseline

  2. Confirm Execution:

     Objective: Ensure that the command was successfully executed to comply with the necessary security standards.

     Command:

     kubectl get ns $ns --show-labels | grep pod-security.kubernetes.io/enforce=baseline

     This command verifies that the namespace label has been successfully updated.

  Notes:

  Namespace Reference: Ensure that the $ns variable is set to the namespace where the Catalog Source is deployed. Adjusting the PSS profile helps in overcoming the restrictions that prevent pod creation.

  The commands ensure that the namespace is set to a less restrictive profile, facilitating the successful deployment of Catalog Source pods.

• Persistent Volume (PV) Access Permissions

  Ensure Correct Access Permissions for Persistent Volumes

  Objective: Verify and adjust the access permissions for Persistent Volumes used by Operators.

  Setup: Set the namespace variable:

  ns=<namespace> # Replace <namespace> with the actual namespace

  Requirements:

  • Access Mode: ReadWriteMany (RWX)
  • Storage: Minimum of 10 GB
  • Accessibility: Accessible by all containers across the cluster
  • Write Access: Owner group must have write access
  • Security Context: Set the fsGroup parameter in the SIPEnvironment custom resource

  Commands:

  1. List Persistent Volumes:

     kubectl get pv -n $ns > pv_list.txt

     Action: Save the list to a file.

  2. Describe Specific Persistent Volume:

     kubectl describe pv <pv-name> > ${pv-name}_description.txt

     Note: Replace <pv-name> with the PV name. Action: Save the description to a file.

  3. List Persistent Volume Claims (PVCs):

     kubectl get pvc -n $ns > pvc_list.txt

     Action: Save the list to a file.

  4. Describe Specific Persistent Volume Claim:

     kubectl describe pvc <pvc-name> -n $ns > ${pvc-name}_description.txt

     Note: Replace <pvc-name> with the PVC name. Action: Save the description to a file.

  5. Check and Adjust Access Permissions:

     • Check Current Access Modes:

       kubectl get pv <pv-name> -o jsonpath='{.spec.accessModes}' -n $ns > ${pv-name}_access_modes.txt

       Action: Save the access modes to a file.

     • Adjust Access Modes (if necessary):

       Note: Adjusting requires creating a new PV and PVC.

       Steps:

       1. Delete Existing PVC (ensure data is backed up):

          kubectl delete pvc <pvc-name> -n $ns

       2. Create New PVC:

          apiVersion: v1 kind: PersistentVolumeClaim metadata: name: <new-pvc-name> namespace: $ns spec: accessModes: - ReadWriteMany # Replace with the required access mode resources: requests: storage: 10Gi # Adjust as necessary storageClassName: <storage-class-name> # Ensure this matches your storage class

          Apply the new PVC:

          kubectl apply -f <new-pvc-file>.yaml

  Notes:

  Namespace Reference: Ensure $ns is set to the correct namespace. Data Backup: Backup data before deleting any PVCs. Access Modes: Use ReadWriteMany (RWX) Security Context: Set fsGroup in the SIPEnvironment custom resource.