IBM Support

c-mdm-redis pod is down

Troubleshooting


Problem

After a Cloud Pak for Data / (Master Data Management)MDM upgrade, MDM/Match 360 redis pod 0 may be found crashing.
c-mdm-redis-xxx-m-0                                  3/4     Running     0
Using the oc describe pod command reveals these events.
Events:
  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  4m10s (x585 over 3h33m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500
The  Openshift UI will show a warning message similar to:
"PVC data-c-mdm-redis-xxx-m-0 utilization has crossed 85%. Free up some space or expand the PVC immediately."
Mdm redis  pod reveals errors:
oc logs c-mdm-redis-xxx  -c mgmt
 - ERROR - cdb.metrics.kubeclient - get_disk_total_from_formation - 38 - Error while reading total disk size
Traceback (most recent call last):
...
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'xxx', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'xxx', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'xxx', 'Date': 'Mon, 26 Jun 2023 20:35:39 GMT', 'Content-Length': '452'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"formations.crd.compose.com \"mdm-redis-\" is forbidden: User \"system:serviceaccount:cpd-instance:mdm-redis-\" cannot get resource \"formations\" in API group \"crd.compose.com\" in the namespace \"cpd-instance\"","reason":"Forbidden","details":{"name":"mdm-redis","group":"crd.compose.com","kind":"formations"},"code":403}

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000UoQtAAK","label":"Administration"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
12 July 2023

UID

ibm17009281

Page Feedback