IBM Support

Cluster is unstable and zen-metastoredb is restarting repeatedly

Troubleshooting


Problem

The cluster is unstable and UIs are opening slowly. One of the zen-metastoredb pods ist being restarted repeatedly. The problem happened after restoring a backup from another cluster. The following error message is displayed in one of the the previous zen-metastoredb logs before the pod was restarted:

2021-04-29T10:13:57.448503711Z I210429 10:13:57.448404 59 server/server.go:943 [n?] Sleeping till wall time 1619691237448332584 to catches up to 1619691237846545097 to ensure monotonicity. Delta: 398.212513ms 2021-04-29T10:13:57.530652709Z F210429 10:13:57.530428 86 server/server.go:252 [n?] clock synchronization error: this node is more than 1s away from at least half of the known nodes (0 of 1 are within the offset) 2021-04-29T10:13:57.530652709Z goroutine 86 [running]: 2021-04-29T10:13:57.530652709Z github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc00052e700, 0xc00052e780, 0x0, 0x3c8b0e0)

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000UoTUAA0","label":"Zen-\u003Emetastoredb"}],"ARM Case Number":"","Platform":[{"code":"PF040","label":"Red Hat OpenShift"}],"Version":"3.5.0"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
01 September 2023

UID

ibm16448960