IBM Support

New Install of OVA not coming up healthy due to ETCD performance.

Troubleshooting


Problem

Upon installing OVA with the ISO and powering up the VM user find that the install says done but finds pods missing or in crashloopback or pending status
One of the following job pods and the job pods would likely show as 0/1 CrashLookBackOff:
  • <managementcluster_CR_name>-up-apim-schema-0-to-<schema_number>*
  • <managementcluster_CR_name>-up-lur-schema-0-to-<schema_number>*
This can cause any of this:
  • could be during apim/lur DB creation
  • could be during schema jobs for apim or lur
  • could be during data-populate jobs for apim or lur
  • all of that above could succeed and we could see that postgres just won’t stay up and keeps restarting

Symptom

Install will not complete successfully. apic health-check will fail.
Pod logs might show
-sql [261262] getConnection
-bhendi:error [261262] lib/db::execute failed, Code='55006', Message='database "apim" is being accessed by other users': database "apim" is being accessed by other users, stack: error: database "apim" is being accessed by other users
    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/src/parser.ts:369:69)
    at Parser.handlePacket (/app/node_modules/pg-protocol/src/parser.ts:188:21)
    at Parser.parse (/app/node_modules/pg-protocol/src/parser.ts:103:30)
  • there could be many messages about "apply request took too long" where it is taking seconds to complete when the expected duration is 100ms
-caller":"etcdserver/server.go:1159","msg":"failed to revoke lease","lease-id":"4bd38ef0015a6704","error":"etcdserver: request timed out"}
-"caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"4.998589206s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/kube-scheduler\" ","response":"","error":"context deadline exceeded"}

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeBXAA0","label":"API Connect-\u003EManagement and Monitoring (MM)-\u003EInstallation"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
02 July 2024

UID

ibm17148626