No new data coming in to IBM Cloud Pak® for Multicloud Management from Infrastructure management

Symptom

Providers are added in Infrastructure management but they are not showing in IBM Cloud Pak for Multicloud Management.

Cause

The access token expired which provides authorization into Infrastructure management from IBM Cloud Pak for Multicloud Management.

To confirm, first check for the BypassAuth annotation in the imconnection connections.infra.management.ibm.com CR in the management-infrastructure-management project or namespace.

metadata:
  annotations:
    BypassAuth: 'false'

If the BypassAuth annotation is not defined or is set to "false" then:

check if spec.cfHost is defined with the correct IM url.

check for the definition of spec.secrets.accessToken.

spec:
cfHost: 'inframgmtinstall.apps.cp4mcm-23.company.com'
secrets:
accessToken:
 secretKeyRef:
   key: accesstoken
   name: ibmconnectionsecret

If accessToken is defined then check the log of the VM operator and look for entries with "TokenRefresh Token has expired" similar to the following:

I0401 13:59:35.553292       1 update.go:34] TokenRefresh Starting update of token.
I0401 13:59:35.555424       1 secret.go:166] TokenRefresh Getting secrets from management-infrastructure-management using name imconnectionsecret
I0401 13:59:35.571513       1 secret.go:173] TokenRefresh got token secret 131462371
I0401 13:59:35.571540       1 update.go:94] TokenRefresh checking if expired.
W0401 13:59:35.571553       1 secret.go:266] TokenRefresh Token has expired. Current time: 2021-04-01 13:59:35.571549714 +0000 UTC m=+29212.072337265, End time: 2021-03-26 18:52:49 +0000 UTC 
E0401 13:59:35.571570       1 update.go:65] refresh token has elapsed not attempting refresh
E0401 13:59:35.571615       1 update.go:27] error getting token using refresh token
I0401 13:59:35.571622       1 configs.go:92] Sleeping for 300.000000 seconds.

Resolving the problem

If the accessToken has expired then update the secret referenced in accessToken.secretKeyRef.name.

Note: For version 2.3 upgraded from 2.2, the default secret name is imconnectionsecret. For a new install of version 2.3 the default secret name is ibm-infra-management-application-connection.

To fix the problem the oauth username and password can be added to the <secret_name> in order for the tokenrefresh process to request a new accesstoken from IAM. Once the accesstoken is obtained the oauth username and password are removed from the secret.

Edit the secret using the command:

oc edit secret <secret_name> -n management-infrastructure-management -o YAML

Add after the line type: Opaque

type: Opaque
stringData:
  oauth_username: <user_name>
  oauth_password: <password>

Where <secret_name> is imconnectionsecret for version 2.3 upgraded from version 2.2.

Where <secret_name> is ibm-infra-management-application-connection for a new installation of version 2.3.

If the BypassAuth annotation is set to "true" then check to make sure spec.cfHost in the imconnection CR is set to the following:

spec:
  cfHost: 'web-service.management-infrastructure-management.svc.cluster.local:3000'

Verifying the solution

Check the log for the sre-inventory-inventory-cfcollector pod in the kube-system project for entries similar to the following:

I0425 00:13:35.835415       7 main.go:41] ============Do synchronization from CloudForm http://web-service.management-infrastructure-management.svc.cluster.local:3000
I0425 00:13:39.922796       7 query_vms.go:147] Totally 333 vms found in cloudform. 333 VMs should be returned and actually 333 returned in this query.
I0425 00:13:40.029993       7 query_providers.go:78] Totally 4 providers found in cloudform. 4 providers should be returned and actually 4 returned in this query.
I0425 00:13:40.114010       7 query_availabilityzone.go:77] Totally 0 availability_zones found in cloudform. 0 availability_zones should be returned and actually 0 returned in this query.
I0425 00:13:40.309270       7 query_hosts.go:112] Totally 31 hosts found in cloudform. 31 hosts should be returned and actually 31 returned in this query.
I0425 00:13:40.387620       7 query_services.go:84] Totally 0 services found in cloudform. 0 services should be returned and actually 0 returned in this query.
I0425 00:13:40.470200       7 query_cloud_volumes.go:104] Totally 0 cloud volumes found in cloudform. 0 cloud volumes should be returned and actually 0 returned in this query.
I0425 00:13:40.627044       7 query_data_stores.go:105] Totally 43 data stores found in cloudform. 43 data stores should be returned and actually 43 returned in this query.
I0425 00:13:41.366165       7 query_cloud_networks.go:101] Totally 112 cloud networks found in cloudform. 112 cloud networks should be returned and actually 112 returned in this query.
I0425 00:13:42.101278       7 query_cloud_subnets.go:97] Totally 342 cloud subnets found in cloudform. 342 cloud subnets should be returned and actually 342 returned in this query.
I0425 00:13:42.176657       7 query_network_routers.go:92] Totally 0 network router found in cloudform. 0 network routers should be returned and actually 0 returned in this query.
I0425 00:13:42.451049       7 query_security_groups.go:95] Totally 112 security groups found in cloudform. 112 security groups should be returned and actually 112 returned in this query.
I0425 00:13:42.959106       7 query_floating_ips.go:96] Totally 287 floating ips found in cloudform. 287 floating ips should be returned and actually 287 returned in this query.
I0425 00:13:42.961653       7 sender.go:206] Sending Resources { request: 99999, add: 853, update: 147, delete: 0, edgedd: 0, edgeDelete:0}
I0425 00:13:43.427390       7 sender.go:206] Sending Resources { request: 99999, add: 0, update: 200, delete: 0, edgedd: 800, edgeDelete:0}
I0425 00:13:46.473678       7 sender.go:206] Sending Resources { request: 99999, add: 0, update: 0, delete: 0, edgedd: 53, edgeDelete:0}
I0425 00:13:46.683429       7 main.go:43] ============Done synchronization this time