Zen reconciles multiple times during PostgreSQL database migration

The Zen reconciles multiple times to complete PostgreSQL database migration when you upgrade from foundational services version 3.x to 4.6. You need to update the network policy of ibm-iam-operator to enable the traffic to mongodb service.

Symptoms

The following errors might be displayed in the ibm-zen-operator pod logs:

stderr: 'W0425 07:10:15.993498    6885 reflector.go:456] k8s.io/client-go/tools/watch/informerwatcher.go:146: watch of *unstructured.Unstructured ended with: an error on the server ("unable to decode an event from the watch stream: no kind \"Pod\" is registered for version \"v1\" in scheme \"k8s.io/client-go/dynamic/scheme.go:29\"") has prevented the request from succeeding'^[[0m^M

stderr: 'error: no matching resources found'^[[0m^M

The following error might be displayed in the ibm-iam-operator logs:

{"level":"error","ts":"2024-05-28T06:41:18Z","logger":"controller_authentication","msg":"Encountered an error while performing the current migration","Request.Namespace":"cp4ba","Request.Name":"example-authentication","subreconciler":"handleMigrations","error":"failure occurred during MongoToV1: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongodb.ibm-common-services.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 172.30.246.180:27017: i/o timeout }, ] }","stacktrace":"github.com/IBM/ibm-iam-operator/controllers/operator.(*AuthenticationReconciler).handleMigrations\n\t/home/prow/go/src/github.com/IBM/ibm-iam-operator/controllers/operator/authentication_controller.go:432\ngithub.com/IBM/ibm-iam-operator/controllers/operator.(*AuthenticationReconciler).Reconcile\n\t/home/prow/go/src/github.com/IBM/ibm-iam-operator/controllers/operator/authentication_controller.go:850\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Co...

Resolving the problem

To resolve the issue, add the MongoDB parameters in the spec.egress section in the network policy of ibm-iam-operator in the namespace where you deployed the foundational services:

- to:
    - podSelector:
        matchLabels:
          app: icp-mongodb
      namespaceSelector: {}

The following is the sample network policy of ibm-iam-operator:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: egress-ibm-iam-operator
spec:
  podSelector:
    matchLabels:
      name: ibm-iam-operator
  egress:
    - to:
      ...
    - to:
        - podSelector:
            matchLabels:
              app: icp-mongodb
          namespaceSelector: {}
  policyTypes:
    - Egress

{: codeblock}