Redis pod fails after installing IBM RPA

After you upgrade or install IBM RPA, the Redis pods might fail to start.

Symptoms

The Redis operator pod fails to start and produces error messages similar to the following code block:

{"level":"error","ts":"2023-12-19T19:43:37Z","logger":"controller-runtime.source","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/source/source.go:148\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:582\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:547\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/source/source.go:136"}
{"level":"info","ts":"2023-12-19T19:43:37Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2023-12-19T19:43:37Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2023-12-19T19:43:37Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2023-12-19T19:43:37Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2023-12-19T19:43:37Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2023-12-19T19:43:37Z","msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/manager/internal.go:555"}
{"level":"error","ts":"2023-12-19T19:43:37Z","logger":"cmd","msg":"Proxy or operator exited with error.","Namespace":"","error":"failed to wait for rediscp-controller caches to sync: timed out waiting for cache to be synced","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.run\n\tmyapp/internal/cmd/ansible-operator/run/cmd.go:287\ngithub.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.NewCmd.func1\n\tmyapp/internal/cmd/ansible-operator/run/cmd.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\tmyapp/cmd/ansible-operator/main.go:40\nruntime.main\n\t/home/go/go/src/runtime/proc.go:267"}

Resolving this issue

To resolve this issue, follow these steps:

  1. Manually install the IBM NamespaceScope operator from the IBM Operator Catalog.
  2. Create the NamespaceScope Operator with a Custom Resource definition.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster.

  2. Install git:

    sudo dnf install git

  3. Clone the NamespaceScope operator repository:

    git clone https://github.com/IBM/ibm-namespace-scope-operator.git

  4. Enter the directory and create the operator in the IBM RPA namespace:

    cd ibm-namespace-scope-operator
oc create ns <RPA NAMESPACE>
    Note:Replace <RPA NAMESPACE> with the name of the namespace that the IBM RPA operator is installed.

  5. Apply the following YAML files from the repository:

    oc apply -f deploy/operator.ibm.com_namespacescopes.yaml
oc -n <RPA NAMESPACE> apply -f deploy/service_account.yaml
oc -n <RPA NAMESPACE> apply -f deploy/role.yaml
oc -n <RPA NAMESPACE> apply -f deploy/role_binding.yaml
oc -n <RPA NAMESPACE> apply -f deploy/operator.yaml

  6. Wait until the files are applied and then create the following NamespaceScope.yaml file:

    apiVersion: operator.ibm.com/v1
kind: NamespaceScope
metadata:
    name: namespacescope
    namespace: <RPA NAMESPACE>
spec:
    configmapName: namespace-scope
    csvInjector:
        enable: false
    namespaceMembers:
        - <COMMON SERVICES NAMESPACE>
        - <RPA NAMESPACE>
        - default
    restartLabels:
        intent: projected

  7. Apply the custom resource:

    oc -n <RPA NAMESPACE> apply -f NamespaceScope.yaml