Installing an active-active configuration on AWS

About this task

When you configure an active-active configuration of the Cloud Pak, you must ensure that the clusters can communicate with each other and synchronize their data. To enable this communication, you update the custom resource (CR) on each cluster with information about the synchronization services that run on the other clusters.

The following services are used to synchronize data between clusters:

Kafka

Data is synchronized by using Kafka MirrorMaker (MM2) on a Red Hat® OpenShift® Container Platform route.

PostgreSQL

Data is synchronized by using SymmetricDS on a Red Hat OpenShift Container Platform route. After you install on cluster 1, that cluster becomes the SymmetricDS registration node. Any other clusters where you install must register with the registration node.

ZooKeeper

Kubernetes load-balancing services are used to synchronize data and balance workloads across all clusters. These services enable ZooKeeper to work as a single ensemble across all clusters.

Vault

Vault data is stored in PostgreSQL, but the decryption keys must be common across all clusters to enable reading of the Vault data.

Before you begin

Complete the steps in Preparing to install an active-active configuration on AWS.

Installing the Cloud Pak in an active-active configuration on AWS

Complete the following sets of steps to install IBM Cloud Pak for Network Automation in an active-active configuration on AWS:

1. On the primary cluster, that is, cluster 1, complete these sets of steps:
   1. Install the orchestration manager operator on cluster 1.
   2. Create the CR for cluster 1.
   3. Optional: Customize the CR for cluster 1.
   4. Create an instance on cluster 1.
   5. Gather data for the installation on the secondary clusters.
2. On each secondary cluster, that is, each instance of cluster <n>, complete these sets of steps:
   1. Install the orchestration manager operator on cluster <n>.
   2. Create Kafka secrets for remote clusters on cluster <n>.
   3. Create the CR for cluster <n>.
   4. Optional: Customize the CR for cluster <n>.
   5. Create an instance on cluster <n>.
   6. Gather the Kafka and ZooKeeper details for cluster <n>.
   7. Add the Kafka and ZooKeeper details for cluster <n> to all other clusters.
3. Reset the replication role setting for the app database role on all clusters.

Install the orchestration manager operator on cluster 1

For information about how to install the IBM Cloud Pak for Network Automation Orchestration Manager operator, see Installing IBM Cloud Pak for Network Automation Orchestration Manager.

Create the CR for cluster 1

You can use the following example to create a CR for cluster 1:

apiVersion: tnc.ibm.com/v1beta1
kind: Orchestration
metadata:
  name: cluster-1
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb  
spec:
  license:
    accept: true
  version: 2.7.6
  featureconfig:
    siteplanner: true
  advanced:
    imagePullPolicy: Always
    podSettings:
      zookeeperlocks:
        replicas: 3
      kafka:
        replicas: 3
    clusters:
      local:
        alias: cluster-1
  storage:
    kafka:
      storageClassName: gp3
    zookeeper:
      storageClassName: gp3
    zookeeperlocks:
      storageClassName: gp3
    zenFile:
      storageClassName: efs-sc
    zenBlock:
      storageClassName: gp3
    postgres:
      storageClassName: gp3

Optional: Customize the CR for cluster 1

You might also want to customize the CR in other ways. For example, you might want to increase the settings for CPU, memory, and replicas for the microservices. For more information, see Custom resources.

Create an instance on cluster 1

On cluster 1, complete these steps:

1. Create an instance of IBM Cloud Pak for Network Automation by running this command:

   oc create -f cluster-cr.yaml

   cluster-cr.yaml is the name of the CR file.

2. After PostgreSQL is fully installed and running, but before the Cloud Pak microservices are running, disable all foreign key checks in the database by completing these steps:
   1. Access the cp4na-o-postgresql pod by running this command:

      oc exec -it $(oc get cluster cp4na-o-postgresql -o jsonpath --template '{.status.targetPrimary}') -- bash

   2. If the postgres user is set up as a superuser, log in to the pod as the postgres user by running this command:

      psql --user postgres

   3. Update the replication role setting for the app role to replica by running this command:

      alter role app SET session_replication_role = 'replica';

3. Verify that the instance is successfully created by running this command:

   oc get orchestration <cp4na_namespace>

   <cp4na_namespace> is the namespace where you installed the orchestration manager operator.

   The instance might take some time to create. When the Status value is Ready, your instance is created.

Gather data for the installation on the secondary clusters

Complete the following steps to gather data from cluster 1 to use during the installation of the Cloud Pak on the secondary clusters:

1. Export the secret cp4na-o-vault-keys to a file.
   1. Run this command to export the keys to a YAML file:

      oc get secret cp4na-o-vault-keys -o yaml > cp4na-o-vault-keys-cluster-1.yaml

   2. Delete the following attributes from the metadata section of the YAML file:
      • labels
      • ownerReferences
      • uid
      • creationTimestamp
      • resourceVersion
2. Cluster 1 is the SymmetricDS registration node, which is the node that other clusters register with when you install them. Get the SymmetricDS registration URL from the cluster 1 orchestration status by running this command:

   oc get orchestration cluster-1 -o jsonpath='{.status.endpoints.symmetricds.registrationURL}'

3. Get the domain name and port details for the ZooKeeper load-balancing servers by running this command:

   oc get svc | grep cp4na-o-zookeeper-locking-ext

   Save the details. You use them later to create the ZooKeeper section in the CRs for the secondary clusters.

4. Get the Kafka password for Kafka MirrorMaker (MM2) by running this command:

   oc get secret cp4na-o-kafka-user -o jsonpath='{.data.password}' -n <cp4na_namespace> | base64 -d

   Save the details. You use them later to create the Kafka section in the CRs for the secondary clusters.

5. Get the external bootstrap service address for Kafka by running this command:

   oc get routes cp4na-o-events-kafka-bootstrap -o jsonpath='{.spec.host}' -n <cp4na_namespace> 

   Save the details. You use them later to create the Kafka section in the CRs for the secondary clusters.

6. Get the SSL certificate for the Kafka bootstrap address and save the certificate to a file.
   1. Inspect the Kafka instance by running this command:

      oc get kafka cp4na-o-events -o yaml

   2. Save the certificate for cp4na-o-events-kafka-bootstrap to a file that is called ca.crt by running this command:

      oc get kafka -o jsonpath='{.items[0].status.listeners[2].certificates[0]}' > ca.crt

Install the orchestration manager operator on cluster <n>

Create Kafka secrets for remote clusters on cluster <n>

On cluster <n>, use the Kafka information that you gathered about remote clusters, such as cluster 1, to create secrets. For each remote cluster, create a secret that contains that cluster's Kafka SSL certificate and bootstrap server password by running this command:

oc create secret generic <remote-cluster-kafka-secret-name> --from-file <remote-kafka-cert_file> --from-literal=password=<remote-kafka-passsword>

<remote-cluster-kafka-secret-name> is a name that you choose to identify this secret.

<remote-kafka-cert_file> is the remote cluster's Kafka SSL certificate file that you created, that is, ca.crt.

<remote-kafka-passsword> is the password for the remote cluster's Kafka bootstrap server.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cp4na-o-kafka-cluster-1
  namespace: lifecycle-manager
data:
  password: Q05RM01BRGMyNDZ4VmhuUEh0TkltS0N2U1dxWXFXZHM=
  ca.crt: |-
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZMVENDQXhXZ0F3SUJBZ0lVZkVCaCtCaTVK
    bHFacTN5b2Yrak1STVh2aGRjd0RRWUpLb1pJaHZjTkFRRU4KQlFBd0xURVRNQkVHQTFVRUNnd0th

Create the CR for cluster <n>

On cluster <n>, create a CR to specify how the orchestration manager operator is installed and configured in your environment.

This example shows the CR for a secondary cluster that is called cluster-2 after the SymmetricDS, Kafka, and ZooKeeper details for a primary cluster that is called cluster-1 are added:

apiVersion: tnc.ibm.com/v1beta1
kind: Orchestration
metadata:
  name: cluster-2
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb  
spec:
  license:
    accept: true
  version: 2.7.6
  featureconfig:
    siteplanner: true
  advanced:
    imagePullPolicy: Always
    podSettings:
      zookeeperlocks:
        replicas: 3
      kafka:
        replicas: 3
    clusters:
      local:
        alias: cluster-2
        kafka:
          mm2:
            replicas: 1
        db:
          symmetricds:
            registrationUrl: https://cp4na-o-symmetricds-lifecycle-manager.apps.cluster-1.26ah.p1.openshiftapps.com/sync/cluster-1
          sequence:
            rangeMultiplier: 1
      remote:
        - alias: cluster-1
          kafka:
            bootstrapServers: cp4na-o-events-kafka-bootstrap-lifecycle-manager.apps.cluster-1.26ah.p1.openshiftapps.com:443
            authentication:
              passwordSecret:
                secretName: cp4na-o-kafka-cluster-1
                password: password
            tls:
              trustedCertificates:
                - secretName: cp4na-o-kafka-cluster-1
                  certificate: ca.crt
          zookeeper:
            server.1: a8605356a4b0d4223bb1030c04b357cd-a0dc7511d3ba8d26.elb.eu-west-1.amazonaws.com:2888:3888
            server.2: a0884ba91b86945a9899e02bd13dd041-9e1adf15da6a3904.elb.eu-west-1.amazonaws.com:2888:3888
            server.3: a90c7fe26e12e48b4a1dc1463d4e0661-385f08d4d4cbdf40.elb.eu-west-1.amazonaws.com:2888:3888
  storage:
    kafka:
      storageClassName: gp3
    zookeeper:
      storageClassName: gp3
    zookeeperlocks:
      storageClassName: gp3
    zenFile:
      storageClassName: efs-sc
    zenBlock:
      storageClassName: gp3
    postgres:
      storageClassName: gp3

Optional: Customize the CR for cluster <n>

On cluster <n>, you might also want to customize the CR in other ways. For example, you might want to increase the settings for CPU, memory, and replicas for the microservices. For more information, see Custom resources.

Create an instance on cluster <n>

On cluster <n>, complete these steps:

1. Create an instance of IBM Cloud Pak for Network Automation by running this command:

   oc create -f cluster-cr.yaml

   cluster-cr.yaml is the name of the CR file.

2. After PostgreSQL is fully installed and running, but before the Cloud Pak microservices are running, disable all foreign key checks in the database by completing these steps:
   1. Access the cp4na-o-postgresql pod by running this command:

      oc exec -it $(oc get cluster cp4na-o-postgresql -o jsonpath --template '{.status.targetPrimary}') -- bash

   2. If the postgres user is set up as a superuser, log in to the pod as the postgres user by running this command:

      psql --user postgres

   3. Update the replication role setting for the app role to replica by running this command:

      alter role app SET session_replication_role = 'replica';

3. Verify that the instance is successfully created by running this command:

   oc get orchestration <cp4na_namespace>

   <cp4na_namespace> is the namespace where you installed the orchestration manager operator.

   The instance might take some time to create. When the Status value is Ready, your instance is created.

Gather the Kafka and ZooKeeper details for cluster <n>

On cluster <n>, complete these steps:

1. Get the Kafka password for Kafka MirrorMaker (MM2) by running this command:

   oc get secret cp4na-o-kafka-user -o jsonpath='{.data.password}' -n <cp4na_namespace> | base64 -d

   Save the details. You add them later to the Kafka section in the CRs for all other clusters.

2. Get the external bootstrap service address for Kafka by running this command:

   oc get routes cp4na-o-events-kafka-bootstrap -o jsonpath='{.spec.host}' -n <cp4na_namespace> 

   Save the details. You add them later to the Kafka section in the CRs for all other clusters.

3. Get the SSL certificate for the Kafka bootstrap address and save the certificate to a file.
   1. Inspect the Kafka instance by running the following command:

      oc get kafka cp4na-o-events -o yaml

   2. Save the certificate for cp4na-o-events-kafka-bootstrap to a file that is called ca.crt by running the following command:

      oc get kafka -o jsonpath='{.items[0].status.listeners[2].certificates[0]}' > ca.crt

4. Get the domain name and port details for the ZooKeeper load-balancing servers by running this command:

   oc get svc | grep cp4na-o-zookeeper-locking-ext

   Save the details. You add them later to the ZooKeeper section in the CRs for all other clusters.

Add the Kafka and ZooKeeper details for cluster <n> to all other clusters

This example shows the CR for a primary cluster that is called cluster-1 after the Kafka and ZooKeeper details for two secondary clusters are added:

apiVersion: tnc.ibm.com/v1beta1
kind: Orchestration
metadata:
  name: cluster-1
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb    
spec:
  license:
    accept: true
  version: 2.7.6
  featureconfig:
    siteplanner: true
  advanced:
    imagePullPolicy: Always
    podSettings:
      zookeeperlocks:
        replicas: 3
      kafka:
        replicas: 3
    clusters:
      local:
        alias: cluster-1
        kafka:
          mm2:
            replicas: 1
      remote:
        - alias: cluster-2
          kafka:
            bootstrapServers: cp4na-o-events-kafka-bootstrap-lifecycle-manager.apps.cluster-2.zp0z.p1.openshiftapps.com:443
            authentication:
              passwordSecret:
                secretName: cp4na-o-kafka-cluster-2
                password: password
            tls:
              trustedCertificates:
                - secretName: cp4na-o-kafka-cluster-2
                  certificate: ca.crt
          zookeeper:
            server.4: a6691ffe18c5b4ce7874872d6d777705-e01b7845d7af5a36.elb.eu-west-1.amazonaws.com:2888:3888
            server.5: ab540e51d6e384891884d8bbd8e32be1-b8d20b05c9212790.elb.eu-west-1.amazonaws.com:2888:3888
            server.6: a5eeea8330f894848b660f0742e001e8-ec12170ceb11a96d.elb.eu-west-1.amazonaws.com:2888:3888
        - alias: cluster-3
          kafka:
            bootstrapServers: cp4na-o-events-kafka-bootstrap-lifecycle-manager.apps.cluster-3.zp0z.p1.openshiftapps.com:443
            authentication:
              passwordSecret:
                secretName: cp4na-o-kafka-cluster-3
                password: password
            tls:
              trustedCertificates:
                - secretName: cp4na-o-kafka-cluster-3
                  certificate: ca.crt
          zookeeper:
            server.7: a6691ffe18c5b4ce7874872d6d777705-b8d20b05c9212790.elb.eu-west-1.amazonaws.com:2999:3999
            server.8: ab540e51d6e384891884d8bbd8e32be1-ec12170ceb11a96d.elb.eu-west-1.amazonaws.com:2999:3999
            server.9: a5eeea8330f894848b660f0742e001e8-e01b7845d7af5a36.elb.eu-west-1.amazonaws.com:2999:3999
  storage:
    kafka:
      storageClassName: gp3
    zookeeper:
      storageClassName: gp3
    zookeeperlocks:
      storageClassName: gp3
    zenFile:
      storageClassName: efs-sc
    zenBlock:
      storageClassName: gp3
    postgres:
      storageClassName: gp3

Reset the replication role setting for the app database role on all clusters

What to do next