Deploying operators in a multi-namespace API Connect cluster

When you have a multiple namespace deployment in 2018, use these instructions to deploy the version 10 operators.

Before you begin

You must have completed the prerequisite steps and obtained the latest API Connect 10.0.1.x-eus software. See Preparing the 2018 deployment for upgrade and Obtaining files for upgrade from 2018.

Procedure

Prepare your environment:
1. Ensure KUBECONFIG is set for the target cluster:
```
export KUBECONFIG=<path_to_cluster_config_YAML_file>
```
  An example path is /Users/user/.kube/clusters/<cluster_name>/kube-config-<cluster_name>.yaml
2. Create namespaces on which subsystems are planned to be installed.
  The required namespaces are:
  - Apiconnect operator namespace
  - Gateway subsystem namespace
  - Management subsystem namespace
  - Portal subsystem namespace
  - Analytics subsystem namespace
  The following example values and deployment are used in this topic:
  - Apiconnect operator is installed in operator namespace
  - DataPower operator and gateway subsystem is installed in gtw namespace
  - Management subsystem is installed in mgmt namespace
  - Portal subsystem is installed in portal namespace
  - Analytics subsystem is installed in a7s namespace
3. Multi-namespace templates are provided in helper_files.zip, inside the release_files.zip you downloaded in Obtaining files for upgrade from 2018:
```
helper_files_unpack/multi-ns/
```
Install cert-manager and configure certificates:
1. Install cert-manager 1.5.1 but do not specify a custom namespace because cert manager will be installed in its own namespace.
```
kubectl apply -f cert-manager-1.7.1.yaml
```
2. Wait for cert-manager pods to enter Running 1/1 status before proceeding. Use the following command to check:
```
kubectl -n cert-manager get po
```
  There are 3 cert-manager pods in total.
3. Install ingress-ca certificate:
  1. Locate ingress-ca certificate at helper_files/multi-ns/ingress-ca-cert.yaml
  2. Create ingress-ca certificate in one of the subsystems. For example, if we choose mgmt namespace:
```
kubectl -n mgmt apply -f ingress-ca-cert.yaml
```
4. Install common issuers on all the namespaces:
  1. Locate common-issuer.yaml in helper_files/multi-ns-support/
  2. Create common-issuer.yaml in all namespaces:
```
kubectl create -f common-issuer.yaml -n mgmt 
kubectl create -f common-issuer.yaml -n gtw 
kubectl create -f common-issuer.yaml -n ptl 
kubectl create -f common-issuer.yaml -n a7s 
```
5. Obtain the ingress-ca secret created by cert manager and create the secret on all namespaces:
  1. Make sure ingress-ca certificate READY status is set to true:
```
kubectl get certificate ingress-ca -n mgmt
NAME         READY   SECRET       AGE
ingress-ca   True    ingress-ca   9m24s
```
  2. Cert-manager creates a secret called ingress-ca in mgmt namespace which represents the certificate we created in Step 2.d.
  3. Use the following command to obtain the yaml format of the secret:
```
kubectl get secret ingress-ca -n <namespace-where-ingress-ca-cert-is-created> -o yaml > ingress-ca-secret-in-cluster.yaml
```
  4. Remove unwanted content from the secret yaml:
```
cat ingress-ca-secret-in-cluster.yaml | grep -v 'creationTimestamp' | grep -v 'namespace' | grep -v 'uid' | grep -v 'resourceVersion' > ingress-ca-secret.yaml
```
  5. Create the secret using the yaml we prepared in previous step on rest of the subsystem namespaces. In this example, in the gtw, a7s and ptl namespaces.
```
kubectl create -f ingress-ca-secret.yaml -n gtw
kubectl create -f ingress-ca-secret.yaml -n ptl
kubectl create -f ingress-ca-secret.yaml -n a7s
```
6. Install management certs in management namespace:
  1. Locate management-certs.yaml in helper_files/multi-ns-support/
  2. Create management-certs.yaml in the mgmt namespace:
```
kubectl create -f management-certs.yaml -n mgmt
```
7. Install gateway certs in the gateway namespace:
  1. Locate gateway-certs.yaml in helper_files/multi-ns-support/
  2. Create gateway-certs.yaml in the gtw namespace:
```
kubectl create -f gateway-certs.yaml -n gtw
```
Install the apiconnect operator:
1. Install the ibm-apiconnect CRDs with the following command. Do not specify a namespace:
```
kubectl apply -f ibm-apiconnect-crds.yaml
```
2. Create a registry secret with the credentials to be used to pull down product images with the following command, replacing <v10_Operator_Namespace> with the desired namespace for the apiconnect operator deployment.
```
kubectl -n <v10_Operator_Namespace> create secret docker-registry apic-registry-secret
              --docker-server=apic-dev-docker-local.artifactory.swg-devops.com
              --docker-username=XXX@ibm.com --docker-password=XXXXXX
              --docker-email=XXX@ibm.com
```
  - For example, replace <v10_Operator_Namespace> with operator.
  - docker-password can be your artifactory API key.
  - -n <v10_Operator_Namespace> may be omitted if default namespace is being used for installation.
3. Locate and open ibm-apiconnect-distributed.yaml in a text editor of choice. Then, find and replace each occurrence of $OPERATOR_NAMESPACE with <v10_Operator_Namespace>, replacing <v10_Operator_Namespace> with the desired namespace for the deployment. In this example, the namespace is operator.
4. Also in ibm-apiconnect-distributed.yaml, locate the image: keys in the containers sections of the deployment yaml right below imagePullSecrets:. Replace the placeholder values REPLACE-DOCKER-REGISTRY of the image: keys with the docker registry host location of the apiconnect operator image (either uploaded to own registry or pulled from public registry).
5. Install ibm-apiconnect-distributed.yaml with the following command
```
kubectl apply -f ibm-apiconnect.yaml -n <v10_Operator_Namespace>
```
Install DataPower operator:
Note: Deployment of the DataPower operator is only needed if you have at least one API Connect 2018 Gateway subsystem (whether v5-compatible or not) to upgrade. If you are using DataPower in a different form factor such as an Appliance, you will not have an API Connect 2018 Gateway subsystem to upgrade, and will not need the DataPower operator for your upgrade.
1. Create a registry secret for the DataPower registry with the credentials to be used to pull down product images with the following command, replacing <namespace> with the desired namespace for the deployment .(In this case gtw):
```
kubectl -n <v2018_gateway_subsystem> create secret docker-registry datapower-docker-local-cred
              --docker-server=datapower-docker-local.artifactory.swg-devops.com
              --docker-username=XXX@ibm.com --docker-password=XXXXXX
              --docker-email=XXX@ibm.com
```
  - For example, replace <v2018_gateway_subsystem> with gtw.
  - docker-password can be your artifactory API key.
  - -n <v2018_gateway_subsystem> may be omitted if default namespace is being used for installation
2. Create a DataPower admin secret, replacing <v2018_gateway_subsystem> with the desired namespace for the deployment. For example, gtw. This secret will be used for $ADMIN_USER_SECRET later in the Gateway CR:
```
kubectl -n <v2018_gateway_subsystem> create secret generic datapower-admin-credentials --from-literal=password=admin
```
  <v2018_gateway_subsystem> may be omitted if default namespace is being used for installation.
3. Locate and open ibm-datapower.yaml in a text editor of choice. Then, find and replace each occurrence of default with <v2018_gateway_subsystem>, replacing <v2018_gateway_subsystem> with the desired namespace for the deployment. For example, gtw.
4. Install ibm-datapower.yaml with the following command:
```
kubectl -n <v2018_gateway_subsystem> apply -f ibm-datapower.yaml
```

Install the API Connect upgrade operator:

Create upgrade CRDs:

kubectl apply -f <path>/ibm-apiconnect-upgrade-crds.yaml

The ibm-apiconnect-upgrade-operator must be installed in the namespace of the ibm-apiconnect-operator.

Open the ibm-apiconnect-upgrade-distributed.yaml in a text editor:

Note:

You can obtain ibm-apiconnect-upgrade-distributed.yaml from the file IBM® API Connect <version> Operator Upgrade Release Files for Containers that you downloaded in Obtaining files for upgrade from 2018.

Replace all the $OPERATOR_NAMESPACE namespaces with the namespace of ibm-apiconnect-operator.
Replace REPLACE-DOCKER-REGISTRY with the root of the docker registry you are working with.

Optionally, you can supply values for the environment variables DEFAULT_IMAGE_REGISTRY and DEFAULT_IMAGE_PULL_SECRET in ibm-apiconnect-upgrade-distributed.yaml

Example:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ibm-apiconnect-upgrade
<...>
      containers:
      - name: ibm-apiconnect-upgrade
        image: REPLACE-DOCKER-REGISTRY/ibm-apiconnect-upgrade-operator@sha256:b1e09c3ed2f52b6724c7ef8a399ac36db63f9f64f19458736fec9a043458435e
        command:
        - ibm-apiconnect-upgrade
        args:
        - --zap-time-encoding=iso8601
        imagePullPolicy: Always
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: ibm-apiconnect-upgrade
        - name: DEFAULT_IMAGE_REGISTRY
          value: <registry-url>
        - name: DEFAULT_IMAGE_PULL_SECRET
          value: <image-pull-secret-name>

Run the following command to get ibm-apiconnect-upgrade-operator up and running in the cluster:
```
kubectl apply -f ibm-apiconnect-upgrade-distributed.yaml -n <v10_Operator_Namespace>
```

ibm-apiconnect-upgrade-operator is used to perform the v2018-v10 upgrade process.

Next, upgrade the subsystems. Continue with Upgrading the Management subsystem from v2018 to v10.

Note: Unless otherwise stated, when performing the individual subsystem upgrades the <namespace> value of example kubectl commands for a given subsystem should be set to the namespace for the particular subsystem component they are acting on.