Deploying operators in a multi-namespace API Connect cluster

Deploy the Kubernetes operator files in multi-namespace cluster.

Before you begin

Ensure you completed the prerequisite tasks for installing operators. See Deploying operators and cert-manager.
Be sure to review your strategy for using certificates with API Connect. See Deployment requirements.

About this task

Note: If you are deploying a single-namespace API Connect cluster, do not use these instructions. Instead, go to Deploying operators in a single-namespace API Connect cluster.

These instructions apply only to native k8s deployments. They do not apply to OpenShift deployments.
The API Connect operator is deployed as a cluster-scoped operator which watches every namespace in the cluster. The API Connect operator can be installed in any namespace, although it is recommended that the operator is installed in its own dedicated namespace.
The DataPower operator must be deployed in a namespace where gateway subsystem install is planned.
Single ingress-ca certificate must be installed and used across the various namespaces where subsystems will be installed.

Procedure

Prepare your environment:
1. Ensure KUBECONFIG is set for the target cluster:
```
export KUBECONFIG=<path_to_cluster_config_YAML_file>
```
  An example path is /Users/user/.kube/clusters/<cluster_name>/kube-config-<cluster_name>.yaml
2. Create namespaces for the subsystems that will be installed:
  - API Connect operator namespace
  - Gateway subsystem namespace
  - Management subsystem namespace
  - Portal subsystem namespace
  - Analytics subsystem namespace
  Note: The following namespaces are reserved and must not be used for your installation:
  - default
  - openshift-*
  - kube-*
  The following example values and deployment are used throughout these instructions:
  - An apiconnect operator is installed in operator namespace
  - A DataPower operator and gateway subsystem is installed in gtw namespace
  - The Management subsystem is installed in mgmt namespace
  - The Portal subsystem is installed in portal namespace
  - The Analytics subsystem is installed in a7s namespace
3. Multi-namespace templates are provided in helper_files.zip, inside the release_files.zip you downloaded in Obtaining product files:
```
helper_files_unpack/multi-ns-support/
```
Install cert-manager and configure certificates:

Cert-manager adds convenience to the generation and management of API Connect certificates. For more information about cert-manager, see Key Concepts: Cert-manager, Issuers, and Secrets.

You can obtain cert-manager v1.12.13 from the API Connect v10 distribution helper_files.zip archive, or download it from https://github.com/cert-manager/cert-manager/releases/tag/v1.12.13.
1. Install cert-manager v1.12.13, do not specify a custom namespace as cert manager will be installed in its own namespace.
  Note: If you are deploying a two data center disaster recovery deployment on Kubernetes, then do not do this step, follow the steps in Installing cert-manager and certificates in a two data center deployment, then return here.
  1. ```
  kubectl apply -f cert-manager-1.12.13.yaml
```
2. Wait for cert-manager pods to enter Running 1/1 status before proceeding. Use the following command to check:
```
  kubectl -n cert-manager get po
```
  There are 3 cert-manager pods in total.
2. Install ingress-ca certificate:
  1. Locate ingress-ca certificate at helper_files/multi-ns-support/ingress-ca-cert.yaml
  2. Create ingress-ca certificate in one of the subsystems. For example, if we choose mgmt namespace:
```
kubectl -n mgmt apply -f ingress-ca-cert.yaml
```
3. Install common issuers on all the namespaces:
  1. Locate common-issuer.yaml in helper_files/multi-ns-support/
  2. Create common-issuer.yaml in all namespaces:
```
kubectl create -f common-issuer.yaml -n mgmt 
kubectl create -f common-issuer.yaml -n gtw 
kubectl create -f common-issuer.yaml -n ptl 
kubectl create -f common-issuer.yaml -n a7s 
```
4. Obtain the ingress-ca secret created by cert manager and create the secret on all namespaces:
  Note: The instructions in this step result in having a selfsigning-issuer in every namespace. This issuer is typically referenced by an API Connect CR in the namespace with:
```
...
  microServiceSecurity: certManager
  certManagerIssuer:
    name: selfsigning-issuer
    kind: Issuer
...
```
  This issuer (and associated root certificate) does not need to be the same in every namespace, nor does it need to be the same for 2 CRs in the same namespace. Each subsystem can use its own selfsigning-issuer, because it is only used for certificates that are internal to the subsystem.
  
  To ensure that the ingress-issuer is in sync across namespaces, it should be backed by the same ingress-ca certificate. This enables, for example, APIM to use a client certificate that matches the CA of the portal-admin ingress endpoint. The sync is achieved by copying around the ingress-ca certificate in each namespace, as shown in the following steps. It does not matter that it is the same issuer, or whether it has a different name. However it is important is that the ingress-ca is the same for the all the "ingress-issuer" that are going to be used by the subsystems.
  1. Make sure ingress-ca certificate READY status is set to true:
```
kubectl get certificate ingress-ca -n mgmt
NAME         READY   SECRET       AGE
ingress-ca   True    ingress-ca   9m24s
```
  2. Cert-manager creates a secret called ingress-ca in mgmt namespace which represents the certificate we created in Step 2.c.
  3. Use the following command to obtain the yaml format of the secret:
```
kubectl get secret ingress-ca -n <namespace-where-ingress-ca-cert-is-created> -o yaml > ingress-ca-secret-in-cluster.yaml
```
  4. Remove unwanted content from the secret yaml:
```
cat ingress-ca-secret-in-cluster.yaml | grep -v 'creationTimestamp' | grep -v 'namespace' | grep -v 'uid' | grep -v 'resourceVersion' > ingress-ca-secret.yaml
```
  5. Create the secret using the YAML that you prepared in previous step on rest of the subsystem namespaces. In this example, in the gtw, a7s and ptl namespaces.
```
kubectl create -f ingress-ca-secret.yaml -n gtw
kubectl create -f ingress-ca-secret.yaml -n ptl
kubectl create -f ingress-ca-secret.yaml -n a7s
```
5. Install management certs in management namespace:
  1. Locate management-certs.yaml in helper_files/multi-ns-support/
  2. Create management-certs.yaml in the mgmt namespace:
```
kubectl create -f management-certs.yaml -n mgmt
```
6. Install gateway certs in the gateway namespace:
  1. Locate gateway-certs.yaml in helper_files/multi-ns-support/
  2. Create gateway-certs.yaml in the gtw namespace:
```
kubectl create -f gateway-certs.yaml -n gtw
```
Install the API Connect operator:
1. Install the ibm-apiconnect CRDs with the following command. Do not specify a namespace:
```
kubectl apply --server-side --force-conflicts -f ibm-apiconnect-crds.yaml
```
2. Create a registry secret with the credentials to be used to pull down product images with the following command, replacing <namespace> with the desired namespace for the API Connect operator deployment.
```
kubectl -n <namespace> create secret docker-registry apic-registry-secret
              --docker-server=<registry_server> 
              --docker-username=<username@example.com> --docker-password=<password>
              --docker-email=<username@example.com>
```
  - For example, replace <namespace> with operator.
  - docker-password can be your artifactory API key.
  - -n <namespace> may be omitted if default namespace is being used for installation
3. Locate and open ibm-apiconnect-distributed.yaml in a text editor of choice. Then, find and replace each occurrence of $OPERATOR_NAMESPACE with <namespace>, replacing <namespace> with the desired namespace for the deployment. In this example, the namespace is operator.
4. Also in ibm-apiconnect-distributed.yaml, replace all the instances of the placeholder string REPLACE-DOCKER-REGISTRY with the docker registry host location of the apiconnect operator image (either uploaded to your own registry or pulled from the public registry).
5. Install ibm-apiconnect-distributed.yaml with the following command
```
kubectl apply -f ibm-apiconnect-distributed.yaml
```
Install DataPower operator:
Deployment of the DataPower operator is only needed if you have at least one API Connect v10 Gateway subsystem (whether v5 compatible or not) to upgrade. If you are using DataPower in a different form factor such as an Appliance, you will not have an API Connect v10 Gateway subsystem to upgrade, and will not need the DataPower operator for your upgrade.

Note: The DataPower Operator supports multiple instances of DataPower in the same cluster, separated by namespace. When deploying into multiple namespaces, ensure that any cluster-scoped resources are created with unique names for separate installations, such that they are not overwritten by subsequent installations. For example, DataPower cluster-scoped resources include ClusterRoleBindings.
1. Create a registry secret for the DataPower registry with the credentials to be used to pull down product images with the following command, replacing <namespace> with the desired namespace for the deployment .(In this case gtw):
```
kubectl -n <namespace> create secret docker-registry datapower-docker-local-cred
              --docker-server=<registry_server> 
              --docker-username=<username@example.com> --docker-password=<password>
              --docker-email=<username@example.com>
```
  - For example, replace <namespace> with gtw.
  - docker-password can be your artifactory API key.
  - -n <namespace> may be omitted if default namespace is being used for installation
2. Create a DataPower admin secret, replacing <namespace> with the desired namespace for the deployment. For example, gtw. This secret will be used for $ADMIN_USER_SECRET later in the Gateway CR:
```
kubectl -n <namespace> create secret generic datapower-admin-credentials --from-literal=password=admin
```
  -n <namespace> may be omitted if default namespace is being used for installation.
3. Locate and open ibm-datapower-distributed.yaml in a text editor of choice. Then, find and replace each occurrence of default with <namespace>, replacing <namespace> with the desired namespace for the deployment. For example, gtw.
4. Install ibm-datapower-distributed.yaml with the following command:
```
kubectl -n <namespace> apply -f ibm-datapower-distributed.yaml
```
  Troubleshooting: If your DataPower operator pod fails to start, see Troubleshooting install.
Next, install the subsystems. Continue with Installing the API Connect subsystems.

Note: Unless otherwise stated, when performing the individual subsystem upgrades the <namespace> value of example kubectl commands for a given subsystem should be set to the namespace for the particular subsystem component they are acting on.