Preparing to upgrade on OpenShift and Cloud Pak for Integration

Prepare your OpenShift or Cloud Pak for Integration deployment for upgrading to the new version of API Connect 10.0.x.

Before you begin

Review the Upgrade considerations on OpenShift and Cloud Pak for Integration to ensure that you are following a supported upgrade path and that you understand important changes that might affect your deployment.

About this task

Complete the following steps to prepare your deployment for upgrading to the newest version of API Connect.

Procedure

Temporarily stop the Analytics data collection to avoid losing event data during the upgrade.
Verify that the pgcluster is healthy:
1. Get the name of the pgcluster:
```
oc get pgcluster -n <APIC_namespace>
```
  The response displays the name of the postgres cluster running in the specified namespace.
2. Check the status of the pgcluster:
```
oc get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3
```
  The response for a healthy pgcluster looks like the following example, where the state is Initialized:
```
status:
    message: Cluster has been initialized
    state: pgcluster Initialized
```
If the cluster is healthy, proceed to the next step.
Important: If the pgcluster is not healthy ( returns any other state):
The cluster is not healthy and an upgrade will fail. Complete the following steps:
1. If there are any ongoing backup or restore jobs, wait until they complete and then check the status again. Do not proceed with the upgrade until the status is Initialized.
2. If all of the background jobs complete but the pgcluster remains unhealthy, contact IBM Support for assistance.
(Upgrading from v10.0.1.2-eus only) Resolve the Pgbouncer DNS error.

If you are running IBM® API Connect 10.0.1.2-eus, your deployment might be experiencing DNS errors in pgbouncer. Complete the following steps to determine whether your deployment is experiencing DNS errors with pgbouncer, and to replace the pgbouncer image if needed.

Note: This issue only affects IBM API Connect 10.0.1.2-eus. If you are running a different version of IBM API Connect (including fix packs), your deployment is not affected, so you can skip this step.

Determine whether your v10.0.1.2-eus deployment is experiencing the problem:
1. Get the pgbouncer pod name:
```
oc get pods -n <APIC_namespace> | grep 'pgbouncer'
```
  where <APIC_namespace> is the namespace where you installed IBM API Connect.
2. Check the pgbouncer log for the server DNS lookup failed error message:
```
oc logs <pgbouncer-pod-name> -n <APIC_namespace> | grep 'server DNS lookup failed'
```
If the server DNS lookup failed message appears in the log, then your deployment is impacted and you must replace the pgbouncer image. If other errors appear, this procedure will not correct them and you should contact IBM Support for assistance. If no errors appear, you can skip the rest of this task and proceed with the upgrade.

Replace the pgbouncer image:
1. Get the new pgbouncer image format from the registry where the 10.0.4.0-ifix3 images were pushed; for example:
```
<registry-name>/ibm-apiconnect-management-crunchy-pgbouncer@sha256:4a5caaf4e5cd4056ccb3de7d39b8e343b0c4ebce7cae694ccbfbe80924d98752
```
  For CP4I, the default registry-name is icr.
2. Get the pgbouncer deployment name:
```
oc get deploy -n <APIC_namespace> | grep 'pgbouncer'
```
3. Edit the pgbouncer deployment:
```
oc edit deploy <pgbouncer-deploy-name> -n <APIC_namespace>
```
4. In the deployment, replace the container image section with the new image that you downloaded.
5. Wait for the pgbouncer pod to restart.
6. Exec into the pgbouncer pod:
```
oc exec -it <pgbouncer-pod> -n <APIC_namespace> -- bash
```
7. Execute pgbouncer --version and make sure the response matches the following information:
```
bash-4.4$ pgbouncer --version
PgBouncer 1.15.0
libevent 2.1.8-stable
adns: evdns2
tls: OpenSSL 1.1.1g FIPS  21 Apr 2020
systemd: yes
```
8. Verify that the server DNS lookup failed no longer appears in the pgbouncer log:
```
oc logs <pgbouncer-pod-name> -n <APIC_namespace> | grep 'server DNS lookup failed'
```
9. Delete back-end microservices to force a restart:
  1. Get the apim microservices pod name: oc get pods -n <APIC_namespace> | grep 'apim'
  2. Delete the apim pod: oc delete pod <apim-pod> -n <APIC_namespace>
  3. Get the lur microservices pod name: oc get pods -n <APIC_namespace> | grep 'lur'
  4. Delete the lur pod: oc delete pod <lur-pod> -n <APIC_namespace>
  5. Get the task manager microservice pod name: oc get pods -n <APIC_namespace> | grep 'task'
  6. Delete the task manager pod: oc delete pod <task-manager-pod> -n <APIC_namespace>
10. Make sure the deployment is up and running.
Run the pre-upgrade health check:
1. Verify that the apicops utility is installed by running the following command to check the current version of the utility:
```
apicops --version
```
  If the response indicates that apicops is not available, install it now. See The API Connect operations tool: apicops in the API Connect documentation.
2. Run the following command to set the KUBECONFIG environment.
```
export KUBECONFIG=/path/to/kubeconfig
```
3. Run the following command to execute the pre-upgrade script:
```
apicops version:pre-upgrade -n <namespace>
```
  If the system is healthy, the results will not include any errors.
(v10.0.1.2-ifix2-eus and earlier) Validate your API definitions.
SwaggerDoc (OpenAPI 2.0) and OpenAPI 3.0 documents that form API definition files are validated at stage or publish time by the API Dev Tools Swagger Parser. Before upgrading, ensure that your existing API definition files can be parsed and are considered valid by the Swagger Parser tool. Note that any API definitions that are already staged or published are not impacted, but will undergo Swagger Parser validation again if they are re-staged or re-published in the future.

You can use either of the following techniques to ensure that existing API definition files are can be parsed and are considered valid by the Swagger Parser:
- Download and install the swagger-parser-cli tool (see https://www.npmjs.com/package/swagger-cli). This tool includes a validate command that can validate local API definition files against the SwaggerDoc (OpenAPI 2.0) and OpenAPI 3.0 schemas, in both JSON and YAML formats.
- Use the browser-based Swagger Parser Validator available at https://apitools.dev/swagger-parser/online/; this accepts a URL to existing files available online, or can accept text copied and pasted into the tool, in both JSON and YAML formats. It can likewise validate against the SwaggerDoc (OpenAPI 2.0) and OpenAPI 3.0 schemas.
Note: Staging or publishing an API in API Connect results in additional syntactic validation of IBM extensions to SwaggerDoc (OpenAPI 2.0) and OpenAPI 3.0, as well as semantic validation to make sure that policies and other resources that the API refers to are valid within the context of the staging or publishing operation. Any errors associated with this additional validation will not be caught by the local or online Swagger Parser tool, nor will they be caught by the apic validate toolkit CLI command.

The Swagger Parser tool will report an invalid OpenAPI document. For example, if a document is missing a mandatory section such as info or paths, the following error is reported:
```
[object Object] is not a valid Swagger API definition
```
If a required property is missing from a given section, a specific error is reported. For example, if the info section is missing the required version property, the following error is reported:
```
Swagger schema validation failed. 
  Missing required property: version at #/info
 
JSON_OBJECT_VALIDATION_FAILED
```
Additionally, if the document contains any local references (that is, values of $ref properties) that do not resolve to an existing reference within the document (known as de-referencing), this is also be flagged as an error. For example, if a document contains the following as a reference:
```
schema:
  $ref: '#/definitions/Pet'
```
then the definition of Pet (that is, #/definitions/Pet) must exist in the same document, at the given path (definitions section in this example), similar to the following:
```
definitions:
  Pet:
    type: object
    required:
    - name
    properties:
      id:
        type: integer
        format: int64
```
If the local reference is not found within the document, the following de-referencing error is reported:
```
Token "Pet" does not exist.
```
Back up the current deployment in case the upgrade fails and you need to perform a roll back.
Cloud Pak for Integration only: If you use the Operations Dashboard with the API management capability, complete the following tasks:
1. Verify that you are using the latest version of the Operations Dashboard.
  
  For information, see "Upgrading Integration tracing" in the Upgrading > Upgrading capabilities section of the Cloud Pak for Integration documentation for the appropriate release.
2. Update the image map paths in the API Connect Cluster CR's spec.gateway.openTracing section.
  
  For information, see step 2 of "Enabling open tracing for API management." in the Installing > Deploying instances of capabilities section of the same documentation.
If you are upgrading to 10.0.4 or later from a version earlier than 10.0.4, update the CR for the new Certificate Manager.

In 10.0.4, the Certificate Manager was upgraded to 1.5.1. In the CR, modify the annotations statements for the endpoint ingress-issuers as follows:
1. Edit the top-level APIConnectCluster CR.
2. In the spec. section, locate the subsection for each API Connect subsystem, and then locate modify the annotations from:
```
    annotations:
        certmanager.k8s.io/issuer: ingress-issuer
```
  to:
```
    annotations:
        cert-manager.io/issuer: ingress-issuer
```
3. Save and apply the change.
Disable the analytics message queue feature.

If you enabled the analytics message queue feature, you must disable it before upgrading. You can re-enable the message queue feature after the profile change is complete.

To disable the message queue:
1. Edit the top-level APIConnectCluster CR.
2. In the spec.analytics section, locate the mq subsection and change the enabled setting to false:
```
spec:
    analytics:
      mq:
        enabled: false
```
3. Save and apply the change.