Upgrading Gateway subsystem on native Kubernetes

Upgrade the Gateway subsystem to the latest version of API Connect.

Before you begin

Complete all steps in Upgrading subsystems on native Kubernetes prior to the step that links to this topic.
Ensure your upgrade path is supported:
- To ensure operator compatibility, upgrade the API Connect management subsystem before you upgrade the DataPower gateway subsystem. This requirement applies to all upgrade scenarios.
- When upgrading from 10.0.1.2-eus to 10.0.4.0-ifix3, you must upgrade the gateway subsystem to 10.0.4.0-ifix3, to match the management subsystem.
- When upgrading from 10.0.3.0 to 10.0.4.0-ifix3, you can either upgrade the gateway subsystem to 10.0.4.0-ifix3 or leave it at 10.0.3.0. Both scenarios are supported. Note that you cannot upgrade the gateway subsystem to 10.0.4.0-ifix3 unless the management subsystem was previously upgraded to 10.0.4.0-ifix3.
When you upgrade a cluster of gateway pods in Kubernetes, a small number of API transactions may fail. During the upgrade, Kubernetes removes the pod from the load balancer configuration, deletes the pod and then starts a new pod. The steps are repeated for each pod. Socket hang ups occur on transactions that are in process at the time the pod is killed.
The number of transactions that fail depends on the rate of incoming transactions and the length of time needed to complete each transaction. Typically the number of failures is a very small percentage. This behavior is expected during an upgrade. If the failure level is not acceptable, schedule the upgrade during an off-hours maintenance window.

Note also that DataPower Gateway supports long-lived connections such as such as GraphQL subscriptions or other websockets connections. These long-lived connections might not be preserved when upgrading. Workloads with long-lived connections are more vulnerable to failed API transactions during upgrading.

Starting with 10.0.3.0, you can limit the number of failed API transactions during the upgrade by using the DataPower Operator's lifecycle property to configure the preStop container lifecycle hook on the gateway pods. This approach mitigates the risk of API failures during the rolling update of the gateway StatefulSet by sleeping the pod for a span of time, allowing in-flight transactions to complete prior to the SIGTERM being delivered to the container. While this feature does not guarantee that no in-flight APIs would fail, it does provide some mitigation for in-flight transactions that can complete successfully within the configured time window. For more information, see Delaying SIGTERM with preStop in the DataPower documentation.
When upgrading a high-availability cluster, ensure that you meet the requirements:
- Gateways must be updated one at a time.
- Before starting the upgrade, a single gateway must be running as primary for all gateway-peering definitions.
- When upgrading multiple gateways, the primary gateway must be upgraded last.
- Ensure that the pod with a name like gwv6-0 or gwv5-0 is the primary because it is the last node to be upgraded.
To determine which gateway is running as primary, use either the show gateway-peering-status command in the DataPower CLI, or use the Gateway Peering Status display in the WebGUI in the API Connect application domain. To move the primary to the DataPower on which you're currently working, you can issue the gateway-peering-switch-primary <peering-object-name> command.

To access the DataPower CLI or WebGUI, use:
```
kubectl attach -it <podname>
```

About this task

When upgrading to a new mod release the version must be changed to the latest mod release version on each CR. This change will be picked up by the operator, and the operator will then start the upgrade.

Procedure

Delete the old Gateway certificates before changing the Gateway operand to 10.0.4.0-ifix3

kubectl delete certs.certmanager.k8s.io gateway-service gateway-peering -n gtw -n <namespace>

Update version: in the CR:
```
kubectl -n $NAMESPACE edit gw <CLUSTERNAME>
```
where CLUSTERNAME is the name specified in the subsystem CR at installation time.
```
version: <version>
```
Example result:
```
version: 10.0.4.0-ifix3
```
Edit the license: setting:
1. Set accept: to true to accept the license. Note that the default value is false. If you do not accept the license, the Operator will not install the subsystem.
2. Set metric: to track your product usage. Enter the unit of measure that is used for your program license:
  - PROCESSOR_VALUE_UNIT - Default value. If you leave the field blank, this value is used.
  - MONTHLY_API_CALL - Applies only to the IBM API Connect Hybrid Entitlement program.
  For information on tracking monthly call volume, see Tracking API volume for auditing and compliance.
3. Set use: to either production or nonproduction, to match the license you purchased.
4. Set license: to the License ID for the version of API Connect that you purchased. See API Connect licenses.
5. In the spec.gateway section, delete the template override section, if it exists. You cannot perform an upgrade if the CR contains an override.
Example entry to accept the license for a production system:
```
  license:
    accept: true
    metric: PROCESSOR_VALUE_UNIT 
    use: production
    license: L-RJON-C2YLGB
```

Example output after upgrading the Gateway subsystem:

kubectl get apic -n <namespace>

NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   <version>  <version-build>  100m

Continue with Enabling gateway peering and verifying cluster status.