Upgrading the Management subsystem from 2018 to v10

You can upgrade the management subsystem from 2018 to the latest version of v10.

Before you begin

Before you can upgrade the management subsystem, you must prepare the API Connect cluster namespaces. If you haven't done so already, see one of the following:

About this task

One of the major changes between 2018 and v10 is the change from Cassandra to Postgres for the management database. The upgrade process uses the following steps to upgrade the management subsystem database:
1. Take management database backup.
  Backup and restore must be configured for management subsystem on 2018.
2. Scale down the 2018 pods except Cassandra database
3. Extract data from Cassandra database.
  This task runs as an extract job. The extract job creates logs.
4. Bring up Postgres database.
5. Load the extracted data into Postgres database and do the necessary transformations.
  This task runs as the load job. The load job creates a log.
To upgrade the management subsystem, you configure an upgrade CR and apply it. When you apply the CR, the upgrade process runs automatically.

Procedure

Take a full backup of the API Connect 2018 Management subsystem. This backup is in addition to the backup taken in Preparing the 2018 deployment for upgrade, which was taken before the pre-upgrade operations. The pre-upgrade operations will have changed some of the contents of the database and so it is recommended to have backups from before and after.

Populate the template upgrade CR for management.apiconnect.ibm.com_v1beta1_upgradefromv2018_cr.yaml with values for your deployment:

Table 1. Variables to set in the Upgrade CR template
Variable	Description
`$NAME`	Name to uniquely identify this resource. Example: `mgmt`
`$MANAGEMENT_SUBSYSTEM_NAME`	Must match the v2018's management subsystem name. Use `apicup subsys list` to find the name. For example: `$ apicup subsys list Name Type Target mgmt ManagementSubsystem kubernetes gwv5 GatewaySubsystem kubernetes gwv6 GatewaySubsystem kubernetes portal PortalSubsystem kubernetes analytics AnalyticsSubsystem kubernetes`
`$TRIGGER_ROLLBACK`	Can be `true` or `false`. Must initially be set to `false` (default value) when creating the resource. After creation can be updated and set to `true` to trigger a management subsystem rollback.
`$IMAGE_REGISTRY`	Provide valid image registry to access the upgrade and version 10 artifacts. Must be provided if `DEFAULT_IMAGE_REGISTRY` is not set.
`$IMAGE_PULL_SECRETS`	Valid image pull secrets to pull images from image registry specified in `$IMAGE_REGISTRY`. Must be provided if `DEFAULT_IMAGE_PULL_SECRET` is not set.
`$PROFILE`	Specifies the deployment profile, see Planning your deployment topology.
`$STORAGE_CLASS_NAME`	Storage class name must be provided for all Volume Claim templates. The `default-class-name` will be used only if cluster's default-class-name has been set.
`$VOLUME_SIZE`	`upgradeVolumeClaimTemplate:` The size of the PVC created for the extract job. Minimum size 50Gi. `databaseVolumeClaimTemplate:` - Minimum size 120 Gi `dbBackupVolumeClaimTemplate:` - Minimum size 120 Gi `dbArchiveVolumeClaimTemplate:` - Minimum size 30 Gi. `messageQueueVolumeClaimTemplate:` - Minimum size of 10 Gi

Edit the license: setting:
1. Set accept: to true to accept the license. Note that the default value is false. If you do not accept the license, the Operator will not install the subsystem.
2. Set metric: to track your product usage. Enter the unit of measure that is used for your program license:
  - PROCESSOR_VALUE_UNIT - Default value. If you leave the field blank, this value is used.
  - MONTHLY_API_CALL - Applies only to the IBM API Connect Hybrid Entitlement program.
    For information on tracking monthly call volume, see Tracking API volume for auditing and compliance.
3. Set use: to either production or nonproduction, to match the license you purchased.
4. Set license: to the License ID for the version of API Connect that you purchased. See API Connect licenses.
Example entry to accept the license for a production system:
```
  license:
    accept: true
    metric: PROCESSOR_VALUE_UNIT 
    use: production
    license: L-RJON-CEBLEH
```

Optional: - Database backup configuration can be specified in the upgrade CR.

You can also specify the database backup config after the management subsystem has been upgraded to version 10. If you choose to specify the database backup config in the management upgrade CR, the backup configuration will be carried forward to the v10 management subsystem. The upgraded management subsystem (version 10) will use this database backup config to perform backups as per the schedule.

Note:

You must:

Use a new location (path) for the version 10 management database backup configuration; do not use the 2018 management backup path.
Ensure that Spec.ManagementSpec.DatabaseBackup.RestartDB is set to false.
Create a management backup secret before configuring database backups.

To review database backup configuration, including creation of a management backup secret, see Configuring backup settings for fresh install of the Management subsystem.

Sample upgrade CR with database backup configuration:

apiVersion: management.apiconnect.ibm.com/v1beta1
   kind: ManagementUpgradeFromV2018
   metadata:
     name: mgmt
   spec:
     subsystemName: mgmt
     triggerRollback: false
     upgradeVolumeClaimTemplate:
       storageClassName: local-storage
       volumeSize: 50Gi
     imageRegistry: apic-dev-docker-local.artifactory.swg-devops.com
     imagePullSecrets:
     - apic-registry-secret
     profile: n1xc4.m16
     license:
       accept: true
       metric: PROCESSOR_VALUE_UNIT 
       use: production
       license: L-RJON-CEBLEH
     certManagerIssuer:
       name: selfsigning-issuer
       kind: Issuer
     managementSpec:
       databaseVolumeClaimTemplate:
         storageClassName: local-storage
         volumeSize: 120Gi
       dbBackupVolumeClaimTemplate:
         storageClassName: local-storage
         volumeSize: 120Gi
       dbArchiveVolumeClaimTemplate:
         storageClassName: local-storage
         volumeSize: 30Gi
       messageQueueVolumeClaimTemplate:
         storageClassName: local-storage
         volumeSize: 10Gi
       databaseBackup:
         credentials: ibmdbcreds
         host: s3.us-east.cloud-object-storage.appdomain.cloud/us-east
         path: $BUCKET_PATH/$BUCKET_NAME
         protocol: objstore
         restartDB:
           accept: false
         s3provider: ibm

Create the 2018 upgrade CR for management:
```
kubectl apply -f management.apiconnect.ibm.com_v1beta1_upgradefromv2018_cr.yaml -n <management-subsystem-namespace>
```
Note: management-subsystem-namespace in a single namespace APIConnect cluster will be same as the namespace as that of all the other subsystem namespaces.

The upgrade CR triggers the upgrade process for management.

Monitor upgrade progress:

Open a new terminal and tail upgrade-operator pod:

kubectl logs <upgrade-operator-pod> -n <management-subsystem-namespace>

Open a new terminal and check the status of the management upgrade:
```
kubectl get up2018m -n <management-subsystem-namespace>
```
Note: up2018m is the short name for management-upgrade-type and can be used to track the management upgrade progress. Each subsystem has a dedicated short name for upgrade:

Upgrade Subsystem Short name

Management up2018m

Analytics up2018a

Portal up2018p

Gateway up2018g

Look for READY and STATUS in the resource's status to track your progress:
```
NAME           READY                      STATUS     AGE
mgmt           14/14: Upgrade completed   Complete   12h
```

Upgrade Subsystem	Short name
Management	`up2018m`
Analytics	`up2018a`
Portal	`up2018p`
Gateway	`up2018g`

Once the upgrade process has completed the extract job, open a new terminal and monitor the progress of version 10 installation:

kubectl get apic -n <management-subsystem-namespace>

When the upgrade succeeds, the upgrade state displays Upgrade is Complete. Sample output:

 NAME                                                              STATUS     MESSAGE                                               AGE 
   managementdbupgrade.management.apiconnect.ibm.com/mgmt-up-4n6gq   Complete   Upgrade is Complete (DB Schema/data are up-to-date)   12h 

   NAME                                                                    READY                      STATUS     AGE
   managementupgradefromv2018.management.apiconnect.ibm.com/mgmt   14/14: Upgrade completed   Complete   12h

   NAME                                                   READY   STATUS    VERSION      RECONCILED VERSION   AGE
   managementcluster.management.apiconnect.ibm.com/mgmt   16/16   Running   10.0.5.1   10.0.5.1-xxxx       12h

If you encounter an error, see Rolling back a management upgrade to v2018.

Restart all nats-server pods by running the following command:

kubectl -n <namespace> delete po -l app.kubernetes.io/name=natscluster

If you did not specify backup settings in the upgrade CR in step 4, you must configure backup settings on the new version 10 management subsystem.
Note:
You must:
- Use a new location (path) for the version 10 management database backup configuration; do not use the 2018 management backup path.
- Ensure that Spec.ManagementSpec.DatabaseBackup.RestartDB is set to false.
- Create a management backup secret before configuring database backups.
- Expect that there will be a brief downtime upon configuring s3 backup settings in a new version 10 cluster.
To review database backup configuration, including creation of a management backup secret, see Configuring backup settings for fresh install of the Management subsystem.
Save the upgrade logs after a successful upgrade. This is not mandatory but is recommended, to retain the ability to track the extract and load jobs.
1. Save the PVC logs after a successful upgrade.
  The upgrade logs are stored in the /upgrade folder in the PVC. See Logs for management upgrade
2. Optionally, you can view logs of orphaned data from database upgrades. In the downloaded PVC contents, the logs are located in logs/apim/orphans and logs/lur/orphans. The data represents records which were deleted or updated because of invalid or missing references to data in other tables. If necessary, you can contact IBM Support to better understand the orphaned data.
3. Save the extract and load jobs logs. See Logs for extract and load jobs
We highly recommend that you retain the encryption secret and database credential secrets in a safe location, in case you need to restore during a disaster recovery scenario. Complete the steps in Preparing the management subsystem for disaster recovery using S3 backups or Preparing the management subsystem for disaster recovery using SFTP backups.

Note: If you do not complete the steps in Preparing the management subsystem for disaster recovery using S3 backups or Preparing the management subsystem for disaster recovery using SFTP backups., you will not be able to restore your management subsystem to a new cluster during a disaster recovery scenario.
Once the management subsystem has been upgraded, clear your browser cache and verify that you can log in to Cloud Manager UI and API manager UI and access the configuration menus.
Next, upgrade the Developer Portal subsystem. Continue with Upgrading the Developer Portal subsystem from 2018 to v10.

Note: If the management subsystem upgraded encountered problems, see the instructions in the links at the end of this page. See also Disaster recovery of a 2018 deployment