Upgrading the Analytics subsystem from v2018 to 10.0.1.x-eus

You can upgrade the Analytics subsystem from v2018 to v10.0.1.2.

Before you begin

An S3 repository called "apic-analytics" must be created and registered with the v2018 deployment before starting the upgrade to v10.0.1.2. Backup and restore should be tested against it. Creating and registering an S3 repository for v2018 is explained in Backing up and restoring the analytics database in the v2018 documentation.

About this task

The Analytics upgrade procedure is based on backing up a v2018 Analytics subsystem and then restoring it onto a newly deployed v10 Analytics cluster.

The steps that the operator goes through to upgrade the Analytics subsystem are as follows:

Record 2018 deployment details.
Take a backup of the v2018 Analytics subsystem.
Scale down the v2018 components.
Create and apply the Analytics v10 Cluster CR.
Create and apply the Analytics v10 Restore CR to restore the v2018 Analytics backup onto the v10 Analytics subsystem.
Remove the v2018 Analytics subsystem resources.
Set the Complete status condition of the Analytics upgrade CR to true.
Notes:
- For deployment profiles with fewer than 3 storage nodes, the restore process is never marked as Complete. The storage cluster status is expected to be yellow and to have more than 0 unassigned shards. In this situation, the restore process is likely to be complete when there are 0 initializing shards.
- There will be a short delay before the restored v2018 data displays in the Analytics user interfaces.

The following Analytics configuration options cannot be copied from the v2018 deployment due to changes in how they are managed in v10:

MQ
Ingestion filters
Ingestion-only
Offloading

After you complete the upgrade, you can configure those options as explained in this procedure.

Procedure

Create a Kubernetes secret that contains the S3 repository credentials.

The secret must be called analytics-backup-secret and must be in the same namespace as the v2018 Analytics deployment. For information on creating the secret, see Configuring backup settings for Analytics.
Make a copy of the analytics.apiconnect.ibm.com_v1beta1_upgradefromv2018_cr.yaml template upgrade CR, and save it with the name: upgradefromv2018.yaml.

Populate the upgradefromv2018.yaml file with values for your deployment as shown in Table 1.

Table 1. Variables to set in the upgradefromv2018.yaml file for Analytics upgrade
Variable	Description
`$NAME`	Name to uniquely identify this resource. Example: `a7s-upgradefromv2018`
`$ANALYTICS_SUBSYSTEM_NAME`	Must match the v2018's management subsystem name. Use `apicup subsys list` to find the name. For example: `$ apicup subsys list Name Type Target mgmt ManagementSubsystem kubernetes gwv5 GatewaySubsystem kubernetes gwv6 GatewaySubsystem kubernetes portal PortalSubsystem kubernetes analytics AnalyticsSubsystem kubernetes`
`$TRIGGER_ROLLBACK`	Can be `true` or `false`. Must initially be set to `false` (default value) when creating the resource. Only set to `true` if you want to trigger an Analytics rollback.
`$PRODUCTION_OR_NONPRODUCTION`	License use type. Must be either `production` or `nonproduction` type of license. Note that license `accept:` must be set to `true` to enable the upgrade to proceed.
`$PROFILE`	Specifies mode of the v10 Analytics installation. Specify `n1xc2.m16` for dev or `n3xc4.m16` for production setup of v10 Analytics.
`$IMAGE_REGISTRY`	Provide valid image registry to access the upgrade and v10 artifacts. Must be provided if `DEFAULT_IMAGE_REGISTRY` is not set.
`$IMAGE_PULL_SECRETS`	Valid image pull secrets to pull images from image registry specified in `$IMAGE_REGISTRY`. Must be provided if `DEFAULT_IMAGE_PULL_SECRET` is not set.
`$STORAGE_CLASS_NAME`	In the `storage data` section, the storage class name must be provided. The `default-class-name` will be used only if the cluster's default-class-name has been set.
`$VOLUME_SIZE`	Volume size must be at least 50Gi for `data`.
`$STORAGE_CLASS_NAME`	In the `storage master` section, the storage class name must be provided. The `default-class-name` will be used only if the cluster's default-class-name has been set.
`$VOLUME_SIZE`	Volume size must be at least 5Gi for `master`.
`"OFFLOAD_SECRET_NAME"`	Optional. If you used a custom certificate for Analytics offloading in v2018 as described in Providing a custom certificate for analytics offload, specify the name of the offload secret in the `external offload` section. The offload configuration will be copied to the v10 Analytics deployment as part of the upgrade. Important: If you provide a secret that does not exist, the upgrade will fail.

Apply the Analytics upgradefromv2018 CR by running the following command:
```
kubectl apply -f analytics.apiconnect.ibm.com_v1beta1_upgradefromv2018_cr.yaml -n <namespace>
```
where <namespace> is the namespace where Analytics is installed.
Monitor the progress of the upgrade CR using one of the following methods:
- Check the current state of the upgrade CR itself by running the following command:
```
kubectl describe analytics.apiconnect.ibm.com_v1beta1_upgradefromv2018_cr.yaml <name> -n <namespace>
```
  where:
  - <name> is the name of the upgrade CR, which you can determine by running the following command:
```
kubectl get analyticsrestore
```
  - <namespace> is the namespace where Analytics is installed.
- Tail the apiconnect-upgrade-operator pod logs by running the following command:
```
kubectl logs -f <apiconnect-upgrade-operator-pod-name> --follow -n <namespace>
```
  where:
  - <apiconnect-upgrade-operator-pod-name> is the name of the pod log.
  - <namespace> is the namespace where Analytics is installed.
When the upgrade is complete, the upgrade CR will enter the status condition "Complete".
Known issues:
- You might see the following error in the operator log during the upgrade from v2018 to v10. The error is harmless and can be ignored:
```
"error": "Elastic search sync-flush returned status code 409 : a replica shard failed to sync-flush"}
```
- Sometimes the upgrade fails with the following message in the Status section of the log:
```
  Conditions:
    Last Transition Time:  <date-time>
    Message:               The analytics restore failed: Unable to trigger restore: Service Unavailable. Please wait and try again, or contact support.
    Reason:                RestoreFailed
    Status:                True
    Type:                  RollbackRequired
```
  If this error occurs, complete the following steps:
  1. Perform a rollback of the upgrade.
  2. Run the following command to get the name of the failed restore:
    kubectl get analyticsrestores.analytics.apiconnect.ibm.com -n <namespace>
  3. Run the following command to delete the failed restore:
    kubectl delete analyticsrestores.analytics.apiconnect.ibm.com <failed_restore_name> -n <namespace>
  4. Run the upgrade again.
Do not proceed to step 6 until the Analytics subsystem is successfully upgraded.
Delete remnant v2018 Analytics cronjob pods.
Two cronjob pods might remain after the upgrade completes. The jobs use names structured as in the following examples:
- r********-analytics-cronjobs-retention-**********-******
- r********-analytics-cronjobs-rollover-**********-******
These are remnant pods from v2018 and are now redundant. Remove them by completing the following steps:
1. List the pods by running the following command:
```
kubectl get pods -n <namespace>
```
2. Delete the pods by running the following command:
```
kubectl delete pods <name-of-remnant-cronjob-pod> -n <namespace>
```
Optional: Configure the Analytics configuration settings that could not be copied from v2018.
The following Analytics configuration settings were not upgraded from v2018, but can be manually configured in v10:
- MQ
- Ingestion filters
- Ingestion-only
- Offloading
Complete the following steps to configure the settings:
1. Manually update the CR of the new v10 Analytics deployment that was created as part of the upgrade.
  Update the CR by running the following command:
```
kubectl edit AnalyticsCluster <name> -n <namespace>
```
  where:
  - name is the name provided in the metadata.name section of the upgrade CR.
  - namespace is the namespace where Analytics is installed.
2. Add the settings for the desired configuration options.
  Refer to the following topics in the v10 documentation for information on the Analytics configuration settings:
  - MQ: Adding a message queue
  - Ingestion filters: Modifying your analytics data
  - Ingestion-only: Disabling internal storage
  - Offloading: Offloading data to a third-party system
Instruct users to clear their cache.

After a successful upgrade, users must clear their browser cache to ensure that the v10 Analytics UI works correctly.