Cleaning up a failed management upgrade

If the management upgrade fails, remove any unneeded artifacts.

About this task

You can remove artifacts left from the failed upgrade, and then either retry the upgrade or stop the upgrade process.

Procedure

  1. Determine if you have any v10 ingresses left after successful rollback, and delete them:
    1. Check for ingresses:
      kubectl get ingress -n <management-subsystem-namespace>

      If any of the following ingresses are found, delete them manually:

      • <management-subsystem-name>-admin
      • <management-subsystem-name>-api-manager
      • <management-subsystem-name>-consumer-api
      • <management-subsystem-name>-platform-api
      Note: v10 ingresses are not deleted if the upgrade operator entered Failed status during any of these upgrade steps:
      • Creating postgres cluster
      • Running load job
      • Bringing up v10 deployments
    2. To manually delete the v10 ingresses after a rollback:
      kubectl delete ingress <ingress-name> -n <management-subsystem-namespace>
      Note: Delete only the v10 ingresses (if any). If not sure, compare the ingresses saved before performing the management upgrade to v10.
  2. Use the following instructions to manually delete the PVC used by the upgrade process:
    Note: Why is this step necessary?

    The PVC used by extract and load job has the same name as that of the management upgrade CR name. Rolling back the management upgrade process does not delete the PVC used by extract/load jobs. This is done to ensure that you can access to the extract and load jobs after a rollback or a successful upgrade.

    1. If applicable, gather extract and load job logs. See Logs for extract and load jobs: 2018 to 10
    2. Gather logs for the upgrade operator. See Logs for management upgrade: 2018 to 10,
    3. Manually delete the PVC:
      Important: Save the PVC data before deleting it.
      1. Delete extract jobs:
        kubectl delete jobs v2018-to-v10-extract -n <management-subsystem-namespace>
      2. Delete load jobs:
        kubectl delete jobs v2018-to-v10-load -n <management-subsystem-namespace>
      3. Get the PVC name. The PVC name is the same as the name of the management upgrade CR.
        kubectl get pvc -n <management-subsystem-namespace>
      4. Delete the PVC:
        kubectl delete pvc <management-upgrade-CR-name> -n <management-subsystem-namespace>
  3. If necessary, delete the postgres PVCs.

    The postgres PVCs must be deleted if you triggered a rollback on or after the upgrade reconciles the v10 cluster with only a postgres database.

    1. List the postgres PVCs:
      kubectl get pvc -n <management-subsystem-namespace>

      The following PVCs are used by postgres and must be deleted manually:

      • mgmt-xxxxxxxx-postgres
      • mgmt-xxxxxxxx-postgres-pgbr-repo
      • mgmt-xxxxxxxx-postgres-wal
    2. Delete the PVCs:
      kubectl delete pvc mgmt-xxxxxxxx-postgres -n <management-subsystem-namespace>
      kubectl delete pvc mgmt-xxxxxxxx-postgres-pgbr-repo -n <management-subsystem-namespace>
      kubectl delete pvc mgmt-xxxxxxxx-postgres-wal -n <management-subsystem-namespace>
  4. Decide whether to troubleshoot and retry the management upgrade to v10, or to stop the upgrade process.
  5. To re-run the management upgrade from v2018 to v10:
    1. Delete management upgrade CR:
      kubectl delete upgradefromv2018.management.apiconnect.ibm.com <management-subsystem-name> -n <management-subsystem-namespace>
    2. Fix the errors in your v2018 installation. See Common errors on management upgradesSee Common errors on management upgrades: version 2018 to version 10.
    3. Re-create the management upgrade v10 CR to trigger v10 management upgrade. See Preparing the 2018 deployment for upgrade.