Troubleshooting the management subsystem upgrade from v2018 to v10

You can troubleshoot the upgrade of the management subsystem from v2018 to v10.

Contents:

Management upgrade states

Upgrade operator uses the following states to log the progress with management subsystem upgrades. The operator can be, but not necessarily be, in more than one state at a given point in time during management upgrade.

To determine the state of the operator, run:

kubectl describe up2018m <management-subsystem-name> -n <management-subsystem-namespace>.
Table 1. Management upgrade states
State Description
Running The management upgrade CR starts with this state. The management upgrade CR reports its progress in Reason and Message fields of this condition.
CleanupInProgress The management upgrade CR enters this state after successfully creating all the v10 microservices. The upgrade CR remains in this state until all v2018 management components have been deleted. To track the progress of the v10 management cluster deployment run:
kubectl get <management-subsystem-name> -n <management-subsystem-namespace>

The next condition set from here is Complete, upon successful upgrade completion.

RollbackRequired

The management upgrade CR enters this state when it hits any error during the upgrade process. Until the error gets fixed or upon setting triggerRollback: true, the upgrade CR will remain in this state.

While in this state, you can either probe into the error and fix it or trigger a rollback by setting spec.triggerRollback: true to continue with the upgrade process. If you choose to trigger a rollback, the subsystem will be rolled-back and the original v2018 management subsystem installation will be brought up.

The next condition set from this state is Failed.

Failed This condition is true when the upgrade process has failed.
Complete This management upgrade CR enters this state upon a successful upgrade of the management subsystem.

Common errors on management upgrades

  • Unable to perform backup due to incorrect backup configuration
  • Unable to perform backup due to improper access rights to the backup folder
  • Extract job stuck or not terminating with an Exit Code = 0
  • Load job stuck or not terminating with an Exit Code = 0
  • Unable to create v10 Management CR

Troubleshooting actions:

Logs for management upgrade

The following logs are must gather logs. This means they are needed in order to diagnosis problems with the upgrade.

You must collect the following logs prior to setting triggerRollback: true in the management upgrade CR. Be sure to redirect the output of each to a suitable file location:

  • Upgrade operator logs:
    kubectl logs <upgrade_operator_pod> -n <management-subsystem-namespace>
  • API Connect operator logs:
    kubectl logs <apiconnect_operator_pod> -n <management-subsystem-namespace>
  • Management CR description:
    kubectl get up2018m <management-subsystem-name> -n <management-subsystem-namespace> -o yaml

Logs for extract and load jobs

  • Example kubectl commands for viewing logs of extract and (or) load jobs:
    • Example for extract pod :
      kubectl logs <mgmt-v2018-to-v10-extract-xxxxx>

      To get information on the extract pod:

      kubectl describe pod <mgmt-v2018-to-v10-extract-xxxxx>
    • Example for load pod:
      kubectl logs <mgmt-v2018-to-v10-load-xxxxx>

      To get information on the load pod:

      kubectl describe pod <mgmt-v2018-to-v10-load-xxxxx>
  • The management upgrade logs the following output for extract and (or) load jobs:
    • Any data that expires after a fixed time, such as access tokens and refresh token.
    • Invitations for creating provider org, catalog, space and consumer org as they have expiration time set.
    • Member invitations created in an admin org, provider org, catalog or space as they have expiration time set.
    • Logs and data directory from the PVC. These are the contents of /upgrade directory. You must have a logging service setup prior to attempting upgrade. For example, https://logdna.com/.
    • The output of apicops preupgrade -m fix --upgrade_to v10
    • The output of apicops preupgrade -m report --upgrade_to v10 which is run before and after the fix option is run.