Troubleshooting the management subsystem upgrade from v2018 to v10
You can troubleshoot the upgrade of the management subsystem from v2018 to v10.
Contents:
- Management upgrade states
- Common errors on management upgrades
- Logs for management upgrade
- Logs for extract and load jobs
Management upgrade states
Upgrade operator uses the following states to log the progress with management subsystem upgrades. The operator can be, but not necessarily be, in more than one state at a given point in time during management upgrade.
To determine the state of the operator, run:
kubectl describe up2018m <management-subsystem-name> -n <management-subsystem-namespace>.
State | Description |
---|---|
Running |
The management upgrade CR starts with this state. The management upgrade CR reports its
progress in Reason and Message fields of this condition. |
|
The management upgrade CR enters this state after successfully creating all the v10
microservices. The upgrade CR remains in this state until all v2018 management components have been
deleted. To track the progress of the v10 management cluster deployment run:
The next condition set from here is |
RollbackRequired |
The management upgrade CR enters this state when it hits any error during the upgrade process.
Until the error gets fixed or upon setting While in this state, you can either probe into the error and fix it or trigger a rollback by
setting The next condition set from this state is |
Failed |
This condition is true when the upgrade process has failed. |
Complete |
This management upgrade CR enters this state upon a successful upgrade of the management subsystem. |
Common errors on management upgrades
- Unable to perform backup due to incorrect backup configuration
- Unable to perform backup due to improper access rights to the backup folder
- Extract job stuck or not terminating with an Exit Code = 0
- Load job stuck or not terminating with an Exit Code = 0
- Unable to create v10 Management CR
Troubleshooting actions:
- Trigger a rollback. See Rolling back a management upgrade to v2018.
- Review Management upgrade states.
- Gather logs. See Logs for management upgrade and Logs for extract and load jobs.
- Update the v2018 backup configuration. See Backing up the management database in a Kubernetes environment
- Restart the v2018 to v10 upgrade. See Preparing the 2018 deployment for upgrade.
- Resolve any inconsistencies with the database. See Preparing the 2018 deployment for upgrade.
Logs for management upgrade
The following logs are must gather logs. This means they are needed in order to diagnosis problems with the upgrade.
You must collect the following logs prior to setting triggerRollback: true
in
the management upgrade CR. Be sure to redirect the output of each to a suitable file location:
- Upgrade operator logs:
kubectl logs <upgrade_operator_pod> -n <management-subsystem-namespace>
- API Connect operator logs:
kubectl logs <apiconnect_operator_pod> -n <management-subsystem-namespace>
- Management CR description:
kubectl get up2018m <management-subsystem-name> -n <management-subsystem-namespace> -o yaml
Logs for extract and load jobs
- Example
kubectl
commands for viewing logs of extract and (or) load jobs:- Example for extract pod
:
kubectl logs <mgmt-v2018-to-v10-extract-xxxxx>
To get information on the
extract
pod:kubectl describe pod <mgmt-v2018-to-v10-extract-xxxxx>
- Example for load
pod:
kubectl logs <mgmt-v2018-to-v10-load-xxxxx>
To get information on the
load
pod:kubectl describe pod <mgmt-v2018-to-v10-load-xxxxx>
- Example for extract pod
:
- The management upgrade logs the following output for extract and (or) load jobs:
- Any data that expires after a fixed time, such as access tokens and refresh token.
- Invitations for creating provider org, catalog, space and consumer org as they have expiration time set.
- Member invitations created in an admin org, provider org, catalog or space as they have expiration time set.
- Logs and data directory from the PVC. These are the contents of /upgrade directory. You must have a logging service setup prior to attempting upgrade. For example, https://logdna.com/.
- The output of
apicops preupgrade -m fix --upgrade_to v10
- The output of
apicops preupgrade -m report --upgrade_to v10
which is run before and after the fix option is run.