Management subsystem two data center upgrade to v10.0.5.4 and previous

How to upgrade your management subsystem in a two data center deployment.

About this task

If you are upgrading to v10.0.5.5 or later, then do not use this page, instead refer to: Management subsystem two data center upgrade to v10.0.5.5 and later.

Follow the steps in this topic, along with the steps documented in Upgrading to the latest release on VMware.

The steps in this topic involve running command-line operations from several locations:
  • apicup operations are run from your API Connect project directory.
    Note: The apicup command is a binary file that is included with each API Connect release. Unless stated otherwise, all operations in this topic that use the apicup command must use the apicup binary from the target release (the release that you want to upgrade to). Be careful to not use the apicup command from your source release.
  • apic and kubectl operations are run from within your management appliance, as the root user. Login to your management virtual appliance with an SSH client and the username apicadm, then switch to the root user with sudo -i.

Procedure

  1. Determine the current version and license of API Connect (the version that you are upgrading from).
    You will need to specify the correct version number and license in the next step.
    1. To get the current version, from within the management appliance run the following command:
      apic version

      In the response, the - subsystem-<subsytem-name> statement displays the installed version. In the following example, the response indicates that the installed version is 10.0.5.2:

      apic version
      
      INFO[0000] Log level: info                              
      Control plane: 1.26.0
      Appliance base 4.6.0
      GitCommitSha:10.0.5.2-55-gb784ac7
      Packages:
      - appliance-base:4.6.0
      - appliance-runtime-4.6.0:1.0.0
      - appliance-setup-4.6.0:1.0.0
      - subsystem-management:1:10.0.5.2
      Important: If you are upgrading from a source version of 10.0.1.x, you must append -eus to the end of all references to that version number; for example 10.0.1.2-ifix2-eus.
    2. To get the current license, run the following apicup command in your project directory:
      apicup licenses list
      in the response, the ID statement displays the installed license, for example:
      Accepted licenses:
      	ID: <source_license>
      	Product Version: <source_version>
      	URL: <license_url>
      Note: If there are multiple accepted licenses, refer only to the <source_license> that aligns with the <source_version> of your system (that is the <source_version> that matches the version that you obtained in step 1.a).
  2. Add source license and version to your extra-values.yaml file.
    Note: If you do not have an extra-values.yaml file, then create one in a directory that the apicup command can access.
    Add the source lines that you identified in step 1.b to your extra-values.yaml as shown:
    spec:
      version: "<source_version>"
      license:
        license: <source_license>
      imageRegistry: 127.0.0.1:8675/<source_version>
    
  3. Apply the extra-values.yaml file.
    Run the following apicup command to set the extra-values file for both the active and warm-standby sites:
    apicup subsys set <mgmt> extra-values-file=<extra-values-file-path>
    where <mgmt> is the name of your management subsystem, which you can get with:
    apicup subsys list

    If you want to validate your <extra-values-file-path>, and the other subsystem settings, before proceeding, you can run:

    apicup subsys get <mgmt> --validate
  4. Start the upgrade process on both active and warm-standby management subsystems.
    apicup subsys install <mgmt> <mgmt_upgrade_tar_archive>
    Note: If any Control Plane files are needed for your upgrade, you must append the files to the install command.
  5. Wait for the management subsystem status to become Running on both sites.
    To determine the status, login to the management appliances on both sites as root user and run:
    kubectl get mgmt
    
    NAME         READY   STATUS    VERSION    RECONCILED VERSION AGE
    management   n/n     Running   10.0.5.2   10.0.5.2-100   5h2m

    The RECONCILED VERSION/VERSION should be the source_version that you retrieved in step 1.

  6. Disable 2DCDR on the warm-standby management subsystem:
    apicup subsys set <warm-standby management subsystem> multi-site-ha-enabled=false
    apicup subsys install <warm-standby management subsystem>
    Conversion from warm-standby to a stand-alone management subsystem can take some time. Run the health check to ensure that the deployment is in a good state before proceeding:
    apicup subsys health-check <warm-standby management subsystem>
    an empty response means no detected problems.
  7. Disable 2DCDR on the active management subsystem.
    apicup subsys set <active management subsystem> multi-site-ha-enabled=false
    apicup subsys install <active management subsystem>
    Conversion from active to a stand-alone management subsystem can take some time. Run the health check to ensure that the deployment is in a good state before proceeding:
    apicup subsys health-check <active management subsystem>
    an empty response means no detected problems.
  8. Wait for the management subsystem status to become Running on both sites.
    To determine the status, login to the management appliances on both sites as root user and run:
    kubectl get mgmt
    
    NAME         READY   STATUS    VERSION    RECONCILED VERSION AGE
    management   n/n     Running   10.0.5.2   10.0.5.2-100   5h2m

    The RECONCILED VERSION/VERSION should be the source_version that you retrieved in step 1.

  9. Remove the source license and version from your extra-values.yaml file.
    Delete the version, license, and imageRegistry lines that you added to your extra-values.yaml file in step 2.
    Apply this change to both sites by running:
    apicup subsys install <mgmt>
  10. Wait for the management subsystems status to become Running on both sites.
    kubectl get mgmt 
    NAME         READY   STATUS    VERSION    RECONCILED VERSION AGE
    management   16/16   Running   10.0.5.3   10.0.5.3-100   5h2m

    The RECONCILED VERSION/VERSION should now be the target release that you are upgrading to.

  11. Run the health check on both clusters to ensure that the upgrade completed.
    apicup subsys health-check <mgmt>
    an empty response means no detected problems.
  12. Enable 2DCDR on the previously active management subsystem.
    apicup subsys set <active management subsystem> multi-site-ha-enabled=true
    apicup subsys install <active management subsystem>
  13. Wait for the management subsystem status to become Running on the active site.
    kubectl get mgmt
    NAME         READY   STATUS    VERSION    RECONCILED VERSION AGE
    management   16/16   Running   10.0.5.3   10.0.5.3-100   5h2m

    The RECONCILED VERSION/VERSION should now be the target release that you are upgrading to.

  14. Enable 2DCDR on the previously warm-standby management subsystem.
    apicup subsys set <warm-standby management subsystem> multi-site-ha-enabled=true
    apicup subsys install <warm-standby management subsystem>
  15. Wait for the management subsystem status to become Running on the warm-standby site.
    kubectl get mgmt 
    NAME         READY   STATUS    VERSION    RECONCILED VERSION AGE
    management   16/16   Running   10.0.5.3   10.0.5.3-100   5h2m

    The RECONCILED VERSION/VERSION should now be the target release that you are upgrading to.

  16. Verify that 2DCDR replication is working.
    To validate that your two data center deployment is replicating data, login to your active management appliance as root user, run kubectl get mgmt:
    kubectl get mgmt
    
    NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
    management   n/n     Running   10.0.5.3-0   10.0.5.3-100         Management is ready.              8m59s
    If you see HA Status Warning in the MESSAGE column, then check the status details with kubectl describe mgmt:
    status:
      haStatus
        {
          "lastTransitionTime": "2023-03-31T19:47:08Z",
          "message": "Replication not working, install or upgrade in progress.",
          "reason": "na",
          "status": "True",
          "type": "Pending"
       }
    Notes:
    • It takes time for the warm-standby site to sync with the active site.
    • When STATUS shows Running, you can reboot (if apicup subsys health-check advises you to do so).