PowerVC offline upgrade

Edit online

The PowerVC Operations Manager (OpsMgr) is a collection of utilities and services that are designed to facilitate user operation of PowerVC clusters like install, upgrade, backup, and restore. In a multinode (HA) environment, all nodes can now be updated while services are offline. Update happens simultaneously on all nodes.

This topic explains the procedure for offline upgrading of opsmgr utility and PowerVC on single node and multinode.

For RHEL
  • You can upgrade to PowerVC 2.2.1.2 from PowerVC 2.2.1 or 2.2.1.1 by using the PowerVC offline upgrade procedure.
  • You can upgrade to PowerVC 2.2.1.1 from PowerVC 2.1.1.2, 2.2.0, or 2.2.1 by using the PowerVC offline upgrade procedure.
  • You must have PowerVC 2.1.1 or 2.1.1.1 installed on the supported RHEL 8.x and 9.x version before proceeding with offline upgrade to PowerVC for Private Cloud 2.2.1. For direct installation procedure, see Initiate installation through PowerVC operations manager.
  • If you have a previous version of PowerVC 2.1.1 or 2.1.1.1 installed and want to upgrade to PowerVC for Private Cloud 2.2.1 by using the rolling upgrade or offline upgrade process, first run the powervc-opsmgr private_cloud --enable OpsMgr command to move to PowerVC for Private Cloud 2.1.1 or 2.1.1.1. Then, upgrade to PowerVC for Private Cloud 2.2.1.
For SLES
  • If you have a previous version of SLES 15 SP5 SAP Edition for Private Cloud 2.2.1 or 2.2.1.1 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP5 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.1 and want to move to SLES 15 SP3 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP3 SAP Edition.
  • If you have a PowerVC 2.1.1.2 or 2.2.0 on SLES 15 SP3 and you want to upgrade to PowerVC 2.2.1.1, you must update the operating system (OS) from SLES 15 SP3 to SLES 15 SP5. PowerVC version 2.2.1.1 is not supported on SLES 15 SP3. For more information about the procedure to upgrade SLES 15 SP3 PowerVC 2.1.1.2 or 2.2.0 to PowerVC 2.2.1.1, see Upgrade PowerVC 2.1.1.2 or 2.2.0 on SLES 15 SP3 to PowerVC 2.2.1.1.
    Note: PowerVC for Private Cloud 2.2.1.1 does not support SLES 15 SP3. For more information about how to upgrade the OS from SLES 15 SP3 to SLES 15 SP5, see OS update from SLES-15 SP3 to SLES-15 SP5.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP3 SAP Edition of PowerVC for Private Cloud 2.2.1, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1 SLES 15 SP3 SAP Edition.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1, you can take a backup and restore to PowerVC for Private Cloud 2.2.1 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP3 PowerVC 2.1.1 or 2.1.1.1 and want to move to SLES 15 SP5, SLES 15 SP3 SAP Edition, or SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1, you can only use the backup and restore procedure. For details, see Backing up.

Prerequisites

  • Before upgrading, make sure that the cluster is in sync and the RabbitMQ service is running by executing these commands.
    • # clustercheck
    • # service rabbitmq-server status
  • It is recommended to update the operating system distribution before you upgrade to PowerVC 2.2.1, 2.2.1.1, or 2.2.1.2. If the operating system distribution is not updated before the update process, then it is mandatory to update the OpenSSH server to the latest version.
    Note: If the OpenSSH server is not updated to the latest version, it might lead to issues during the upgrade process.
    Before you proceed with PowerVC upgrade, perform the following steps to update the OpenSSH server to the latest version:
    1. Run the following command:
      yum update openssh-server
    2. Restart the SSHD service.
Notes:
  • Manually take a backup of existing PowerVC, before you upgrade opsmgr utility.
  • Before upgrading to PowerVC version 2.2.1, migrate any hosts running on Compute plane node to PowerVC controller by using /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST>. After upgrade process is completed, you can migrate the hosts back to Compute plane node by using /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST>.
Notes:
  • If upgrade is interrupted at any point for various reasons, you can always re-trigger the upgrade procedure.
  • If network disconnections are expected, try to upgrade by using nohup command.
  • PowerVC is not accessible when upgrade is in progress.

Upgrading opsmgr utility

  1. Visit Fix Central to download and install any fix packs that are available. For more information see the Getting fixes from Fix Central topic.
  2. Extract the tar file that matches your environment to the location you want to run the installation script from:
    • Upgrading to PowerVC 2.2.1.2 from PowerVC 2.2.1 or 2.2.1.1

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.2.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.2.tgz, where download_location is the directory where the tar file was downloaded to.

    • Upgrading to PowerVC 2.2.1.1 from PowerVC 2.1.1.2, 2.2.0, or 2.2.1

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.1.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.1.tgz, where download_location is the directory where the tar file was downloaded to.

    • Upgrading to PowerVC 2.2.1 from PowerVC 2.1.1 or 2.1.1.1 versions

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.tgz, where download_location is the directory where the tar file was downloaded to.

  3. Change the directory to: extract location/powervc-opsmgr-2.2.1, where extract location is the directory you extracted the tar file to in step 2.
  4. Run the update_opsmgr.sh script on the primary or bootstrap node to upgrade the following PowerVC versions:
    • PowerVC version 2.1.1 or 2.1.1.1 to 2.2.1
    • PowerVC version 2.2.1 to 2.2.1.1
    • PowerVC version 2.2.1 or 2.2.1.1 to 2.2.1.2

    Alternatively, you can run <path>/update_opsmgr.sh -s to perform silent installation.

Upgrading PowerVC on single node or multinode

After upgrading OpsMgr, you can upgrade PowerVC on single node and later add multiple nodes (up to five nodes) if needed.
Notes:
  • Run powervc-opsmgr update --help for details on update sub-commands.
  • Before upgrading from 2.1.1 or 2.1.1.1 to 2.2.1.x, upgrade to the latest version of NovaLink.
Run powervc-opsmgr update -o/--offline-mode -c/--cluster <cluster_name> on primary / bootstrap node to upgrade.
Notes:
  • If there are any deploy failures post PowerVC update, it is recommended to restart PowerVC services and retry deploy operation.
  • After the upgrade is complete, and you see the host state as unknown, check if IPv6 is enabled in the node by running the sysctl -a 2>/dev/null | grep disable_ipv6 command. If IPv6 is enabled, disable IPv6 in all the nodes by running the sudo sysctl -w -p net.ipv6.conf.all.disable_ipv6=1 command, and then reboot the system.
  • Occasionally, after an upgrade, the virtual IP might become unreachable. You can confirm if the virtual IP is reachable or not by a ping test. This issue might occur due to an IP conflict or high load during the upgrade. If this issue occurs, run crm_resource -r virtualip --restart command after the upgrade to reactivate the virtual IP.\
Upgrade PowerVC 2.1.1.2 or 2.2.0 on SLES 15 SP3 to PowerVC 2.2.1.1
To upgrade PowerVC 2.1.1.2 or 2.2.0 on SLES 15 SP3 to PowerVC 2.2.1.1 on both single node and multinode, complete the following steps:
  1. Place all the nodes in standby mode starting from the non-primary nodes. The primary node must be placed in standby mode only after other nodes are already in standby mode. To place the nodes in standby mode, enter the following command:
    crm_standby -v on --node <Nodename>
    where the Nodename is the name of the node that must be placed in standby mode.
  2. Update the OS for all the hosts to SLES 15 SP5, set the correct Python version and reboot the system. For more information about the procedure to update the OS to SLES 15 SP5, see OS update from SLES-15 SP3 to SLES-15 SP5.
  3. Remove all the nodes from standby mode after the OS is successfully updated to SLES SP5 for all the nodes and the system reboot is complete. Ensure that the primary node is removed from standby mode first, followed by all the other nodes. To remove the nodes from standby mode, enter the following command:
    crm_standby -v off --node <Nodename>
    where the Nodename is the name of the node that must be removed from standby mode.
  4. Extract the PowerVC 2.2.1.1 compressed file and save it on the primary node.
  5. To run the update_opsmgr.sh script file on the primary node, enter the following command:
    cd powervc-opsmgr-<version> 

./update_opsmgr.sh
    where version is the old version of PowerVC.
    Note: Few of the resources that are available in the crm_mon -1A utility fails in the start as the OS is already upgraded to SLES 15 SP5 whereas PowerVC is yet to be updated.
  6. To start the offline update of PowerVC 2.1.1.2 or 2.2.0 to PowerVC 2.2.1.1, enter the following command on the primary node:
    $ nohup powervc-opsmgr update -c <cluster-name> -o &