PowerVC rolling upgrade

Edit online

The PowerVC Operations Manager (OpsMgr) is a collection of utilities and services that are designed to facilitate user operation of PowerVC clusters like install, upgrade, backup, and restore.

This topic explains the procedure for upgrading opsmgr utility and PowerVC on single node and multinode.

For RHEL
  • You can upgrade to PowerVC 2.2.1.2 from PowerVC 2.2.1 or 2.2.1.1 by using the PowerVC rolling upgrade procedure.
  • You can upgrade to PowerVC 2.2.1.1 from PowerVC 2.2.0 or 2.2.1 by using the PowerVC rolling upgrade procedure.
  • You must have PowerVC 2.1.1 or later installed on the supported 2.2.1 RHEL 8.x and 9.x version before proceeding with rolling upgrade to PowerVC for Private Cloud 2.2.1. For direct installation procedure, see Initiate installation through PowerVC operations manager.
  • If you have a previous version of PowerVC 2.1.1 or 2.1.1.1 installed and want to upgrade to PowerVC for Private Cloud 2.2.1 by using the rolling upgrade or offline upgrade process, first run the powervc-opsmgr private_cloud --enable OpsMgr command to move to PowerVC for Private Cloud 2.1.1 or 2.1.1.1. Then, upgrade to PowerVC for Private Cloud 2.2.1.
For SLES
  • If you have a previous version of SLES 15 SP5 SAP Edition for Private Cloud 2.2.1 or 2.2.1.1 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP5 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.1 and want to move to SLES 15 SP3 SAP Edition of PowerVC for Private Cloud 2.2.1.2, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1.2 SLES 15 SP3 SAP Edition.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP3 SAP Edition of PowerVC for Private Cloud 2.2.1, you can perform a rolling upgrade, offline upgrade, or take a backup and restore to PowerVC for Private Cloud 2.2.1 SLES 15 SP3 SAP Edition.
  • If you have a previous version of SLES 15 SP3 SAP Edition for Private Cloud 2.2.0 and want to move to SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1, you can take a backup and restore to PowerVC for Private Cloud 2.2.1 SLES 15 SP5 SAP Edition.
  • If you have a previous version of SLES 15 SP3 PowerVC 2.1.1 or later and want to move to SLES 15 SP5, SLES 15 SP3 SAP Edition, or SLES 15 SP5 SAP Edition of PowerVC for Private Cloud 2.2.1, you can only use the backup and restore procedure. For details, see Backing up.

Prerequisites

  • Before upgrading, make sure that the cluster is in sync and the RabbitMQ service is running by executing these commands.
    • # clustercheck
    • # service rabbitmq-server status
  • It is recommended to update the operating system distribution before you upgrade to PowerVC 2.2.1, 2.2.1.1, or 2.2.1.2. If the operating system distribution is not updated before the update process, then it is mandatory to update the OpenSSH server to the latest version.
    Note: If the OpenSSH server is not updated to the latest version, it might lead to issues during the upgrade process.
    Before you proceed with PowerVC upgrade, perform the following steps to update the OpenSSH server to the latest version:
    1. Run the following command:
      yum update openssh-server
    2. Restart the SSHD service.
Notes:
  • Legacy install of PowerVC is not supported for add node, replace node, resync node, and compute plane node registration.
  • PowerVC rolling upgrade can be performed on PowerVC 2.1.1 or later installed nodes.
  • Manually take a backup of existing PowerVC, before you upgrade opsmgr utility.
  • Before upgrading to PowerVC version 2.2.1, migrate any hosts running on Compute plane node to PowerVC controller by using /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST>. After upgrade process is completed, you can migrate the hosts back to Compute plane node by using /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST>.
  • If upgrade is interrupted at any point for various reasons, you can always re-trigger the upgrade procedure.
  • Until all three nodes in a multinode environment are upgraded, you will not be able use the import or export functions on both CLI and API for images and CLI operations that change associated configuration options.
  • PowerVC GUI operations will not work when there are inadequate members in the MongoDB replica set to elect a primary node.
  • If upgrade for a remote node is being run, PowerVC OpsMgr will be upgraded automatically.
  • If network disconnections are expected, try to upgrade by using nohup command.
  • During VIP movement, you may experience downtime.
  • If there are any deploy failures post PowerVC update, it is recommended to restart PowerVC services and retry deploy operation.
  • To maintain PowerVC cluster consistency, restart controller node one at a time. Make sure that you allow 30 minutes before you restart the next node.
  • During the second node upgrade of PowerVC, you may experience downtime.
  • If there are hosts in maintenance mode and if it is not a single node cluster, exit the hosts from maintenance mode before upgrade. After the upgrade is completed, you can move the hosts to maintenance mode.
  • After the upgrade process is started, it is not recommended to leave the cluster in a partially upgraded state for a long period. Instead, it is recommended to upgrade all three nodes at once.

Upgrading opsmgr utility

  1. Visit Fix Central to download and install any fix packs that are available. For more information see the Getting fixes from Fix Central topic.
  2. Extract the tar file that matches your environment to the location you want to run the installation script from:
    • Upgrading to PowerVC 2.2.1.2 from PowerVC 2.2.1 or 2.2.1.1

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.2.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.2.tgz, where download_location is the directory where the tar file was downloaded to.

    • Upgrading to PowerVC 2.2.1.1 from PowerVC 2.2.0 or 2.2.1

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.1.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.1.tgz, where download_location is the directory where the tar file was downloaded to.

    • Upgrading to PowerVC 2.2.1 from PowerVC 2.1.1 or later versions

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.2.1.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.2.1.tgz, where download_location is the directory where the tar file was downloaded to.

  3. Change the directory to: extract location/powervc-opsmgr-2.2.1, where extract location is the directory you extracted the tar file to in step 2.
  4. Run the update_opsmgr.sh script on the primary or bootstrap node to upgrade the following PowerVC versions:
    • PowerVC version 2.1.1 or 2.1.1.1 to 2.2.1
    • PowerVC version 2.2.1 to 2.2.1.1
    • PowerVC version 2.2.1 or 2.2.1.1 to 2.2.1.2

    Alternatively, you can run <path>/update_opsmgr.sh -s to perform silent installation.

Upgrading PowerVC on single node or multinode

After upgrading OpsMgr, you can upgrade PowerVC on single node and later add multiple nodes (up to five nodes) if needed.
Notes:
  • For multinode, upgrade happens node by node.
  • Run powervc-opsmgr update --help for details on update sub-commands.
  • There are two ways to upgrade to the latest version of NovaLink during the upgrade of the latest version of PowerVC:
    • After the first node upgrade of PowerVC, perform the NovaLink upgrade to the latest version. After the NovaLink upgrade, proceed with the remaining nodes upgrade in PowerVC.
    • You can use the -sr option to skip remote node upgrade during the PowerVC upgrade. After the PowerVC upgrade is complete, you can upgrade the remote nodes.
  • If upgrading from PowerVC 2.1.1 or later to PowerVC 2.2.1, after the management node upgrade, you can continue to use PowerVC 2.2.1 with the NovaLink 2.1.1 or later version, and later update to NovaLink 2.2.1 version.
  • You can use the skip remote node -sr option for NovaLink upgrade, Compute plane node, and cinder backup node.
  1. For single node upgrade, run powervc-opsmgr update -c <cluster_name> -n <node_ip/hostname> to upgrade PowerVC to version 2.2.1.x.
  2. Upgrade of multinode must be done one node at a time.

    Run powervc-opsmgr update -c <cluster_name> -n <node_ip/hostname> to upgrade non primary / bootstrap node.

Note:
  • After the upgrade is complete, and you see the host state as unknown, check if IPv6 is enabled in the node by running the sysctl -a 2>/dev/null | grep disable_ipv6 command. If IPv6 is enabled, disable IPv6 in all the nodes by running the sudo sysctl -w -p net.ipv6.conf.all.disable_ipv6=1 command, and then reboot the system.
  • Occasionally, after an upgrade, the virtual IP might become unreachable. You can confirm if the virtual IP is reachable or not by a ping test. This issue might occur due to an IP conflict or high load during the upgrade. If this issue occurs, run crm_resource -r virtualip --restart command after the upgrade to reactivate the virtual IP.\