PowerVC rolling upgrade

Edit online

The PowerVC Operations Manager (OpsMgr) is a collection of utilities and services that are designed to facilitate user operation of PowerVC clusters like install, upgrade, backup, and restore.

This topic explains the procedure for upgrading opsmgr utility and PowerVC on single node and multinode.

Note: Direct rolling upgrade from PowerVC 2.2.0 to PowerVC 2.3.0 is not supported.
For RHEL
  • You can upgrade to PowerVC 2.3.0 from PowerVC 2.2.1 or 2.2.1.1 by using the PowerVC rolling upgrade procedure.
  • You must have PowerVC 2.2.1 or 2.2.1.1 installed on the supported RHEL 8.x and 9.x version before proceeding with rolling upgrade to PowerVC for Private Cloud 2.3.0. For direct installation procedure, see Initiate installation through PowerVC operations manager.
  • PowerVC rolling upgrade is supported only for multinode environments.
For SLES
  • Direct rolling upgrade to PowerVC 2.3.0 is not supported. However, you can upgrade to PowerVC 2.3.0 by using backup and restore . For more information about backup, see Backing up. For more information about restore, see Restoring to PowerVC version 2.3.0.

Prerequisites

  • Before upgrading, make sure that the cluster is in sync and the RabbitMQ service is running by executing these commands.
    • # clustercheck
    • # service rabbitmq-server status
  • It is recommended to update the operating system distribution before you upgrade to PowerVC 2.3.0, or later versions. If the operating system distribution is not updated before the update process, then it is mandatory to update the OpenSSH server to the latest version.
    Note: If the OpenSSH server is not updated to the latest version, it might lead to issues during the upgrade process.
    Before you proceed with PowerVC upgrade, perform the following steps to update the OpenSSH server to the latest version:
    1. Run the following command:
      yum update openssh-server
    2. Restart the SSHD service.
Notes:
  • PowerVC rolling upgrade can be performed on PowerVC 2.2.1 or 2.2.1.1 installed nodes.
  • Manually take a backup of existing PowerVC, before you upgrade opsmgr utility.
  • Before you upgrade to PowerVC version 2.3.0, migrate any hosts that runs on Compute plane node to PowerVC controller by using the /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST> command. After upgrade process is complete, you can migrate the hosts back to Compute plane node by using the /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST> command.
  • If upgrade is interrupted at any point for various reasons, you can always re-trigger the upgrade procedure.
  • Until all three nodes in a multinode environment are upgraded, you will not be able use the import or export functions on both CLI and API for images and CLI operations that change associated configuration options.
  • PowerVC GUI operations will not work when there are inadequate members in the MongoDB replica set to elect a primary node.
  • If upgrade for a remote node is being run, PowerVC OpsMgr will be upgraded automatically.
  • If network disconnections are expected, try to upgrade by using nohup command.
  • During VIP movement, you may experience downtime.
  • If there are any deploy failures post PowerVC update, it is recommended to restart PowerVC services and retry deploy operation.
  • To maintain PowerVC cluster consistency, restart controller node one at a time. Make sure that you allow 30 minutes before you restart the next node.
  • During the second node upgrade of PowerVC, you may experience downtime.
  • If there are hosts in maintenance mode and if it is not a single node cluster, exit the hosts from maintenance mode before upgrade. After the upgrade is completed, you can move the hosts to maintenance mode.
  • After the upgrade process is started, it is not recommended to leave the cluster in a partially upgraded state for a long period. Instead, it is recommended to upgrade all three nodes at once.

Upgrading opsmgr utility

  1. Visit Fix Central to download and install any fix packs that are available. For more information see the Getting fixes from Fix Central topic.
  2. Extract the tar file that matches your environment to the location you want to run the installation script from:
    • Upgrading to PowerVC 2.3.0 from PowerVC 2.2.1 or 2.2.1.1 versions

      For ppc64le, extract download_location/powervc-opsmgr-<rhel or sles>-ppcle-2.3.0.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.3.0.tgz, where download_location is the directory where the tar file was downloaded to.

  3. Change the directory to: extract location/powervc-opsmgr-2.3.0, where extract location is the directory you extracted the tar file to in step 2.
  4. To upgrade from 2.2.1 or 2.2.1.1 versions to 2.3.0, run update_opsmgr.sh script on the primary or bootstrap node.

    Alternatively, you can run <path>/update_opsmgr.sh -s to perform silent installation.

Upgrading PowerVC on multinode

After upgrading OpsMgr, you can upgrade PowerVC on multiple nodes (up to five nodes).
Notes:
  • For multinode, upgrade happens node by node.
  • Run powervc-opsmgr update --help for details on update sub-commands.
  • There are two ways to upgrade to the latest version of NovaLink during the upgrade of the latest version of PowerVC:
    • After the first node upgrade of PowerVC, perform the NovaLink upgrade to the latest version. After the NovaLink upgrade, proceed with the remaining nodes upgrade in PowerVC.
    • You can use the -sr option to skip remote node upgrade during the PowerVC upgrade. After the PowerVC upgrade is complete, you can upgrade the remote nodes.
    • When upgrading from PowerVC 2.2.1 if skip remote node upgrade option was used, clone VM fails with VSN related error until Novalink is updated to 2.3.0 version.
  • If upgrading from PowerVC 2.2.1 or 2.2.1.1 to PowerVC 2.3.0, after the management node upgrade, you can continue to use PowerVC 2.3.0 with the NovaLink 2.2.1 or 2.2.1.1 version, and later update to NovaLink 2.3.0 version.
  • You can use the skip remote node -sr option for NovaLink upgrade, Compute plane node, and cinder backup node.

Upgrade of multinode must be done one node at a time.

Run powervc-opsmgr update -c <cluster_name> -n <node_ip/hostname> to upgrade non primary / bootstrap node.

Note:
  • After the upgrade is complete, and you see the host state as unknown, check if IPv6 is enabled in the node by running the sysctl -a 2>/dev/null | grep disable_ipv6 command. If IPv6 is enabled, disable IPv6 in all the nodes by running the sudo sysctl -w -p net.ipv6.conf.all.disable_ipv6=1 command, and then reboot the system.
  • Occasionally, after an upgrade, the virtual IP might become unreachable. You can confirm if the virtual IP is reachable or not by a ping test. This issue might occur due to an IP conflict or high load during the upgrade. If this issue occurs, run crm_resource -r virtualip --restart command after the upgrade to reactivate the virtual IP.\