PowerVC rolling upgrade

Edit online

The PowerVC Operations Manager (OpsMgr) is a collection of utilities and services that are designed to facilitate user operation of PowerVC clusters like install, upgrade, backup, and restore.

This topic explains the procedure for upgrading opsmgr utility and PowerVC on single node and multinode.

For RHEL
  • You can upgrade to PowerVC 2.3.3 from PowerVC 2.3.1 or 2.3.2 by using the PowerVC rolling upgrade procedure.
  • You must have PowerVC 2.3.1 or 2.3.2 installed on the supported RHEL 8.x and 9.x version before proceeding with rolling upgrade to PowerVC for Private Cloud 2.3.3. For direct installation procedure, see Initiate installation through PowerVC operations manager.
  • PowerVC rolling upgrade is supported only for multinode environments.

Prerequisites

Notes:
  • PowerVC rolling upgrade can be performed on PowerVC 2.3.1 or later installed nodes.
  • Manually take a backup of existing PowerVC, before you upgrade opsmgr utility.
  • Before you upgrade to PowerVC version 2.3.3, migrate any hosts that runs on Compute plane node to PowerVC controller by using the /opt/ibm/powervc/bin/powervc-manage -o migratehmchost --hmchostname <MTMS HOST> command. After upgrade process is complete, you can migrate the hosts back to Compute plane node by using the same command.
  • If upgrade is interrupted at any point for various reasons, you can always re-trigger the upgrade procedure.
  • Until all three nodes in a multinode environment are upgraded, you will not be able use the import or export functions on both CLI and API for images and CLI operations that change associated configuration options.
  • PowerVC GUI operations will not work when there are inadequate members in the MongoDB replica set to elect a primary node.
  • If upgrade for a remote node is being run, PowerVC OpsMgr will be upgraded automatically.
  • If network disconnections are expected, try to upgrade by using nohup command.
  • During VIP movement, you may experience downtime.
  • If there are any deploy failures post PowerVC update, it is recommended to restart PowerVC services and retry deploy operation.
  • To maintain PowerVC cluster consistency, restart controller node one at a time. Make sure that you allow 30 minutes before you restart the next node.
  • During the second node upgrade of PowerVC, you may experience downtime.
  • If there are hosts in maintenance mode and if it is not a single node cluster, exit the hosts from maintenance mode before upgrade. After the upgrade is completed, you can move the hosts to maintenance mode.
  • After the upgrade process is started, it is not recommended to leave the cluster in a partially upgraded state for a long period. Instead, it is recommended to upgrade all three nodes at once.

Upgrading opsmgr utility

  1. Visit Fix Central to download and install any fix packs that are available. For more information see the Getting fixes from Fix Central topic.
  2. Extract the tar file that matches your environment to the location you want to run the installation script from:
    • Upgrading to PowerVC 2.3.3 from PowerVC 2.3.1 or 2.3.2 versions

      For ppc64le, extract download_location/powervc-opsmgr-<rhel>-ppcle-2.3.3.tgz, where download_location is the directory where the tar file was downloaded to.

      For x86_64, extract the download_location/powervc-opsmgr-rhel-x86-2.3.3.tgz, where download_location is the directory where the tar file was downloaded to.

  3. Change the directory to: extract location/powervc-opsmgr-2.3.3, where extract location is the directory you extracted the tar file to in step 2.
  4. Run the following pre-update opsmgr health check command before you perform the update opsmgr operation:
    python3 powervc_opsmgr_health_check.py
  5. To upgrade from PowerVC 2.3.1 or 2.3.2 versions to PowerVC 2.3.3, run update_opsmgr.sh script on the primary or bootstrap node.

    Alternatively, you can run <path>/update_opsmgr.sh -s to perform silent installation.

Pre-upgrade recommendation for PowerVC multinode setup (20 or more registered hosts)

Before you upgrade PowerVC in a multinode environment with more than 20 registered hosts, it is recommended to update the galera cache settings to prevent potential downtime during the upgrade process, especially if one of the three database nodes is offline for an extended period or if the database capacity is high.

Configuration change
Update the following galera cache settings in the /etc/my.cnf.d/server.cnf configuration file under the [galera] section:
wsrep_provider_options='gcache.size=2G; gcache.recover=yes;'
You can also use the following crudini command to set the gcache field in the /etc/my.cnf.d/server.cnf configuration file:
crudini --set /etc/my.cnf.d/server.cnf galera wsrep_provider_options "'gcache.size=2G; gcache.recover=yes;'"
Restart the database service
After the configuration change are applied, restart the database service on each node individually by using the following command:
powervc-services db restart --advanced <node-ip>

Upgrading PowerVC on multinode

After upgrading OpsMgr, you can upgrade PowerVC on multiple nodes (up to five nodes).
Notes:
  • For multinode, upgrade happens node by node.
  • Run powervc-opsmgr update --help for details on update sub-commands.
  • There are two ways to upgrade to the latest version of NovaLink during the upgrade of the latest version of PowerVC:
    • After the first node upgrade of PowerVC, perform the NovaLink upgrade to the latest version. After the NovaLink upgrade, proceed with the remaining nodes upgrade in PowerVC.
    • You can use the -sr option to skip remote node upgrade during the PowerVC upgrade. After the PowerVC upgrade is complete, you can upgrade the remote nodes.
  • If upgrading from PowerVC 2.3.1 or 2.3.2 to PowerVC 2.3.3, after the management node upgrade, you can continue to use PowerVC 2.3.3 with the NovaLink 2.3.1 or 2.3.2 version, and later update to NovaLink 2.3.3 version.
  • Upgrade the multi nodes one after the other in this order.
    1. Primary node should always be updated first.
    2. Update the non-VIP running node.
    3. Update the VIP running node.
  • You can use the skip remote node -sr option for NovaLink upgrade, Compute plane node, and cinder backup node.

Upgrade of multinode must be done one node at a time.

You must run the powervc-health-check update command before you perform the update operation. For more information about the health check commands, see the powervc-health-check command in the CLI commands page. After the update health check is complete run the powervc-opsmgr update -c <cluster_name> -n <node_ip/hostname> command to upgrade non primary / bootstrap node.

Note:
  • After the upgrade is complete, and you see the host state as unknown, check if IPv6 is enabled in the node by running the sysctl -a 2>/dev/null | grep disable_ipv6 command. If IPv6 is enabled, disable IPv6 in all the nodes by running the sudo sysctl -w -p net.ipv6.conf.all.disable_ipv6=1 command, and then reboot the system.
  • Occasionally, after an upgrade, the virtual IP might become unreachable. You can confirm if the virtual IP is reachable or not by a ping test. This issue might occur due to an IP conflict or high load during the upgrade. If this issue occurs, run crm_resource -r virtualip --restart command after the upgrade to reactivate the virtual IP.