Convert IBM Cloud Infrastructure Center standalone deployment to multi-node cluster deployment

After upgrading to the 1.2.1 release for standalone IBM® Cloud Infrastructure Center following the standalone upgrade docs Getting started with the upgrade to release 1.2.1 you may want to convert your environment to the multi-node cluster.

The overall steps are the following:

  1. Backup your current 1.2.1 environment

  2. Convert environment to "single-node" cluster temporarily

  3. Migrate Glance images.

  4. Add two more nodes to have a 3 node multi-node cluster setup

The following goes into detail for each step of the conversion:

  1. First you must upgrade your standalone IBM Cloud Infrastructure Center to the 1.2.1 release and make sure it is online and healthy before going further. Note: The exported images and volume backups cannot be migrated to the IBM Cloud Infrastructure Center multi-node cluster. If you need to continue to use them on multi-node cluster, for the exported images, you need to import them before you convert your standalone IBM Cloud Infrastructure Center to multi-node cluster. Then, export them if needed after the multi-node cluster is available. In the same way, for volume backups, you need to restore them before the conversion and backup them if needed after the multi-node cluster is available.

  2. Run a backup of your current standalone environment, this is only used in a disaster recovery scenario: icic-backup --all --targetdir <backup_folder>. After the backup verify that it is correctly generated in the <backup_folder> where this is a directory of your choosing.

  3. Extract the IBM Cloud Infrastructure Center package and navigate to the directory icic-opsmgr-1.2.1.0/

  • per-vm package:

     tar -xzvf IBM_Cloud_Infrastructure_Center_1.2.1.tar.gz
     tar -xzvf icic-opsmgr-s390x-rhel-1.2.1.0.tgz
     cd icic-opsmgr-1.2.1.0
  • per-engine package:

     tar -xzvf IBM_Cloud_Infrastructure_Center_per_engine_1.2.1.tar.gz
     tar -xzvf icic-opsmgr-s390x-rhel-1.2.1.0.tgz
     cd icic-opsmgr-1.2.1.0
  1. Now, begin converting to the multi-node cluster, run ./setup_opsmgr.sh --icic-to-ha that configures the packages to /opt/ibm/icic-opsmgr and installs some of the neccessary dependencies. This also ensures that you have already taken a backup prior to going further.

Example:

 ```
 ./setup_opsmgr.sh --icic-to-ha
 ################################################################################
 Starting the IBM Cloud Infrastructure Center:setup_prepnode 1.2.1.0 installation on:
 2023-06-05T22:17:06-04:00
 ################################################################################

 2023-06-05T22:17:06.10-04:00 [INFO]:[] IBM Cloud Infrastructure Center Opsmgr Standalone to HA conversion requires a backup to exist in the case of disaster recovery.
 2023-06-05T22:17:06.10-04:00 [INFO]:[] The more recent the backup the better to mitigate any data loss in the case of disaster recovery and we need to use this backup.
 2023-06-05T22:17:06.11-04:00 [INFO]:[] The following backup files were found at /var/opt/ibm/icic/backups/
 /var/opt/ibm/icic/backups/etc
 /var/opt/ibm/icic/backups/20230605214815065571
 2023-06-05T22:17:06.13-04:00 [WARNING]:[] These backups would only be used in the case of diaster recovery. Do these backups look OK and are the most recent? y
 2023-06-05T22:17:09.51-04:00 [WARNING]:[] IBM Cloud Infrastructure Center Opsmgr will be installed and will set the default python to 3.6 at system level
 2023-06-05T22:17:09.51-04:00 [INFO]:[VALIDATION] BASE DIRECTORY : /opt/ibm/icic-opsmgr
 2023-06-05T22:17:09.52-04:00 [INFO]:[VALIDATION] INSTALL SCRIPT : /root/icic-opsmgr-1.2.1.0/setup_opsmgr.sh
 2023-06-05T22:17:09.52-04:00 [INFO]:[VALIDATION] LOG FILE : /opt/ibm/icic-opsmgr/log/prep_node/prep_node_2023-06-05-2217.log
 2023-06-05T22:17:09.53-04:00 [INFO]:[VALIDATION] OPERATING SYSTEM: Red Hat Enterprise Linux.
 2023-06-05T22:17:09.54-04:00 [INFO]:[VALIDATION] User ID : ****
 International Program License Agreement

 Part 1 - General Terms

 BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, CLICKING ON
 AN "ACCEPT" BUTTON, OR OTHERWISE USING THE PROGRAM,
 LICENSEE AGREES TO THE TERMS OF THIS AGREEMENT. IF YOU ARE
 ACCEPTING THESE TERMS ON BEHALF OF LICENSEE, YOU REPRESENT
 THAT YOU HAVE FULL AUTHORITY TO BIND LICENSEE TO THESE
 TERMS.

 IF YOU DO NOT AGREE TO THESE TERMS OR DO NOT HAVE
 AUTHORITY: i) DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, CLICK
 ON AN "ACCEPT" BUTTON, OR USE THE PROGRAM; AND ii) PROMPTLY
 RETURN THE UNUSED MEDIA, DOCUMENTATION, AND PROOF OF

 Press Enter to continue viewing the license agreement, or
 enter "1" to accept the agreement, "2" to decline it, "3"
 to print it, "4" to read non-IBM terms, or "99" to go back

 to the previous screen. 1
 2023-06-05T22:17:10.09-04:00 [INFO]:[VALIDATION] License accepted.
 2023-06-05T22:17:10.10-04:00 [INFO]:[VALIDATION] Ansible not detected, Installing ansible-2.9.20
 2023-06-05T22:17:26.06-04:00 [INFO]:[VALIDATION] Configuring DNF repository for ICIC-Opsmgr...
 2023-06-05T22:17:29.06-04:00 [INFO]:[VALIDATION] Installing IBM Cloud Infrastructure Center Ops Manager...
 2023-06-05T22:17:29.12-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] Installing IBM Cloud Infrastructure Center Opsmgr packages for RHEL8: sshpass python3-netaddr rsync python3-oslo-i18n python3-crudini net-tools python3-icic-opsmgr python3-ansible-runner python3-pexpect python3-daemon python3-lockfile python3-backports-ssl_match_hostname python3-certifi python3-chardet python3-ipaddress python3-jmespath python3-netaddr python3-paramiko python3-prettytable python3-requests python3-psutil python3-ptyprocess python3-PyYAML python3-prettytable python3-iniparse python3-six python3-backports-ssl_match_hostname
 2023-06-05T22:18:04.86-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] Python version (python36) check passed!
 2023-06-05T22:18:04.86-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] Installing python3-lxml python3-requests python3-urllib3 python3-cffi
 2023-06-05T22:18:07.57-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] Installing python3-pytz python3-pyparsing python3-cryptography python3-jinja2 python3-decorator python3-stevedore python3-lxml python3-cffi
 2023-06-05T22:18:10.29-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] Installing python3-PyYAML python3-PyNaCl python3-bcrypt python3-cryptography
 2023-06-05T22:18:13.43-04:00 [INFO]:[INSTALL_ICIC_OPSMGR] IBM Cloud Infrastructure Center Opsmgr packages installed successfully.

 ################################################################################
 IBM Cloud Infrastructure Center:setup_prepnode installation completed at 2023-06-05T22:18:13-04:00.
 Refer to
 /opt/ibm/icic-opsmgr/log/prep_node/prep_node_2023-06-05-2217.log
 for more details.
 ################################################################################
 ```
  1. Now use the icic-opsmgr command suite to create our inventory file

    icic-opsmgr inventory -c <cluster_name>

    Tips:

    1. When prompted Configure Firewall (y/n) choose y if your current standalone IBM Cloud Infrastructure Center environment uses a firewall configuration. If y it proceeds to ask Firewall Type where you choose what firewall you desire. If you choose firewalld, then later in install process if iptables is running it stops the service and vice-versa. For more information about the option of whether to use the Swift Object Storage to store the images, image backups, and volume backups, you can refer to the Create multi-node cluster in the inventory creation document.

    2. When prompted Enter the number of nodes this must be 1

    3. When prompted Enter IP/hostname for Node 1 this would be the ip address of the node where we are running.

    4. When prompted Enter virtual IP, this is a non pingable IP address of your selection that is setup.

  2. After the previous successful steps you can now run icic-opsmgr install -c <cluster_name> --icic-to-ha. This starts the deployment to the multi-node cluster. This step takes 45+ minutes with a lot of output to the console. You must use the --icic-to-ha option.

  3. Run icic-services restart and wait about 45 seconds for the services to stabalize.

  4. Migrate Glance images:

    • If you use a shared storage on the standalone IBM Cloud Infrastructure Center, such as: NFS or IBM Storage Scale, then skip this step.

    • If the Swift Object Storage is used, convert the images by using the command: icic-config image swift-store --enable, for more details, refer to Configure Image Swift Store.

    • If you use a local storage of the file system for the Glance images, and you want to use the shared storage in the multi-node cluster, you need to mount the shared storage to a directory on the management node. Then, configure the images store directory by using the command: icic-config image file-store-datadir. For more information, refer to Configure image store directory.

  5. Now, add 2 more nodes to make the configuration a fully operation multi-node cluster following the document: Adding nodes to a single node cluster.

  6. After converting to the multi-node cluster, if you turn on the monitoring services (ceilometer, gnocchi) and manually pack the /var/lib/gnocchi directory to one local store directory(for example: /tmp/gnocchi_backup) before converting, you need to check the historical meter data before converting. Restore it from the local store directory with the command: rsync -a /tmp/gnocchi_backup /var/lib/gnocchi, grant the gnocchhi ownership with the command: chmod -R gnocchi:gnocchi /var/lib/gnocchi, and modify the following option in the [storage] section of the /etc/gnocchi/gnocchi.conf to change storage backend driver from swift to file. Now, check meter data.

    [storage]
    driver = swift   -> file
  7. When complete, login to the UI using the Virtual IP specified.