Migrating IBM Red Hat CloudForms and IBM Cloud Pak for Multicloud Management appliances to Infrastructure Automation

This topic describes the process of doing a full appliance replacement upgrade when a new release of Infrastructure Automation – Infrastructure Management is available.

The following migration paths are supported:

  • IBM Cloud Pak for Multicloud Management 2.3.x - Infrastructure Management to IBM Cloud Pak for AIOps 4.6.1 - Infrastructure Automation
  • IBM Red Hat CloudForms 4.7 through 5.0 to IBM Cloud Pak for AIOps 4.6.1 - Infrastructure Automation

Ordering migrations

You must complete all migrations of the databases before you configure replication and high availability. The reason is because you deploy new appliances for migrating to Infrastructure Management.

Important: Do not run evmserverd if a database is in a partially migrated state.

Modifying the PostgreSQL configuration

This procedure modifies the strategy that is used for updating the PostgreSQL configuration. Previously, changes were made directly to the postgresql.conf file in the data directory. After you update the new configuration, changes will be made by editing a separate file in the /etc/manageiq/postgresql.conf.d directory. This modification allows future PostgreSQL configuration changes to be made more easily.

Note: The 01_miq_overrides.conf file in the /etc/manageiq/postgresql.conf.d directory is reserved for application overrides and is re-created on application updates. Important: This file must not be edited. More files are read by the PostgreSQL server process according to the documentation here: PostgreSQL - Managing Configuration File Contents

  1. Back up the postgresql.conf file so that you have any configuration changes:

    cp $APPLIANCE_PG_DATA/postgresql.conf <file_location>
    
  2. Copy the updated postgresql.conf file to the pg_data directory:

    cp $APPLIANCE_SOURCE_DIRECTORY/TEMPLATE/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf $APPLIANCE_PG_DATA/postgresql.conf
    
  3. Optional: If you updated the postgresql.conf file before this procedure, create a new 02_user_overrides.conf file, and add any modifications that you have from the original postgresql.conf file. Example layout of the 02_user_overrides.conf file:

    #----------------------------------------------
    # WRITE AHEAD LOG
    #----------------------------------------------
    
    wal_level = 'logical'
    wal_log_hints = on
    wal_buffers = 16MB
    min_wal_size = 1GB
    max_wal_size = 2GB
    checkpoint_completion_target = 0.9
    

    After the 02_user_overrides.conf file is modified and saved, restart the postgresql service:

    systemctl restart $APPLIANCE_PG_SERVICE
    

All postgresql.conf customizations are now made to files in the /etc/manageiq/postgresql.conf.d directory due to the following include_dir line in the default configuration. The settings in these new files override ones in the postgresql.conf file.

include_dir = '/etc/manageiq/postgresql.conf.d`

Migrating to the Infrastructure Management appliance

Overview

This procedure describes the process of migrating to the Infrastructure Management appliance. This procedure does not necessarily include migration of all possible customer modifications, so it is recommended that you fully test any modifications before you migrate your environment.

Important:

  • Read through all of the steps in this procedure before you begin the migration process.
  • Infrastructure Management appliances require 12 GB memory. However, before you migrate your appliances, adjust resources in your environment to avoid performance issues.

You can classify the migration into three groups of appliances:

  • VMDB appliance - An appliance with running workers that also hosts its own database that other appliances can also connect to.
  • Non-VMDB appliance - An appliance with running workers that does not host a database. The appliance is connected to an external database.
  • Dedicated database appliance - A Infrastructure Management appliance or non-Infrastructure Management virtual machine with no running workers on it: the appliance contains only a database for other appliances to connect to.

Migration workflow summary

In summary, the migration process follows this workflow:

  1. Prepare appliances:

    Note: This step applies only for a multi-region Infrastructure Management environment. Remove all the subscriptions from the global region before you migrate the global database since the subscriptions must be dropped and re-created.

  2. Back up the database of your appliance.

  3. Migrate appliances:

    1. Provision new appliances to replace each appliance.

    2. On the database and nondatabase appliances, copy the GUID file from the version of the same appliance to the same location.

    3. On the database and nondatabase appliances, use the appliance console menu to stop the evmserverd service, manually remove the pre-configured database and key, and fetch the v2_key from an appliance.

    4. Copy the appliance database backup file and restore the database on each VMDB and dedicated database appliance.

    5. Load the new version of the pglogical library on the VMDB and dedicated database appliances.

      Note: This step applies only for a multi-region Infrastructure Management environment.

    6. Migrate the VMDB appliance databases and update the Automate Model.

    7. Restart evmserverd on the VMDB appliances.

  4. Configure replication after the migration process is complete and appliances are running again.

Backing up current appliances

These steps do not affect the operations of your appliance infrastructure. However, they help ensure that you are able to roll back if required and replicate the network settings.

  1. Find the size of the databases by using the following command. Then, ensure that your volume has enough space for the backup file.

    psql -d vmdb_production -c "SELECT pg_size_pretty(pg_database_size('vmdb_production'));"
    
  2. If it’s a multi-region environment, ensure you removed all subscriptions from the global database appliance.

  3. Stop the evmserverd process.

    systemctl stop evmserverd
    
  4. Back up the databases of your appliances in the root directory. Take a snapshot if possible.

    pg_dumpall -c --if-exists | gzip > vmdb.pg.gz
    

    Notes:

    • This command creates a compressed text-based backup. pg_dumpall is a utility for writing out (“dumping”) all PostgreSQL databases of a cluster into one script file. For more information, see pg_dumpall.
    • When you run a pg_dumpall, the pg_replication_slots is not copied over causing the slots to become inactive. Also, the size of the global database appliance is reduced allowing for quicker backup and restore.
    • Even a compressed database backup can be large so consider storing it on a file system with adequate space such as /var/tmp. Review your file system available space by using the df command.
    • You must also provide backup for configuration files such as /var/www/miq/vmdb/certs/v2_key, /var/www/miq/vmdb/config/database.yml, and /var/www/miq/vmdb/GUID.
  5. Back up the following files and take note of the appliance that each file comes from. These files are used to replace the old appliances in the database and for disaster recovery.

    • /var/www/miq/vmdb/GUID
  6. During the upgrade, the iptables configuration file /etc/sysconfig/iptables is removed. If you changed the iptables configuration from the default, run the iptables --list -n command to see the current configuration. Run the following command to back up the iptables configuration:

    iptables-save > /etc/iptables.conf
    

    You can restore your iptables configuration file with the following command:

    iptables-restore < /etc/iptables.conf
    

    You can also add this command to /etc/rc.local to reload the rules at every restart.

Steps to follow on newly deployed Infrastructure Management 4.6.1 Appliances

Complete the following steps on your appliances as specified to migrate to Infrastructure Management.

Note: Some steps are run on certain appliances. Ensure you wait for each command to finish before you move to the next step.

  1. Provision and start the new appliances.

  2. On the VMDB and non-VMDB appliances, run the following commands as the root user:

    1. Copy the GUID file from the appliance.

      scp root@$IP_of_old_appliance:/var/www/miq/vmdb/GUID /var/www/miq/vmdb/
      

      Notes:

      • Complete these steps on VMDB appliances first. The non-VMDB appliances don't have a database to connect to if they’re done first.
      • Complete this step only if you want to keep the same appliance/miq_servers records in the database and inherit that appliance’s existing settings. If you plan on deploying new appliances with new IP addresses and hostname and not inherit the settings, you might prefer creating new records by not copying the GUID.
    2. Stop the evmserverd process.

      systemctl stop evmserverd
      
    3. Remove the previously generated encryption key.

      rm /var/www/miq/vmdb/certs/v2_key
      
    4. Copy the default database configuration file, ovewriting the previously generated one that used the deleted encryption key.

      scp root@<IP_of_old_appliance>:/var/www/miq/vmdb/config/database.yml /var/www/miq/vmdb/config/database.yml
      
    5. Drop the previously generated database.

      dropdb vmdb_production
      
    6. For VMDB appliances, configure the application, fetch the key from remote system, and Reset Configured Database.

    7. For VMDB appliances, select the same region number and database password as the old installation. You can find the region number that is used for an installation that is listed as Database/Region: in the appliance_console summary information.

    8. For non-VMDB appliances, configure the application by using the join region in external database option. Note as part of the database configuration step, fetch the v2_key from the appliances.

    9. Allow the appliance to start.

  3. On the new VMDB and new dedicated database appliances:

    1. Stop the evmserverd process.

      systemctl stop evmserverd
      
    2. Drop the default database.

      dropdb vmdb_production
      
    3. Copy the database dump from the appliance. Update the following location to reflect the source file location of the database dump and copy it to a location with enough available space.

      scp root@$IP_of_old_appliance:/var/tmp/vmdb.pg.gz /var/tmp
      
    4. Decompress and restore the database dump backup file.

      gunzip < vmdb.pg.gz | psql postgres
      
    5. Change to the vmdb directory.

      cd /var/www/miq/vmdb/
      
    6. Run the command appropriate to your environment to migrate everything in the database to work with the latest configuration. If you run into an error, contact IBM Support.

      rake db:migrate
      
  4. Update the Automate Model to the latest version. This resets the ManageIQ and IBM domains (base domains) to a new and upgraded version. In a single (stand-alone database) or replicated environment, run this command on each VMDB appliance.

    rake evm:automate:reset
    
  5. Start the evmserverd process. In a single (stand-alone database) or replicated environment, run this command on each VMDB appliance:

    systemctl start evmserverd
    

The migration process is now complete. Once appliances are running again, you can configure database replication in your environment. For more information, see Configuring Database Replication and Centralized Administration in the General Configuration guide.