Upgrading to Infrastructure Management appliance

This topic describes the process of doing a full appliance replacement upgrade when there are new releases of IBM Cloud Pak® for Multicloud Management – Infrastructure Management.

Ordering migrations

You must perform all migrations of the databases before configuring replication and high availability. This is because you will deploy new appliances for migrating to IBM Cloud Pak® for Multicloud Management – Infrastructure Management.

Important: Do not run evmserverd if a database is in a partially migrated state.

Modifying the PostgreSQL configuration

This procedure modifies the strategy used for making changes to the PostgreSQL configuration. Previously, changes were made directly to the postgresql.conf file in the data directory. After switching to the new configuration, changes will be made by editing a separate file in the /etc/manageiq/postgresql.conf.d directory. This modification allows future PostgreSQL configuration changes to be made more easily.

Note: The 01_miq_overrides.conf file in the /etc/manageiq/postgresql.conf.d directory is reserved for application overrides and will be recreated on application updates; this file should not be edited. Additional files will be read by the PostgreSQL server process according to the documentation here: PostgreSQL - Managing Configuration File Contents

Back up the postgresql.conf file so that you have any configuration changes:
```
cp $APPLIANCE_PG_DATA/postgresql.conf <file_location>
```

Copy the updated postgresql.conf file to the pg_data directory:

cp $APPLIANCE_SOURCE_DIRECTORY/TEMPLATE/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf $APPLIANCE_PG_DATA/postgresql.conf

Optional: If you have made changes to the postgresql.conf file prior to following this procedure, create a new 02_user_overrides.conf file, and add any modifications you have from the original postgresql.conf file. Example layout of the 02_user_overrides.conf file:
```
#----------------------------------------------
# WRITE AHEAD LOG
#----------------------------------------------

wal_level = 'logical'
wal_log_hints = on
wal_buffers = 16MB
min_wal_size = 1GB
max_wal_size = 2GB
checkpoint_completion_target = 0.9
```
Once the 02_user_overrides.conf file has been modified and saved, restart the postgresql service:
```
systemctl restart $APPLIANCE_PG_SERVICE
```

All postgresql.conf customizations will now be made to files in the /etc/manageiq/postgresql.conf.d directory due to the following include_dir line in the default configuration. The settings in these new files will override ones in the postgresql.conf file.

include_dir = '/etc/manageiq/postgresql.conf.d`

Migrating to the Infrastructure Management appliance

Overview

This procedure describes the process of migrating to the Infrastructure management appliance. This procedure does not necessarily include migration of all possible customer modifications, so it is recommended that you fully test any modifications before migrating your environment.

Important:

Read through all of the steps in this procedure before beginning the migration process.
Infrastructure management appliances require 12 GB memory. However, before migrating your appliances, adjust resources in your environment to avoid performance issues.

You can classify the migration into three groups of appliances:

VMDB appliance - An appliance with workers running, which also hosts its own database that other appliances can also connect to.
Non-VMDB appliance - An appliance with workers running which does not host a database. The appliance is connected to an external database.
Dedicated database appliance - A CloudForms appliance or non-CloudForms virtual machine with no workers running on it: the appliance contains only a database for other appliances to connect to.

Migration workflow summary

In summary, the migration process follows this workflow:

Prepare appliances:

Note: This step only applies for a multi-region CloudForms environment. Remove all the subscriptions from the global region before migrating the global database since the subscriptions will have to be dropped and recreated.
Back up the database of your appliance.
Migrate appliances:
1. Provision new appliances to replace each appliance.
2. On the database and non-database appliances, copy the GUID file from the version of the same appliance to the same location.
3. On the database and non-database appliances, use the appliance console menu to stop the evmserverd service, manually remove the pre-configured database and key, and fetch the v2_key from an appliance.
4. Copy the appliance database backup file and restore the database on each VMDB and dedicated database appliance.
5. Load the new version of the pglogical library on the VMDB and dedicated database appliances.
  
  Note: This step only applies for a multi-region CloudForms environment.
6. Migrate the VMDB appliance databases and update the Automate Model.
7. Restart evmserverd on the VMDB appliances.
Configure replication after the migration process is complete and appliances are running once again.

Backing up current appliances

These steps will not affect the operations of your appliance infrastructure. However, they will help ensure that you are able to roll back if required and replicate the network settings.

Find the size of the databases by using the following command. Then, ensure that your volume has enough space for the backup file.
```
psql -d vmdb_production -c "SELECT pg_size_pretty(pg_database_size('vmdb_production'));"
```
If it’s a multi-region environment, ensure you have removed all subscriptions from the global database appliance.
Stop the evmserverd process.
```
systemctl stop evmserverd
```
Back up the databases of your appliances in the root directory. Take a snapshot if possible.
```
pg_dumpall -c --if-exists | gzip > vmdb-5.11.pg.gz
```
Notes:
- This will create a compressed text-based backup. pg_dumpall is a utility for writing out (“dumping”) all PostgreSQL databases of a cluster into one script file. For more information, see pg_dumpall.
- When doing a pg_dumpall the pg_replication_slots will not get copied over causing the slots to become inactive. This will also reduce the size of the global database appliance allowing for quicker backup and restore.
- Even a compressed database backup can be large so consider storing it on a filesystem with adequate space such as /var/tmp. Review your filesystem free space using the df command.
Back up the following files noting which appliance each comes from. This is to replace the old appliances in the database and for disaster recovery.
- /var/www/miq/vmdb/GUID
- /var/www/miq/vmdb/REGION
During the upgrade, the iptables configuration file /etc/sysconfig/iptables is removed. If you have changed the iptables configuration from the default, run the iptables --list -n command to see the current configuration. Run the following command to back up the iptables configuration:
```
iptables-save > /etc/iptables.conf
```
You can restore your iptables configuration file with the following command:
```
iptables-restore < /etc/iptables.conf
```
You can also add this command to /etc/rc.local to reload the rules at every reboot.

Steps to follow on newly deployed Infrastructure management 2.x Appliances

Perform the following steps on your appliances as specified to migrate to Infrastructure management.

Note: Some steps are run on certain appliances. Ensure you wait for each command to finish before going to the next step.

Provision and start the new appliances.
On the VMDB and non-VMDB appliances, run the following commands as the root user:
1. Copy the GUID file from the appliance.
```
scp root@$IP_of_5.11_appliance:/var/www/miq/vmdb/GUID /var/www/miq/vmdb/
```
  Notes:
  - Perform these steps on VMDB appliances first. The non-VMDB appliances won’t have a database to connect to if they’re done first.
  - Perform this step only if you want to keep the same appliance/miq_servers records in the database and inherit that appliance’s existing settings. If you plan on deploying new appliances with new IP addresses and host name and not inherit the settings, you may prefer creating new records by not copying the GUID.
2. Stop the evmserverd process.
```
systemctl stop evmserverd
```
3. Remove the previously generated encryption key.
```
rm /var/www/miq/vmdb/certs/v2_key
```
4. Copy the default database configuration file, ovewriting the previously generated one that used the just deleted encryption key.
```
cp /var/www/miq/vmdb/config/database.pg.yml /var/www/miq/vmdb/config/database.yml
```
5. Drop the previously generated database.
```
dropdb vmdb_production
```
6. For VMDB appliances, configure the application with the Reset Configured Database option and fetch the v2_key from the appliance.
7. For VMDB appliances, select the same region number and database password as the old installation. You can find the region number used for an installation listed as Database/Region: in the appliance_console summary information.
8. For non-VMDB appliances, configure the application using the join region in external database option. Note that as part of the database configuration step, fetch the v2_key from the appliances.
9. Allow the appliance to start.
On the new VMDB and new dedicated database appliances:
1. Stop the evmserverd process.
```
systemctl stop evmserverd
```
2. Drop the default database.
```
dropdb vmdb_production
```
3. Copy the database dump from the appliance. Update the location below to reflect the source file location of the database dump and copy it to a location with adequate free space.
```
scp root@$IP_of_5.11_appliance:/var/tmp/vmdb-5.11.pg.gz /var/tmp
```
4. Decompress and restore the database dump backup file.
```
gunzip < vmdb-5.11.pg.gz | psql postgres
```
5. Change to the vmdb directory.
```
cd /var/www/miq/vmdb/
```
6. Run the command appropriate to your environment to migrate everything in the database to work with the latest configuration. If you run into an error, contact IBM Support.
```
rake db:migrate
```
Update the Automate Model to the latest version. This resets the ManageIQ and IBM domains (base domains) to a new and upgraded version. Run the command appropriate to your environment to update the Automate Model:
1. In a single (standalone database) or replicated environment, run this command on each VMDB appliance.
```
rake evm:automate:reset
```
Start the evmserverd process.
1. In a single (standalone database) or replicated environment, run this command on each VMDB appliance:
```
systemctl start evmserverd
```

The migration process is now complete. Once appliances are running again, you can configure database replication in your environment. For more information, see Configuring Database Replication and Centralized Administration in the General Configuration guide.