Location

The command can be found under /usr/bin/ of the management node.

/usr/bin/icic-migrate

Parameters

-h, --help (Optional) Show the help message and exit.

-i, --ignorelog (Optional) Specifies the silence of the log.

-t TARGET, --target TARGET (Required) Specifies the target node to which the service will be migrated. The node refers to one compute node.

  • Run the 'icic-services remote list' command from the management node, and use the value listed in the 'Name' column as the input for this argument.

# example
[root@112-mgt-wdl ~]# icic-services remote list
Running on remote nodes.
Name                   Display Name
====                   ============
BOEM5403               113-cmp-wdl-m5403
BOEM5401               113-cmp-wdl-m5401

-f, --forceRemove Specify this option when the source storage agent node is down. The storage agent service related files are not removed from the source agent node.

--storageHost STORAGEHOST (Required) Specifies the hostname of the storage to be migrated. To determine valid storage providers, follow two steps beneath:

  1. run the 'source /opt/ibm/icic/icicrc' command, specify the username and the password of an administrator of IBM® Cloud Infrastructure Center when prompted.

  2. run the 'openstack volume service list' command, use the value listed in the 'Host' column of cinder-volume as input for this argument.

# example
[root@112-mgt-wdl ~]# openstack volume service list | egrep 'Host|cinder-volume|\-\-'
+------------------+-------------+---------------+---------+-------+----------------------------+
| Binary           | Host        | Zone          | Status  | State | Updated At                 |
+------------------+-------------+---------------+---------+-------+----------------------------+
| cinder-volume    | v7k60       | Default Group | enabled | up    | 2021-02-17T13:43:00.000000 |
| cinder-volume    | ds8k60c2    | AZ_a          | enabled | up    | 2021-02-17T13:43:00.000000 |
+------------------+-------------+---------------+---------+-------+----------------------------+

Prerequisites

During the migration, the management node copies some relevant configuration files to the source and the target nodes via the Secure Shell(SSH) protocol. Hence, the management node must have TCP/IP connections to the source and the target nodes.

If the source storage agent node is down, you need to specify -f or --forceRemove to force migrate the storage provider.

Ensure the selected target node has the FC connections to the storage provider and the target node is in the same availability zone as the storage provider.

If posix is used as the enabled backup service type for the storage provider, you must ensure the backup folder that is assigned as the backup_posix_path exists on the target node. Create the folder if not exist.

  • Run below commmand on the management node to get the enabled backup service type of one storage provider and backup folder information.

    icic-config storage backup-service show
  • If the backup_posix_path is on a local storage or on a shared storage but does not exist on the target node, besides creating the folder on the target node, you must copy the volume backup files and image backup files from the source node to the target node.

  • Ensure that the cinder user is allowed to access the backup_posix_path on the target node, use the following command to grant the access if needed.

    chown -R cinder:cinder <backup_posix_path>

    where:

    • <backup_posix_path> is the backup folder when posix is used as the enabled backup service type and to get its value use the command icic-config storage backup-service show.

Notes:

  • Before using the tool, you must ensure that the storage provider related operations are completed. When the migration is being executed, the storage provider is not available to be used. Any storage related operation (for example: editing storage provider, updating volume, image export/import) should not be performed before the migration is completed successfully.

  • The command can only be run from the management node by any user with sudo permission.

  • If the storage provider is connected to a fabric, after migration, the storage provider is still connected to the fabric. The allowlist and blocklist configured before the migration are also migrated along with the storage provider.

Common usage scenarios

  1. Before removing a compute node where a storage provider has been registered, the storage provider must be either deregistered from IBM® Cloud Infrastructure Center or migrated from this compute node to another node. If deregistration is not preferred, then use icic-migrate to perform the migration.

  2. Bringing down a compute node where a storage provider has been registered, causes the IBM® Cloud Infrastructure Center to be unable to use the storage provider. If the outage of the storage provider is not preferred, then use icic-migrate forceRemove to perform the migration.

Migration cases

No. source node target node
case 1 compute node compute node

Stages Introduction

There are five stages during the migration of the agent service of a registered storage provider.

stage 1: Pre-check.

stage 2: Copy relevant configuration files to target node.

stage 3: Create the agent service of the storage provider on the target node.

stage 4: Remove the agent service of the storage provider from the source node.(This step is skipped by specifiying -f or --forceRemove.)

stage 5: Start the agent service of the storage provider on the target node.

stage 6: (Multinode cluster deployment environment only) Synchronize the storage provider configuration files between management nodes.

Rollback

To rollback a migration from one node(node_A) to another node(node_B), migrate back from node_B to node_A.

Once the stage of Pre-check is passed, if the migration is interrupted or aborted in the later stages(such as due to the network connection dropped or a manual abortion by Ctrl+C), it leaves the agent service in an error state.

To recover the agent service, the migration with the same parameters must be performed again after fixing the problems that interrupt the previous migration.

Invocation examples

An example of migrating from one compute node(BOEM5401) to another compute node(BOEM5403) in standalone environment

# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-18-233233.log

------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node BOEM5401 found in database
Pass: source_node BOEM5401 found in conf /etc/cinder/cinder-v7k60.conf
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak created.

------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: /etc/cinder/fabrics.conf transfered to target_node
Pass: /etc/cinder/rsakeys/cinder_v7k60_rsa transfered to target_node
Pass: temp conf /tmp/icic_temp_gwolpiqg.conf created on management node
Pass: temp conf /tmp/icic_temp_gwolpiqg.conf copied to target_node

------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 created on target_node
Pass: service openstack-cinder-backup-v7k60 created on target_node

------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-v7k60 removed on source_node
Pass: service openstack-cinder-backup-v7k60 removed on source_node
Pass: set node_name as BOEM5403 in database

------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 started on target_node
Pass: service openstack-cinder-backup-v7k60 started on target_node
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak deleted.

------ Completed ! -------

An example of migrating from one compute node(BOEM5407) to another compute node(BOEM5403) in multinode cluster deployment environment

/opt/ibm/icic/bin/icic-migrate --storageHost ds8k63c1 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2023-06-02-032312.log

------ Pre_check -------
Pass: /etc/cinder/cinder-ds8k63c1.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node BOEM5407 found in database
Pass: source_node BOEM5407 found in conf /etc/cinder/cinder-ds8k63c1.conf
Pass: backup_conf /etc/cinder/cinder-ds8k63c1.conf_migr_bak created.

------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: temp conf /tmp/icic_temp_28iqhtp6.conf created on management node
Pass: /tmp/icic_temp_28iqhtp6.conf transfered to target_node
Pass: /opt/ibm/icic/data/certs/ds8k63c1_boeblingen_de_ibm_com.crt transfered to target_node
Pass: temp conf /tmp/icic_temp_fc_qegr5.conf created on management node
Pass: temp conf /tmp/icic_temp_fc_qegr5.conf copied to target_node

------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-ds8k63c1 created on target_node
Pass: service openstack-cinder-backup-ds8k63c1 created on target_node

------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-ds8k63c1 removed on source_node
Pass: service openstack-cinder-backup-ds8k63c1 removed on source_node
Pass: set node_name as BOEM5403 in database

------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-ds8k63c1 started on target_node
Pass: service openstack-cinder-backup-ds8k63c1 started on target_node
Pass: backup_conf /etc/cinder/cinder-ds8k63c1.conf_migr_bak deleted.

------ sync_storage_conf_on_management_nodes -------
Pass: storage conf /etc/cinder/cinder-ds8k63c1.conf synchronized on management nodes

------ Completed ! -------

An example of pre-check failure

# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5401
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-084119.log

------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5401 is valid.
Pass: source_node BOEM5401 found in database
Pass: source_node BOEM5401 found in conf /etc/cinder/cinder-v7k60.conf
Migration abort due to pre-check fails.
Cause: Source node and target node are the same.

An example of aborting migration after passing the Pre-check stage

# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-090705.log

------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node management found in database
Pass: source_node management found in conf /etc/cinder/cinder-v7k60.conf
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak created.

------ sftp_cinder_files_to_target_node -------
^C
Traceback (most recent call last):
...
KeyboardInterrupt

As stated in the previous section of Rollback, to recover the agent service, the migration with the same parameters must be performed again after fixing the problems that interrupt the previous migration.

# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5401
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-090818.log

------ Pre_check -------
Warning: Last migration failure is detected!
         Retry now!
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 found in backup_conf
Pass: target_node BOEM5403 is valid.
Pass: source_node management found in backup_conf

------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: /etc/cinder/fabrics.conf transfered to target_node
Pass: /etc/cinder/rsakeys/cinder_v7k60_rsa transfered to target_node
Pass: temp conf /tmp/icic_temp_4i5ag6d1.conf created on management node
Pass: temp conf /tmp/icic_temp_4i5ag6d1.conf copied to target_node

------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 created on target_node
Pass: service openstack-cinder-backup-v7k60 created on target_node

------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-v7k60 removed on source_node
Pass: service openstack-cinder-backup-v7k60 removed on source_node
Pass: set node_name as BOEM5403 in database

------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 started on target_node
Pass: service openstack-cinder-backup-v7k60 started on target_node
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak deleted.

------ Completed ! -------