Introduction of icic-migrate command tool
The icic-migrate command is used to migrate the agent service of a registered storage provider from a source node to a target node. The target node is one compute node.
Location
The command can be found under /usr/bin/ of the management node.
/usr/bin/icic-migrate
Parameters
-h, --help (Optional) Show the help message and exit.
-i, --ignorelog (Optional) Specifies the silence of the log.
-t TARGET, --target TARGET (Required) Specifies the target node to which the service will be migrated. The node refers to one compute node.
- Run the 'icic-services remote list' command from the management node, and use the value listed in the 'Name' column as the input for this argument.
# example
[root@112-mgt-wdl ~]# icic-services remote list
Running on remote nodes.
Name Display Name
==== ============
BOEM5403 113-cmp-wdl-m5403
BOEM5401 113-cmp-wdl-m5401
-f, --forceRemove Specify this option when the source storage agent node is down. The storage agent service related files are not removed from the source agent node.
--storageHost STORAGEHOST (Required) Specifies the hostname of the storage to be migrated. To determine valid storage providers, follow two steps beneath:
- run the 'source /opt/ibm/icic/icicrc' command, specify the username and the password of an administrator of IBM® Cloud Infrastructure Center when prompted.
- run the 'openstack volume service list' command, use the value listed in the 'Host' column of cinder-volume as input for this argument.
# example
[root@112-mgt-wdl ~]# openstack volume service list | egrep 'Host|cinder-volume|\-\-'
+------------------+-------------+---------------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+-------------+---------------+---------+-------+----------------------------+
| cinder-volume | v7k60 | Default Group | enabled | up | 2021-02-17T13:43:00.000000 |
| cinder-volume | ds8k60c2 | AZ_a | enabled | up | 2021-02-17T13:43:00.000000 |
+------------------+-------------+---------------+---------+-------+----------------------------+
Prerequisites
During the migration, the management node copies some relevant configuration files to the source and the target nodes via the Secure Shell(SSH) protocol. Hence, the management node must have TCP/IP connections to the source and the target nodes.
If the source storage agent node is down, you need to specify
-f or --forceRemove to force migrate the
storage provider.
Ensure the selected target node has the Fibre Channel connection to the storage provider. The target node and storage provider can be in the same or different Availability Zone.
If posix is used as the enabled backup service type
for the storage provider, you must ensure the backup folder that is
assigned as the backup_posix_path exists on the target
node. Create the folder if not exist.
- Run below commmand on the management node to get the enabled
backup service type of one storage provider and backup folder
information.
icic-config storage backup-service show - If the
backup_posix_pathis on a local storage or on a shared storage but does not exist on the target node, besides creating the folder on the target node, you must copy the volume backup files and image backup files from the source node to the target node. - Ensure that the cinder user is allowed to access the
backup_posix_pathon the target node, use the following command to grant the access if needed.
where:chown -R cinder:cinder <backup_posix_path>-
<backup_posix_path>is the backup folder whenposixis used as the enabled backup service type and to get its value use the commandicic-config storage backup-service show.
-
Notes:
- Before using the tool, you must ensure that the storage provider related operations are completed. When the migration is being executed, the storage provider is not available to be used. Any storage related operation (for example: editing storage provider, updating volume, image export/import) should not be performed before the migration is completed successfully.
- The command can only be run from the management node by any user with sudo permission.
- If the storage provider is connected to a fabric, after migration, the storage provider is still connected to the fabric. The allowlist and blocklist configured before the migration are also migrated along with the storage provider.
Common usage scenarios
-
Before removing a compute node where a storage provider has been registered, the storage provider must be either deregistered from IBM® Cloud Infrastructure Center or migrated from this compute node to another node. If deregistration is not preferred, then use icic-migrate to perform the migration.
-
Bringing down a compute node where a storage provider has been registered, causes the IBM® Cloud Infrastructure Center to be unable to use the storage provider. If the outage of the storage provider is not preferred, then use icic-migrate forceRemove to perform the migration.
Migration cases
| No. | source node | target node |
|---|---|---|
| case 1 | compute node | compute node |
Stages Introduction
There are five stages during the migration of the agent service of a registered storage provider.
stage 1: Pre-check.
stage 2: Copy relevant configuration files to target node.
stage 3: Create the agent service of the storage provider on the target node.
stage 4: Remove the agent service of the storage provider from
the source node.(This step is skipped by specifiying
-f or --forceRemove.)
stage 5: Start the agent service of the storage provider on the target node.
stage 6: (Multinode cluster deployment environment only) Synchronize the storage provider configuration files between management nodes.
Rollback
To rollback a migration from one node(node_A) to another node(node_B), migrate back from node_B to node_A.
Once the stage of Pre-check is passed, if the migration is interrupted or aborted in the later stages(such as due to the network connection dropped or a manual abortion by Ctrl+C), it leaves the agent service in an error state.
To recover the agent service, the migration with the same parameters must be performed again after fixing the problems that interrupt the previous migration.
Invocation examples
An example of migrating from one compute node(BOEM5401) to another compute node(BOEM5403) in standalone environment
# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-18-233233.log
------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node BOEM5401 found in database
Pass: source_node BOEM5401 found in conf /etc/cinder/cinder-v7k60.conf
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak created.
------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: /etc/cinder/fabrics.conf transfered to target_node
Pass: /etc/cinder/rsakeys/cinder_v7k60_rsa transfered to target_node
Pass: temp conf /tmp/icic_temp_gwolpiqg.conf created on management node
Pass: temp conf /tmp/icic_temp_gwolpiqg.conf copied to target_node
------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 created on target_node
Pass: service openstack-cinder-backup-v7k60 created on target_node
------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-v7k60 removed on source_node
Pass: service openstack-cinder-backup-v7k60 removed on source_node
Pass: set node_name as BOEM5403 in database
------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 started on target_node
Pass: service openstack-cinder-backup-v7k60 started on target_node
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak deleted.
------ Completed ! -------
An example of migrating from one compute node(BOEM5407) to another compute node(BOEM5403) in multinode cluster deployment environment
/opt/ibm/icic/bin/icic-migrate --storageHost ds8k63c1 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2023-06-02-032312.log
------ Pre_check -------
Pass: /etc/cinder/cinder-ds8k63c1.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node BOEM5407 found in database
Pass: source_node BOEM5407 found in conf /etc/cinder/cinder-ds8k63c1.conf
Pass: backup_conf /etc/cinder/cinder-ds8k63c1.conf_migr_bak created.
------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: temp conf /tmp/icic_temp_28iqhtp6.conf created on management node
Pass: /tmp/icic_temp_28iqhtp6.conf transfered to target_node
Pass: /opt/ibm/icic/data/certs/ds8k63c1_boeblingen_de_ibm_com.crt transfered to target_node
Pass: temp conf /tmp/icic_temp_fc_qegr5.conf created on management node
Pass: temp conf /tmp/icic_temp_fc_qegr5.conf copied to target_node
------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-ds8k63c1 created on target_node
Pass: service openstack-cinder-backup-ds8k63c1 created on target_node
------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-ds8k63c1 removed on source_node
Pass: service openstack-cinder-backup-ds8k63c1 removed on source_node
Pass: set node_name as BOEM5403 in database
------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-ds8k63c1 started on target_node
Pass: service openstack-cinder-backup-ds8k63c1 started on target_node
Pass: backup_conf /etc/cinder/cinder-ds8k63c1.conf_migr_bak deleted.
------ sync_storage_conf_on_management_nodes -------
Pass: storage conf /etc/cinder/cinder-ds8k63c1.conf synchronized on management nodes
------ Completed ! -------
An example of pre-check failure
# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5401
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-084119.log
------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5401 is valid.
Pass: source_node BOEM5401 found in database
Pass: source_node BOEM5401 found in conf /etc/cinder/cinder-v7k60.conf
Migration abort due to pre-check fails.
Cause: Source node and target node are the same.
An example of aborting migration after passing the stage
# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5403
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-090705.log
------ Pre_check -------
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 is valid.
Pass: source_node management found in database
Pass: source_node management found in conf /etc/cinder/cinder-v7k60.conf
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak created.
------ sftp_cinder_files_to_target_node -------
^C
Traceback (most recent call last):
...
KeyboardInterrupt
As stated in the previous section of Rollback, to recover the agent service, the migration with the same parameters must be performed again after fixing the problems that interrupt the previous migration.
# sudo /opt/ibm/icic/bin/icic-migrate --storageHost v7k60 -t BOEM5401
LOG file: /opt/ibm/icic/log/icic_migrate_2021-02-17-090818.log
------ Pre_check -------
Warning: Last migration failure is detected!
Retry now!
Pass: /etc/cinder/cinder-v7k60.conf found on management node
Pass: target_node BOEM5403 found in backup_conf
Pass: target_node BOEM5403 is valid.
Pass: source_node management found in backup_conf
------ sftp_cinder_files_to_target_node -------
Pass: /opt/ibm/icic/data/known_hosts transfered to target_node
Pass: /etc/cinder/fabrics.conf transfered to target_node
Pass: /etc/cinder/rsakeys/cinder_v7k60_rsa transfered to target_node
Pass: temp conf /tmp/icic_temp_4i5ag6d1.conf created on management node
Pass: temp conf /tmp/icic_temp_4i5ag6d1.conf copied to target_node
------ create_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 created on target_node
Pass: service openstack-cinder-backup-v7k60 created on target_node
------ remove_service_on_source_node -------
Pass: service openstack-cinder-volume-v7k60 removed on source_node
Pass: service openstack-cinder-backup-v7k60 removed on source_node
Pass: set node_name as BOEM5403 in database
------ start_service_on_target_node -------
Pass: service openstack-cinder-volume-v7k60 started on target_node
Pass: service openstack-cinder-backup-v7k60 started on target_node
Pass: backup_conf /etc/cinder/cinder-v7k60.conf_migr_bak deleted.
------ Completed ! -------