Deploy VM from SCSI image failed with error: qemu-img: Unable to initialize gcrypt

Issue description

Deploy a virtual machine from SCSI image failed with the following error:

Deploy of virtual machine rhel82_ext4_scsi on host pok_152 failed with exception: Build of instance 88c69d9e-563e-4234-9b4c-147c1276c5d7 aborted: Volume 12d922c1-4286-4000-a232-0c62a29b894d did not finish being created even after we waited 24 seconds or 9 attempts. And its status is error.

Error log in /var/log/cinder/volume-xxx.log:

2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     raise value
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/retrying.py", line 200, in call
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1035, in _create_from_image
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     image_service)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 912, in _create_from_image_cache_or_download
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     data = image_utils.qemu_img_info(tmp_image)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 138, in qemu_img_info
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     prlimit=QEMU_IMG_LIMITS)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server Command: /usr/libexec/platform-python -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/conversion/tmp5r0h5wf8sv7016
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server Exit code: 1
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server Stdout: ''
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server Stderr: 'qemu-img: Unable to initialize gcrypt\n'
2021-02-24 23:13:32.708 2461 ERROR oslo_messaging.rpc.server

Explanation

The root cause is that the qemu-img package and libgcrypt package version are inconsistent. This is caused by the OS version and the configured repositories' latest version are inconsistent in the agent node where the storage provider is registered. For example, the OS version is RHEL 8.2 but the latest package version in your configured repository is RHEL 8.3. The qemu-img package is installed from RHEL repo as a dependency package when installing the IBM® Cloud Infrastructure Center and the libgcrypt package is a RHEL iso built-in package.

For example:

[root@cic112m ~]# rpm -qa | grep qemu
qemu-img-4.2.0-34.module+el8.3.0+7976+077be4ec.s390x   <-----RHEL 8.3
[root@cic112m ~]# rpm -qa | grep gcrypt
libgcrypt-1.8.3-4.el8.s390x                            <------RHEL 8.2

Resolution

Run dnf update on your agent node of storage provider and then make sure these two packages versions are consistent.

Note: The dnf update updates the host os version.