Swift Object Storage for multi-node cluster deployment

Before the IBM Cloud Infrastructure Center multi-node cluster is installed, when you create the inventory with command icic-opsmgr inventory -c {cluster_name}, an installation option for Swift needs to be set:

Use the object storage provided by the IBM Cloud Infrastructure Center to store the images, image backups and volume backups? If 'n' is selected, it means you have external shared storage available and you must configure the IBM Cloud Infrastructure Center to store images, image backups and volume backups to your share storage after installation done. (y/n)

When you choose y, Swift is configured as the backend storage for Glance images and Cinder backup automatically, then Glance images and volume backup are stored to Swift. After the IBM Cloud Infrastructure Center multi-node cluster is installed, a 20G sparse file is created and it is mounted to Swift storage path /srv/node/partition1 through the loop device. You can check it with the following command. It uses the file system space, but the disk blocks are not allocated until some data is stored.

# df -h /srv/node/partition1
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       20G  6.0G   14G  30% /srv/node/partition1

If you have an external shared storage, for example: NFS or IBM Storage Scale, you can choose n, and Swift is not configured as the backend storage for Glance images and Cinder backup services. After the IBM Cloud Infrastructure Center multi-node cluster is installed, you can manually configure the storage path to your shared storage for Glance images and Cinder backup services. Refer to the guide of config the data directory of Glance Image and config the data directory of Cinder backup.

If you choose n for the Swift option, but you want to use Swift for Glance Images or Cinder backup after installation, you can refer to the guide of config Swift as Glance storage and config swift as Cinder backup storage to configure them.

1. Cannot upload image to other project.
2. Cannot delete the images that belong to other projects, even though the images are public.
3. Swift Object Storage is only supported in the IBM Cloud Infrastructure Center multi-node cluster.
4. When the Swift storage is configured as the loop device by the IBM Cloud Infrastructure Center, the back file is /srv/node/drives/swift.img, and it is a sparse file. The used space grows when data is stored to Swift. But it cannot automatically shrink. Even though you delete all the data that is stored in Swift, the space cannot be released.

If you want to use Logical Volume Manager (LVM) as Swift storage, you need to prepare the storage device before the installation. Refer to the guide about how to prepare storage devices for Swift.

Note: After you uninstall the IBM Cloud Infrastructure Center multi-node cluster, the data stored in LVM are not deleted. You can either mount the Logical Volume to a directory to manually delete the objects, or remove the Logical Volume directly.

With more and more data to be stored to Swift, the storage space might be insufficient. Then, you can extend its size. Before the extension, ensure the partition that /srv/nova/partition1 stands on has enough free space for the extension.

1. When you use LVM as Swift storage, you can extend the size of the Volume Group and Logical Volume, manually.
2. When you use a default way to configure the Swift store device provided by the IBM Cloud Infrastructure Center, Swift may need more space than 20G, you can follow the commands to extend the storage size. For example, you can increase another 10G on the Swift store device.

   # truncate -s +10G /srv/node/drives/swift.img
   # losetup -c /dev/loop0
   # xfs_growfs /dev/loop0
   

   1. For the command to extend the storage size, the loop device must be same as the one that is used in your environment. You can get the device path from the column Filesystem of the output of the command df -h /srv/node/partition1.
   2. When the available space of the file system is not enough, the commands to extend storage size can still succeed. But with more and more data to be stored to Swift, the file system might be used up and the write operation to Swift might fail.

When you use the Swift Object Storage with the default configuration for a while, some data is stored to Swift, then you want to move Swift storage from loop device to Logical Volume Manager, you can follow these steps:

1. Create a Logical Volume.
2. Mount the Logical Volume to a temporary directory, for example: /mnt/swift
3. Stop Swift services by command

   icic-services swift stop
   

4. Copy all the Swift data to the temporary directory with the command:

   cp -a /srv/node/partition1/* /mnt/swift
   

5. Unmount Swift storage path with the command:

   umount /srv/node/partition1
   

6. Mount the Logical Volume to /srv/node/partition1
7. Check the Swift storage path by command ll -Z /srv/node and ensure the permission and ownership of /srv/node/partition1 are correct:
   1. The permissions are 755. If it is not correct, change it with command:

      chmod -R 755 /srv/node/partition1
      

   2. The ownership is swift:swift. If it is not correct, change it with command:

      chown -R swift:swift /srv/node/partition1
      

   3. The SELinux context is system_u:object_r:swift_data_t:s0. If it is not correct, change it with command:

      restorecon -R /srv/node/partition1
      

8. Edit /etc/fstab, remove the line /srv/node/drives/swift.img /srv/node/partition1 xfs defaults 0 0 and add the mount path:

   /dev/{vg_name}/{lv_name} /srv/node/partition1 xfs defaults 0 0
   

9. Start Swift services by command:

   icic-services swift start
   

By default, after the IBM Cloud Infrastructure Center multi-node cluster is installed, the Swift services are enabled. If you do not plan to use the Swift Object Storage as the backend for Glance images and Cinder backup services, and Gnocchi services are not enabled, you can disable Swift to save the resources. Run this command on one management node to disable Swift services in the cluster:

icic-config swift manage --disable

When you want to enable all Swift services, run the following command on one management node:

icic-config swift manage --enable