Achieving data consistency for virtual machine workloads

IBM Storage Fusion backs up the data of the Red Hat® OpenShift® Virtualization virtual machines consistently.

During virtual machine backups, consider the following options to provide data consistency within the virtual machine.

  • Crash consistency- the backup includes data that is written to a disk. It is the equivalent to a virtual machine crash before a backup. To achieve this consistency, take a snapshot of the virtual machine PVCs without any additional orchestrations or hooks within the virtual machine.
  • File system consistency- the file system freezes, writes, and flushes to the disk before a backup. The backup does not capture any data in the application buffers that is not yet written to the file system. Achieve this data consistency by using tools inside the virtual machine, which interacts with the file system to freeze write before a snapshot (backup) operation.
  • Application consistency- data in memory buffers is written to disk before the application is backed up. It ensures that all application data is included in the backup. You can achieve this consistency by tools inside the virtual machine, which interacts with an application to take an application-specific action such as locking a database.

File system consistency

The following example diagram shows a virtual machine that runs within a namespace. Each virtual machine has a virt-launcher pod that runs the virtual machine instance. The virt-launcher pod runs an instance of libvirt, which is used to manage the virtual machine process.

When a namespace backup is requested, a virtual machine snapshot operation gets run. The virt-launcher pod includes Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations to run the kubevirt tool virt-freezer. It communicates to a file system driver inside the Linux® or Microsoft® Windows virtual machine guest to freeze and unfreeze i/o operations around the virtual machine snapshot.

Configure and validate file system consistency for Linux virtual machines:
  1. Ensure that virt-launcher pod has the appropriate Velero annotations:
    pre.hook.backup.velero.io/command:
              '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse",
              "--namespace", "test-vm-ns"]'
    post.hook.backup.velero.io/command:
              '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse",
              "--namespace", "test-vm-ns"]
  2. Ensure that the QEMU guest agent (qemu-guest-agent) is installed in the Linux virtual machine. See OpenShift documentation.
  3. Add the default service account security context constraint kubevirt-controller:
    oc adm policy add-scc-to-user
              kubevirt-controller -z default -n <namespace>
  4. Disable SELinux inside the virtual machine.
Configure and validate file system consistency for Microsoft® Windows virtual machines:
  1. Ensure that virt-launcher pod has the appropriate Velero annotations:
    pre.hook.backup.velero.io/command:
              '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse",
              "--namespace", "test-vm-ns"]'
    post.hook.backup.velero.io/command:
              '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse",
              "--namespace", "test-vm-ns"]

  2. Ensure that the QEMU guest agent (qemu-ga-x86_64) is installed in the Microsoft Windows virtual machine and that the “QEMU Guest Agent” network service is running. See OpenShift documentation.

  3. Ensure that the Volume Shadow Copy service is running in the Microsoft Windows virtual machine.
  4. Add the default service account security context constraint “kubevirt-controller“:
    oc adm policy
              add-scc-to-user kubevirt-controller -z default -n
          <namespace>

Crash consistency

You can achieve crash consistency by taking a snapshot of the virtual machine without engaging the QEMU guest agent to do a file system freeze. It an be done in one of following two ways:
Note: QEMU guest agent is a helper daemon. It is installed in the guest to exchange information between the host and guest, and to run command in the guest.
  1. Remove the Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations from the virt-launcher pod. This change does not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created along with a virtual machine and must be removed for each new instantiation of the virt-launcher pod.
  2. Ensure that the QEMU guest agents are not installed or running in a virtual machine guest.

Application consistency

If you have a specific application consistency requirement that cannot be met by using the file system consistency, you can create a backup workflow that communicates directly with the application by using the virtual shell tool (virsh) instead of the virt-freezer tool. The following diagram depicts the scenario:
Consider the following alternative approaches run an application command by using virsh:
  • Velero annotations to run an application command by using virsh:
    • You can directly modify the Velero annotations on the virt-launcher pod to run virsh (/usr/bin/virsh) instead of virt-freezer (/usr/bin/virt-freezer).
    • The modifications to the Velero annotations do not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created along with the virtual machine and must be modified for each new instantiation of the virt-launcher pod.
  • Using IBM Storage Fusion backup and restore recipes to run an application command by using virsh:
    Alternatively, you can create an IBM Storage Fusion backup and restore recipe to run an application command by using virsh. Even though no differences exist in the outcome as compared to virsh by using the Velero annotations, consider the following points:
    • There might be instances wherein you must run more than one command to produce the wanted results. For example, issuing a command that locks a database for write and a second command to monitor or validate the lock. In these cases, a IBM Storage Fusion recipe might be easier to work with.
    • If you decide to use an IBM Storage Fusion recipe, the Velero annotations are also acceptable but it might produce undesirable results. For this reason, consider the removal of Velero annotations from the virt-launcher pod.
    • Use the recipe exec hooks to run virsh in the wanted sequence.