How to simulate the impact of a Compute Node failure on running VMs in PureApplication?
Hendrik van Run 310001AHXF Comments (2) Visits (5370)
When designing and building patterns on PureApplication, most clients need to demonstrate the impact of hardware failure of a Compute Node on VMs running software. For example, what happens when the Compute Node hosting the primary DB2 VM fails? Deploying a pattern instance and pulling out a Compute Node from a PureApplication System in the data center is usually not an option (as suggested in this
The issue with (1) is that this is very specific to the actual software. When performing (2), the OS actually performs a graceful shutdown (so again not realistic). Option (3) looks like the best option, however we found that the command to "Power off" a VM actually send a signal to the OS to perform a graceful shutdown as well.
We will describe a mechanism here to perform a more "realistic" and abrupt "Power off" of the VM here. We can achieve this by making sure that this signal that is sent to the VM simply does not arrive or does not perform a graceful shutdown. Under the covers, VMWare Tools is installed on every VM in PureApplication (on Intel). VMWare Tools is running a set of services within the OS of the VM, which can be used to receive the call from VMWare to perform a graceful shutdown of the OS. In order to prevent this for testing purposes, we can simply stop the VMWare Tools services:
Note: Please keep in mind that performing this simulation of an abrupt shutdown of a VM may lead to data corruption of the filesystem(s) within the VM!
With the VMWare Tools services stopped, issuing a "Power off" from the PureApplication UI for the VM will effectively be an immediate shutdown. You can validate this by examining the file messages in /var/log after the "Power off", it should show a message that recovery is required on a readonly filesystem:
Jan 5 08:11:29 pure-9-3-172-232 kernel: dracut: Scanning devices sda2 for LVM logical volumes vg_root/LogVol00 vg_root/LogVol01 Jan 5 08:11:29 pure-9-3-172-232 kernel: dracut: inactive '/de
Another approach here would be to simply uninstall the plugin from VMWare Tools that facilitates the call from VMWare to the OS for a graceful shutdown. The name of this module is vmwa
yum uninstall vmwa
The advantage of this approach is that you will have the remaining VMWare Tools modules still in place, which in general optimise performance of the OS running inside the VM. Should you wish to re-install the module vmwa
Note: You can determine the version of ESX by performing a REST GET call to http
software_version": "VMware ESXi 6.0.0 build-3620759",
You can find more information about the VMWare Tools Operating System Specific Packages here.