Security Bulletin: Publicly disclosed vulnerabilities from Kernel affect IBM Netezza Host Management

Workarounds and Mitigations

Mitigation of the reported CVEs : CVE-2020-12655, CVE-2020-12652, blocklisting kernel module xfs and mptctl to prevent them from loading automatically on PureData System for Analytics N200x and N3001 is as follows:

  1. Change to user nz:
       [root@nzhost1 ~]# su – nz

  2. Check to see if Call Home is enabled:
      [nz@nzhost1 ~]$ nzcallhome -status
      If enabled, disable it:
      [nz@nzhost1 ~]$ nzcallhome –off
Note: Ensure that nzcallhome returns status as disabled. If there are errors in the callHome.txt configuration file, errors are listed in the output, and call-Home is disabled.

  3. Check the state of the Netezza system:
      [nz@nzhost1 ~]$ nzstate

  4. If the system state is online, stop the system using the command:
      [nz@nzhost1 ~]$ nzstop

  5. Wait for the system to stop, using the command:
      [nz@nzhos1t ~]$ nzstate

      System state is 'Stopped'.

  6. Exit from the nz session to return to user root:
      [nz@nzhost1 ~]$ exit

  7. Logged into the active host as root, type the following commands to stop the heartbeat processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service heartbeat stop
      [root@nzhost1 ~]# /sbin/service heartbeat stop

  8. Run below commands as a root user to disable heartbeat from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig heartbeat off
      [root@nzhost1 ~]# /sbin/chkconfig heartbeat off

  9. Type the following commands to stop the DRBD processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service drbd stop
      [root@nzhost1 ~]# /sbin/service drbd stop

  10. Run below commands as a root user to disable drbd from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig drbd off
      [root@nzhost1 ~]# /sbin/chkconfig drbd off

Execute below steps using "root" user on both ha1/ha2 hosts

Step 1: Check if xfs and mptctl are loaded in the hosts

lsmod | grep xfs
lsmod | grep mptctl

example:
[root@nzhost1 ~]# lsmod | grep xfs
xfs 1137495 0
exportfs 4300 2 xfs,nfsd
[root@nzhost1 ~]# lsmod | grep mptctl
mptctl 31785 0
mptbase 93807 1 mptctl

Note: If there is no output skip Step 2, and proceed with Step 3

Step 2: Unload xfs and mptctl module

modprobe -rv xfs
modprobe -rv mptctl

example:
[root@nzhost1 ~]# modprobe -rv xfs
rmmod /lib/modules/2.6.32-754.31.1.el6.x86_64/kernel/fs/xfs/xfs.ko
[root@nzhost1 ~]# modprobe -rv mptctl
rmmod /lib/modules/2.6.32-754.31.1.el6.x86_64/kernel/drivers/message/fusion/mptctl.ko
rmmod /lib/modules/2.6.32-754.31.1.el6.x86_64/kernel/drivers/message/fusion/mptbase.ko

The output shows that xfs and mptctl, their dependent modules are unloaded in the reverse order that they are loaded, given that no processes depend on any of the modules being unloaded.

Step 3: To prevent a module from being loaded directly you add the blocklist line to a configuration file specific to the system configuration.

echo "blocklist xfs" >> /etc/modprobe.d/local-blocklist.conf
echo "blocklist mptctl" >> /etc/modprobe.d/local-blocklist.conf

example :
[root@nzhost1 ~]# echo "blocklist xfs" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "blocklist mptctl" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep xfs
blocklist xfs
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep mptctl
blocklist mptctl

Step 4: Kernel modules can be loaded directly or loaded as a dependency from another module
To prevent installation as a dependency from another module follow below step:

echo "install xfs /bin/false" >> /etc/modprobe.d/local-blocklist.conf
echo "install mptctl /bin/false" >> /etc/modprobe.d/local-blocklist.conf

example:
[root@nzhost1 ~]# echo "install xfs /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "install mptctl /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep xfs
blocklist xfs
install xfs /bin/false
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep mptctl
blocklist mptctl
install mptctl /bin/false

The install line simply causes /bin/false to be run instead of installing a module.

Step 5: Make a backup copy of your initramfs.

cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak

Example:
[root@nzhost1 ~]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
[root@nzhost1 ~]# uname -r
2.6.32-754.31.1.el6.x86_64
[root@nzhost1 ~]# ll /boot/initramfs-2.6.32-754.31.1.el6.x86_64.img.08-17-105347.bak
-rw------- 1 root root 21881438 Aug 17 10:53 /boot/initramfs-2.6.32-754.31.1.el6.x86_64.img.08-17-105347.bak

Step 6: If the kernel module is part of the initramfs (boot configuration), rebuild your initial ramdisk image, omitting the module to be avoided

dracut --omit-drivers xfs -f
dracut --omit-drivers mptctl -f

example:
[root@nzhost1 ~]# dracut --omit-drivers xfs -f
[root@nzhost1 ~]# dracut --omit-drivers mptctl -f
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.31.1.el6.x86_64.img | grep xfs
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.31.1.el6.x86_64.img | grep mptctl

Step 7: Append module_name.blocklist to the kernel cmdline. We give it an invalid parameter of blocklist and set it to 1 as a way to preclude the kernel from loading it.

sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ xfs.blocklist=1/' /etc/grub.conf
sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ mptctl.blocklist=1/' /etc/grub.conf

example :
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ xfs.blocklist=1/' /etc/grub.conf
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ mptctl.blocklist=1/' /etc/grub.conf

Step 8: blocklist the kernel module in kdump's configuration file.

echo "blocklist xfs" >> /etc/kdump.conf
echo "blocklist mptctl" >> /etc/kdump.conf

example:
[root@nzhost1 ~]# echo "blocklist xfs" >> /etc/kdump.conf
[root@nzhost1 ~]# echo "blocklist mptctl" >> /etc/kdump.conf
[root@nzhost1 ~]# cat /etc/kdump.conf | grep xfs
blocklist xfs
[root@nzhost1 ~]# cat /etc/kdump.conf | grep mptctl
blocklist mptctl

Note: Perform Step 9 if kexec-tools is installed and kdump is configured else continue with Step 10.
Perform below commands to check if kexec-tools is installed and Kdump is operational
[root@nzhost1 ~]# rpm -qa | grep kexec-tools
[root@nzhost1 ~]# service kdump status

Step 9: Restart the kdump service to pick up the changes to kdump's initrd.

service kdump restart

example:
[root@nzhost1 ~]# service kdump restart
Stopping kdump: [ OK ]
Starting kdump: [ OK ]

Step 10: Reboot the system at a convenient time to have the changes take effect.
Make sure the secondary host is up by pinging or logging in before rebooting the primary host.

/sbin/shutdown -r now

example:
[root@nzhost1 ~]# /sbin/shutdown -r now
Make sure the primary server comes up and is reachable before performing Mitigation steps on the secondary server.

After applying the mitigation:

  1. Start the services using following:
      [root@nzhost1 ~]# service heartbeat start
      [root@nzhost1 ~]# ssh ha2 service heartbeat start
      [root@nzhost1 ~]# service drbd start
      [root@nzhost1 ~]# ssh ha2 service drbd start

  2. Check the stat of the system. Type:
      [root@nzhost1 ~]# crm_mon -i5

      Result: When the cluster manager comes up and is ready, status appears as follows.
      Make sure that nzinit has started before you proceed. (This could take a few minutes.)
      Node: nps61074 (e890696b-ab7b-42c0-9e91-4c1cdacbe3f9): online
      Node: nps61068 (72043b2e-9217-4666-be6f-79923aef2958): online
      Resource Group: nps
      drbd_exphome_device(heartbeat:drbddisk): Started nps61074
      drbd_nz_device(heartbeat:drbddisk): Started nps61074
      exphome_filesystem(heartbeat::ocf:Filesystem): Started nps61074
      nz_filesystem (heartbeat::ocf:Filesystem): Started nps61074
      fabric_ip (heartbeat::ocf:IPaddr): Started nps61074
      wall_ip (heartbeat::ocf:IPaddr): Started nps61074
      nzinit (lsb:nzinit): Started nps61074
      fencing_route_to_ha1(stonith:apcmaster): Started nps61074
      fencing_route_to_ha2(stonith:apcmaster): Started nps61068

  3. From host 1 (ha1), press Ctrl+C to break out of crm_mon.

  4. Turn on heartbeat and DRBD using the chkconfig:
      ssh ha2 /sbin/chkconfig drbd on
      /sbin/chkconfig drbd on
      ssh ha2 /sbin/chkconfig heartbeat on
      /sbin/chkconfig heartbeat on