Security Bulletin: Publicly disclosed vulnerabilities from Kernel affect IBM Netezza Host Management

Workarounds and Mitigations

Mitigation of the reported CVEs : CVE-2021-3609, CVE-2021-37159 blocklisting kernel modules can-bcm, hso to prevent them from loading automatically on PureData System for Analytics N200x and N3001 is as follows:

  1. Change to user nz:
      [root@nzhost1 ~]# su – nz

  2. Check to see if Call Home is enabled:
      [nz@nzhost1 ~]$ nzcallhome -status
      If enabled, disable it:
      [nz@nzhost1 ~]$ nzcallhome –off
      Note: Ensure that nzcallhome returns status as disabled. If there are errors in the callHome.txt configuration file, errors             are listed in the output, and call-Home is disabled.

  3. Check the state of the Netezza system:
      [nz@nzhost1 ~]$ nzstate

  4. If the system state is online, stop the system using the command:
      [nz@nzhost1 ~]$ nzstop

  5. Wait for the system to stop, using the command:
      [nz@nzhos1t ~]$ nzstate
      System state is 'Stopped'.

  6. Exit from the nz session to return to user root:
      [nz@nzhost1 ~]$ exit

  7. Logged into the active host as root, type the following commands to stop the heartbeat processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service heartbeat stop
      [root@nzhost1 ~]# /sbin/service heartbeat stop

  8. Run below commands as a root user to disable heartbeat from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig heartbeat off
      [root@nzhost1 ~]# /sbin/chkconfig heartbeat off

  9. Type the following commands to stop the DRBD processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service drbd stop
      [root@nzhost1 ~]# /sbin/service drbd stop

  10. Run below commands as a root user to disable drbd from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig drbd off
      [root@nzhost1 ~]# /sbin/chkconfig drbd off

Execute below steps using "root" user on both ha1/ha2 hosts

Step 1: Check if kernel modules can-bcm, hso are loaded in the hosts

lsmod | grep can
lsmod | grep hso

example:
[root@ nzhost1 ~]# lsmod | grep can
can_bcm                12192  0
can                    32029  1 can_bcm
[root@ nzhost1 ~]# lsmod | grep hso
hso                    34520  0
rfkill                 19319  1 hso

Note: No output on Step 1 for any module indicates, that modules are not loaded hence skip Step 2 for that module, and proceed with Step 3

Step 2: Unload kernel modules are can-bcm, hso if they are loaded

modprobe -rv can-bcm
modprobe -rv hso

example:
[root@nzhost1 ~]# modprobe -rv can-bcm
rmmod /lib/modules/2.6.32-754.41.2.el6.x86_64/kernel/net/can/can-bcm.ko
rmmod /lib/modules/2.6.32-754.41.2.el6.x86_64/kernel/net/can/can.ko
[root@nzhost1 ~]#  modprobe -rv hso
rmmod /lib/modules/2.6.32-754.41.2.el6.x86_64/kernel/drivers/net/usb/hso.ko
rmmod /lib/modules/2.6.32-754.41.2.el6.x86_64/kernel/net/rfkill/rfkill.ko

Kernel modules and their dependent modules will be unloaded in the reverse order that they are loaded, given that no processes depend on any of the modules being unloaded.

Step 3: To prevent modules from being loaded directly you add the blocklist line to a configuration file specific to the system configuration.

echo "blocklist can-bcm" >> /etc/modprobe.d/local-blocklist.conf
echo "blocklist hso" >> /etc/modprobe.d/local-blocklist.conf

example :
[root@nzhost1 ~]# echo "blocklist can-bcm" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "blocklist hso" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep can-bcm
blocklist can-bcm
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep hso
blocklist hso

Step 4: Kernel modules can be loaded directly or loaded as a dependency from another module
To prevent installation as a dependency from another module follow below step:

echo "install can-bcm /bin/false" >> /etc/modprobe.d/local-blocklist.conf
echo "install hso /bin/false" >> /etc/modprobe.d/local-blocklist.conf

example:
[root@nzhost1 ~]# echo "install can-bcm /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "install hso /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep can-bcm
blocklist can-bcm
install can-bcm /bin/false
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep hso
blocklist hso
install hso /bin/false

The install line simply causes /bin/false to be run instead of installing a module.

Step 5: Make a backup copy of your initramfs.

cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak

Example:
[root@nzhost1 ~]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
[root@nzhost1 ~]# uname -r
2.6.32-754.41.2.el6.x86_64
[root@nzhost1 ~]# ll /boot/initramfs-2.6.32-754.41.2.el6.x86_64.img.10-01-041951.bak
-rw------- 1 root root 22126152 Sep 17 03:52 /boot/initramfs-2.6.32-754.41.2.el6.x86_64.img.10-01-041951.bak

Step 6: If the kernel module is part of the initramfs (boot configuration), rebuild your initial ramdisk image, omitting the module to be avoided

dracut --omit-drivers can-bcm -f
dracut --omit-drivers hso -f

example:
[root@nzhost1 ~]# dracut --omit-drivers can-bcm -f
[root@nzhost1 ~]# dracut --omit-drivers hso -f
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img | grep can-bcm
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img | grep hso

Step 7: Append module_name.blocklist to the kernel cmdline. We give it an invalid parameter of blocklist and set it to 1 as a way to preclude the kernel from loading it.

sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ can-bcm.blocklist=1/' /etc/grub.conf
sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ hso.blocklist=1/' /etc/grub.conf

example :
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ can-bcm.blocklist=1/' /etc/grub.conf
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ hso.blocklist=1/' /etc/grub.conf

Step 8: Blocklist the kernel module in kdump's configuration file.

echo "blocklist can-bcm" >> /etc/kdump.conf
echo "blocklist hso" >> /etc/kdump.conf

example:
[root@nzhost1 ~]# echo "blocklist can-bcm" >> /etc/kdump.conf
[root@nzhost1 ~]# echo "blocklist hso" >> /etc/kdump.conf
[root@nzhost1 ~]# cat /etc/kdump.conf | grep can-bcm
blocklist can-bcm
[root@nzhost1 ~]# cat /etc/kdump.conf | grep hso
blocklist hso

Note: Perform Step 9 if kexec-tools is installed and kdump is configured else continue with Step 10.
Perform below commands to check if kexec-tools is installed and Kdump is operational
[root@nzhost1 ~]# rpm -qa | grep kexec-tools
[root@nzhost1 ~]# service kdump status

Step 9: Restart the kdump service to pick up the changes to kdump's initrd.

service kdump restart

example:
[root@nzhost1 ~]# service kdump restart
Stopping kdump: [ OK ]
Detected change(s) the following file(s):

/etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-754.31.1.el6.x86_64kdump.img
Starting kdump: [ OK ]

Step 10: Reboot the system at a convenient time to have the changes take effect.
Make sure the secondary host is up by pinging or logging in before rebooting the primary host.

/sbin/shutdown -r now

example:
[root@nzhost1 ~]# /sbin/shutdown -r now
Make sure the primary server comes up and is reachable before performing Mitigation steps on the secondary server.

  After applying the mitigation:

      1. Start the services using following:
         [root@nzhost1 ~]# service heartbeat start
         [root@nzhost1 ~]# ssh ha2 service heartbeat start
         [root@nzhost1 ~]# service drbd start
         [root@nzhost1 ~]# ssh ha2 service drbd start

      2. Check the stat of the system. Type:
         [root@nzhost1 ~]# crm_mon -i5

          Result: When the cluster manager comes up and is ready, status appears as follows.
          Make sure that nzinit has started before you proceed. (This could take a few minutes.)
          Node: nps61074 (e890696b-ab7b-42c0-9e91-4c1cdacbe3f9): online
          Node: nps61068 (72043b2e-9217-4666-be6f-79923aef2958): online
          Resource Group: nps
          drbd_exphome_device(heartbeat:drbddisk): Started nps61074
          drbd_nz_device(heartbeat:drbddisk): Started nps61074
          exphome_filesystem(heartbeat::ocf:Filesystem): Started nps61074
          nz_filesystem (heartbeat::ocf:Filesystem): Started nps61074
          fabric_ip (heartbeat::ocf:IPaddr): Started nps61074
          wall_ip (heartbeat::ocf:IPaddr): Started nps61074
          nzinit (lsb:nzinit): Started nps61074
          fencing_route_to_ha1(stonith:apcmaster): Started nps61074
          fencing_route_to_ha2(stonith:apcmaster): Started nps61068

      3. From host 1 (ha1), press Ctrl+C to break out of crm_mon.

      4. Turn on heartbeat and DRBD using the chkconfig:
          ssh ha2 /sbin/chkconfig drbd on
          /sbin/chkconfig drbd on
          ssh ha2 /sbin/chkconfig heartbeat on
          /sbin/chkconfig heartbeat on