IBM Support

Leap Second bug in older Linux kernels may affect IBM MQ and WebSphere MQ

Flashes (Alerts)


Abstract

Leap seconds are a periodic one-second adjustment of Coordinated Universal Time (UTC) required to keep a system's time of day close to the mean solar time. 

A leap second is scheduled for 31 December 2016 at 23h 59m 60s UTC.

In the past, the addition of leap seconds caused some systems running older Linux kernels to fail. IBM MQ and WebSphere MQ queue managers running on these systems generated repeated FDC files showing Probe Id XY348010 or XC272003 from xcsCreateThread, with error code xecP_E_NO_RESOURCE. Other FDCs are also possible.

Content

On 31 December 2016, the Network Time Protocol (NTP) daemon will schedule a leap second to occur at midnight, making the final minute of the day 61 seconds long. Prior leap second insertions, including the 30 June 2016 leap second, caused some Linux systems to exhibit unexplained high CPU usage.


IBM MQ and WebSphere MQ use the Native POSIX Thread Library (NPTL) which relies on futuxes ("fast userspace mutexes") for synchronization between threads. Futexes can exhibit high CPU spins after leap seconds on Linux kernel versions 2.6.22 through 3.5-rc7, inclusive, although some Linux distributions shipped patches for kernels in this range:



Contact your Linux vendor to check the status of other distributions, or check the Linux kernel version using "uname -r" and confirm that your system is older than 2.6.22 or running 3.5 or later.

If you encounter this problem, the following workaround may resolve the high CPU usage. As root, run:
    date -s "$(LC_ALL=C date)"

Internal Use Only

Backup from 22 December 2016:
Operating System Related Leap Second may cause Linux to freeze not a IBM MQ Issue

=========================================================

Leap seconds are a periodic one-second adjustment of Coordinated Universal Time(UTC) in order to keep a system's time of day close to the mean solar time. 

The next leap second will occur on 2016 December 31, 23h 59m 60s UTC

Previously several leap second were added. Due to this Linux it self froze and , it was subsequently observed that a WebSphere MQ queue manager, running on Linux OS, generated many FDC files related to resource issues or constraints; these FDCs were commonly reporting rc=xecP_E_NO_RESOURCE.

You may also see your queue manager hang or freeze or there may be high cpu. The FDCs are being generated on a daily basis and may have probes of XY348010 or XC272003 from xcsCreateThread but there could be other FDCs with different probes also.

=========================================================


On December 31 2016, the Network Time Protocol (NTP) daemon scheduled a leap second to occur at midnight, meaning that the final minute of the day was 61 seconds long. We have seen several problems with otherwise unexplainable high CPU usage on Linux systems caused by the leap second at the end of June.

WebSphere MQ does not directly make calls which experience the problem, but we do use the pthreads library (NPTL), which in turn uses futexes ("fast userspace mutexes"), which can hit this problem. Busy systems running WebSphere MQ and other products are susceptible to this problem. You can read more about the problem at these links:

Anyone else experiencing high rates of Linux server crashes during a leap second day?

Leap Seconds in Red Hat Enterprise Linux

Leap second: Linux can freeze

This problem is solved by either applying Operating System (Linux) patches, resetting the date or rebooting the system. The resolution is dependent on your level of Linux and your environment. Please consult your Linux provider for details of the solution appropriate for your system.

As a workaround you can follow these steps

  1. Check the Linux kernel version. In theory only 2.6.22 and newer levels should be affected:
    All: uname -r
  2. Switch to root or log in as root at the console
  3. Check to see if NTP is running:
    RHEL: service ntpd status
    SLES: /etc/init.d/ntp status
  4. If NTP is running, disable it:
    RHEL: service ntpd stop
    SLES: /etc/init.d/ntp stop
  5. Set the system clock to the current time:
    All sntp -P no -r pool.ntp.org
    Or: ntpdate 0.us.pool.ntp.org
  6. If NTP was running, reenable it:
    RHEL: service ntpd start
    SLES: /etc/init.d/ntp start

[{"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Component":"ABEND \/ FDC","Platform":[{"code":"PF016","label":"Linux"}],"Version":"9.0.1;9.0;8.0;7.5;7.1;7.0;6.0","Edition":""}]

Product Alias/Synonym

IBM MQ WMQ WebSphere MQ MQSeries

Document Information

Modified date:
15 June 2018

UID

swg21602521