Leap seconds are a periodic one-second adjustment of Coordinated Universal Time (UTC) required to keep a system's time of day close to the mean solar time.
A leap second is scheduled for 31 December 2016 at 23h 59m 60s UTC.
In the past, the addition of leap seconds caused some systems running older Linux kernels to fail. IBM MQ and WebSphere MQ queue managers running on these systems generated repeated FDC files showing Probe Id XY348010 or XC272003 from xcsCreateThread, with error code xecP_E_NO_RESOURCE. Other FDCs are also possible.
On 31 December 2016, the Network Time Protocol (NTP) daemon will schedule a leap second to occur at midnight, making the final minute of the day 61 seconds long. Prior leap second insertions, including the 30 June 2016 leap second, caused some Linux systems to exhibit unexplained high CPU usage.
IBM MQ and WebSphere MQ use the Native POSIX Thread Library (NPTL) which relies on futuxes ("fast userspace mutexes") for synchronization between threads. Futexes can exhibit high CPU spins after leap seconds on Linux kernel versions 2.6.22 through 3.5-rc7, inclusive, although some Linux distributions shipped patches for kernels in this range:
- Red Hat Enterprise Linux (RHEL): Leap Second to be added on December 31st 2016
SUSE Linux Enterprise Server (SLES): General information on Leap Second 2016 as well as the preceding article Fixes and workarounds to avoid issues caused by Leap Second 2015
Ubuntu: Addition of leap second causes spuriously high CPU usage and futex lockups
Contact your Linux vendor to check the status of other distributions, or check the Linux kernel version using "uname -r" and confirm that your system is older than 2.6.22 or running 3.5 or later.
If you encounter this problem, the following workaround may resolve the high CPU usage. As root, run:
- date -s "$(LC_ALL=C date)"
Internal Use Only
Backup from 22 December 2016:
Operating System Related Leap Second may cause Linux to freeze not a IBM MQ Issue
Leap seconds are a periodic one-second adjustment of Coordinated Universal Time(UTC) in order to keep a system's time of day close to the mean solar time.
The next leap second will occur on 2016 December 31, 23h 59m 60s UTC
Previously several leap second were added. Due to this Linux it self froze and , it was subsequently observed that a WebSphere MQ queue manager, running on Linux OS, generated many FDC files related to resource issues or constraints; these FDCs were commonly reporting rc=xecP_E_NO_RESOURCE.
You may also see your queue manager hang or freeze or there may be high cpu. The FDCs are being generated on a daily basis and may have probes of XY348010 or XC272003 from xcsCreateThread but there could be other FDCs with different probes also.
On December 31 2016, the Network Time Protocol (NTP) daemon scheduled a leap second to occur at midnight, meaning that the final minute of the day was 61 seconds long. We have seen several problems with otherwise unexplainable high CPU usage on Linux systems caused by the leap second at the end of June.
WebSphere MQ does not directly make calls which experience the problem, but we do use the pthreads library (NPTL), which in turn uses futexes ("fast userspace mutexes"), which can hit this problem. Busy systems running WebSphere MQ and other products are susceptible to this problem. You can read more about the problem at these links:
Anyone else experiencing high rates of Linux server crashes during a leap second day?
Leap Seconds in Red Hat Enterprise Linux
Leap second: Linux can freeze
This problem is solved by either applying Operating System (Linux) patches, resetting the date or rebooting the system. The resolution is dependent on your level of Linux and your environment. Please consult your Linux provider for details of the solution appropriate for your system.
As a workaround you can follow these steps
- Check the Linux kernel version. In theory only 2.6.22 and newer levels should be affected:
All: uname -r
- Switch to root or log in as root at the console
- Check to see if NTP is running:
RHEL: service ntpd status
SLES: /etc/init.d/ntp status
- If NTP is running, disable it:
RHEL: service ntpd stop
SLES: /etc/init.d/ntp stop
- Set the system clock to the current time:
All sntp -P no -r pool.ntp.org
Or: ntpdate 0.us.pool.ntp.org
- If NTP was running, reenable it:
RHEL: service ntpd start
SLES: /etc/init.d/ntp start
IBM MQ WMQ WebSphere MQ MQSeries
15 June 2018