LKU(Live Kernel Update ): 1020-154 Resource /dev/kmem is not checkpointable error.

Content

We know that /dev/kmem is character special file provide access to a pseudo device driver that allows read and write access to system memory or I/O address space. Typically, these special files are used by operating system utilities , third party software, commands such as sar , iostat, vmstat, sendmail to obtain status and statistical  information about the system. The process that uses the /dev/kmem can not be checkpointed or interrupted. 

The process that uses the /dev/kmem cannot be checkpointed or interrupted. So, while a process using /dev/kmem, when we try to run Live Kernel Update (LKU) which will fail because LKU must checkpoint every process running at that point in time to migrate them to the surrogate LPAR. The error we get when during such failure is as below:

Moving workload to surrogate LPAR.

1020-007 System call mcrk_sys_spl_getfd error (50). [03.156.0064]

1020-154 Resource /dev/kmem is not checkpointable (pid: 5308846) (pname: sendmail).

Query returned 19 The specified device does not exist. [02.229.0462]

1020-089 Internal error detected. [02.335.0458]

........1430-030 An error occurred while moving the workload.

1430-045 The live update operation failed.

Initiating log capture.

Running live update with /dev/kmem opened is not supported  and it is documented as a limitation in AIX Documentation.

LKU could checkpoint and restart the processes which are started trough System Resource Controller (SRC). So, a process running under SRC could open /dev/kmem. The sendmail being an SRC Subserver, thought it use /dev/kmem, we could do LKU when sendmail is running. Basically, any process which is running as an SRC Subsystem or Subserver could use /dev/kmem without objecting LKU. If we run sendmail directly without being under SRC, that can block LKU. srcmstr (SRC master) daemon being the parent for SRC subsystems, the below method could use to check whether the user of /dev/kmem is started under SRC or not

ps being the command to display information about the active processes, and lssrc being the command to display the status of all SRC subsystmes, they can be run on the victim to check as below. Also note that, these commands output are captured by snap and kept in the file kernel/kernel.snap.

.....    ps -ef

.....

     UID      PID     PPID   C    STIME    TTY  TIME CMD

     root  3146084        1   1   Mar 06      -  1:20 /usr/sbin/srcmstr  =======>

    root  3670514  3146084   0   Mar 06      -  2:34 /usr/sbin/syslogd

    root  5308846        1   0   Mar 06      -  0:01 sendmail: Queue runner@00:30:00 for /var/spool/mqueue=============================================>not started by srcmstr                                                                                                                                              

    root  5505494  3146084   0   Mar 06      -  0:03 /usr/sbin/portmap

    root  5833208  3146084   0   Mar 06      -  0:54 sendmail: accepting connections ======> started by srcmstr

.....

.....    lssrc -a

.....

Subsystem         Group            PID          Status

 syslogd          ras              3670514      active

 sendmail         mail           5833208      active =====> started by srcmstr

or

From command line :

#lssrc -a | grep sendmail           // probably sendmail might be not active or active

sendmail         mail             5833208      active =====> started by srcmstr
 

and
 

ps -ef | grep sendmail // sendmail is running

 root  5308846        1   0   Mar 06      -  0:01 sendmail: Queue runner@00:30:00 for /var/spool/mqueue 

 root  5833208  3146084   0   Mar 06      -  0:54 sendmail: accepting connections ====> started by   srcmstr

ps -ef | grep srcmstr    // To know sendmail with pid is 5833208  is started by srcmstr 

           root  3146084        1   1   Mar 06      -  1:20 /usr/sbin/srcmstr

This check is not  restricted only to sendmail process , But  it should be done for any process, which failed  with  “Resource /dev/kmem is not checkpointable” error.

Workaround and Solutions : 

If sendmail is the only process blocking LKU, then it can be stopped before LKU and can be restarted after LKU in the surrogate LPAR. But if other process also open /dev/kmem, then this workaround is not enough to solve the problem. The below method can be used to identify all the processes which are opening /dev/kmem and stopping all of them will resolve the problem:

fuser /dev/kmem

/dev/kmem : <pid of sendmail >

/dev/kmem : < pid of other utilities>

Once we confirm these processes are not started by SRC and if they support to start by SRC (in the case of sendmail), that will solve the problem. Otherwise we can possibly register scripts to stop and restart these processes by using Live Update event lvupdateRegScript.

The lvupdateRegScript -l  will list several script and Customer  can refer them to develop start and stop the problematic utilities ( in this case sendmail).

        Information Needed by IBM support for Review :

Some commands or utilities might be using /dev/kmem to read/write system memory or I/O address space, the same can be achieved by other services as well. To analyse the alternative solutions and arrive at the right decision, IBM Support needs the below information:

7. Snap collected during LKU failure.
8. Details of the process which uses /dev/kmem ( Pids shown up in Fuser

-> fuser  /dev/kmem

/dev/kmem:  2752890 4981170 8782200

root@cypress3:/

• tpid of all the process shown up in fuser (tpid -d <pid>) 

• stack trace of the all pvthread got in step (a)

• Switch to all pvthread got in step (a) and get the file descriptor  opened kmem

(0)> sw  pvthread+086300

Switch to  thread: <pvthread+086300>

(0)> u | grep fp.

save.... @ F00000002FF47600   fpr..... @ F00000002FF47900

_thrfp         :0000000000000000        _contextp    :0000000000000000

effpriv[0]..FFFFF7FE   effpriv[1]..7FFFFFFF   effpriv[2]..87FFFF7F

effpriv[3]..FFFFFFFF   effpriv[4]..FFFFFFFF   effpriv[5]..FFFFFFFF

effpriv[6]..FFFFFFFF   effpriv[7]..FFFFFFFF

fd      0 fp..F100111F50055700 count..00000000 flags. ALLOCATED FP

fd      1 fp..F100111F50056F00 count..00000000 flags. ALLOCATED FP

fd      2 fp..F100111F50056B00 count..00000000 flags. ALLOCATED FP

fd      3 fp..F100111F5005D900 count..00000000 flags. EXCLOSE ALLOCATED FP

fd      4 fp..F100111F50055700 count..00000001 flags. ALLOCATED FP

fd      5 fp..F100111F5005B300 count..00000000 flags. ALLOCATED FP

fd      6 fp..F100111F5005EF00 count..00000000 flags. EXCLOSE ALLOCATED FP

fd      7 fp..F100111F50063000 count..00000001 flags. EXCLOSE ALLOCATED FP

fd      8 fp..F100111F5005ED00 count..00000000 flags. FDLOCK ALLOCATED FP

(0)>

(0)> file F100111F5005D900 | grep kmem

f_fnamep....... F100111F5005D988 kmem

(0)> file F100111F50055700 | grep kmem

(0)> file F100111F5005B300 |grep kmem

(0)>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Author: Chetan Gaonkar
Operating System: AIX and VIOS
Hardware: Power
Feedback: aix_feedback@wwpdl.vnet.ibm.com, cgaonkar@in.ibm.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~