IBM Support

How to find which process is causing the increase of rmsock.log file.

Troubleshooting


Problem

rmsock.log file is growing

Symptom

Customer reported the size of file /var/adm/ras/rmsock.log was over 800M, and was increasing. 
Want to know the reason.

Diagnosing The Problem

Checked the content of file /var/adm/ras/rmsock.log:
socket 0xf1000f00002ac808 held by process 7012626 (sshd) can't be removed.
socket 0xf1000f00002ab008 held by process 7012626 (sshd) can't be removed.
socket 0xf1000f00003e0808 held by process 6160892 (httpd) can't be removed.
socket 0xf1000f00003f3808 held by process 3670430 (snmpdv3ne) can't be removed.
socket 0xf1000f00003e1808 held by process 6160892 (httpd) can't be removed.
socket 0xf1000f0004316008 held by process 4587970 (inetd) can't be removed.
socket 0xf1000f00003fa808 held by process 6881552 (rmcd) can't be removed.
# fuser -u /var/adm/ras/rmsock.log
/var/adm/ras/rmsock.log: 16843222(root)
# ps -ef | grep 16843222
Can not get any result.
Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket. Note that the second argument of rmsock is the protocol. tcpcb indicates that the protocol is TCP. The results of the command are  logged to file /var/adm/ras/rmsock.log.
So, if we use fuser to check /var/adm/ras/rmsock.log, the reported process should be rmsock command, in spite that we couldn't see it in output of "ps -ef".
So, the problem here is to find out which process invokes the rmsock command, instead of which process accesses the file /var/adm/ras/rmsock.log.

Resolving The Problem

audit subsystem can be used to trace this issue.
  • Before audit, I do some testing about rmsock command:
# netstat -@aA|grep LISTEN
Global f1000e000016cbb8 tcp4      0     0 *.daytime            *.*                  LISTEN
Global f1000e00015a5bb8 tcp       0     0 *.ftp                *.*                  LISTEN
Global f1000e000158bbb8 tcp6      0     0 *.ssh                *.*                  LISTEN
Global f1000e000158cbb8 tcp4      0     0 *.ssh                *.*                  LISTEN
Global f1000e000016bbb8 tcp       0     0 *.login              *.*                  LISTEN
...
# rmsock f1000e00015a5bb8 tcpcb
The socket 0xf1000e00015a5808 is being held by proccess 3276958 (inetd).
# rmsock f1000e000158bbb8 tcpcb
The socket 0xf1000e000158b808 is being held by proccess 5243060 (sshd).
After this, when I checked the file /var/adm/ras/rmsock.log, we can see:
# more /var/adm/ras/rmsock.log
socket 0xf1000e00015a5808 held by process 3276958 (inetd) can't be removed.
socket 0xf1000e000158b808 held by process 5243060 (sshd) can't be removed.
  • Configure and start audit subsystem
1- Back up the existing audit configuration files.
# cd /etc/security/audit
# tar -cvf ./auditfiles.tar *
2- In the /etc/security/audit/config file, set up stream auditing start:
       binmode = off
       streammode = on
And the existing "kernel" event group will catch process info. So down in the "users" stanza and set it up like
users:
       default = kernel
3 -  Include the command rmsock as an object. So in the objects file, add it to the bottom:
/usr/sbin/rmsock:
        x = "RMSOCK_EXEC"
So the event we just made up called RMSOCK_EXEC will be triggered when someone executes /usr/sbin/rmsock.
4 -  In the events file,  include the new event to have some extra output:
RMSOCK_EXEC = printf "%s"
5- Since we're doing stream auditing, I like to add more output flags than the normal output. In the streamcmds file, change it from:
/usr/sbin/auditstream | auditpr > /audit/stream.out &
to
/usr/sbin/auditstream | auditpr -t1 -v -helRtcrpP > /audit/stream.out &
6 -  Start auditing
# audit start
  • After configuring and starting audit subsystem, I run another rmsock command:
    # rmsock f1000e000016bbb8 tcpcb
    The socket 0xf1000e000016b808 is being held by proccess 3276958 (inetd).
    After that, we can see, a new line was output to rmsock.log
    # more /var/adm/ras/rmsock.log
    socket 0xf1000e00015a5808 held by process 3276958 (inetd) can't be removed.
    socket 0xf1000e000158b808 held by process 5243060 (sshd) can't be removed.
    socket 0xf1000e000016b808 held by process 3276958 (inetd) can't be removed. 
  • Stop auditing by:
    # audit shutdown
  • After that, check audit output file /audit/stream.out, we can see
    event          login   status     time                    command                                      real      process         parent
    --------------- -------- ----------- ------------------------ ------------------------------- -------- -------- --------
    PROC_Create    root    OK         Thu Sep 19 03:38:32 2019 ksh                            root    6488108     3473416
           forked child process 7209088
    RMSOCK_EXEC    root    OK         Thu Sep 19 03:38:32 2019 rmsock                         root    7209088 6488108
           audit object exec event detected /usr/sbin/rmsock
    PROC_Execute   root    OK         Thu Sep 19 03:38:32 2019 rmsock                         root    7209088 6488108
           euid: 0 egid: 0 epriv: ffffffff:ffffffff name rmsock f1000e000016bbb8 tcpcb
    PROC_Delete    root    OK         Thu Sep 19 03:38:32 2019 rmsock                         root    7209088 6488108
           exited child process 7209088, rc: 1, filename: rmsock
     
    The ksh is the process for my login console.

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
15 September 2021

UID

ibm11135732

Page Feedback