Troubleshooting
Problem
Potential Linux kernel reboot when running Trendmicro Deep Security Agent and Guardium STAP on the same Linux server.
Reboot happens when these conditions are met.
1. Guardium installed
2. TrendMicro installed
3. Reboot triggers when Real Time SCAN of TrendMicro Ends.
2. TrendMicro installed
3. Reboot triggers when Real Time SCAN of TrendMicro Ends.
Note: The order of installation doesn't matter.
Symptom
Operating system crash
Cause
The crash is caused by a NULL pointer of inode->i_pipe in pipe_release. It is a hook conflict between Guardium and Trendmicro. Both hook the same system call.
Environment
The problem has been reported in the following environment, but may apply to other versions:
Oracle Linux 6 64-bit
IBM Security Guardium Stap Agent 10.6.0.2_r106401
Trendmicro Deep Security Agent 11.0 U11 (11.0.0.760)
Diagnosing The Problem
The crash is caused by a NULL pointer of inode->i_pipe in pipe_release.
Crash Dump Analysis:
crash> bt
PID: 13141 TASK: ffff880237842040 CPU: 0 COMMAND: "sleep"
.....
#8 [ffff8802379e7c60] page_fault at ffffffff8155d265
[exception RIP: pipe_release+65]
RIP: ffffffff811ac211 RSP: ffff8802379e7d18 RFLAGS: 00010286
RAX: ffff8802379e4000 RBX: 0000000000000000 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880239b53d70
RBP: ffff8802379e7d48 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880239b53cb8
R13: ffff880239b53d70 R14: 0000000000000000 R15: 0000000000000001
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
.....
crash> dis pipe_release
0xffffffff811ac1d0 <pipe_release>: push %rbp
0xffffffff811ac1d1 <pipe_release+1>: mov %rsp,%rbp
0xffffffff811ac1d4 <pipe_release+4>: sub $0x30,%rsp
0xffffffff811ac1d8 <pipe_release+8>: mov %rbx,-0x28(%rbp)
0xffffffff811ac1dc <pipe_release+12>: mov %r12,-0x20(%rbp)
0xffffffff811ac1e0 <pipe_release+16>: mov %r13,-0x18(%rbp)
0xffffffff811ac1e4 <pipe_release+20>: mov %r14,-0x10(%rbp)
0xffffffff811ac1e8 <pipe_release+24>: mov %r15,-0x8(%rbp)
0xffffffff811ac1ec <pipe_release+28>: nopl 0x0(%rax,%rax,1)
0xffffffff811ac1f1 <pipe_release+33>: lea 0xb8(%rdi),%r13
0xffffffff811ac1f8 <pipe_release+40>: mov %rdi,%r12
0xffffffff811ac1fb <pipe_release+43>: mov %esi,%r14d
0xffffffff811ac1fe <pipe_release+46>: mov %edx,%r15d
0xffffffff811ac201 <pipe_release+49>: mov %r13,%rdi
0xffffffff811ac204 <pipe_release+52>: callq 0xffffffff8155a930 <mutex_lock+32>
0xffffffff811ac209 <pipe_release+57>: mov 0x1d0(%r12),%rbx
0xffffffff811ac211 <pipe_release+65>: sub %r14d,0x28(%rbx)
0xffffffff811ac1d0 <pipe_release>: push %rbp
0xffffffff811ac1d1 <pipe_release+1>: mov %rsp,%rbp
0xffffffff811ac1d4 <pipe_release+4>: sub $0x30,%rsp
0xffffffff811ac1d8 <pipe_release+8>: mov %rbx,-0x28(%rbp)
0xffffffff811ac1dc <pipe_release+12>: mov %r12,-0x20(%rbp)
0xffffffff811ac1e0 <pipe_release+16>: mov %r13,-0x18(%rbp)
0xffffffff811ac1e4 <pipe_release+20>: mov %r14,-0x10(%rbp)
0xffffffff811ac1e8 <pipe_release+24>: mov %r15,-0x8(%rbp)
0xffffffff811ac1ec <pipe_release+28>: nopl 0x0(%rax,%rax,1)
0xffffffff811ac1f1 <pipe_release+33>: lea 0xb8(%rdi),%r13
0xffffffff811ac1f8 <pipe_release+40>: mov %rdi,%r12
0xffffffff811ac1fb <pipe_release+43>: mov %esi,%r14d
0xffffffff811ac1fe <pipe_release+46>: mov %edx,%r15d
0xffffffff811ac201 <pipe_release+49>: mov %r13,%rdi
0xffffffff811ac204 <pipe_release+52>: callq 0xffffffff8155a930 <mutex_lock+32>
0xffffffff811ac209 <pipe_release+57>: mov 0x1d0(%r12),%rbx
0xffffffff811ac211 <pipe_release+65>: sub %r14d,0x28(%rbx)
crash> sym pipe_release
ffffffff811ac1d0 (t) pipe_release /usr/src/debug/kernel-2.6.32-754.17.1.el6/linux-2.6.32-754.17.1.el6.x86_64/fs/pipe.c: 707
static int
pipe_release(struct inode *inode, int decr, int decw)
{
struct pipe_inode_info *pipe; mutex_lock(&inode->i_mutex);
pipe = inode->i_pipe;
pipe->readers -= decr;
pipe->writers= decw; if (!pipe>readers && !pipe->writers) {
free_pipe_info(inode);
} else {
wake_up_interruptible_sync_poll(&pipe->wait, POLLIN | POLLOUT | POLLRDNORM | POLLWRNORM | POLLERR | POLLHUP);
kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
}
mutex_unlock(&inode->i_mutex); return 0;
}
Dereferencing it is a problem on line pipe = inode->i_pipe; pipe is a NULL pointer.
ffffffff811ac1d0 (t) pipe_release /usr/src/debug/kernel-2.6.32-754.17.1.el6/linux-2.6.32-754.17.1.el6.x86_64/fs/pipe.c: 707
static int
pipe_release(struct inode *inode, int decr, int decw)
{
struct pipe_inode_info *pipe; mutex_lock(&inode->i_mutex);
pipe = inode->i_pipe;
pipe->readers -= decr;
pipe->writers
free_pipe_info(inode);
} else {
wake_up_interruptible_sync_poll(&pipe->wait, POLLIN | POLLOUT | POLLRDNORM | POLLWRNORM | POLLERR | POLLHUP);
kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
}
mutex_unlock(&inode->i_mutex); return 0;
}
Dereferencing it is a problem on line pipe = inode->i_pipe; pipe is a NULL pointer.
Resolving The Problem
To avoid the issue, Trendmicro provided workaround to avoid hooking from both sides.
Workaround :
1. Disable Deep Security Anti-Malware filesystem hooking
2. $ echo "/opt/ds_agent/lib/libvmpd_dsa_rtscan.so=rtscan_hook_enable=1,rtscan_hook_kern_method=2" > /var/opt/ds_agent/am/ds_am.ini
3. $ systemctl restart ds_agent
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSMPHH","label":"IBM Security Guardium"},"Component":"IBM Security Guardium STAP","Platform":[{"code":"PF016","label":"Linux"}],"Version":"10.6","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Product Synonym
IBM Guardium
Was this topic helpful?
Document Information
Modified date:
10 January 2020
UID
ibm11170778