Attempts to stop the Guardium Data Encryption Expert agent fail. The kernel module may be busy and still hold open files or may be bound in a system call chain.
The Guardium Data Encryption Expert (GDEE) module fails to unload and attempts to stop the agent fail. Failure to stop the agent may result in blocked shutdown or reboot attempts.
Multiple causes exist.
Kernel Module Busy
The kernel module may be busy when a process maintains an open file descriptor of a file within the guard point. When an open descriptor exists within the kernel module, the module must wait for the process to close the descriptor or exit before it can de-allocate and release the file descriptor memory. Failure to de-allocate and release leads to system instability.
A kernel module implementing a file system may also report as busy if another filesystem is mounted over the existing file system ("nested file systems"), similar to the example shown below:
/dev/hda1 on / type ext3 (rw)
/dev/hda2 on /home type ext3 (rw)
/dev/hda3 on /home/user type ext3 (rw)
In the example shown above, /home will report as busy even if there are no active processes.
Kernel Module Still Bound
The kernel maintains a table of system calls and addresses of functions that implement the system call; this is known as the "syscall table". Kernel modules may intercept certain system calls from the syscall table and insert a custom implementation. Upon unload, a well-behaved kernel module will check the syscall table to verify that the function pointed to by the syscall table matches itself. Failure to match indicates that an additional kernel module has intercepted the system call, thus linking the current module function into a chain (syscall chaining), making it unsafe for the current module to unload.
Diagnosing The Problem
To determine if this problem is occurring:
1. system fails to shutdown OR the GDEE agent cannot be stopped manually
2. VMD log (/var/log/vormetric/vorvmd_root.log) shows restart of agent
3. /var/log/vormetric/secfsd.log shows re-guard of file systems
To determine if the module is busy, use the fuser or lsof command to determine if files are still open inside of the guard point. A process that blocks stat() access to the guard point will cause both fuser and lsof to fail. fuser lists the PIDs of processes holding files open. lsof lists processes and object paths.
# fuser /guarded_path
# lsof | grep /guarded_path
bash 2199 root cwd DIR 253,0 4096 10977282 /guarded_path
fuser and lsof have many additional options which may be more relevant depending upon operating system. Consult the man pages or User's Manual before relying on the information provided.
If the kernel module is not busy, no processes or descriptors are found to be in the guard point directories, the module may be bound in a syscall chain. On AIX systems, the agent will begin the shutdown process and display an error unloading the module. This failure is followed by a restart of the user-space processes:
# /etc/init.d/secfs stop
Stopping the Vormetric Encryption Expert File System Agent.
Stopping the vmd...done
Stopping the secfsd...done
Failed to unload kernel extension agent/secfs/.sec/mod/secfs2 error = 1
Starting the secfsd...done
Starting the vmd...done
To display a list of loaded kernel modules in linux, use the lsmod command. The lsmod command lists loaded kernel modules as well as kernel module dependencies. Bear in mind that kernel module dependencies refer to symbol resolution and are NOT indicative of syscall chaining.
To display a list of loaded kernel modules in AIX, use the genkex command. The genkex command displays loaded kernel module in the order in which they have been loaded with the most recently loaded module at the top of the list. While not a deterministic indicator of syscall chaining, modules loaded before the current module are unlikely to be problematic.
Definitive diagnosis of syscall chaining can be found in the file /opt/vormetric/DataSecurityExpert/agent/secfs/tmp/secfs.log. The following message is from an AIX system. There will be a message similar, if not the same, as this on other platforms:
# tail /opt/vormetric/DataSecurityExpert/agent/secfs/tmp/secfs.log
<date time> mod_fini: Intercepts superceded - failing unload
Resolving The Problem
If the kernel module is busy, identify which running processes are maintaining the open descriptors. Try to stop the processes gracefully. If graceful shutdown is not possible, kill the processes with either the fuser or kill command. After using kill, please allow the Operating System a few moments to perform garbage collection.
If the kernel module is bound, determine which processes load modules on top of the existing kernel module and stop those processes using the appropriate start/stop script. Stopping the processes with the kill command may leave the module with incorrect reference counts.
To validate, manually stop the GDEE agent using the stop/start script. On most systems, it is located in the /etc/init.d directory.
Was this topic helpful?
16 June 2018