All computer operating systems verify whether the file systems that they mount at boot time are consistent, meaning that there are no errors in their internal data structures and the associated storage that they map to. UNIX, Linux®, and other UNIX-like operating systems take a clever approach to determining whether the consistency of a file system needs to be verified (typically by using the fast command). When these systems mount a file system, they set a value in the file system header that marks the file system as DIRTY, meaning that it is in use and may be transiently inconsistent as updates are being written to it. When file systems are unmounted as part of a system shutdown, they are marked as CLEAN. When the system reboots, only file systems that are still marked as DIRTY need to be checked for consistency.

File systems are automatically unmounted as part of the system shutdown process, which usually occurs after all non-system processes have been terminated. Regardless, unmounting a file system can still fail with the traditional message:

$ sudo umount /mnt/NAS
umount: /mnt/NAS: device is busy

In this case, busy means that a process is writing to or running from that file system. The fact that you cannot unmount a file system in either of these cases is one of the basic rules of computer systems. If this were not the case, you could unmount a file system while some process is writing to a file that it contains, which could leave the file in an inconsistent state while the file system itself is marked as CLEAN.

The standard Linux version of the umount command includes a lazy unmount option, -l, to help unmount file systems that are in use. This command requires Linux kernel version 2.4.11 or greater, which isn't much of a problem today. Executing umount -l /name/of/file system detaches the specified file system from the system's directory hierarchy so that no new processes can use the file system and then unmounts the file system when all of the processes that were accessing it terminate. This can be handy but is not exactly what you want to use when you need to unmount a file system now.

If you need to unmount a file system now, and that file system reports that it is busy, you still have some options. If you are the only user of a system, terminating the processes that are preventing you from unmounting a file system is as easy as looking through all your windows for suspended or background processes that are writing to the partition in question or using it as their current working directory and terminating them. However, on multi-user systems with many local and remote users this approach isn't practical. Luckily, the open source community offers some commands that make it easy to identify and terminate such processes.

Locating open files with lsof

The lsof (list open files) command displays a list of all open files and the processes associated with them on a specific file system, directory, or device. The lsof command is available for most UNIX and UNIX-like systems, including IBM® AIX®, Berkeley Software Distribution (BSD®), Hewlett Packard UNIX (HP-UX®), Linux, and Solaris®. See Resources for information about obtaining lsof for your system.

By default, the lsof command lists all files, shared libraries, and directories that are currently open and provides as much information as possible about each of them. The output from this is huge, even on a lightly loaded system, so you typically either supply the name of a directory as a command-line argument or use a pipe to filter its output. For example, suppose that you want to unmount a file system that is mounted on the /opt2 directory. To see all of the processes associated with the /opt2 directory, you execute a command such as the one shown in Listing 1.

$ lsof /opt2
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    23334  wvh  cwd    DIR   8,17     4096    2 /opt2
more    23402  wvh  cwd    DIR   8,17     4096    2 /opt2
more    23402  wvh    3r   REG   8,17    10095  264 /opt2/resume.txt

You need to terminate all of these processes before you can unmount the /opt2 partition. Because none of the processes in this listing can be writing any files, you could use the kill command to terminate the process IDs (PIDs) that are listed in the second column and then unmount the partition with no problems. Note that PID 23402 is associated with the last two lines—the first line indicates that the more command is running with a current working directory (cwd) of /opt2, and the second indicates that the more command has the /opt2/resume.txt file open.

However, suppose that the output of the lsof command looks like Listing 2.

$ lsof /opt2
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    23334  wvh  cwd    DIR   8,17     4096    2 /opt2
more    23402  wvh  cwd    DIR   8,17     4096    2 /opt2
more    23402  wvh    3r   REG   8,17    10095  264 /opt2/resume.txt
bash    21343  djf  cwd    DIR   8,17     4096    2 /opt2
emacs   21405  djf  cwd    DIR   8,17     4096    2 /opt2

The first three commands associated with the /opt2 directory are the same, but the last two are being run by another user. Of these, the emacs command is designed for editing files, so you might want to have the user listed in the USER column save and exit before you terminate that process.

Back to top

Customizing lsof output

The previous section showed how to identify open files and directories on a local device, but you can just as easily get the same information about a mounted remote file system.

To provide a consistent set of examples for this article, all of the command and output examples refer to mounted partitions from the system shown in Listing 3.

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            230528596 201462232  17356188  93% /
/dev/sdb1            240362656  12533532 215619324   6% /opt2
//nas.vonhagen.org/writing  
                     100790048  75945920 197241926  80% /mnt/NAS
192.168.6.166:/mnt/disk1
                     714854640 386972432 291569696  58% /mnt/yellowmachine

As shown in Listing 3, /mnt/NAS is the mountpoint for a Samba share called writing that is shared from the device nas.vonhagen.org. Specifying the name of the mountpoint as an argument to the lsof command creates output similar to Listing 2 but specific to that device and directory, as shown in Listing 4.

$ lsof /mnt/NAS
COMMAND   PID     USER   FD   TYPE DEVICE SIZE    NODE NAME
bash    23236 wvh  cwd  DIR   0,27 4096 6406145 /mnt/NAS/writing \
   (nas.vonhagen.org:/writing)

The lsof command also provides options that enable you to restrict its output to reporting open files and directories on specific types of devices. For example, as shown in Listing 3, the /mnt/yellowmachine directory is a mountpoint for a Network File System (NFS) mount of the /mnt/disk1 directory on the 192.168.6.166 device. You can easily supply the name of the mountpoint for this device as an argument to the lsof command, as shown in Listing 5.

$ lsof /mnt/yellowmachine
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    23334  wvh  cwd    DIR   0,23     4096    2 /mnt/yellowmachine \
   (192.168.6.166:/mnt/disk1)

You can also use the lsof command's -N option to list only files and directories that are in use on NFS-mounted devices, as shown in Listing 6.

$ lsof -N
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash      23334  wvh  cwd    DIR   0,23     4096      2 /mnt/yellowmachine
  (192.168.6.166:/mnt/disk1)

The lsof command has many more options that can help you identify open files and directories on different types of file systems, as well as the processes that have network sockets open, who is using specific binaries, and more. The downside of the lsof command is that you always have to either contact users and ask them to terminate certain processes or manually terminate them yourself. The fuser command is a more cryptic but also more powerful command that can do much of your process termination work for you when run as the root user.

Back to top

Finding user processes with fuser

The fuser (find user processes) command is another open source application that can help identify processes that are preventing you from unmounting file systems. The fuser command finds user processes that are associated with whatever files, directories, or file system mountpoints that you supply as command-line arguments. This article focuses on using fuser with file system mountpoints. For more generic information about the fuser command, see its online reference information. The fuser command requires that your system supports the /proc file system. Therefore, it's available for all Linux distributions and FreeBSD systems. See Resources for information about obtaining the source code for the fuser command.

As with the lsof command, supplying the name of a file system mountpoint as a command-line argument is the simplest way to use the fuser command to identify processes that are preventing you from unmounting a file system:

$ fuser /mnt/yellowmachine
/mnt/yellowmachine:  23334c 23697c

The output of the fuser command simply identifies the PIDs of processes that are using the specified mountpoint. Each PID is followed by a single letter that identifies the way in which the process associated with that PID is using the specified mountpoint. The most common of these is the letter c, shown in the previous example, which indicates the specified process is using a directory on that file system as its current working directory.

Unfortunately, the default output of the fuser command isn't end-user friendly, even by Linux standards. The fuser command provides a -v option that adds some output that is similar to the output of the standard ps command to the output of the fuser command, as shown in Listing 7.

$ fuser -v /mnt/yellowmachine
                     USER        PID ACCESS COMMAND
/mnt/yellowmachine:  wvh       23334 ..c.. bash
                     wvh       23697 ..c.. emacs

This is handier because it at least identifies what the processes are. After you obtain the PID information from the fuser command, you can always use a combination of the standard ps and egrep commands to get as much detail as possible about the processes before terminating them, as shown in Listing 8.

# ps alxww |egrep '23334|23697'
4 1000 23334 23332 20 0 18148  2076 wait   Ss pts/13  0:00 -bash
0 1000 23697 23334 20 0 75964 12352 poll_s S+ pts/13  0:00 emacs -nw file2.txt
0    0 23703 23665 20 0  6060   632 -      R+ pts/16  0:00 egrep
23334|23697

You can then use the standard kill command to terminate the specified processes manually or, as explained in the next section, use some of the advanced capabilities of the fuser command to terminate them automatically.

Back to top

Terminating processes with fuser

The fuser command's -k option automatically terminates processes that it detects are using a mountpoint that you specify as an argument. You must, of course, execute the fuser command as root to be able to terminate processes that may be owned by other users, as shown in Listing 9.

# fuser -k /mnt/yellowmachine
/mnt/yellowmachine:  23334c 23697c
Could not kill process 23697: No such process

In this case, the second process (emacs) was a child of the first (the bash shell), and therefore terminated when the first was killed by the fuser command.

If you want to specify the name of an underlying physical device rather than simply the mountpoint for the file system that it contains, you must also specify the -m option, as shown in Listing 10.

# fuser -v /opt2
                     USER        PID ACCESS COMMAND
/opt2:               wvh       23712 ..c.. bash
                     wvh       23753 ..c.. emacs
# fuser -v /dev/sdb1
# fuser -vm /dev/sdb1
                     USER        PID ACCESS COMMAND
/dev/sdb1:           wvh       23712 ..c.. bash
                     wvh       23753 ..c.. emacs

The first command returns the output that you would expect because it references the mountpoint for a file system. The second command shows that you cannot directly query the underlying device by using the standard fuser options. The third illustrates that the -m option enables you to specify a device directly. You could add the -k option to either of the first or third commands in this example to terminate the processes in the file system that is located on the /dev/sdb1 device.

Back to top

Conclusion

At some point, every Linux or UNIX systems administrator needs to unmount a partition in response to some emergency or simply to remove a device such as a mounted CD-ROM or DVD. When the system won't let you unmount a device because that device is busy, examining every process on the system is both irritating and slow. The lsof and fuser commands make it easy to identify the processes that are preventing you from unmounting a file system. The fuser command even terminates them for you if you're in a hurry.

Resources

Learn

• The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX? Visit the New to AIX and UNIX page to learn more.
  
  
• Browse the technology bookstore for books on this and other technical topics.
  
  

Get products and technologies

• The lsof command is installed by default on most Linux systems, and source packages for lsof are available in the repositories used by those distributions. The latest version of lsof is available via anonymous File Transfer Protocol (FTP) from lsof.itap.purdue.edu in the pub/tools/unix/lsof directory. You can also get lsof from an FTP URL: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof.
  
  
• The fuser command is part of the PSmisc package, which also includes commands such as pstree and killall.
  
  
• The Linux Foundation provides a simple script called Forced Unmount, or fumount, that tries to automatically combine a number of passes of fuser with the appropriate unmount commands to forcibly unmount a specified partition.
  
  

Discuss

• Check out developerWorks blogs and get involved in the developerWorks community.
  
  
• Follow developerWorks on Twitter.
  
  
• Get involved in the My developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems management
  • IBM Support Assistant Forum
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Locating open files with lsof
• Customizing lsof output
• Finding user processes with fuser
• Terminating processes with fuser
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers