System Administration Toolkit: Monitoring disk space and usage

Look at methods for determining disk usage across multiple UNIX® systems and how to create a simple warning system to alert you of potential problems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital process in the day-to-day management of your UNIX systems. In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how to make use of quotas and other solutions to find the information you need.

Share:

Martin Brown (mc@mcslp.com), Freelance Writer, Freelance Developer

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertise spans myriad development languages and platforms -- Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®, Solaris, Linux, BeOS, Mac OS X and more -- as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) for Microsoft and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The Apple Blog, and other sites. You can contact him through his Web site.



13 June 2006

Also available in Russian

About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.

Getting started: using df

Monitoring disk space is a vital part of your job as a UNIX administrator. This article gives you the tools you need to be successful, including the use of df, du, find, and even the use of quotas. Let's get started by taking a look at how useful df can be.

Use of the df tool can be a bit like a nervous twitch for many administrators, as it gives you a snapshot view of the storage space used and available across all of your file systems through a single command. Depending on your UNIX environment, the default output from df might contain a range of different information.

Most modern df variants show disk space, usage and availability, and usually the mountpoint (and sometimes the device). For example, Mac OS X, based on BSD, shows the information in Listing 1.

Listing 1. Default disk space information for Mac OS X
Filesystem    512-blocks      Used     Avail Capacity  Mounted on
/dev/disk0s2   268435456 119741344 148182112    45%    /
devfs                195       195         0   100%    /dev
fdesc                  2         2         0   100%    /dev
<volfs>             1024      1024         0   100%    /.vol
/dev/disk0s3    43474520   7251448  36223072    17%    /Volumes/Untitled

Note that the size is displayed in 512 byte blocks, not kilobytes. Nearly all traditional UNIX operating systems report the size in 512 byte blocks rather than kilobytes by default. To get the information in 1K blocks, use the -k command-line option, as shown in Listing 2.

Listing 2. Using the -k option with the df command
$ df -k
Filesystem    1K-blocks      Used     Avail Capacity  Mounted on
/dev/disk0s2  134217728  59870704  74091024    45%    /
devfs                97        97         0   100%    /dev
fdesc                 1         1         0   100%    /dev
<volfs>             512       512         0   100%    /.vol
/dev/disk0s3   21737260   3625724  18111536    17%    /Volumes/Untitled

Some variants might also support alternative block sizes, such as megabytes (use -m) and gigabytes (use -g), as demonstrated in Listing 3.

Listing 3. Using the -g option with the df command
$ df -g
Filesystem    1G-blocks Used Avail Capacity  Mounted on
/dev/disk0s2        128   57    70    45%    /
devfs                 0    0     0   100%    /dev
fdesc                 0    0     0   100%    /dev
<volfs>               0    0     0   100%    /.vol
/dev/disk0s3         20    3    17    17%    /Volumes/Untitled

Obviously, as you increase the block size shown, the level of detail in the information starts to reduce, but using alternative sizes can be a useful way to quickly monitor disks, particularly very large ones. Other versions of the df tool (notably Solaris) report the bare block and file availability information by default. Most file systems have a limited number of files that can be stored (the number is so large that you are unlikely to reach the limit), so it is possible to fill up a system with files, but still have disk capacity and be unable to store any more files.

You can see a sample of the default output of Solaris in Listing 4.

Listing 4. Default output of Solaris
$ df
/                  (/dev/dsk/c0t0d0s0 ):14877208 blocks   914042 files
/devices           (/devices          ):       0 blocks        0 files
/system/contract   (ctfs              ):       0 blocks 2147483618 files
/proc              (proc              ):       0 blocks    16109 files
/etc/mnttab        (mnttab            ):       0 blocks        0 files
/etc/svc/volatile  (swap              ): 5737216 blocks   147177 files
/system/object     (objfs             ):       0 blocks 2147483532 files
/usr               (/dev/dsk/c0t0d0s3 ): 9076010 blocks   863695 files
/dev/fd            (fd                ):       0 blocks        0 files
/var               (/dev/dsk/c0t0d0s4 ): 8110796 blocks   483714 files
/tmp               (swap              ): 5737216 blocks   147177 files
/var/run           (swap              ): 5737216 blocks   147177 files
/export/home       (/dev/dsk/c0t0d0s7 ):69362510 blocks  4272812 files

Using the -k command-line option switches the display to one similar to that shown with earlier examples, where the data is summarized into a more easily readable format (see Listing 5).

Listing 5. Using the -k option
$ df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s0    7644629  206026 7362157     3%    /
/devices                   0       0       0     0%    /devices
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 2868600    1016 2867584     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
/dev/dsk/c0t0d0s3    8261237 3723232 4455393    46%    /usr
fd                         0       0       0     0%    /dev/fd
/dev/dsk/c0t0d0s4    4130238   74849 4014087     2%    /var
swap                 2867584       0 2867584     0%    /tmp
swap                 2867624      40 2867584     1%    /var/run
/dev/dsk/c0t0d0s7    35611388  930133 34325142     3%    /export/home

All variants of df accept a directory or path, and then the information about that disk space for the file system that contains that path is displayed. For example, the following shows you how to get the space on the root file system:

$ df -k /
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s0    7644629  206026 7362157     3%    /

Or, the following uses df to show the space on a user's home directory:

$ df -k ~mc
Filesystem             kbytes    used    avail capacity  Mounted on
/dev/dsk/c0t0d0s7    35611388  930133 34325142     3%    /export/home

Using the df output

The df tool is actually short for disk free, and that is precisely what the tool shows. If you look at some sample output again, you can extract the important individual elements:

Filesystem             kbytes    used    avail capacity  Mounted on
/dev/dsk/c0t0d0s7    35611388  930133 34325142     3%    /export/home

The first column, usually with a header according to the block size data, shows the total size of the disk. The used column shows the number of blocks that have been used on the file system or device. The avail column shows the number of unused (available) blocks on the file system.

The capacity column gives a percentage calculation of how much disk space out of the total size has actually been used. On its own, this information can give a quick indication of the space available.

By default, UNIX creates file systems that can be filled up to 90 percent capacity with data by users. The remaining 10 percent is reserved by the system for use by root to perform any emergency maintenance. If a user tries to add or create a file that goes beyond that level, an error is returned (out of file system space).

That emergency maintenance space is there so it can be used to recover from an overfull disk. For example, as an administrator, you could use the space to create a quick backup, or to compress existing data without having to use a secondary file system for the purpose. You can adjust the amount of free space that is reserved (using the tunefs tool), or when creating the file system. This is vital for large disks, where a 10 percent buffer could equal many gigabytes of potential space. It is generally a good idea to reserve at least one percent to give you a minimal buffer before you completely run out of space. Before you get that far, especially if a lot of your disk space is being used, you should determine who is using all of that space.


Using du

The du command, instead of showing disk free space, shows disk usage information. The du tool is used to determine the disk usage for files and directories. To use it, change to a directory and run the tool (see Listing 6).

Listing 6. du command
$ cd /var
$ du
16      ./lost+found
4       ./sadm/install/admin
22      ./sadm/install/logs
28448   ./sadm/install
4       ./sadm/pkg/SUNWocfd/install
4       ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd/install
16      ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd
18      ./sadm/pkg/SUNWocfd/save/pspool
20      ./sadm/pkg/SUNWocfd/save
28      ./sadm/pkg/SUNWocfd
4       ./sadm/pkg/SUNWcsu/inst
...

The output, shown in Listing 6 above, has been trimmed. By default, du shows the file usage for every file and directory beneath the current, or specified, directory. The value given is the size of the file in the default block size for that system, just as that used with df. This might, or might not, be 1K; you can force the display to 1K blocks using the -k command-line option.

You will probably want to summarize the information by the top-level file, or directory, that you are examining. Use the -s option to switch on the summary view. Here's a summarized version, shown in the /var directory, of a Solaris installation:

$ du -sk  
70818   .

Note that it shows the summary information for the current directory (.). To get summary information for all of the files and directories, use the * wildcard (see Listing 7).

Listing 7. Using the * wildcard to get summary information
$ du -sk *
382     adm
950     apache
683     apache2
6837    appserver
1       audit
162     cache
3       cc-ccr
2       crash
4       cron
31      dmi
22      dt
6       fm
2       imq
1       inet
3       krb5
4       ld
1       ldap
937     lib
6       log
8       lost+found
2       lp
2       mail
1       mysql
1       news
3       nfs
38      nis
2       ntp
10034   opt
1       preserve
96      run
49687   sadm
15      saf
3       samba
2       sma_snmp
131     snmp
39      spool
4       statmon
663     svc
14      tmp
10      uucp
24      yp

Be careful when using that tool, particularly in user directories, as the information shown will not include hiddenfiles -- in other words, those files and directories with a single period prefix. You might want to use the following line in a user directory to get all of the summary information:

$ du -sk * .[a-zA-Z0-9]*

One final very useful option for du is -d, which prevents du from following file system boundaries. For example, to determine the disk usage on the root file system but not any other file systems, you would use the -d option:

$ du -dsk /

Some systems do not have this option but do have the -x command-line option, which only includes the file usage for files on the same device or file system as the path you specify.


Finding disk usage for a specific user

To find the disk space being used by a single user, you need to combine du with the find command to only report disk usage for a specific user:

$ find . -user mc -type f -exec du -k {} \;

The -user option allows you to specify that find will only report files that are owned by the specified user. The -type option forces find to only return the path of items of a specific type (in this case, files); this prevents du from including directories, which might be owned by one user, but contain files for many users. Then, for each found path, the du command is executed to report the disk usage.

To get summary information, in other words, the total space used by a specific user, you can use awk to total the information and print out the final value:

$ find . -user mc -type f -exec du -k {} \;|awk '{ s = 
     s+$1 } END { print "Total used: ",s }'
Total used:  123721

You use the same principle with groups using the -group option to find:

$ find . -group mcslp -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print
 "Total used: ",s }'
Total used:  542485

There is, however, an easier way, if you enable disk quotas.


Using quotas

The quota system automatically monitors the disk usage for users on a file system by a file system basis. The quota environment not only enables you to monitor disk usage, but you can also set limits for usage that provide a warning and, more explicitly, prevent users from using disk space over their assigned quota value. The lower limit (warning) is called the soft limit and the higher limit (which prevents creation of files over that level) is called the hard limit. Some systems might also allow you to control the number of files owned by each user.

The precise method required to enable quotas on your machine is dependent on the operating system you are using. Most UNIX systems include quota support by default. Linux® systems might need to build a new kernel that includes quota support. Most, however, use a single file on each file system, usually called quotas, into which you place the limits for each user.

To enable quotas, you should first create this file and ensure that the quotas file is only editable by root:

$ touch /export/home/quotas
$ chmod 600 /export/home/quotas

Then switch on quotas using the quotaon command:

$ quotaon /export/home

Finally, you must edit the appropriate quota for each user. You can do this by using the edquota command, specifying the user:

$ edquota mc

Your default editor (or vi, if you haven't set an alternative) will be opened with a simple form for setting quota values. You can see below that quotas have been enabled on the users home directory file system with a soft limit of 200,000KB and a hard limit of 400,000KB. The settings of zero for file limits indicate that there is no set limit and the user can set and create as many files as they like.

fs /export/home blocks (soft = 200000, hard = 400000) inodes (soft = 0,
 hard = 0)

If you have more file systems with quotas, then you will need to configure more rows for the file systems.

If you want to configure the quota for multiple users, first set the quota for a single user, then use the -p command-line option to edquota. This uses the specified user as the basis for the new users. For example, to set the quota using the settings for mc, for usernames slp, tw, and sh, do the following:

$ edquota -p mc slp tw sh

File limit warnings with quotas

When the user creates a file that goes over their soft limit, they will get the following warning:

quota_ufs: Warning: over disk limit (pid 1738, uid 101, inum 94, fs /export/home)

Note that the user is given seven days to correct the problem; you can alter the duration by using edquota -t.

If the user tries to create a file that pushes them over the hard limit, the system terminates the write process and truncates the file to the limit:

$ mkfile 210000k overlimit
quota_ufs: over hard disk limit (pid 1843, uid 101, inum 130, fs
 /export/home)
overlimit: initialized 191873024 of 215040000 bytes: Disc quota exceeded

Any user can check their own quota limits and disk usage by running the quota command:

$ quota
Over disk quota on /export/home, remove 199993K within 7.0 days

Quota administration

The systems administrator can check the quota of any user using the quota command; you should use the -v command-line option to provide a full report on the file system, usage, and limit information (see Listing 8).

Listing 8. Using the -v option
$ quota -v mc
Disk quotas for mc (uid 101):
Filesystem     usage  quota  limit    timeleft  files  quota  limit   
 timeleft
/export/home  399993 200000 400000    6.9 days    151      0      0

To get a report about disk and quota usage use on a file system basis detailing all users, use the repquota command, specifying the file system to report (see Listing 9).

Listing 9. Using the repquote command
$ repquota -v /export/home 
/dev/dsk/c0t0d0s7 (/export/home):
                      Block limits                  File limits
User           used   soft   hard  timeleft  used   soft   hard timeleft
mc        +- 399993 200000 400000  6.9 days   151      0      0

To ensure that the quota information is kept up to date, you should use the quotacheck command. This validates the file storage data with the quota information used to report quota information. You should run this automatically using cron -- once a day should be fine (the process is comparatively time consuming).


Quota tricks and traps

The quota system provides the best combination of monitoring and automatic administration, but you should be careful about the file systems on which you enable quotas and the limits that you set, as they might actually end up hindering the work of your users rather than controlling their disk usage.

For example, programmers might need significantly more space to actually build an application than you might have configured for their use. You can get around this issue by providing an unrestricted area for compilation (by setting an alternative temporary directory) while retaining quotas on their home directories, or by setting appropriate soft limits while setting the hard limits very high -- maybe even the maximum size of the file system.

The result should be a warning when the soft limit is reached, but no enforcement of the hard limit. Users should still be able to build and create temporary files but, as they are naturally deleted during the build process, they should never be prevented from creating the files they need.

Quotas can also be used to help monitor and alert you to usage of special user accounts. I have, in the past, used quotas to monitor Web user accounts, so called nobody accounts and others, to ensure that they do not have the ability to write files to file systems they should not otherwise have access to. To do this, set the hard limit to 1KB for the file system and user you want to protect.


Automatic monitoring

Monitoring disk space usage by hand is fine, but you don't want to be constantly running df (or even du) to determine the disk space being used, or that is available. You can automate the process and automatically send the administrator (or an administration group) an e-mail when the available space gets below a certain level. The script in Listing 10 monitors disk space, and you can set the warning (warninglimit) and emergency (lowlimit) limits, as well as the list of file systems that are checked.

Listing 10. Monitoring disk space
#!/bin/sh

warninglimit=500000
lowlimit=250000

filesystems="/export/data /export/home /"

for fs in $filesystems
do
        size=`df -k $fs|grep $fs|awk '{ print $4; }'`
        if [ $size -le $lowlimit ]
        then
                mailx -s "URGENT: Low disk space for $fs ($size)"
                break
        fi
        if [ $size -le $warninglimit ]
        then
                mailx -s "WARNING: Low disk space for $fs ($size)"
fi
done

The key is the line that extracts the amount of free disk space for each file system:

size=`df -k $fs|grep $fs|awk '{ print $4; }'`

The script uses df, extracts only the line you want with grep, and then uses awk to extract the fourth column of data, which is the amount of free space.

You can then check the free space against the warning/low limits and generate a suitable error. To prevent the system from raising both a low and a warning error, the script checks for a low warning first and uses break to skip to the next file system in the loop before the warning test is tried.


Summary

Monitoring disk space is a vital part of your job as an administrator; run out of disk space and you could seriously impair the work of your users and, in severe cases, lose data, or crash systems as they run out of the vital disk space required to operate.

You can determine the free space and usage for entire file systems using df, but this only gives you part of the picture. To determine where that disk space is being used, you need to use the du tool to study individual directories. Through the use of find, you can even use du to find out how much space specific users are using. For more automated user-level disk usage and control, quotas are a much better option.

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=128464
ArticleTitle=System Administration Toolkit: Monitoring disk space and usage
publish-date=06132006