Disk or memory-related problem

Edit online

Just as a large fraction of real memory is available for buffering files, the system's page space is available as temporary storage for program working data that has been forced out of RAM.

Suppose that you have a program that reads little or no data and yet shows the symptoms of being I/O dependent. Worse, the ratio of real time to user + system time does not improve with successive runs. The program is probably memory-limited, and its I/O is to, and possibly from the paging space. A way to check on this possibility is shown in the following vmstatit shell script:

vmstat -s >temp.file   # cumulative counts before the command
time $1                # command under test
vmstat -s >>temp.file  # cumulative counts after execution
grep "pagi.*ins" temp.file >>results   # extract only the data
grep "pagi.*outs" temp.file >>results  # of interest

The vmstatit script summarizes the voluminous vmstat -s report, which gives cumulative counts for a number of system activities since the system was started.

If the shell script is run as follows:

# vmstatit "cp file1 file2"  2>results

the result is as follows:

real    0m0.03s
user    0m0.01s
sys     0m0.02s
     2323 paging space page ins
     2323 paging space page ins
     4850 paging space page outs
     4850 paging space page outs

The before-and-after paging statistics are identical, which confirms our belief that the cp command is not paging-bound. An extended variant of the vmstatit shell script can be used to show the true situation, as follows:

vmstat -s >temp.file
time $1
vmstat -s >>temp.file
echo "Ordinary Input:"               >>results
grep "^[ 0-9]*page ins"    temp.file >>results
echo "Ordinary Output:"              >>results
grep "^[ 0-9]*page outs"   temp.file >>results
echo "True Paging Output:"           >>results
grep "pagi.*outs"          temp.file >>results
echo "True Paging Input:"            >>results
grep "pagi.*ins"           temp.file >>results

Because file I/O in the operating system is processed through the VMM, the vmstat -s command reports ordinary program I/O as page ins and page outs. When the previous version of the vmstatit shell script was run against the cp command of a large file that had not been read recently, the result was as follows:

real    0m2.09s
user    0m0.03s
sys     0m0.74s
Ordinary Input:
    46416 page ins
    47132 page ins
Ordinary Output:
   146483 page outs
   147012 page outs
True Paging Output:
     4854 paging space page outs
     4854 paging space page outs
True Paging Input:
     2527 paging space page ins
     2527 paging space page ins

The time command output confirms the existence of an I/O dependency. The increase in page ins shows the I/O necessary to satisfy the cp command. The increase in page outs indicates that the file is large enough to force the writing of dirty pages (not necessarily its own) from memory. The fact that there is no change in the cumulative paging-space-I/O counts confirms that the cp command does not build data structures large enough to overload the memory of the test machine.

The order in which this version of the vmstatit script reports I/O is intentional. Typical programs read file input and then write file output. Paging activity, on the other hand, typically begins with the writing out of a working-segment page that does not fit. The page is read back in only if the program tries to access it. The fact that the test system has experienced almost twice as many paging space page outs as paging space page ins since it was booted indicates that at least some of the programs that have been run on this system have stored data in memory that was not accessed again before the end of the program. Memory-limited programs provides more information. See also Memory performance.

To show the effects of memory limitation on these statistics, the following example observes a given command in an environment of adequate memory (32 MB) and then artificially shrinks the system using the rmss command (see Memory requirements assessment with the rmss command). The following command sequence

# cc -c ed.c
# vmstatit "cc -c ed.c" 2>results

first primes memory with the 7944-line source file and the executable file of the C compiler, then measures the I/O activity of the second execution:

real    0m7.76s
user    0m7.44s
sys     0m0.15s
Ordinary Input:
    57192 page ins
    57192 page ins
Ordinary Output:
   165516 page outs
   165553 page outs
True Paging Output:
    10846 paging space page outs
    10846 paging space page outs
True Paging Input:
     6409 paging space page ins
     6409 paging space page ins

Clearly, this is not I/O limited. There is not even any I/O necessary to read the source code. If we then issue the following command:

# rmss -c 8

to change the effective size of the machine to 8 MB, and perform the same sequence of commands, we get the following output:

real    0m9.87s
user    0m7.70s
sys     0m0.18s
Ordinary Input:
    57625 page ins
    57809 page ins
Ordinary Output:
   165811 page outs
   165882 page outs
True Paging Output:
    11010 paging space page outs
    11061 paging space page outs
True Paging Input:
     6623 paging space page ins
     6701 paging space page ins

The following symptoms of I/O dependency are present:

Elapsed time is longer than total CPU time
Significant amounts of ordinary I/O on the nth execution of the command

The fact that the elapsed time is longer than in the memory-unconstrained situation, and the existence of significant amounts of paging-space I/O, make it clear that the compiler is being hampered by insufficient memory.

Note: This example illustrates the effects of memory constraint. No effort was made to minimize the use of memory by other processes, so the absolute size at which the compiler was forced to page in this environment does not constitute a meaningful measurement.

To avoid working with an artificially shrunken machine until the next restart, run

# rmss -r

to release back to the operating system the memory that the rmss command had sequestered, thus restoring the system to its normal capacity.