Question & Answer
This document describes sparse files, exposure due to sparse files, and the effects of certain commands on sparse files. This document applies to all versions of AIX.
Creating a sparse file
The effect of certain commands on sparse files
Many applications, particularly databases, maintain data in sparse files. A sparse file is a file with empty space, or gaps, left open for future addition of data. If the empty spaces are filled with the ASCII null character and the spaces are large enough, the file will be sparse, and disk blocks will not be allocated to it.
This creates an exposure: a large file will be created, but the disk blocks will not be allocated. Then, as data is added to the file, the disk blocks will be allocated but there may not be enough free disk blocks in the file system. Then the file system will be full and writes to any file in the file system will fail.
You can prevent these problems by either assuring that you have no sparse files on your system or by planning to have enough free space in the file system for the future allocation of the blocks.
You also need to be aware of how you manipulate sparse or potentially sparse files because you can easily change them from sparse to not sparse or vice-versa.
An example sparse file can be created fairly easily. To do this, open the file, seek to a large address, and write some data. This can be demonstrated with the dd command, as follows:
- First, create a regular file:
date > notsparse ls -l
The output of the ls command will be similar to:
total 8 -rw-r--r-- 1 root sys 29 Dec 21 08:12 notsparse
- Use the
fileplacecommand to see how many allocated and deallocated blocks are included in the file notsparse.
(NOTE: perfagent.tools must be installed to run the
fileplacecommand at AIX 4.x and 5.x.)
The output will look similiar to:
File: notsparse Size: 29 bytes Vol: /dev/lv03 Blk Size: 4096 Frag size: 4096 Nfrags: 1 Compress: no Logical Fragment ---------------- 00716 1 frags 4096 bytes, 100.0%
(NOTE: Performance Analysis and Control Commands [perfagent.tools] must be installed to enable the
- The du command will also reflect how many 512-byte blocks a file occupies.
du -rs *
- Now create a sparse file using the regular file notsparse as input:
touch sparse.1 dd if=notsparse of=sparse.1 seek=100
dd: 0+1 records in. dd: 0+1 records out.
The dd command takes the data from the regular file and places it 100 512-byte blocks into the sparse.1 file. Note that nothing is written to the initial 99 512-byte blocks. The following steps show the characteristics of the resulting file.
- The ls command reports the distance from block zero to the last block in the file:
total 16 -rw-r--r-- 1 root sys 29 Dec 21 08:12 notsparse -rw-r--r-- 1 root sys 51229 Dec 21 08:13 sparse.1
fileplacecommand tells the story accurately - there are 12 deallocated 4K blocks and one allocated 4K block in the file:
File: sparse.1 Size: 51229 bytes Vol: /dev/lv03 Blk Size: 4096 Frag Size: 4096 Nfrags: 1 Compress: no Logical Fragment ---------------- unallocated 12 frags 49152 Bytes, 0.0% 0000769 1 frags 4096 Bytes, 100.0%
- The du command reports the number of allocated blocks the file takes:
du -rs *
8 notsparse 8 sparse.1
backup and/or restore (by name and inode)
The restore command aggressively preserves sparseness. In fact, the restore command will deallocate any blocks filled with zeroes, thus making a file sparse.
cp command does not preserve the sparseness of a file.
When used within the same filesystem, the mv command simply renames the file. The inode and data blocks are not moved or altered. If used between two filesystems, however, the mv command does not preserve the sparseness of a file.
If you create a backup using the
cpio command on sparse files, you will need to use the pax command to restore that data. Using the
cpio command to restore the data will not preserve sparseness.
Using the dd command on the file itself does not preserve sparseness. However, using dd on the file system device does preserve the state of the individual files.
Example: Backing up a logical volume:
dd if=/dev/datalv of=/dev/rmt0 ibs=4096 obs=1024 conv=sync
The mksysb command uses backup/restore. See the section on backup/restore.
NOTE: The pax command can read tar archives and can read
cpio archives if the c flag was used.
The pax command aggressively preserves sparseness. In fact, the pax command will deallocate any blocks filled with zeroes, thus making a file sparse.
Sysback will use either backup by name or inode to backup the data on the system. See the section on backup.
If you create a backup using the tar command on sparse files, you will have to use the pax command to restore that data. Using the tar command to restore the data will not preserve sparseness.
03 December 2019