File synchronization performance tuning

Edit online

There are several ways to enhance file synchronization.

JFS file I/Os that are not sequential accumulates in memory until certain conditions are met:

  • The free list shrinks to minfree, and page replacement must occur.
  • The syncd daemon flushes pages at regularly scheduled intervals.
  • The sync command is issued.
  • Random-write behind flushes the dirty pages after random-write behind threshold is reached.

If many pages accumulate before one of these conditions occur, then when pages do get flushed by the syncd daemon, the i-node lock is obtained and held until all dirty pages have been written to disk. During this time, threads trying to access that file gets blocked because the i-node lock is unavailable. The fuser command is also blocked for the file, because thefuser command needs the inode lock to provide the required information. Remember that the syncd daemon currently flushes all dirty pages of a file, but one file at a time. On systems with large amount of memory and large numbers of pages getting modified, high peaks of I/Os can occur when the syncd daemon flushes the pages.

AIX has a tunable option called sync_release_ilock. The ioo command with the -o sync_release_ilock=1 option allows the i-node lock to be released while dirty pages of that file are being flushed. This can result in better response time when accessing this file during a sync() call.

This blocking effect can also be minimized by increasing the frequency of syncs in the syncd daemon. Change /sbin/rc.boot where it starts the syncd daemon. Then reboot the system for it to take effect. For the current system, kill the syncd daemon and restart it with the new seconds value.

A third way to tune this behavior is by turning on random write behind using the ioo command (see Sequential and random write behind performance tuning).

JFS2 Synchronization Tunables

The file system synchronization operation might not be effective in situations when there is random I/O activity to a large file. When a sync occurs, all reads and writes from user programs to the file are blocked. With many dirty pages in the file, a considerable amount of time is required to finish writing to the disk. The following JFS2 tunable parameters can be used in such scenarios:
  • j2_syncPageCount: Limits the number of modified pages that are scheduled to be written by sync in one pass for a file. When this tunable is set, the file system writes the specified number of pages without blocking I/O to the rest of the file. The sync call iterates on the write operation until all modified pages are written.
  • j2_syncPageLimit: Overrides j2_syncPageCount parameter when a threshold is reached. Use this parameter to ensure that the sync operation is complete for a file. The tunables are maintained in the standard way.

The tunables are manipulated by using the ioo command.

The j2_syncPageCount and j2_syncPageLimit tunables are added to the list of values that are controlled by the ioo command.

Use the –o flag to display or change the value, and to view the help option use the –h flag. 
# ioo –h j2_syncPageCount
Sets the maximum number of modified pages of a file that is written to disk by the sync system call in a single operation.

Values: Default: 0 Range: 0-65536

Type: Dynamic

Unit: 4 KB pages

Tuning: When running an application that uses file system caching and does large numbers of random writes, it is necessary to adjust this setting to avoid lengthy application delays during sync operations. The values must be in the range of 256 to 1024. The default value is zero that results in the normal sync behavior of writing all dirty pages in a single call. If small values for the tunables are set, it results in longer sync times and shorter delays in application response time. If larger values are set, then response time delays are longer and sync times are shorter. 

# ioo –h j2_syncPageLimit
Sets the maximum number of times the sync system call uses the j2_syncPageCount, to limit pages that are written to improve the sync operation performance.

Values: Default: 256 Range: 16-65536

Type: Dynamic

Unit: Numeric

Tuning: Is set when j2_syncPageCount is set and must be increased, if the effect of the j2_syncPageCount change is insufficient. The acceptable values are in the range of 250 to 8000. j2_syncPageLimit has no effect if j2_syncPageCount is 0.

This tunable must be set when j2_syncPageCount is set and must be increased, so that the effect of thej2_syncPageCount change does not reduce the application response time.

The values must be in the range of 1 to 8000. Optimum value for these tunables is dependent on the memory size and I/O bandwidth. A neutral starting point is to set both these tunables to 256.

JFS2 synchronization period and concurrency

File system synchronization is managed by the sync daemon (syncd). Use JFS2 tunable parameters to allow the file system to handle synchronization without using the syncd.

JFS2 sync handler spreads the sync process such that all cached data is not written to disk at the same time. The sync is performed on a single file system at a time. Each file system is scheduled to start the next sync operation after the previous operation ends. You can also increase the number of threads that are handling sync operation when multiple file systems to must be processed.

Use following tunable parameters for JFS2 file system synchronization:

j2_syncByVFS:
Specifies use of the JFS2 sync handler and sets the interval between sync operation for each file system.
j2_syncConcurrency:
Sets the number of threads that are handling file system synchronization. This value indicates the number of file systems on which sync operation must be performed concurrently. Only one sync thread performs sync operation on an individual file system.

ioo command manages I/O tunable parameters. Refer documentation of ioo command for more information.

# ioo –h j2_syncByVFS

Purpose: Specifies the number of seconds to wait between system calls to sync a JFS2 file system. This value supersedes the value specified by the syncd command.

Values: Default: 0, Range: 0-86400.

Type: Dynamic.

Units: Seconds.

Tuning: This value indicates the number of seconds between iterations of the sync process. The JFS2 sync handler until time specified by the j2_syncByVFS tunable parameter before initiating calls to the syncvfs subroutine for JFS2 file systems. A value of 0 indicates that the normal syncd processing should be used. A non-zero value overrides the time that is specified on the syncd command and causes the JFS2-specific file sync handler to be used.

# ioo –h j2_syncConcurrency

Purpose: Sets the number of threads to be used for JFS2 sync operations.

Values: Default: 1, Range: 1-128.

Type: Dynamic.

Unit: Numeric.

Tuning: The sync daemon start sync operations concurrently for the number of file systems set by the j2_syncConcurrency tunable parameter. This vale is effective only when the j2_syncByVFS tunable parameter is nonzero.