Client memory requirements and ulimit settings for incremental backups

The amount of memory that is used for normal incremental backups is proportional to the number of files that are being examined for backup. When you start an incremental backup operation that backs up numerous files, one way to prevent the client from running out of usable memory is by setting the operating system ulimit data value (ulimit -d) to unlimited.

If your business policies do not support a ulimit value of unlimited, you can estimate the system memory that is required for incremental backups by using the following calculations, and then set the ulimit to an appropriate value. Alternatively, you can set the MEMORYEFFICIENT DISKCACHEMETHOD client option or use journal-based backups to reduce memory requirements.

Tip: Your file system configuration can affect memory use.

To estimate the memory that is needed for an incremental backup, which is based on the number of objects (files and directories) that exist on the file system, complete the following steps:

Multiply the total number of objects by 700 to get an estimated number of bytes in the file system. The 700 is an estimate for the number of bytes in each path. For example, if the number of objects is 500000, then 500000 x 700=350000000.
Tip: The multiplier that is used in these calculations (700) is an estimate of the amount of memory that is needed per object. If your files and directories have file names that are longer than 80 characters, you might need extra memory.
Round up the value by 33% or to the next 100 MB from the previous step. In this example, round the 350 MB value to 500 MB. Convert this value to KB (400 x 1024=409600 KB).
If you have more than one file system, estimate the memory requirements for each file system, and use the highest of these estimates to set the ulimit data value.

This estimate works when the client RESOURCEUTILIZATION option is set to 4 or less. The estimation is part of a producer session. A producer session is a producer thread that scans the file system to search for changed, new, or deleted files. A producer session uses memory, and the number of producer sessions is important for calculating random access memory (RAM).

If you use a RESOURCEUTILIZATION value of 5, 6, or 7, you can have up to two concurrent producer sessions. RESOURCEUTILIZATION values of 8 and 9 can have up to three concurrent producer sessions. If RESOURCEUTILIZATION 10 is set, you can have up to four concurrent producers. You must base the ulimit value on the sum of the number of objects in each of the two, three, or four file systems that have the highest number of objects.

For example, in the following scenario, you have:

/fs1 contains 500000 objects
/fs2 contains 400000 objects
/fs3 contains 50000 objects
/fs4 contains 40000 objects

If you specify RESOURCEUTILIZATION 6, you can have up to two concurrent producer sessions. Therefore, you must calculate the ulimit value for the file systems with the highest number of objects (500000 and 400000):

(500000 + 400000) * 700 = 630000000 ~= 630 MB. Round up to 700 MB, then convert to KB = 716800. Set the ulimit value to 716800.

Tip: If the number of objects on the file system increases, you must readjust the ulimit value to accommodate the growth.

If you do not set the ulimit value to unlimited, or to an estimated value, you can use the following methods to reduce the memory that is needed for incremental backups:

MEMORYEFFICIENTBACKUP DISKCACHEMETHOD: This method uses disk space as if it were system memory. You might be able to use the default ulimit value, but you need free disk space that is available to process objects. For more information about estimating the disk space that is needed for this option, see the backup-archive client documentation. If disk space is limited, you can use the memoryefficientbackup yes option setting. This option uses less disk space that the DISKCACHEMETHOD option, but it does decrease incremental backup performance.
Journal-based backup: This method uses journal-based backups. The journal daemon records changes to an object or its attributes in a journal database. During a journal-based backup, the client obtains a list of files that are eligible for backup from the journal database instead of by scanning the entire file system. Journal-based backups reduce the memory that is needed to process incremental backups.