Avoiding process errors with large trace or log files by using log file rotation
If trace or log files become too large, processes might quit unexpectedly. If too many files are created, you might run out of disk space. Configure the number and size of trace and log files by configuring log file rotation.
You can control the size of log and trace files using one of the following criteria:
- Maximum size
- Time of day
You can configure the number of log files by configuring a pool of files, that is, a number of files that are written to in succession. The process of using new files when triggered is called log file rotation. All log files can be rotated, but not all components have log files. Some components have only trace files, which can also be rotated.
Setting the maximum size of log and trace files
Large log or trace files can cause process errors. The default maximum size for a log file is 1 GB on UNIX systems. As a guidance estimate for log files, assuming that each log file is 1 GB in size and six processes are set to full debug level, you would require 24 GB of disk space. (6 processes x 4 log or trace files each = 24 log or trace files x 1 GB = 24 GB).
Set the NDE_LOGFILE_MAXSIZE environment variable to determine the maximum size that a log or trace file can reach for a process. This setting applies to all Network Manager processes. When the log file reaches the maximum size, the next log file is used.
-
If log file rotation is turned off for a process, only two files are used.
-
If log file rotation is turned on for a process, the number of files configured in the pool are used.
You can either rotate log files according to file size or time of day. If log file rotation according to time of day is configured, the settings for file size are ignored.
setenv NDE_LOGFILE_MAXSIZE 1073741824
You must restart Network Manager processes after you change the log file rotation environment variables so that the processes use the updated variables.
Setting a time of day for log file rotation
To set log file rotation to occur at a certain time of day, set the NDE_LOGFILE_ROTATION_TIME environment variable to the time required, and set the NDE_LOGFILE_ROTATION_FORMAT environment variable to the required naming format. If you set a time of day, this setting takes precedence over the maximum file size setting.
The NDE_LOGFILE_ROTATION_TIME environment variable specifies the time after which a log file rotation occurs each day. Log files are rotated when the first write is made to the file after the time that is set by the NDE_LOGFILE_ROTATION_TIME environment variable. This environment variable is an integer, with a default of 0000.
setenv NDE_LOGFILE_ROTATION_FORMAT "%Y%m%d-%H%M"
setenv NDE_LOGFILE_ROTATION_TIME 0000
"old"
setenv NDE_LOGFILE_ROTATION_FORMAT \'old\'
setenv NDE_LOGFILE_ROTATION_TIME 0000
The NDE_LOGFILE_ROTATION_FORMAT environment variable configures how the log files are created.
- If NDE_LOGFILE_ROTATION_FORMAT is set to "old", and a pool of log files is not configured for the process in question, two files are used.
- If a pool of log files is configured for a process, the number of files in the pool is used.
- If NDE_LOGFILE_ROTATION_FORMAT is set to a supported time stamp format, and a pool of log files
is not configured for the process in question, a new file is created each day. Using a time stamp
format without configuring a pool of log files can result in an unlimited number of log files.Restriction: The only time stamp format that is supported for use by Network Manager processes is the POSIX format. The following options are supported:
- %a
- Abbreviated weekday name, for example,
Thu
- %A
- Full weekday name, for example,
Thursday
- %b
- Abbreviated month name, for example,
Aug
- %B
- Full month name, for example,
August
- %d
- Day of the month, zero-padded, (01-31), for example,
23
- %e
- Day of the month, space-padded, ( 1-31), for example,
23
- %h
- Equivalent to %b.
- %H
- Hour in 24h format (00-23), for example,
14
- %I
- Hour in 12h format (01-12), for example,
02
- %j
- Day of the year (001-366), for example,
235
- %m
- Month as a decimal number (01-12), for example,
08
- %M
- Minute (00-59), for example,
55
- %p
- AM or PM designation, for example,
PM
- %R
- 24-hour HH:MM time, equivalent to %H:%M, for example,
14:55
- %S
- Second (00-61), for example,
02
- %T
- ISO 8601 time format (HH:MM:SS), equivalent to %H:%M:%S, for example,
14:55:02
- %u
- ISO 8601 weekday as number with Monday as 1 (1-7), for example,
4
- %V
- ISO 8601 week number (00-53), for example,
34
- %y
- Year, last two digits (00-99), for example,
01
- %Y
- Year, for example,
2001
- %Z
- Timezone name.
For example, %Y%m%d-%H%M
generates rotate log files with the year, month, day,
hour, and minute added. An example file is ncp_model.NCOMS.log_20100430-0000
.
You must restart Network Manager processes after you change the log file rotation environment variables so that the processes use the updated variables.
Configuring the number of log files created per process
The number of log files created by a process is determined on a per-process basis by the
command-line variables -logfileusepool
and -logfilepoolsize
. If a
Network Manager process is
started with the -logfileusepool
command line option, then the process uses a pool
of log files. The number of files in the pool is defined by the value of the
-logfilepoolsize
command line option. If the -logfilepoolsize
command line option is not specified, the default value of 10 is used. The minimum value of
-logfilepoolsize
is 2 and the maximum is 99.
The log files in the pool are deleted each time the process is restarted by the user or by the system. If you want to keep historical logs you need to use the NDE_LOGFILE_ROTATION_FORMAT without a log pool.
Do not start the ncp_g_event or nco_p_ncpmonitor processes with the
-logfileusepool
command line option if you have set the NDE_LOGFILE_ROTATION_FORMAT
environment variable. Using log file pooling and a timed log file rotation together for these
processes is not supported.
Example of starting a process with log file pool specified
The following example excerpt from the CtrlServices.cfg configuration file configures the ncp_disco process to use a pool of 5 log files:
insert into services.inTray (
serviceName,
binaryName,
servicePath,
domainName,
argList,
dependsOn,
retryCount
)
values (
"ncp_model",
"ncp_model",
"$PRECISION_HOME/platform/$PLATFORM/bin",
"$PRECISION_DOMAIN",
[ "-domain" , "$PRECISION_DOMAIN", "-latency" , "100000" , "-debug", "0-", "-messagelevel", "warn", "-logfileusepool", "-logfilepoolsize", "5" ],
[ "ncp_config", "ncp_store", "ncp_class" ],
5
);
What happens when log files are rotated
- A number of log files are created for that process when it starts. The number of files that are
opened is configured by the value of the
-logfilepoolsize
command line option. The process keeps the log files open for as long the process runs. By contrast,.trace
files are only created when they are needed, and are not kept open. Files in the pool are named using the formatprocess[.domain].log_ID
orprocess[.domain].trace_ID
, whereID
is a two-digit number that starts from 01, and ranges up to the maximum number specified for the-logfilepoolsize
option. - If they already exist, the first N .log files are truncated (set to zero size), and the first N
.trace
files are deleted, where N is the number of log files specified for the-logfilepoolsize
option for the process. If there are more than N .log or .trace files, for example from a previous time when the log pool size was greater, the remaining files are not affected. If you want to be sure to keep all previous .log or.trace
files, copy them to a different location before starting or restarting the process.Note: An exception is the ncp_disco process. When the ncp_disco process is restarted (including on every full discovery) the old ncp_disco.DOMAIN.log_n files are renamed with an extension of .old appended to the file name - The file with the lowest available file number is used for logging, regardless of which file was last used.
- When the criteria for log rotation are met, the logging system writes to the next file.
- When all the files in the pool are at maximum size, the logging system truncates the first file in the pool and starts writing to it again.
- The ncp_ctrl process renames the log file from logfilename.log to the format set by the NDE_LOGFILE_ROTATION_FORMAT environment variable, or to logfilename.log_old if the NDE_LOGFILE_ROTATION_FORMAT environment variable is not set.
- The ncp_ctrl process generates a new log file named logfilename.log.
- When the new logfilename.log file reaches the maximum size, the ncp_ctrl process overwrites logfilename.log_old.