S3 logs

The mms3 command logs

The mms3 command writes all log messages to the /var/mmfs/s3 folder.

/var/mmfs/s3/mms3-<hostname>.log

An example of logs:

{"ts": 2024-07-22 11:08:19.911443, "level":error, "caller":cmd/config.go:596, "msg":Failed to backup as S3 _ces_s3.master_keys file does not exist at path: /mnt/cesSharedRoot/ces/_ces_s3.master_keys}

{"ts": 2024-07-22 11:08:47.945496, "level":error, "caller":cmd/config.go:482, "msg":Failed to fetch master keys: error fetching master keys: get file failed: Name not found on get (err 804)

}

{"ts": 2024-07-22 11:08:47.945773, "level":error, "caller":cmd/config.go:602, "msg":Failed to backup as S3 _ces_s3.master_keys file is empty at path: /mnt/cesSharedRoot/ces/_ces_s3.master_keys}

{"ts": 2024-07-23 11:25:07.423525, "level":debug, "caller":utils/utils.go:76, "msg":CallerPath of File utils/utils.go}

{"ts": 2024-07-23 11:25:07.423603, "level":debug, "caller":utils/utils.go:156, "msg":Running Command /usr/lpp/mmfs/bin/mmlsconfig cesSharedRoot}

{"ts": 2024-07-23 11:25:07.612137, "level":debug, "caller":utils/utils.go:275, "msg":Running Command /usr/lpp/mmfs/bin/mmces service list}

{"ts": 2024-07-23 11:25:08.000118, "level":debug, "caller":utils/utils.go:340, "msg":Running Command /usr/lpp/mmfs/bin/mmces node list}

{"ts": 2024-07-23 11:25:08.358902, "level":debug, "caller":utils/utils.go:77, "msg":Running Command: env LC_ALL=C /usr/local/bin/noobaa-cli bucket 2>/dev/null list >> /tmp/tempBucket-20240723_112507.json}

{"ts": 2024-07-23 11:25:25.887730, "level":error, "caller":definitions/buckets.go:279, "msg":Failed to list Bucket LOG_LEVEL=info: Bucket does not exist}

S3-related log information is written to the standard GPFS log files as part of the overall CES infrastructure. This information relates to the S3 service management and recovery orchestration within CES.

To display the currently configured CES log level, use the following command:
```
 mmces log level
```
A sample output is as follows:
```
CES log level is currently set to 0
```
The log file is /var/adm/ras/mmfs.log.latest. By default, the log level is 0 and other possible values are 1, 2, and 3.
To increase the log level, use the following command:
```
 mmces log level 1
```
To display log levels, use the following command:
```
 mms3 config list | grep DEBUGLEVEL
```
A sample output is as follows:
```
DEBUGLEVEL: default
```
By default, the log level is set to default. Supported log debugging levels are as follows:
- default: For default debugging
- warn: For only warning or errors debugging
- nsfs: For more nsfs detailed debugging
- all: For all detailed debugging
  Note: The all level increases the size of the log file. Use it in the production mode only if instructed by the IBM® Support.

LOG_TO_SYSLOG_ENABLED : true: By using these parameters, you can view the log messages in /var/log/messages. This flag enables syslog logging for the application.
LOG_TO_STDERR_ENABLED : true: By using these parameters, you can view the log messages in /var/log/noobaa.log. This flag specifies whether logs will be pushed stderr or not.

Logrotate for the mms3 command logs

The mms3 logrotate logging is enabled with default configurations as shown in the following example. The logrotate mms3-config.json configuration file is located under the ces directory of the CES shared root path.

mms3-config.json

***************

{

  "MaxSize": 5,

  "MaxBackups": 50,

  "MaxAge": 180,

  "Compress": true,

  "EnableDebug": false

}

MaxSize: Is the maximum size in megabytes of the log file before it gets rotated. The default value is 5 MB.
MaxBackups: Is the maximum number of old log files to retain. The default value is 100.
MaxAge: Is the maximum number of days the old log files will be retained based on the timestamp encoded in their filename.
Compress: Determines if the rotated log files should be compressed by using gzip.
EnableDebug: If EnableDebug is true, the log is generated with the enabled debug mode. If EnableDebug is false, the debug message are be logged.

S3 service logs

The S3 service that is running on each CES node writes all log messages to the /var/log/noobaa.log file.

/var/log/noobaa.log

An example of logs:

Jul 24 10:21:51 challenger-21 node[2018336]: [/2018336] [LOG] CONSOLE:: read_rand_seed: closing fd ...
Jul 24 10:21:51 challenger-21 node[2018336]: [/2018336] [LOG] CONSOLE:: init_rand_seed: seeding with 32 bytes
Jul 24 10:21:52 challenger-21 node[2018516]: [/2018516] [LOG] CONSOLE:: read_rand_seed: reading 32 bytes from /dev/urandom ...
Jul 24 10:21:52 challenger-21 node[2018516]: [/2018516] [LOG] CONSOLE:: read_rand_seed: got 32 bytes from /dev/urandom, total 32 ...
Jul 24 10:21:52 challenger-21 node[2018516]: [/2018516] [LOG] CONSOLE:: read_rand_seed: closing fd ...
Jul 24 10:21:52 challenger-21 node[2018516]: [/2018516] [LOG] CONSOLE:: init_rand_seed: seeding with 32 bytes