Significant disk usage or full /var file path, or SQL30108N error

This scenario illustrates a failure of the IBM® Tivoli® System Automation for Multiplatforms recovery resource manager daemon, which serves as the decision engine for Tivoli SA MP and is identified as IBM.RecoveryRM in the system. If the daemon fails, symptoms include significant disk usage or a full /var file path, or an SQL30108N

There are two cases for this scenario.

Case 1: Significant disk usage (or full) '/var' file system

The following scenario shows the symptoms for this error:

If the IBM Tivoli System Automation for Multiplatforms recovery resource manager daemon fails during runtime, Tivoli SA MP will automatically try to restart the daemon so that end users or applications are not affected and database processing will continue. There will not be any noticeable symptoms. Tivoli SA MP will write diagnostic information into /var/ct/db2domain/log/mc/ (error logs) and /var/ct/db2domain/run/mc/ (core dumps) and /tmp/db2_cluster_manager_spooling (default trace directory). Check the var/... directories for an accumulation of diagnostic data.

The following instructions give details on diagnosis and resolution:

If you see a continuous accumulation of diagnostic data written into the aforementioned directories, it is safe to archive old diagnostic data to an archive location. For example
- mv /var/ct/db2domain/run/mc/IBM.RecoveryRM/trace.6.sp /archivepath where /var/ct/db2domain/run/mc/IBM.RecoveryRM/trace.6.sp is a Tivoli SA MP diagnostic destination path
  Note: /archivepath is an arbitrary file system
It is important to monitor the /var/ct/db2domain/log/mc/ and /var/ct/db2domain/run/mc/ directories on a regular basis and maintain free space of at least 3 GB for the /var file system.
IBM service and development teams use trace and core files for troubleshooting. For IBM Technical Support to analyze the diagnostic data, obtain a db2support package by running the following command on each member in the cluster
- db2support output_directory -d database_name -s
Follow these instructions to upload data to IBM Technical Support:
- Submitting diagnostic information to IBM Technical Support for problem determination
The IBM Technical Support website is a good source of information, where you can identify known problems based on symptoms or error log messages
- Linux® UNIX and Windows support.

Case 2: SQL30108N

This case occurs if restarting the daemon fails with an error, resulting in a restart light onto another host in the cluster.

The following scenario shows the symptoms for this error:

The application returns an SQL30108N error message.
Check for an accumulation of diagnostic data under the /var file system. Tivoli SA MP will write diagnostic information into /var/ct/db2domain/log/mc/ (error logs) and /var/ct/db2domain/run/mc/ (trace and core dumps) and /tmp/db2_cluster_manager_spooling (default trace directory). .

The following instructions give details on diagnosis and resolution:

Check the db2diag log file for messages similar to the following one:
- ```
DATA #6 : String, 48 bytes
Line # : 6884---2610-403 The resource is stale.
```
  or
- ```
DATA #6 : String, 142 bytes
Line # : 9578---2610-422 Cannot execute the command on
 node coralxib38.torolab.ibm.com. The resource manager 
IBM.RecoveryRM is not available.
```
  If you see the previous errors, this indicates that the Tivoli SA MP recovery resource manager daemon experienced a problem. Diagnostic data will be written by Tivoli SA MP to diagnose the problem.
If there is a continuous accumulation of diagnostic data written into the /var/ct/db2domain/log/mc/ and /var/ct/db2domain/run/mc/ directories, it is safe to archive old diagnostic data to an archive location. For example
- mv /var/ct/db2domain/run/mc/IBM.RecoveryRM/trace.6.sp /archivepath where /var/ct/db2domain/run/mc/IBM.RecoveryRM/trace.6.sp is a Tivoli SA MP diagnostic destination path
  Note: /archivepath is an arbitrary archive file system
It is important to monitor the /var/ct/db2domain/log/mc/ and /var/ct/db2domain/run/mc/ directories on a regular basis and maintain free space of at least 3 GB for the /var file system.
IBM service and development teams use trace and core files for troubleshooting. If you would like IBM Technical Support to analyze the diagnostic data, obtain a db2support package by running the following command on each node in the cluster
- db2support output_directory -d database_name -s
Follow these instructions to upload data to IBM Technical Support:
- Submitting diagnostic information to IBM Technical Support for problem determination
The IBM Technical Support website is a good source of information, where you can identify known problems based on symptoms or error log messages
- Linux UNIX and Windows support.