Message Broker Agent: Troubleshooting large history files filling up the file system

Body

The history files for the ITCAM Message Broker agent can become large very quickly. Warehousing is enabled for history collection, and Warehouse Proxy agent is working fine, but all history files continue to grow.

A fast growing history file can fill up file system. Agent needs sufficient disk space to perform trimming of the history files, and when this space is not available, trimming fails, and all history files start growing, making the problem worse. To control the history file size, the history collection settings need to be modified to control the problem history file size.

Following 'ls -l' output shows file sizes of some of the history files for a IIB agent (qi)
/opt/IBM/ITM/ls3263/qi/hist/AGT

-rw-r--r--     1  mqm mqm  223387648 May 22 23:00 KQITRSJV                   
-rwxrwxrwx 1 mqm mqm        510 May  8 21:42 KQITRSJV.hdr              
-rw-r--r--    1 mqm mqm 3078807552 May 22 23:00 KQITRSPS           <--- large file       
-rwxrwxrwx 1 mqm mqm        608 May  8 21:42 KQITRSPS.hdr               
-rw-r--r--     1 mqm mqm  362799104 May 22 23:00 KQITRSSP                   
-rwxrwxrwx 1 mqm mqm        550 May  8 21:42 KQITRSSP.hdr               
                                                                        
host_qi_hexnumber-01.log has following log entry, indicating that it is unable to trim history files.

(573B2559.0011-6:khdxhist.cpp,2367,"trimHistoryFile") Unable to trim history file. Table <KQITRSSP>. CopyHistoryFile status = 29

Troubleshooting

1) In the history files listed above, the KQITRSPS file is the largest in size, and corresponding history collection configuration needs to be investigated. The next step is to identify the attribute group which corresponds to this history table.

The attribute group and corresponding history file name are found in Historical data tables section of the agent User Guide:
https://www.ibm.com/support/knowledgecenter/SS3JRN_7.2.1.1/com.ibm.itcama.doc_7.2.1.1/kqi_userguide/r-dskspace_table.html

You can also find this information in the summarization and pruning agent log named host_sy_java*-01.log on the sy agent installation host - which lists all associations for the table and attribute groups, such as:
............Table : KQITRSPS, Parsers_Resource_Statistics
............Table : KQITRSSP, SOAPInput_Resource_Statistics
............Table : KQITSTBR, Broker_Statistics

Once the attribute group which is causing large history file is identified, Parsers_Resource_Statistics in this case, locate the corresponding collection in history configuration panel. Review the collection settings, then determine whether increasing the collection interval or disabling the history collection for the agent is preferred. Delete the large history file from the file system before restarting collection with increased interval, or after disabling the history collection.         

2) To prevent the history files from growing indefinitely, set limit on short term history file size as described in the following document:                                                      
     Limiting the growth of short-term history files:                        
     https://www.ibm.com/support/knowledgecenter/SSTFXA_6.3.0.2/com.ibm.itm.doc_6.3fp2/adminuse/historyshortterm_limitgrowth.htm                          

In every agent configuration file where agent history files are to be limited in size, set KHD_TOTAL_HIST_MAXSIZE and KHD_HISTSIZE_EVAL_INTERVAL environment variables as described.

This will prevent the history file size from increasing indefinitely. Gaps in history will occur if the file size reaches size limit set, but the problem of file system out of space will be prevented.