ItmWarehouse: Short Term History files - less is more

Body

You have just bought your first smart phone. You then have to decide on how to use 'less data' using your data plan(unless you are on an unlimited plan). You google 'how to use less data' and see there are thousands of posts on how to do this. In terms of Tivoli ITM warehousing, the same concept applies, how to keep the short term history(STH) files at a reasonable size. This requires for large data intensive agents to think about the configuration of the historical collection interval, warehousing upload interval, and retention period. All three of these are factors in the size of the STH file.

The historical collection interval is the time period that the agent saves data in the STH file. This setting is probably the most important setting in that the other two factors are for removing the data from the STH file. The historical collection interval is best determined by using the Warehouse Planning spreadsheet, discussed in this blog entry: ItmWarehouse: Big Data and Tivoli Data Warehousing.

The warehousing upload interval is the time for uploading the collected short term historical data to the warehouse database. Once the data is uploaded, it then can be deleted from the STH file based on the retention period. Again the Warehouse Planning spreadsheet is invaluable in helping with this setting. Another good article to read for over all historical collection practices is: Historical Collection Best Practices

The STH file retention interval is the time that the data is kept in the STH file before being pruned. The default is 24 hours. For data intensive tables this usually is to long a period to keep the data. For some customers, the STH file retention time is only 1 hour, so as to prune the data as quickly as possible. Now what is a reasonable size for the STH file? Based on over all performance studies, it has been found that the STH file size should not be greater than 1Gig bytes. The reason for this limit is that three processes are interacting with the STH file, the agent saving the data, the Warehouse Proxy agent uploading the data, and the pruning process deleting the data in the STH file. Once the STH file grows larger than 1Gig bytes then these processes start failing to complete and the STH file starts growing out of control.

Thus, in the world of ITM historical collection there really is no unlimited data plan. In my next blog entry I will detail more on how to manage the STH file size in your infrastructure.