As ITM 6.3 fix pack 3 has just been released, I know many of you will be planning upgrades in the near future. Here are a few tips and features that can really improve the speed, capacity usage and capabilities of warehousing in ITM. There is even a neat little feature to send data directly to analytics tools outside of the warehouse entirely, thus offsetting load on your database (win, win).
This blog is tacking a slightly different tact to the how to/deep dive information I usually post, but I feel this information if worth blogging about, as I have set this up and seen it in action on big databases and it makes HUGE positive difference to the warehousing process. I presented the below information and the London Tivoli user group this year, so if you have any questions post them below.
Here is an overview of what I will run through:
Write incoming data to CSV files
Easily consumable data for analytics tools
Ability to export historical data as CSV files, before writing to database.
Faster data uploads to TDW
Low latency, high frequency data uploads for analytics tools
Ability to specify faster data upload interval (1 min and 5 min)
Historical data to .CSV files
- Offset load from the warehouse of your analytics tools
I know this is geeky, but this is cool! This feature allows you to send data to the Tivoli Data Warehouse (TDW) and/or a .CSV file. Still thinking why is this cool...?
This feature offsets the resource and workload that data hungry analytics tools use. As their data is being read from a CSV file (at a location of your choice) there is no added load on your already busy TDW server. Anyone that has analytics tools running agaianst the TDW, I would recomend giving this a try. set it up in one of your Dev/test environments and you will see an improvement on response times and load compared to when the anaylitics tools are feeding from the TDW.
Normal historical data flow model is:
Agents -> WPA -> Warehouse <-- Data tools access the data from the TDW
New Data model
New CSV alibility allows the WPA to send data to both the warehouse and output to a CSV file
Agents -> WPA -> Warehouse & CSV file at the same time (or just to one CSV or TDW)
Please note: A TDW connection is still required, even if only writing data to CSV files
How to set up the WPA to use CSV files
In the file hd.ini set:
–Data sent to CSV only :
–Data sent only to TDW:
–Data sent to CSV and TDW:
Mark's notes on setting this up from on site
Make sure you have:
- Have sufficient disk space on the file system where the files will be written.
- Have an external application set up to consume and delete the files (I have seen files grow to silly sizes so this is important to check).
- Monitor the file system to ensure data is being consumed correctly with no time gap in the data.
Key parameters broken down in detail
Required. Use Y to enable CSV output and N to disable CSV output. The default is N
This is optional.
If specified, only data that has been tagged for analytic use is written to the CSV files. The KHD_CSV_OUTPUT_LIST variable is ignored if this variable is enabled. The default is Y.
Optional. The maximum total size in MB of the CSV files.
When this threshold is exceeded, the Warehouse Proxy agent stops writing data to the CSV files at the next evaluation interval. In IBM® Tivoli Monitoring V6.3 and later, a warning message is logged in the Warehouse Proxy agent logs and an audit message is displayed. Files must be deleted to resume writing.
This checking cannot be disabled. The default is 10 MB.
Required. The directory where the CSV files are written. If the directory is not specified or does not exist, a message is logged to the trace file, an audit message is generated, and the CSV output is disabled
Further reading on warehousing and CSV usage:
Link to IBM documentation on this feature
Faster Uploads to the TDW
Before ITM 6.3 FP02 the lowest upload interval you could set was 15 minutes for sampled data (being uploaded to the warehouse) .
This wasn't cutting it for the data hungry analytitics tools out there. They were all chanting "how low can you go" agian and again. Luckily the ITM development team like a challenge and they have now managed to reduce the the intervals to 1 or 5 minutes in both autonomous and non autonomous modes.
Mark's notes from setting this up:
If data is being used only for TDW or analytical consumption, there is no need to retain it at the agent as you will not be making TEPS calls for data. it is now possible and recommended with short upload intervals to delete the short term history file using KHD_HISTRETENTION=0 .This will help reduce the resource utilisation in the agent (CPU and disk).