IBM Common Data Provider for z Systems - Performance History and Improvements: 4Q18
CRF 3100006QVH Visits (3609)
The IBM Common Data Provider for z Systems allows you to stream operational data from z/OS to analytics engines such as Splunk and Elastic. I could go on and on about how this product works, but today I’m here to discuss the performance improvements that have been made over the past year, and how it impacts our customers. For more general information regarding IBM Common Data Provider for z Systems, I’ll
Configuration and Buffering:
One of the ways that we’ve made improvements that are immediately visible is in our data buffering. Previously, the JCL procedure that started the SDE component of IBM Common Data Provider for z Systems included a statement that was used to tune the buffer size available to the address space. The effort of tuning this, however, was not always simple – and was required for each individual environment that the SDE would run on. Through investigation and test efforts, we were able to make changes that negate the need to tune this parameter and removed the statement from the SDE startup procedure.
Further changes to buffering include the use of buffering in memory rather than on disk in the case of being unable to reach the Data Streamer’s subscriber. Doing this saves valuable CPU time by avoiding additional read/write operations
Some major changes have been made in the past year to improve data throughput and CPU consumption. The largest change made from a performance standpoint was how we package and send records from the SDE. In our original implementation of the SDE, records were sent to the Data Streamer address space individually. Even in scenarios where the IBM_SDE_INTERVAL was set to a large delay, such as 1 MINUTES, the buffered data would be individually packaged and sent to the Data Streamer address space 1-by-1. We changed this so that the SDE will package records into larger payloads, sending multiple records at once, which is much more efficient in terms of CPU consumption and network traffic.
Our customers' environments all differ vastly from each other, so we had to create a product that works for every environment without complicating required setup procedure.
The SDE is able to process SMF records, but these records can be complicated. Some of the SMF record types and subtypes may contain information that is valuable for one customer, while being not so important to another. What we’ve done is created a configuration that tells the SDE exactly which records to process based on the individual needs of our customers. Now, instead of processing each piece of every SMF record, we target only the information that a user desires, as indicated in the configuration UI. For simple SMF record types such as SMF_030, this isn’t incredibly noticeable. In other more complicated SMF record types, such as SMF_110, the power to selectively process these records is clearly impactive. The reduction in data volume saves a sizeable amount of CPU time on the host, and disk on the subscriber.
2018 has been a good year for performance improvements in IBM Common Data Provider for z Systems, and we're gearing up to continue this through next year and into the future. Hopefully the information discussed here was able to provide some insight towards our dedication in making performance improvements.
Of course, as always, I’m happy to answer any questions you have – so reach out in the comments and let’s have a conversation!