Forwarding logs to analytics platforms

You can use Transaction Analysis Workbench to forward logs in comma-separated values (CSV) or JavaScript Object Notation (JSON) format to various analytics platforms.

About this task

To convert logs to CSV or JSON, use the CSV or JSON command of the Transaction Analysis Workbench report and extract utility.

The CSV and JSON commands can write CSV or JSON data to MVS data sets, z/OS® UNIX files, or TCP sockets. Writing to a TCP socket is also known as streaming. To stream data, you combine the CSV or JSON command with a STREAM command.

To run these commands, you can either write JCL yourself or use option 5 Analytics of the Transaction Analysis Workbench ISPF dialog to create JCL for you.

The dialog option offers only a small subset of the log record types supported by Transaction Analysis Workbench:

Several of the most common and useful SMF record types.
IMS transaction index records, which consolidate multiple IMS log records into one record per transaction.

Furthermore, the JCL created by the dialog extracts only a subset of fields from those records.

You can tailor the JCL created by the dialog, or write JCL yourself, to extract any of the fields from any of the log record types supported by Transaction Analysis Workbench.

ISPF dialog option 5 Analytics offers suboptions for different forwarding methods and destinations:

1 Stream: Creates JCL that streams log data in JSON Lines format over a TCP network, optionally with security (SSL/TLS). Use this option for Splunk, Elastic, and any other destination that can be configured to listen on a TCP port for JSON Lines from Transaction Analysis Workbench.
2 DB2: Creates JCL that extracts log data in CSV format, and then loads the data into a DB2® table.
3 Data Set: Creates JCL that converts logs to either CSV or JSON format, and then saves the output without forwarding over a network or loading into a DB2 table. Use this option for testing CSV or JSON output from Transaction Analysis Workbench before forwarding.

Remember: When using Transaction Analysis Workbench, it is essential to note that the input logs you provide may contain sensitive information. Therefore, taking the necessary steps to protect any data you forward from these logs is crucial, just as you would secure the input log data.

You must ensure that all access to the data you forward is restricted to authorized personnel. You can also implement appropriate security measures such as encryption and access controls to secure against unauthorized access or disclosure of sensitive information.