Streaming logs as JSON Lines over TCP to an analytics platform

You can run Transaction Analysis Workbench batch jobs that stream log data in JSON Lines format over a network to a TCP socket on an analytics platform, such as Elastic or Splunk. The TCP socket can be secure or unsecure.

Before you begin

You must configure the analytics platform to listen on a TCP port for JSON Lines. You need to know the analytics platform host name (or IP address) and port number.

For secure TCP, you need to know the corresponding security details that Transaction Analysis Workbench will require: for example, the location on z/OS® of the security certificates, such as a SAF key ring.

You need to know the location on z/OS of the logs that you want to forward.

About this task

Streaming gets data off z/OS without the storage and processing costs of writing data to temporary staging files on z/OS, and then transferring those files off z/OS.

You can configure some analytics platforms, such as Elastic and Splunk, to listen on a TCP port and ingest data as it arrives.

You can configure some log forwarding tools, such as Logstash (from the Elastic Stack), to listen on a TCP port and forward data to a variety of destinations. The destinations can include analytics platforms that do not natively ingest data via TCP. For example, you can configure Logstash to receive JSON Lines input over TCP, and then forward the data in a different, platform-specific output format over HTTP, to Elasticsearch or other destinations. Such log forwarding tools expand the range of destinations for data from Transaction Analysis Workbench.

Many analytics platforms ingest data in JSON Lines format. In some cases, a key-value (KV) pair data format such as JSON Lines is the preferred format.

To stream logs in JSON Lines format to a TCP socket, use the JSON and STREAM commands of the Transaction Analysis Workbench report and extract utility.

You can either write JCL yourself to run these commands, or you can use Transaction Analysis Workbench ISPF dialog option 1 Stream to create the JCL for you.

The dialog option offers only a small subset of the data sources and record types supported by Transaction Analysis Workbench. You can tailor the JCL created by the dialog, or write JCL yourself, to forward data from any of the data sources and record types supported by Transaction Analysis Workbench.

The following procedure uses the dialog option to create JCL. If you prefer to write JCL yourself, skip to the example JCL following the procedure.

Procedure

  1. On the Transaction Analysis Workbench Primary Option Menu, select option 5 Analytics.
  2. Select a data source: SMF or IMS.
    1. Enter the data set name of the input SMF data set or IMS log.
    2. For SMF, select one or more record types.

    The JCL will contain a JSON command for each selected record type.

  3. Select option 1 Stream.
  4. Enter the host name and the port number on which the analytics platform (for example, Logstash or Splunk) is listening.
  5. Optionally, enter a timeout value.
  6. If the analytics platform TCP port uses security (TLS), enter the security details.

    For more information about the security details, press the Help function key (F1) or see the corresponding security parameters of the STREAM command.

  7. Enter SUB on the command line to create the JCL.

    The JCL is displayed in an ISPF edit panel.

Example

The following JCL extracts selected fields from CICS monitoring facility performance class records in a dumped SMF data set, converts them to JSON Lines in ASCII, and then forwards them over TCP in a single stream to a listening analytics platform, such as Elastic (specifically, Logstash) or Splunk. The analytics platform is installed on a computer with host name analytics, listening on unsecure TCP port number 6789.

//UIDFUW   JOB NOTIFY=&SYSUID  1 
//FUWBATCH EXEC PGM=FUWBATCH
//STEPLIB  DD DISP=SHR,DSN=<FUW HLQ>.SFUWLINK  2 
//SMFIN    DD DISP=SHR,DSN=HLQ.SMF.DAILY
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
OUTZONE(Z)  3 
STREAM NAME(STREAM1) HOST(analytics) PORT(6789) +
       TIMEFORMAT(ISO8601)  4 
JSON CODE(CMF) STREAM(STREAM1) 
FIELDS(  5 
       TRAN
       USRCPUT
       /* More fields...
      )
/*

Notes on the JCL:

 1 
If you use the ISPF dialog to create JCL, then the job statement is specified by the Job Statement Information field under dialog option 0.1 Workbench Personal Settings.
 2 
If you use the ISPF dialog to create JCL, then the Transaction Analysis Workbench executable load module library (STEPLIB) is specified by the Workbench Load Library field under dialog option 0.1 Workbench Personal Settings.
 3 
The OUTZONE(Z) command causes time stamps to be output in UTC. The TIMEFORMAT(ISO8601) parameter on the subsequent STREAM command causes time stamps to be output in ISO 8601 format. Hence, the time stamps in the output have a trailing Z zone designator, indicating UTC.
 4 
Some parameters of the JSON command, such as TIMEFORMAT, can be specified on the STREAM command, ensuring consistency between JSON commands that write to the same stream.
 5 
The FIELDS command is optional. If you omit the FIELDS command, the output includes all fields from the selected records.

Here is a line from the output stream:

{"time":"2015-11-30T08:00:00.000001Z","type":"cmf-6e13","tran":"TRNA","usrcput":0.003456,…}

For a secure TCP port, add the following parameters to the STREAM command in the previous JCL listing:

SECURITY(TLS*) FIPS KEYRING(my/fuw.stream)

where my/fuw.stream identifies the RACF® key ring that contains the CA certificate used by Splunk or Logstash.

You can forward multiple record types in the same stream. For example, to add SMF type 30 job termination records to the stream, append the following lines to the SYSIN data set:

JSON CODE(SMF:30.) STREAM(STREAM1)
FIELDS(                     
       SMF30JBN
       SMF30CPT
       /* More fields...
      )
CODE(SMF:30.)                                    
  COND SMF30STP EQ 5 /* Filter to select job termination records

Here is a corresponding output line:


{"time":"2015-11-30T08:00:01.000002Z","type":"smf-30","smf30jbn":"PRDJOB1","smf30cpt":0.123006,…}

The ISPF dialog creates JCL that forwards logs from a single data source, such as a single SMF data set. You can tailor that JCL, or write JCL yourself, to forward logs from multiple data sources. For example, to add IMS transaction index records to the stream in the previous example JCL, add a DD statement that refers to an IMS system log data set (SLDS):


//LOGIN    DD DISP=SHR,DSN=IMSP.SLDS

then append the following lines to SYSIN:


IMSVRM=154           
IMSINDEX             
JSON CODE(IMS:CA01) STREAM(STREAM1)  
FIELDS(              
       TRANCODE
       PROGRAM
       USERID
       IMSID
       /* More fields...
      )

where 154 identifies the release level of the IMS system.

Each CSV or JSON command can only write to a single stream. However, you can specify multiple STREAM commands in a SYSIN data set; CSV and JSON commands in the same SYSIN data set can write to different streams.