JSON data ingestion by using generic data streams

You can configure generic data streams to ingest JSON-formatted data from z/OS® sources and stream it to common subscribers. This capability enables the setup of generic data streams to ingest JSON data from various sources and forward it to shared subscribers for processing or analysis. You can also include Z Common Data Provider metadata in the JSON body. The metadata is passed directly to all supported subscribers for processing or analysis.

Generic data streams support the following JSON structures:
  • Object
  • Array

Subscriber-specific configuration

For subscribers such as Logstash and Splunk through Data Receiver, define the schema within the respective subscriber configuration files.

Logstash
To ingest JSON data into Logstash, update the Elastic Stack configuration based on the data format. Create a configuration file that defines field name annotations. Use the following format:

N_<data_stream_name>.lsh

Splunk through Data Receiver

To ingest JSON data into Splunk through Data Receiver, update the following configuration files that are located in <$SPLUNK_HOME>/<your_app_location>/default/:

  • inputs.conf
    [monitor://$CDPDR_PATH/CDP-<your data type>-*.cdp]
disabled = false
index = zosdex
sourcetype = <your datasource type>
  • props.conf
    [<Your Datasource Type>]
INDEXED_EXTRACTIONS = JSON

TIMESTAMP_FIELDS = datetime, timezone

TIME_FORMAT= %D %H:%M:%S:%3N %z

KV MODE = none

AUTO_KV_JSON = false
Kafka, Fluentd, Humio, and Splunk HTTP Event Collector (HEC)
No additional configuration is required for these subscribers. JSON data is forwarded directly.