Preparing to send data to Splunk via the Data Receiver

To send data from Z Common Data Provider to Splunk, configure and run an Z Common Data Provider Data Receiver on the system where the Splunk Enterprise server or heavy forwarder is installed. In Splunk, you must also install the Z Common Data Provider Buffered Splunk Ingestion App.

Procedure

In preparation for sending data to Splunk, complete the following steps:

  1. Configure the Data Receiver, as described in Configuring the Data Receiver.
    Important: The Data Receiver working directory and output directory must also be available to Splunk. If you want to set these directories as environment variables, verify that the Data Receiver working directory is assigned to the environment variable CDPDR_HOME, and that the Data Receiver output directory is assigned to the environment variable CDPDR_PATH, as described in Setting up a working directory and an output directory for the Data Receiver. If you do not want to change your system environment variables, you can specify CDPDR_HOME and CDPDR_PATH in SPLUNK_HOME/etc/splunk-launch.conf.
  2. Start the Data Receiver, as described in Running the Data Receiver.
  3. Define a policy with the Data Receiver as the subscriber.
  4. From the product's or suite's installation directory, download the Splunk Ingestion App in binary mode.
    The following files contain different versions of the App based on the intended platform on which Splunk runs.
    Table 1. Downloading the Splunk Ingestion App in binary mode
    Platform on which Splunk runs File name for Buffered Splunk Ingestion App
    UNIX ibm_cdpz_buffer_nix.spl
    Windows ibm_cdpz_buffer_win.spl
    Cloud ibm_cdpz_buffer_cloud.spl
  5. To install the Buffered Splunk Ingestion App in Splunk, complete the following steps:
    1. Log in to Splunk.
    2. Click the gear icon that is next to the word Apps.
    3. Select Install app from file.
    4. Browse for the file that you downloaded in step 4, and select that file.
    5. When you are prompted, select Enable now.
    Important: If you are sending data to Splunk Cloud via the Data Receiver, you must first install ibm_cdpz_buffer_nix.spl or ibm_cdpz_buffer_win.spl on the forwarder where the Data Receiver is installed, and then install ibm_cdpz_buffer_cloud.spl on the Splunk Cloud instance.

    If you are using a Splunk heavy forwarder, you do not have to index the data locally. You can use the system, sysplex, and host attributes to route the data to an appropriate indexer.

    If you want to split the indexing locally, you can refine the monitor stanzas in the inputs.conf file by extending them to add the sysplex component of the file name. Then, duplicate the monitor stanza for each sysplex from which you want to ingest data, and change the index value on the monitor stanzas to indicate the index in which the data is to be kept. These indexes must be created within Splunk. If you update the Z Common Data Provider Buffered Splunk Ingestion App, this customization is deleted.

Results

You can see the data that is loaded into Splunk by using a simple search. For example, the following search shows you all ingested z/OS® SYSLOG events in the zosdex index:
index=zosdex sourcetype=zOS-SYSLOG-Console

If you expand an event, you can see the individual fields for which extraction rules are set.

The following search example shows you the z/OS SYSLOG messages that are issued by the CICS35 job that is running on your production sysplex and are in the zosdex index:
index=zosdex sysplex=PRODPLEX jobname=CICS35 sourcetype=zOS-SYSLOG-Console
You can also use Splunk analytics tools to analyze the data, or write your own deep analysis tools.
Tip: Currently the Buffered Splunk Ingestion App supports only the following log data types for indexing:
  • Job log
  • z/OS UNIX log file
  • Entry-sequenced VSAM cluster
  • z/OS SYSLOG
  • IBM Z® NetView messages
  • IBM® WebSphere® Application Server for z/OS HPEL log
  • Resource Measurement Facility Monitor III reports
  • z/OS sequential data set

For more information, see Configuration reference for data gathered by Log Forwarder.

Searches for other types of data will not yield any results, although the data is in the output directory that is specified by the environment variable CDPDR_PATH. To use this data in the Z Common Data Provider, you can edit the Buffered Splunk Ingestion App, which is installed in the directory SPLUNK_HOME/etc/apps/ibm_cdpz_buffer/.

Splunk indexers can generally ingest data up to 300GB per day. Further data volumes require multiple indexers and search heads. See recommendations of Splunk on scaling and capacity planning for more information.