Including CSV headers information within transmitted data

Edit online

You can configure the Data Streamer to include CSV headers information within the data transmitted to certain subscribers. This feature significantly enhances the flexibility and customization of data handling in streaming scenarios.

About this task

The 'CSV Header Included' function is specifically designed for streaming data in CSV format. When you create a policy, you can configure the Data Streamer to send headers of CSV data within data to certain subscribers including Splunk via Data Receiver, Logstash, and Generic HTTP subscriber.
  • For Splunk via Data Receiver, headers are sent along with the data to the Data Receiver. The Data Receiver appends the CSV header to the initial line of the output data file. The data is then ingested into Splunk with the auto-detect header feature, which is enabled in the Splunk ingestion kit.
  • For Logstash and Generic HTTP subscribers, if the 'CSV Header Included' function is enabled, headers are sent to the subscriber along with the data packet. Logstash or other Generic HTTP subscribers can automatically detect the header directly from the streaming input data.
Note: The 'CSV Header Included' function was introduced in UJ94825 (updated in April 2024).

Procedure

  1. Configure the subscriber.
    • If you choose Logstash (secure) as the subscriber to receive data containing the header information, you need to copy and update the Logstash configuration file to ingest the data into Elasticsearch.
      1. Log in to the Logstash server and extract the Z Operational Log and Data Analytics Elastic Stack ingestion kit, which is in the file ZLDA-IngestionKit-raw-v.r.m.f.zip. By default, the files are extracted into the zlda-config-raw directory. For more information about how to get the package, see Obtaining and preparing the installation files.
        Note: Logstash processes all *.conf files that are included in the configuration file directory zlda-config-raw. Processing is done in lexicographical order.
      2. Update the Logstash configuration files for your environment.
      3. Start the Elastic Stack.
    • If you choose Splunk via Data Receiver (secure) as the subscriber to receive the data containing the header information, you must install the Z Operational Log and Data Analytics application on Splunk and configure the property file cdpdr.properties of your Data Receiver.
      1. Install the IBM Z® Log and Data Analytics Buffered Header Data Ingestion application, which specializes in processing data that contains header information. The installation files are named as:
        • ibm_cdpz_buffer_header_nix.spl
        • ibm_cdpz_buffer_header_win.spl
        • ibm_cdpz_buffer_header_cloud.spl
        Select the appropriate file according to your environment for installation. For more information about how to get the package, see Obtaining and preparing the installation files.
        Note: If the Splunk already has the IBM Z Log and Data Analytics Buffered Data Ingestion application installed, which processes data without header information, you need to first disable it before you install the IBM Z Log and Data Analytics Buffered Header Data Ingestion application.
      2. After you complete the installation, ensure to enable this newly installed application.
      3. Restart Splunk.
      4. Update the csv_header parameter in the cdpdr.properties file of the Data Receiver. This parameter determines whether the Data Receiver writes header information from the data streams into the output data files.

        Setting this parameter to a lowercase y activates the 'CSV Header Included' function and enables the Data Receiver to write CSV header information into the output data files. Conversely, a lowercase n deactivates this function.

        Header information will be written into the output data files and then received by Splunk only when you set the csv_header parameter in the cdpdr.properties file to y and also set the CSV Header Included parameter in the policy to Yes.

      5. Start the Data Receiver.

        For more information about installing the Z Operational Log and Data Analytics application on Splunk, seeDeploying the Z Operational Log and Data Analytics application on a single Splunk Enterprise system and Deploying the Z Operational Log and Data Analytics application in a clustered Splunk environment.

  2. Enable the CSV Header Included parameter when you create a policy in the Configuration Tool.
    1. After you choose the data stream that you want to collect, you can select the subscriber from the following options: Splunk via Data Receiver (secure), Logstash (secure), or Generic HTTP (secure). These options allow you to enable the 'CSV Header Included' function.
    2. After you select any of these subscribers, the CSV Header Included parameter is displayed in the subscriber window. If you want to embed header information in your data, set this parameter to Yes; otherwise, leave it to No.
  3. Start the Data Streamer by running the following command: 
    START HBODSPRO