Customizing events used in Event Search using the DSV toolkit

You must use the DSV toolkit to add events to Event Search if you are running Netcool® Operations Insight® 1.4.1.1 or earlier, and are therefore using Tivoli® Netcool/OMNIbus Insight Pack 1.3.0.2.

Before you begin

You can use the DSV toolkit to generate a customized insight pack. The DSV toolkit is provided with the Operations Analytics - Log Analysis product, in $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4. In the properties file, you can change the index configurations to meet your requirements

A new source type with an updated index configuration is created when you install the insight pack. An insight pack contains the following elements:

  • An index configuration: defines how the fields are indexed in Operations Analytics - Log Analysis.
  • A splitter: splits the ingested data into individual log entries.
  • An annotator: splits the log entries into fields to be indexed.

The Netcool/OMNIbus insight pack requires the newlineSplitter.aql custom splitter, and the insight pack can use an annotator built using the DSV toolkit. To modify the index configuration, and generate a data source to ingest the data with the new index, you need to create a new insight pack using the DSV toolkit and modify it to use the Netcool Operations Insight newlineSplitter.aql.

  • See the DSV toolkit documentation in the $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/docs directory for information about specifying field properties and generating insight packs.
  • See the Gateway for Message Bus documentation for information about mapping event fields to insight pack properties. Testing insight packs requires a Gateway for Message Bus to transfer events from Netcool/OMNIbus to Operations Analytics - Log Analysis.

About this task

Use the DSV toolkit to generate an insight pack that contains a new rule set (annotator and splitter) for the Netcool/OMNIbus event fields that you want.

The procedure describes how to create an insight pack called ITEventsInsightPack_V1.1.0.1, based on the Tivoli Netcool/OMNIbus Insight Pack 1.3.0.2. Use your own naming as appropriate.

Procedure

  1. Make a copy of the omnibus1100.properties file, which is in the docs directory of the Tivoli Netcool/OMNIbus Insight Pack installation directory ($UNITY_HOME/unity_content/OMNIbusInsightPack_v1.3.0.2), and rename it.
    For example, rename it to ITEvents.properties.
  2. Copy the ITEvents.properties file that you created in step 1 to the DSV toolkit directory $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4.
  3. Edit the ITEvents.properties file.
    For example, change the default value of the aqlModuleName field to ITEvents, and add, modify, or remove event field properties as required. To obtain the version number 1.1.0.1, change the version property to 1.1.0.1.
  4. If you added or removed fields from the file, change the value of the totalColumns field so that it specifies the total number of fields in the file.
  5. Use the following command to generate an insight pack:
    python dsvGen.py ITEvents.properties -o
    The insight pack is named ITEventsInsightPack_V1.1.0.1.
  6. Add the customized splitter into the insight pack as follows:
    1. Create the following directory for the splitter: $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build/ITEventsInsightPack_v1.1.0.1/extractors/ruleset/splitter
    2. Extract the Netcool/OMNIbus insight pack and copy the following file:

      Insight Pack Extract Directory/OMNIbusInsightPack_v1.3.0.2/extractors/ruleset/splitter/newlineSplitter.aql

      to

      $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build/ITEventsInsightPack_v1.1.0.1/extractors/ruleset/splitter/

    3. Open the file $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build/ITEventsInsightPack_v1.1.0.1/metadata/filesets.json and remove the following text:

      ,{"name":"ITEvents-Split","type":0,"fileType":0,"fileName":"Dsv.jar","className":"com.ibm.tivoli.unity.content.insightpack.dsv.extractor.splitter.DsvSplitter"}

    4. Open the file $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build/ITEventsInsightPack_v1.1.0.1/metadata/ruleset.json and add the following text:

      [{"name":"ITEvents-Split","type":0,"rulesFileDirectory":"extractors\/ruleset\/splitter"}]

    5. Open the file $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build/ITEventsInsightPack_v1.1.0.1/metadata/sourcetypes.json and change the following text:

      "splitter":{"fileSet":"ITEvents-Split","ruleSet":null,"type":1}

      to

      "splitter":{"fileSet":null,"ruleSet":"ITEvents-Split","type":1}

    6. Go to $UNITY_HOME/unity_content/DSVToolkit_v1.1.0.4/build and compress the contents of the insight pack directory using the zip command utility to create a new insight pack. Ensure you run the command from the /build directory to preserve the directory structure in the resulting .zip file (in this example, the directory is /ITEventsInsightPack_v1.1.0.1, so the file would be ITEventsInsightPack_v1.1.0.1.zip).
      For example:
      zip -r ITEventsInsightPack_v1.3.0.2.zip ITEventsInsightPack_v1.3.0.2
    7. Install the insight pack using the $UNITY_HOME/utilities/pkg_mgmt.sh command as described in Installing the Tivoli Netcool/OMNIbus Insight Pack.
  7. Test the insight pack:
    1. Create a temporary data source in Operations Analytics - Log Analysis for the new Source Type and Collection created by the DSV toolkit.
    2. Change the Gateway for Message Bus map file to match the fields that you defined in the ITEvents.properties file.
      Important: The order of the column entries must match exactly the order of the alert field entries in the gateway map file.
      See the Gateway for Message Bus documentation for information about configuring the map file.
    3. In the gateway scalaTransport.properties file, modify the values of the jsonMsgHostname and jsonMsgLogPath properties to match the attributes of the new data source that you created in step 7.a.
    4. Test the new configuration.
  8. Create a new data source called "omnibus" by using the new source type defined in the ITEvents Insight Pack.
    Important: You cannot rename an existing data source to the default name omnibus or use an existing data source that is named omnibus. You must delete the existing data source, then create the new data source and name it omnibus.

What to do next

Test the new insight pack in the Operations Analytics - Log Analysis UI.