Configuring Splunk Enterprise as an event source

Edit online

Splunk Enterprise is an on-premises version of Splunk that you can use to monitor and analyze machine data from various sources. You can set up an integration with Netcool® Operations Insight® to receive alert information from Splunk Enterprise.

Before you begin

The following event types are supported for this integration:
  • Splunk App for Infrastructure Monitoring
    • Monitoring for Linux®/UNIX
    • Monitoring for Windows
Note: You can use the Splunk App to define the mapping of Splunk fields with event management fields.
Warning: Splunk Enterprise does not provide a means of downgrading to previous versions. If you want to revert to an older Splunk release, uninstall the upgraded version and reinstall the version you want. The Splunk App for UNIX/Linux is currently not supported beyond version 7.2.x.

About this task

Using a package of installation and configuration files provided by Netcool Operations Insight, you set up an integration with Splunk Enterprise. The alerts generated by Splunk Enterprise are sent to the Netcool Operations Insight service as events.

Procedure

  1. Click Administration > Integrations with other systems.
  2. Click New integration.
  3. Go to the Splunk Enterprise tile and click Configure.
  4. Enter a name for the integration.
  5. Click Download file to download and decompress the ibm-cem-splunk.zip file. The compressed file contains the savedsearches.conf file for both the UNIX and Windows systems, and the ibm-cem-alert.zip file which contains the file for installing the Splunk App for Netcool Operations Insight.
    • splunk_app_for_nix/local/savedsearches.conf
    • splunk_app_windows_infrastructure/local/savedsearches.conf
    • ibm-cem-alert.zip
    Important: The download file contains credential information and should be stored in a secure location.
  6. Install the Splunk App using the ibm-cem-alert.zip file.
    1. Log in to your Splunk Enterprise browser UI as an administrator.
    2. Select App then click Manage Apps.
    3. Click Install app from file.
    4. Click Browse to locate the ibm-cem-alert.zip file.
    5. Click Upload.
  7. Log in to your Splunk Enterprise server host and copy the savedsearches.conf file to $SPLUNK_HOME/etc/apps/<app_name>/local.
    UNIX:
    sudo cp ibm-cem-splunk/splunk_app_for_nix/local/savedsearches.conf
 $SPLUNK_HOME/etc/apps/splunk_app_for_nix/local/savedsearches.conf
    Windows:
    copy ibm-cem-splunk\splunk_app_windows_infrastructure\local\savedsearches.conf
 %SPLUNK_HOME%\etc\apps\splunk_app_windows_infrastructure\local
    Important: If you already have an existing Splunk app installed, then you already have settings defined in a savedsearches.conf file. Merge your existing savedsearches.conf file with the one downloaded from Netcool Operations Insight. You can merge the files manually, or use the Splunk Enterprise browser UI by clicking the Alerts tab, expanding the selected alert section, clicking Edit > Edit Alerts, and editing the fields under section IBM Cloud Event Management Alert. You can use the savedsearches.conf file to check the mapping for the values of the fields.
  8. Restart the Splunk Enterprise instance to ensure the new alerts are available.
    UNIX:
    sudo $SPLUNK_HOME/bin/splunk restart
    Windows:
    %SPLUNK_HOME%\bin\splunk.exe restart
  9. Log in to the Splunk Enterprise UI as an administrator and check that the alerts defined in savedsearches.conf are available:

    For UNIX systems, go to Search & Reporting > Splunk App for Unix > Core Views > Alerts.

    For Windows systems, go to Search & Reporting > Splunk App for Windows Infrastructure > Core Views > Alerts.

    Note: If you modify the trigger conditions for the alerts, ensure you do not set a trigger interval that is too frequent. For example, if you set the Edit > Edit Alerts > Trigger Conditions to trigger an alert once every minute when the result count is greater than 0, the resulting number of events can overload event management. To limit the trigger frequency, set the greater than value to a higher number than 0, and set it to be triggered 5 times in every hour, for example. You can also use the Throttle option to suspend the triggering of events for a set period after an event is triggered.
  10. Optional: To receive resolution events from Splunk Enterprise, add the resolution:true value to the action.ibm_cem_alert.param.cem_custom parameter in the savedsearches.conf file, for example: 
    # Example
## Automation mapping for IO Utilization Exceeds Threshold Alert
## using IBM Event Management custom webhook alert
[IO_Utilization_Exceeds_Threshold]
action.ibm_cem_alert = 1
action.ibm_cem_alert.param.cem_custom = statusOrThreshold:$result.bandwidth_util$,resolution:true
action.ibm_cem_alert.param.cem_event_type = $name$
action.ibm_cem_alert.param.cem_resource_name = $result.host$
action.ibm_cem_alert.param.cem_resource_type = Server
action.ibm_cem_alert.param.cem_severity = Major
action.ibm_cem_alert.param.cem_summary = $result.host$: IO utilization exceeds $bandwidth_util$ threshold
action.ibm_cem_alert.param.cem_webhook = {{WEBHOOK_URL}}/{{WEBHOOK_USER}}/{{WEBHOOK_PASSWORD}}
disabled = 0
    Tip: You can also add the resolution setting using the UI. Open Edit > Edit Alerts under section IBM Cloud Event Management Alert, and add resolution:true to the Additional mapping (optional) field.
  11. Click Save to save the integration in Netcool Operations Insight.
  12. To start receiving alert notifications from Splunk Enterprise, ensure that Enable event management from this source is set to On..