Creating Splunk integrations

About this task

Before you create a Splunk integration, consider the following details:

• Load: To prevent the integration from placing an inordinate load on your data source and potentially impacting your logging operations, the integration connects to only one API with a default data frequency of 60 seconds. This frequency is controlled by the Sampling rate setting that is configured when you create the integration.

• Access: Custom data sources are cloud-based REST APIs. Access is configured by using the authentication methods that are specified in the Authentication type setting when you create the integration.

• Data volume: Data volume depends on the application, and is not a set value. Therefore, it does not appear in the settings.

Procedure

To create a Splunk integration, complete the following steps:

1. Log in to IBM Cloud Pak for AIOps console.

2. Expand the navigation menu (four horizontal bars), then click Define > Integrations.

3. On the Integrations page, click Add integration.

4. From the list of available integrations, find and click the Splunk tile.

   Note: If you do not immediately see the integration that you want to create, you can filter the tiles by type of integration. Click the type of integration that you want in the Category section, or alternatively start typing the integration name in the searchbar and the corresponding tile appears.

5. On the side-panel, review the information and when ready to continue, click Get Started.

6. On the Add integration page, define the general integration details:

   • Name: Set a display name for your integration. Use a name that uniquely identifies this integration.

   • Description (optional): Provide a description for the integration.

   • Splunk service URL: Enter the URL for your Splunk instance. For example https://myinstance.mydomain.com:8089.

     Note: If you have any restrictive EgressNetworkPolicies in place, ensure that they are updated to allow for this integration.

   • Authentication type: Select one of the following options:

     • User ID/Password: The Splunk instance has a user ID and password as authentication. You must enter both in the integration configuration that appears.

     • Token: The Splunk instance is authenticated with a temporary token field that appears.

       

   • Splunk dashboard URL: Enter the dashboard URL for your Splunk instance. For example https://myinstance.mydomain.com:8080.

   • Deployment options: Select one of the two options, Local or Remote. Use Local to deploy in the same cluster as IBM Cloud Pak for AIOps console. Select Remote to deploy anywhere, such as on a different network region, on SaaS, or on a remote on-premises system (VM, cluster, container).

     Note: Remote deployment only supports metric integration. If you need to use the log integration, you must select Local. If you select Remote, after adding the integration, you are provided with a bootstrap command to finish the deployment.

7. Click Next.

8. On the Configure Proxy (Optional) page complete the following fields:

   • Connect to Splunk via a proxy server: Set the toggle button to on.

   • URL of HTTP proxy server: Enter a URL for the HTTP proxy server. Note: The URL for the metrics-only integration should have the protocol in the URL. Only unauthenticated HTTP proxy is supported.

   • Port number of proxy server: Enter a port. The default is 8080.

   • Connect to proxy server with username and password: Toggle to on if username or password is required for the proxy.

   • Certificate (optional): Provide a certificate to verify the SSL/TLS connection to the REST service.

   • To verify the SSL/TLS configuration:

     1. Set the Verify the SSL/TLS certificate for the REST service toggle to on.
     2. Either enter a CA certificate used to verify the SSL/TLS integration to the REST service or click Fetch SSL certificate to fetch the certificate automatically.

     

9. Click Test connection (required). The test can take some time to complete.

10. Click Next.

11. On the Collect log data page, define how you want to collect log data for use in AI training:

    

• Data collection: Use the toggle to turn on or off data collection.

  • Set the toggle to Off (default) if you do not want to collect log data. With this setting, no data is collected through the integration, which results in no anomalies being detected as no AI model is trained. You must turn the toggle to On to begin any data flow through the integration.

  • Set the toggle to On to turn on data collection.

    Note: Log data collection is not supported for a remote deployment. If you want to collect log data, you must deploy a local integration.

• Mode: Select how you want data to be collected through the integration for AI training:

  • Live data for continuous AI training and anomaly detection: This option enables continuous live data collection through the integration to train AI models and analyze data for anomalous behavior.

    Important: Before you use this option, you must first set Mode to Historical data for initial AI training and collect a minimum amount of data. Then, you need to run AI training on that data. For more information, see Planning data loading and training.

  • Live data for initial AI training: This option collects a single set of training data to define your AI model. Data collection takes place over a specified time period that starts when you create your integration.

    Note: If you select this option, you must disable your integration when you collect enough data for training. If you do not disable the integration, it continues to collect data. For more information about AI model training, including the minimum and ideal data quantities, see Configuring AI training. For more information about disabling and enabling the integration, see Enabling and disabling Splunk integrations.

  • Historical data for initial AI training: This option collects a single set of training data to define your AI model. Historical data is harvested from existing logs in your integration over a specified time period in the past. You must specify the start and end dates for this data collection, along with the parallelism of your source data.

    • Start date: Select a start date from the calendar. The start date must not exceed 31 days from the present as the maximum time period for historical data collection is 31 days. The recommended time period is two weeks.

    • End date: Select the end date from the calendar. The recommended time period is two weeks.

    • Source parallelism (1-50): Select a value to specify the number of requests that can run in parallel to collect data from the source. Generally, you can set the value to equal the number of days of datat that you want to collect. When you are setting this value, consider the number of requests that are allowed by the source in a minute. For example, if only 1-2 requests are allowed, set the value to be low.

  Important: Keep in mind the following considerations when you select your data collection type:

  • Anomaly detection for your integration occurs if you select Live data for continuous AI training and anomaly detection.

  • Different types of AI models have different requirements to properly train a model. Make sure that your settings satisfy minimum data requirements. For more information about how much data you need to train different AI models, see Configuring AI training.

  • Mapping: Enter the field-mapping information.

  You can improve search performance by mapping your implementation fields to the standard IBM Cloud Pak for AIOps fields. For more information about this mapping, including how to clean data for use in IBM Cloud Pak for AIOps, see, Mapping data from incoming sources.

  The following code shows the supported data schema for the integration:

  {
    "rolling_time": 10,
    "instance_id_field": "sourcetype",
    "log_entity_types": "source",
    "message_field": "_raw",
    "timestamp_field": "_time",
    "resource_id": "sourcetype"
  }
  

* **Filters (optional)**: Enter source, app, or log-level filters to apply to the event data. For more information, see [Search Tutorial ![Opens in a new tab](../images/icons/launch-glyph.svg "Opens in a new tab")](https://docs.splunk.com/Documentation/Splunk/8.1.0/SearchTutorial/Usethesearchlanguage){: external} in the Splunk documentation.

* **Base parallelism (1-50)**: Select a value to specify the number of Flink jobs that can run in parallel. These jobs run to process and normalize the collected data. The default value is 1. However, it is recommended to use a higher value than 1 so that you can process data in parallel. This value cannot exceed the total available free Flink slots. In a small environment, the available flinks slots are 16, while in a large environment, the maximum available slots are 32. If you are collecting historical data with this integration, you can set this value to be equal to the **source** parallelism.

  **Note:** To improve data throughput, you can increase the base parallelism value incrementally. For more information about maximum base parallelism for starter and production deployments, see [Improving data streaming performance for log anomaly detection](../configuring/improving_data_streaming_performance.html).

* **Sampling rate (1-150 seconds)**: Select a rate at which data is sampled from live sources (in seconds). The default value is `60`.

* **JSON processing option**: Select a JSON processing option from the following options:

    * **None**: The default option. The JSON is not processed or modified.
    * **Flatten**: This option flattens the JSON object by removing the opening and closing braces.
    * **Filter**: This option extracts the JSON object and replaces it with an empty string.

 For more information about the options, see [Managing embedded JSON](../configuring/managing_embedded_json_lad.html).

12. Click Next.

13. On the Collect metric data page, define how you want to collect metric data for monitoring.

    • Enable Data collection: Set the toggle to on to enable the data collection. If data flow is disabled, a new integration is created, but metrics are not collected.

    • Select an index to see all associated source types: Pick a splunk index from the list to see its related source types.

    • Select a source type to see all associated metrics: Choose a source type from the list to see its related metrics in the filter table.

    • Search all metrics for the current selection: Search through all the metrics that are associated with the specified index and source type and select the wanted metrics from the table.

    • Hosts: Select the hosts you want to get metric data for. All hosts in Splunk of the specifed source type, that are not marked as too_small and that have data from within the last 48 hours, will be listed.

    • Aggregation interval (minutes): Enter a value in minutes. The value represents the frequency at which data is collected. The default value is 5 minutes.

    Complete the settings to use either historical or live data collection:

    • To use only live data collection, complete the following settings:

      Select the checkbox Only collect live data once the integration is established.

      Note: If you choose to collect only live data, you cannot collect historical data after live data collection starts.

    • To use historical data collection, complete the following settings:

      • Do not select the checkbox Only collect live data once the integration is established.

      • Set the Start date. Select a start date from the calendar and enter the time in hh:mm (hours and minutes) format. Select the time zone from the dropdown list, for example (Localtime or UTC).

        Note: The start date must not exceed 31 days from the present as the maximum time period for historical data collection is 31 days. The recommended time period is two weeks.

      • (Optional) Set an end date for the data collection. If you do not specify the end date, then live data collection follows the historical data collection. If you do not want to set an end date, click Remove end date.

        To add the end date, click Add end date. For the End date, select an end date from the calendar and enter the time in hh:mm format. The end date uses the same time zone as the start date.

14. Click Next.

15. The UI displays the Resource requirements page.

    

    Note: If logs data collection is set to Off and metrics data collection is set to On, you can skip this page.

    On the Resource requirements page, you can review the slot usage for your log integrations to see if there are enough slots to fully support the integration for multizone high availability.

    If you set the Data collection toggle to On, you will see the resource management overview.

    • If your current usage and other usage are less than the provisioned slots, but the high availability slots exceed the provisioned slots, you will be able to create the integration, but will see a warning that you do not have enough slots. The integration will not have multizone high availability.

    • If your projected usage exceeds the provisioned slots, you will not be able to create the integration because you do not have enough slots on your system for log data integrations.

    • If your total slots, including high availability slots, are within the provisioned slots, the integration will have multizone high availability.

      Note: High availability operation assumes high availability for three zones.

      If you set the Data collection toggle to Off, you will see a message stating that you need to enable logs data collection to see the resource management overview. no slots are used by that integration.

16. Click Done. You created a Splunk integration in your instance.

After you create your integration, enable data collection to use your integration with the AI of IBM Cloud Pak for AIOps. For more information, see Enabling and disabling data collection.

For more information about performance considerations for metric data collection, see Performance considerations for metric data collection.

To create more integrations (such as a ChatOps integration), see Configuring Integrations.

For more information about working with the insights provided by your integrations, see ChatOps insight management.