Configuring Splunk Enterprise integrations

To collect CPU, memory, disk, network metrics, and events from Splunk Enterprise, install the Splunk Enterprise integration.

Gathering data

The Splunk Enterprise integration uses a remote-only sensor that connects to a Splunk Enterprise instance and collects the following information:
  • Host entities and metrics: cpu, memory, disk

  • Host metrics: cpu, memory, network, disk

  • Events

Verifying prerequisites

You must have your API token for your Splunk Enterprise endpoint handy to complete the integration. For more information about accessing the Splunk API keys, see Splunk API token.

Installing

  1. Verify the public GA image path of the integration for Splunk Enterprise (for example: cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest).
  2. Log in as a root user on a Linux host machine that has network access to Splunk Enterprise. The Splunk Enterprise integration pulls information from Splunk Enterprise by using a remote TCP connection.
  3. To log in before downloading the public image of integration for Splunk Enterprise, run the podman login <cdc-mm ga-image-path> command.
  4. Create a new directory to store the integration-related configuration file and bash script.
    mkdir -p /root/cdc
    cd /root/cdc
  5. To define connection information to the Metric Manager API, create a Metric Manager backend configuration file with the name: com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg.
    # Metric Manager configuration file
    # Metric Manager's URL
    host=http://<metricManagerHost>.ibm.com
    
    # Metric Manager's port
    port=18080
    
    # Metric Manager's username for REST API
    username=system
    
    # Metric Manager's password for REST API 
    # password has been mask ****
    password=**********
    
    # Metric Manager's tenant id
    tenant_id=APM
  6. Create the configuration-.yaml sensor configuration file. Define the Splunk Enterprise endpoint, API key, and the metric entities information as in the following example configuration-splunkent.yaml file for a Splunk Enterprise sensor.
    com.instana.plugin.splunkent:                                  
      api_token: <splunk_api_token>
      endpoint: https://splunk.endpoint.com:8089                 
      enabled: true                                            
      metrics:                                                       
        enabled: true                                             
        entities:                                                      
          <splunk-alert-topic-entity>:                                         
            metrics:                                                       
              - <metric_name1>                                                       
            attributes:                                                    
              - <attribute1>                                                     
              - <attribute2>                                                      
              - <attribute3>                                                      
            poll_rate: 300                                                  
            resource_id: ${<attribute1>}-${<attribute2>}-${<attribute3>}                       
            report_name: <splunk_alert_or_report_name>
  7. Create a bash script with execution permission, as in the following example bash script for a Splunk Enterprise sensor.
    podman run \
      -itd \
      --name instana-agent-metric-manager-ga \
      --volume /var/run:/var/run \
      --volume /run:/run \
      --volume /dev:/dev:ro \
      --volume /sys:/sys:ro \
      --volume /var/log:/var/log \
      --volume <cdc-root-path>/configuration-splunkent.yaml:/opt/instana/agent/etc/instana/configuration-splunkent.yaml \
      --mount type=bind,source=<cdc-root-path>/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg,target=/opt/instana/agent/etc/instana/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg \
      --privileged \
      --net=host \
      --pid=host \
      --env INSTANA_PRODUCT_NAME="metric-manager" \
      --env AGENT_MAX_MEM=6G \
      <IBM-CDC-Public-GA-Image-Path>/ibm-mm-cdc-conn:4.5-latest
  8. If you want to use vault, complete the following steps:
    1. Add the app secret information to the vault server.
    2. Mount the vault PEM file in the image.
    3. Run the bootstrap script to start up the docker image.
    4. Run the docker ps command to check the container id and access to the container by the docker exec -ti <container_id> bash command.
    5. In the container, add the vault IP address into the /etc/hosts file.
      9.x.x.159 Vault
    6. Check the connection to the vault server.
      ping vault
      Note: If ping isn't available, run the dnf install iputils -y command.
    7. Go to the path where the Splunk Enterprise configuration YAML file is located.
    8. Edit the configuration.yaml to add the vault configuration.
      com.instana.configuration.integration.vault:
        connection_url: 'https://Vault:8200' # Mapping through hosts file since PEM ca cert does not contain hostname
        token: '<vault_token>'
        path_to_pem_file: '/root/agentdev/agent-installer/instana-agent/etc/instana/vault-ca.pem'
        secret_refresh_rate: 24
        kv_version: 2
    9. Modify the sensor configuration to use the vault type in the configuration-splunkent.yaml file. ...
      com.instana.plugin.splunkent:
        api_token:
          configuration_from:
            type: vault
            secret_key:
              path: cem/splunk
              api_token: api_token
        endpoint: https://splunk.endpoint.com:8089         
        enabled: true
    10. Restart the integration and check whether the Splunk Enterprise sensor can connect and receive metrics.
  9. Run the bash script to set up and configure the instance for the integration.
Note: If you don't want to monitor everything in your Splunk Enterprise integration, or if you have a large number of management zones, you might want to specify the zones that you do want to monitor in your configuration file. If you have a large number of zones, you may encounter an Out of Memory error when the integration reports on every one of your Splunk Enterprise zones. You can set the zones when you configure your integration by adding values to the zone field of your configuration. For more information about zones, or if you want to make other changes to the default configuration, see the Configuring section. For example, if you monitor approximately 200 hosts, you might not need to specify zones in your configuration. Conversely, if you monitor 5000 hosts that are grouped into hundreds of management zones, it's likely worthwhile to narrow them down.
The Splunk Enterprise integration is installed and set up on the Linux host.

Verifying the installation

  1. Verify whether the integration instance is up and running.
    $ podman ps
    CONTAINER ID   IMAGE                                                                                                                                 COMMAND                  CREATED        STATUS        PORTS     NAMES
    3c75a6d23ca8   cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest   "/usr/local/bin/tini…"   2 weeks ago  Up 2 weeks ago             instana-agent-metric-manager-ga     
  2. Check the logs to confirm that Splunk Enterprise metrics are forwarded to Metric Manager.
    $ podman logs -f <container_id>
    Example logs, which show that the metrics are forwarded:
    2023-10-05T12:12:09.543+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : MetricManagerConfig{Host=http://test.ibm.com, Port=18080, Username=system 
    2023-10-05T12:12:09.544+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : metricManagerURL : http://test.ibm.com:18080/metrics/api/1.0/metrics
    2023-10-05T12:12:10.026+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | Successfully sent payload to Metric Manager
    2023-10-05T12:12:10.026+00:00 | WARN  | tana-agent-scheduler-thread-13-2 | SensorTicker     | com.instana.agent - 1.1.697 | Sending metrics with 1260411 chars took 255815 ms

Configuring

You can edit the configuration-.yaml file to further configure your Splunk Enterprise integration.
  1. Navigate to your configuration-.yaml file on the Linux host machine where you installed your Splunk Enterprise integration.
  2. Open the file with your preferred text editor and find the Splunk Enterprise section. By default, it should look like the following example, except optional fields are empty.
    # Splunk Enterprise
    com.instana.plugin.splunkent:
      api_token: 'myToken123'                                      # Required
      endpoint: https://renlei-vm1.fyre.ibm.com:8089               # Required
      enabled: true                                                # Required
      metrics:                                                     # Required
        enabled: true                                              # Required
        entities:                                                  # Required
    
        ...
  3. Edit the values that you want to change, and save the file. The following table lists the configurable options for Splunk Enterprise.
    Variable Description Type Default value Required or optional
    api_token The access token to use for connecting to Splunk Enterprise. For more information about accessing the Splunk API tokens, see Splunk API token.     Required
    enabled Set to true or false depending on whether you want the integration to collect data. Boolean true Required
    endpoint The URL for the Splunk Enterprise REST APIs. String N/A Required
    metrics Metrics configuration for the sensor. Node N/A Required
    metrics: enabled Set to true or false to enable or disable the metrics integration. If metrics.enabled is set to false, metrics collection will be disabled even if the enabled value for the sensor is true. Boolean true Required
    metrics: entities A list of entities for metric integration. String N/A Required