Configuring Splunk Enterprise integrations
To collect CPU, memory, disk, network metrics, and events from Splunk Enterprise, install the Splunk Enterprise integration.
Gathering data
The Splunk Enterprise integration uses a remote-only sensor that connects to a Splunk Enterprise
instance and collects the following information:
-
Host entities and metrics: cpu, memory, disk
-
Host metrics: cpu, memory, network, disk
-
Events
Verifying prerequisites
You must have your API token for your Splunk Enterprise endpoint handy to complete the integration. For more information about accessing the Splunk API keys, see Splunk API token.
Installing
- Verify the public GA image path of the integration for Splunk Enterprise (for example:
cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest
). - Log in as a root user on a Linux host machine that has network access to Splunk Enterprise. The Splunk Enterprise integration pulls information from Splunk Enterprise by using a remote TCP connection.
- To log in before downloading the public image of integration for Splunk Enterprise, run the
podman login <cdc-mm ga-image-path>
command. - Create a new directory to store the integration-related configuration file and bash script.
mkdir -p /root/cdc cd /root/cdc
- To define connection information to the Metric Manager API, create a Metric Manager backend
configuration file with the name:
com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg
.# Metric Manager configuration file # Metric Manager's URL host=http://<metricManagerHost>.ibm.com # Metric Manager's port port=18080 # Metric Manager's username for REST API username=system # Metric Manager's password for REST API # password has been mask **** password=********** # Metric Manager's tenant id tenant_id=APM
- Create the
configuration-.yaml
sensor configuration file. Define the Splunk Enterprise endpoint, API key, and the metric entities information as in the following exampleconfiguration-splunkent.yaml
file for a Splunk Enterprise sensor.com.instana.plugin.splunkent: api_token: <splunk_api_token> endpoint: https://splunk.endpoint.com:8089 enabled: true metrics: enabled: true entities: <splunk-alert-topic-entity>: metrics: - <metric_name1> attributes: - <attribute1> - <attribute2> - <attribute3> poll_rate: 300 resource_id: ${<attribute1>}-${<attribute2>}-${<attribute3>} report_name: <splunk_alert_or_report_name>
- Create a bash script with execution permission, as in the following example bash script for a
Splunk Enterprise sensor.
podman run \ -itd \ --name instana-agent-metric-manager-ga \ --volume /var/run:/var/run \ --volume /run:/run \ --volume /dev:/dev:ro \ --volume /sys:/sys:ro \ --volume /var/log:/var/log \ --volume <cdc-root-path>/configuration-splunkent.yaml:/opt/instana/agent/etc/instana/configuration-splunkent.yaml \ --mount type=bind,source=<cdc-root-path>/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg,target=/opt/instana/agent/etc/instana/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg \ --privileged \ --net=host \ --pid=host \ --env INSTANA_PRODUCT_NAME="metric-manager" \ --env AGENT_MAX_MEM=6G \ <IBM-CDC-Public-GA-Image-Path>/ibm-mm-cdc-conn:4.5-latest
- If you want to use vault, complete the following steps:
- Add the app secret information to the vault server.
- Mount the vault PEM file in the image.
- Run the bootstrap script to start up the docker image.
- Run the
docker ps
command to check the container id and access to the container by thedocker exec -ti <container_id> bash
command. - In the container, add the vault IP address into the /etc/hosts
file.
9.x.x.159 Vault
- Check the connection to the vault server.
ping vault
Note: If ping isn't available, run thednf install iputils -y
command. - Go to the path where the Splunk Enterprise configuration YAML file is located.
- Edit the configuration.yaml to add the vault
configuration.
com.instana.configuration.integration.vault: connection_url: 'https://Vault:8200' # Mapping through hosts file since PEM ca cert does not contain hostname token: '<vault_token>' path_to_pem_file: '/root/agentdev/agent-installer/instana-agent/etc/instana/vault-ca.pem' secret_refresh_rate: 24 kv_version: 2
- Modify the sensor configuration to use the vault type in the
configuration-splunkent.yaml file.
...
com.instana.plugin.splunkent: api_token: configuration_from: type: vault secret_key: path: cem/splunk api_token: api_token endpoint: https://splunk.endpoint.com:8089 enabled: true
- Restart the integration and check whether the Splunk Enterprise sensor can connect and receive metrics.
- Run the bash script to set up and configure the instance for the integration.
Note: If you don't want to monitor everything in your Splunk Enterprise integration, or if you
have a large number of management zones, you might want to specify the zones that you do want to
monitor in your configuration file. If you have a large number of zones, you may encounter an Out
of Memory error when the integration reports on every one of your Splunk Enterprise zones. You
can set the zones when you configure your integration by adding values to the zone field of your
configuration. For more information about zones, or if you want to make other changes to the default
configuration, see the Configuring section. For example, if you monitor
approximately 200 hosts, you might not need to specify zones in your configuration. Conversely, if
you monitor 5000 hosts that are grouped into hundreds of management zones, it's likely worthwhile to
narrow them down.
The Splunk Enterprise integration is installed and set up on the Linux
host.Verifying the installation
- Verify whether the integration instance is up and running.
$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c75a6d23ca8 cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest "/usr/local/bin/tini…" 2 weeks ago Up 2 weeks ago instana-agent-metric-manager-ga
- Check the logs to confirm that Splunk Enterprise metrics are forwarded to Metric
Manager.
Example logs, which show that the metrics are forwarded:$ podman logs -f <container_id>
2023-10-05T12:12:09.543+00:00 | INFO | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : MetricManagerConfig{Host=http://test.ibm.com, Port=18080, Username=system 2023-10-05T12:12:09.544+00:00 | INFO | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : metricManagerURL : http://test.ibm.com:18080/metrics/api/1.0/metrics 2023-10-05T12:12:10.026+00:00 | INFO | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | Successfully sent payload to Metric Manager 2023-10-05T12:12:10.026+00:00 | WARN | tana-agent-scheduler-thread-13-2 | SensorTicker | com.instana.agent - 1.1.697 | Sending metrics with 1260411 chars took 255815 ms
Configuring
You can edit the
configuration-.yaml
file to further configure your Splunk
Enterprise integration.- Navigate to your
configuration-.yaml
file on the Linux host machine where you installed your Splunk Enterprise integration. - Open the file with your preferred text editor and find the Splunk Enterprise section. By
default, it should look like the following example, except optional fields are empty.
# Splunk Enterprise com.instana.plugin.splunkent: api_token: 'myToken123' # Required endpoint: https://renlei-vm1.fyre.ibm.com:8089 # Required enabled: true # Required metrics: # Required enabled: true # Required entities: # Required ...
- Edit the values that you want to change, and save the file. The following table lists the
configurable options for Splunk Enterprise.
Variable Description Type Default value Required or optional api_token
The access token to use for connecting to Splunk Enterprise. For more information about accessing the Splunk API tokens, see Splunk API token. Required enabled
Set to true
orfalse
depending on whether you want the integration to collect data.Boolean true Required endpoint
The URL for the Splunk Enterprise REST APIs. String N/A Required metrics
Metrics configuration for the sensor. Node N/A Required metrics: enabled
Set to true
orfalse
to enable or disable the metrics integration. Ifmetrics.enabled
is set tofalse
, metrics collection will be disabled even if theenabled
value for the sensor istrue
.Boolean true Required metrics: entities
A list of entities for metric integration. String N/A Required