Monitoring Splunk Observability (private preview)

Splunk Observability is an application performance and infrastructure monitoring tool. To display the monitored hosts and their associated metrics and events in the Instana UI, you can use the Splunk Observability sensor for Instana.

After you install the Instana host agent, the Splunk Observability sensor is automatically installed. You can view the metrics of Splunk Observability-monitored hosts in the Instana UI after you configure the sensor. For more information about the configuration, see Configuring the Splunk Observability sensor.

Supported information

Supported operating systems

The supported operating systems of the Splunk Observability sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of each host agent, such as Supported operating systems for Unix.

Configuring the Splunk Observability sensor

To connect to the Splunk Observability instance, you must configure the Splunk Observability with the endpoint and API token. You must configure the following fields in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml:

com.instana.plugin.splunk:
  enabled: true
  endpoint: 'https://api.<realm>.signalfx.com'
  api_token: 'redacted'
  poll_rate: 30

The following table contains the supported configuration options:

Name Type Required Default Description
enabled boolean Yes False This parameter verifies whether the sensor is enabled.
endpoint string Yes NA The endpoint of your Splunk Observability API.
api_token string Yes NA The API token that you use to access the endpoint.
poll_rate integer No 30 The number of seconds between metric queries. You might adjust this time to account for any poll rate limit imposed by your endpoint.
target_zone string No NA The name of the generic zone where your Splunk Observability monitored hosts are displayed on the Instana Infrastructure Map. If the zone is not specified, hosts are displayed in Undefined Zone.

Splunk Observability endpoint

The Instana Splunk Observability sensor uses the Splunk Observability API to collect information about the hosts that Splunk Observability monitors. The Splunk Observability API endpoint is required to communicate with Splunk Observability. The following example shows the format of the Splunk Observability endpoint in the configuration yaml file: https://api.<realm>.signalfx.com

You must include the entire endpoint with <realm> to replace the realm that was assigned by Splunk.

For more information about the Splunk Observability API, see the Splunk Observability documentation.

Splunk Observability API token

An API token is required to authenticate Splunk Observability. For more information about how to create an API token, see Splunk Observability documentation.

Viewing metrics

To view the metrics, complete the following steps:

  1. From the navigation menu in the Instana UI, select Infrastructure.
  2. Click a specific monitored host.

You can see a remote host dashboard with all the collected metrics.

Configuration data

  • OS (name, version, and architecture)
  • CPU model and count
  • Total Memory
  • Hostname and FQDN
  • Disk Capacity

Performance metrics

  • CPU Used %
  • CPU Load
  • Memory Used
  • Disk (used, reads, and writes)
  • Network RX and TX Bytes

Viewing events

Splunk Observability manages incidents that are composed of one or more events. The Splunk Observability sensor imports all events to Instana as issues and changes based on the event severity. To view the issues or changes, complete the following steps:

  1. From the navigation menu in the Instana UI, select Events.
  2. To filter events that are not issues or changes, click the Issues or Changes tab.
  3. To filter issues or changes that are imported from Splunk Observability, enter the following query in the search box: entity.remote.host.source:Splunk

You can see a list of issues that are imported from Splunk Observability. An open Splunk Observability event creates an Instana issue, which remains open until the Splunk Observability event is closed. When the Splunk Observability event closes, the Instana issue or change closes within a few minutes.

Troubleshooting

Agent monitoring issues

If the Splunk Observability sensor cannot communicate with the Splunk Observability endpoint, an agent monitoring issue is created. The issue describes incorrect configuration parameters. For example, if the sensor receives an HTTP 401 or 403 error from Splunk Observability, the API token is likely incorrect or expired. The following configuration parameters can lead to agent monitoring issues:

  • endpoint
  • api_token (HTTP 401 or 403)
  • poll_rate (HTTP 429)

Agent OutOfMemoryError

If you are connected to the Splunk Observability instance that monitors many hosts, you might need to increase the JVM heap size of the Instana agent. If you find an OutOfMemoryError in the agent log, you must increase the maximum heap size of the Instana agent. For more information about the JAVA_OPTS environment variable, see Environment variable reference.