Splunk

Log management
Alert channel
- Configuration
Show 5 minute AP/services metrics in Splunk
- Add-on & app

Log management

Configuration

To configure Splunk log management, select "Settings > Log Management > Splunk".

Splunk Form

By default, Splunk log management is not enabled. To enable, set the toggle to show Splunk link on Hosts, Container and Pods.

Enter the following parameters:

Parameter	Description
Splunk Instance	The URL or the IP address (including the port number) of the deployed instance where the logs are stored.
Index (optional)	Optionally, the name of the index you have configured in the Splunk platform.

Accessing Splunk

Or if there are multiple log managements enabled

Splunk Buttons

To access Splunk, click Splunk which is at the upper left of each of these dashboards:

Kubernetes:
- Host
- Pod
- Docker container
Host
Docker container

Accessing Instana from Splunk

There are two option to enable accessing Instana related entities from your logs:

Adjust your current dashboards: add the _raw field to the panel's query and the drilldown to the panel section

<drilldown>
        <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
</drilldown>

Create a new dashboard: Go to the Dashboards section in Splunk. Click Create New Dashboard, enter a name, and Save. Click Edit Dashboard, select Source, and paste the following content:

<form theme="light">
  <label>Instana</label>
  <fieldset submitButton="false" autoRun="true">
      <input type="time" token="myTime" searchWhenChanged="true">
          <label></label>
          <default>
              <earliest>-1@h</earliest>
              <latest>now</latest>
          </default>
      </input>
  </fieldset>
  <row>
      <panel>
          <title>Events</title>
          <table>
              <search>
                  <query>sourcetype = * | table host docker.container_id kubernetes.pod_name _raw
                  </query>
                  <earliest>$myTime.earliest$</earliest>
                  <latest>$myTime.latest$</latest>
              </search>
              <option name="count">15</option>
              <option name="drilldown">row</option>
              <option name="refresh.display">progressbar</option>
              <option name="rowNumbers">false</option>
              <option name="totalsRow">true</option>
              <option name="wrap">false</option>
              <fields>["host","docker.container_id","kubernetes.pod_name","_raw"]</fields>
              <drilldown>
                  <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
              </drilldown>
          </table>
      </panel>
  </row>
</form>

Alert channel

Configuration

Once you have the Add-On and App from Splunkbase installed, you're ready to set up the Instana integration.

To configure, head over to "Settings > Team Settings > Events & Alerts > Alert Channels > Add Alert Channel":

splunk alert channel

The following Splunk events are received as an HTTP POST to the configured URLs (HTTP or HTTPS).

On open issues/incidents

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "issue",
    "state": "OPEN",
    "start": 1460537793322,
    "severity": 5,
    "text": "Garbage Collection Activity High (11%)",
    "suggestion": "Tune your Garbage Collector, reduce allocation rate through code changes",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}

On close issues/incidents

{
  "issue": {
    "id": "6596e1c9-d6e4-4a8e-85fd-432432eddac3",
    "state": "CLOSED",
    "end": 1460537777478
  }
}

On offline/online/change events

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "presence",
    "start": 1460537793322,
    "text": "online",
    "description": "Java virtual machine on Host host1.demo.com",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}

Show 5 minute AP/services metrics in Splunk

Add-on & app

Leverage the Instana and Splunk integration to access Instana metrics directly within Splunk.

splunk events