Host agent REST API

Infrastructure correlation on Linux

Starting from host agent 1.1.582, metrics and trace data that is received by the host agent are correlated with the process that sends the data, if the following preconditions are met:

1. The host agent is running on a Linux system with the following commands available: lsns, nsenter, and ss.
2. The process that sends the metrics and tracing data is running on the same host as the host agent.
3. The process that sends the metrics and tracing data is not ignored by using the Ignore Processes capability.
4. Metrics and traces are reported directly to the host agent, rather than running over a proxy (for example, for OpenTelemtry that uses OpenTelemetry Collector). If the metrics and tracing data go over a proxy, the proxy process is associated with the metrics and traces instead.

For tracing data, the Application Perspective Service dashboard correctly references any process from which related traces were ingested, including, for example, infrastructure changes that took place for this service.

If any of these preconditions is not fulfilled, the metrics and tracing data is correlated with the host on which the host agent is running.

For subsequent invocations of the corresponding API, you are highly recommended to reuse a once-established connection toward the agent's API endpoint. Otherwise, the process lookup is triggered for every new API request. If this cannot be guaranteed and you encounter issues (for example, high-load scenarios with high trace rates), the process lookup can be disabled. Metrics and traces from the API will still be correlated with the host that ingests the data, but not correlated with the corresponding process entity.{: note} To disable the process lookup, add the following snippet to the agent's configuration.yaml file:

com.instana.processlookup:
  enabled: false

OpenTelemetry metrics and traces endpoint

For OpenTelemetry metrics and trace ingestion, see OpenTelemetry documentation.

Prometheus metrics endpoint

For Prometheus metrics ingestion, see Prometheus remote write documentation.

Event SDK web service

Endpoint: http://<agent_ip>:42699/com.instana.plugin.generic.event

By using the Event SDK REST web service, it is possible to integrate custom health checks and other event sources into Instana. Each one running the Instana agent can be used to feed in manual events. The agent has an endpoint that listens on http://<agent_ip>:42699/com.instana.plugin.generic.event and accepts, for example, the following JSON with a POST request:

{
  "title": <string>,
  "text": <string>,
  "severity": <integer>,
  "timestamp": <integer>,
  "duration": <integer>,
}

The following event parameters are available:

• title: The title that is shown in the Instana events view.

• text: The description that is shown in the details section of the event.

• severity: An optional integer with one of the following values depending on the intended event type:

  • Change event: -1 (default)
  • Warning event: 5
  • Critical event: 10

  Furthermore, a change event is considered as a presence event if the title field contains the keyword "online" or "offline".

• timestamp: The timestamp of the event in milliseconds since the UNIX epoch. If not present, the current time is used instead.

• duration: Optional duration of the event in milliseconds. If the value is not designated, the event is shown as an "instant" event with no duration. Furthermore, "change" events or "presence" events are always closed after the duration time. To extend the duration of an event, use the path parameter to identify an open event that has the same value as what's set on the path parameter.

• incident: Optional boolean that turns it into an Incident triggering event if set to true. This requires severity to be positive.

• path: Optional identifier that can be defined in case an event might need to be extended with a later request to this endpoint using the same path value.

The following example shows an instant event:

{
    "title": "Instant critical event on the host",
    "text": "Description of the event that occurred",
    "severity": 10
}

The following example shows an event with duration of 5 minutes, which can be extended by using the path parameter. If no subsequent event with the same path identifier gets sent, the event gets closed automatically after 5 minutes:

{
    "title": "WARNING: This event started to happen",
    "text": "It will be active for 5 minutes",
    "severity": 5,
    "duration": 300000,
    "path": "unique-event-identifier"
}

You can send subsequent events with the same path parameter to extend the lifetime of the event. The following example event extends the event from the previous example by another 5 minutes:

{
    "title": "WARNING: This event continues to happen",
    "text": "It will be active for another 5 minutes",
    "severity": 5,
    "duration": 300000,
    "path": "unique-event-identifier"
}

The endpoint also accepts a batch of events when given an array:

[
  {
    "title": "First event",
    ...
  },
  {
    "title": "Second event",
    ...
  }
]

duration = (Time.now.to_f * 1000).floor - deploy_start_time_in_ms
payload = {}
payload[:title] = 'Deployed MyApp'
payload[:text] = 'pglombardo deployed MyApp@revision'
payload[:duration] = duration

uri = URI('http://localhost:42699/com.instana.plugin.generic.event')
req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
req.body = payload.to_json
Net::HTTP.start(uri.hostname, uri.port) do |http|
  http.request(req)
end

curl -XPOST http://localhost:42699/com.instana.plugin.generic.event -H "Content-Type: application/json" -d '{"title":"Custom API Events ", "text": "Failure Redeploying Service Duration", "duration": 60000, "severity": 5}'

Invoke-RestMethod
	-Uri http://localhost:42699/com.instana.plugin.generic.event
	-Method POST
	-Body '{"title":"PowerShell Event ", "text": "You used PowerShell to create this event!", "severity": -1}'

Invoke-WebRequest
  -Uri http://localhost:42699/com.instana.plugin.generic.event
  -Method Post
  -Body '{"title":"PowerShell Event ", "text": "You used PowerShell to create this event!", "severity": -1}'

Trace SDK web service

Endpoint: http://<agent_ip>:42699/com.instana.plugin.generic.trace

Using the Trace SDK REST Web Service, it is possible to integrate Instana into any application regardless of language. Each active Instana agent can be used to feed manually captured traces into the Web Service, which can be joined with automatically captured traces or be completely separate. The agent offers an endpoint that listens on the http://<agent_ip>:42699/com.instana.plugin.generic.trace URL and accepts the following JSON by using a POST request:

{
  "spanId": <string>,
  "parentId": <string>,
  "traceId": <string>,
  "timestamp": <64 bit long>,
  "duration": <64 bit long>,
  "name": <string>,
  "type": <string>,
  "error": <boolean>,
  "data": {
    <string> : <string>
  }
}

spanId is the unique identifier for any particular span. The trace is defined by a root span, that is, a span that does not have a parentId. The traceId needs to be identical for all spans that belong to the same trace, and is allowed to be overlapping with a spanId. traceId, spanId, and parentId are 64-bit unique values encoded as hex string like b0789916ff8f319f. Spans form a hierarchy by referencing the spanId of the parent as parentId. An example of a span hierarchy in a trace is shown as follows:

	root (spanId=1, traceId=1, type=Entry)
	  child A (spanId=2, traceId=1, parentId=1, type=Exit)
	    child A (spanId=3, traceId=1, parentId=2, type=Entry)
	      child B (spanId=4, traceId=1, parentId=3, type=Exit)
	child B (spanId=5, traceId=1, parentId=4, type=Entry)

The timestamp and duration fields are in milliseconds. The timestamp must be the epoch timestamp coordinated to Coordinated Universal Time.

The name field can be any string that is used to visualize and group traces, and can contain any text. However, simplicity is recommended.

The type field is optional, when absent is treated as ENTRY. Options are ENTRY, EXIT, INTERMEDIATE, or EUM. Setting the type is important for the UI. It is assumed that after an ENTRY, child spans are INTERMEDIATE or EXIT. After an EXIT, an ENTRY follows. This is visualized as a remote call.

The data field is optional and can contain arbitrary key-value pairs. The behavior of supplying duplicate keys is unspecified.

The error field is optional and can be set to true to indicate an erroneous span.

The endpoint also accepts a batch of spans, which then need to be given as an array:

[
  {
    // span as above
  },
  {
    // span as above
  }
]

For traces that are received by using the Trace SDK Web Service, the same rules regarding Conversion and Service/Endpoint Naming are applied as for the Java Trace SDK. In particular these key-value pairs in data are used for naming: service, endpoint, and call.name.

The optional Instana agent service that is provided on Kubernetes with the Instana agent Helm Chart is very useful in combination with the Trace Web SDK support. This is because it ensures that the data is pushed to the Instana agent that runs on the same Kubernetes node, and ensure that the Instana agent can correctly fill in the infrastructure correlation data.

#!/bin/bash

curl -0 -v -X POST 'http://localhost:42699/com.instana.plugin.generic.trace' -H 'Content-Type: application/json; charset=utf-8' -d @- <<EOF
[
  {
    "spanId": "8165b19a37094800",
    "traceId": "1368e0592a91fe00",
    "timestamp": 1591346182000,
    "duration": 134,
    "name": "GET /my/service/asdasd",
    "type": "ENTRY",
    "error": false,
    "data": {
      "http.url": "https://orders.happyshop.com/my/service/asdasd",
      "http.method": "GET",
      "http.status_code": 200,
      "http.path": "/my/service/asdasd",
      "http.host": "orders.happyshop.com"
    }
  },
  {
    "spanId": "7ddf6b31b320cc00",
    "parentId": "8165b19a37094800",
    "traceId": "1368e0592a91fe00",
    "timestamp": 1591346182010,
    "duration": 97,
    "name": "GET /orders/asdasd",
    "type": "EXIT",
    "error": false,
    "data": {
      "http.url": "https://crm.internal/orders/asdasd",
      "http.method": "GET",
      "http.status_code": 200,
      "http.path": "/orders/asdasd",
      "http.host": "crm.internal"
    }
  }
]
EOF