Custom integration setup with Firehose

Edit online

To enable AWS metric streams CloudWatch metrics gathering, you need to apply the Custom setup with Firehose integration. For more information about setting up other prerequisites, see Set up a metric stream guide.

The integration process consists of:
  • Setting up the AWS metric stream instance that provides a necessary AWS CloudWatch metric data.
  • Setting up the AWS Firehose delivery stream instance that delivers AWS CloudWatch metric data to the Instana API by using the custom HTTP endpoint configuration.

The rest of the section describes how to implement the integration process by using AWS Management console.

Set up an AWS metric stream by using the AWS management console

For more information about setting up the AWS metric stream instance by using the AWS CloudWatch Management Console, see the AWS CloudWatch Management console setup guide.

The rest of the section focuses only about the configuration parameters that need to be configured specifically to achieve a near-real-time observability with Instana.

Table 1. Configuration parameters
Configuration parameter name Configuration parameter required value
Change output format - optional OpenTelemetry 1.0
Add additional statistics - optional Choose only the default statistics (Minimum, Maximum, SampleCount, and Sum) since no additional statistics are supported currently.
Metrics to be streamed Choose Select metrics and pick only namespaces that are currently support by Instana.

Set up an AWS Firehose delivery stream by using the AWS management console

For more information about setting up the AWS Firehose delivery stream instance by using the AWS Firehose Management console, see the AWS Firehose Management console setup guide.

The rest of the section focuses only about the configuration parameters that need to be configured specifically to achieve a near-real-time observability with Instana.

Configure source and destination

To configure the AWS Firehose delivery stream data delivery settings, use the following parameters:

Table 2. Configuration parameters for source and destination
Configuration parameter name Configuration parameter required value
Source Direct PUT
Destination HTTP endpoint

Configure destination settings

To configure the AWS Firehose delivery stream destination HTTP endpoint, use the following parameters:

Table 3. Configuration parameters for destination settings
Configuration parameter name Configuration parameter required value
HTTP endpoint URL

For Self-Hosted:

https://otlp-http.{base_domain}/v1/metrics/cloudwatch

For SaaS:

https://otlp-http-{region}-saas.instana.io/v1/metrics/cloudwatch
Note: For more information about the region value and the HTTP OTLP acceptor, see Endpoints of the Instana backend otlp-acceptor.

Configure authentication settings

Use either Use access key or Use AWS Secrets Manager - new configuration settings to provide a valid Instana agent key so that your Firehose delivery stream requests can authenticate against the Instana endpoint.

If you choose Use access key:

Table 4. Configuration parameters for Use access key
Configuration parameter name Configuration parameter required value
Access key - optional A valid Instana agent key

If you choose Use AWS Secrets Manager - new, while creating a new secret:

Table 5. Configuration parameters for Use AWS Secrets Manager - new
Configuration parameter name Configuration parameter required value
Secret type Other type
Key-value pairs

secret key: api_key

secret value: A valid Instana agent key

Configure content encoding settings

Choose GZIP to compress the request body before sending it to the destination. Using GZIP encoding is required for Instana to process your data and to control the overall data transfer cost.

Configure retry duration settings

To achieve near-real-time data processing, the retry duration must be short. A short retry duration ensures that delayed data points do not disrupt data continuity in the Instana time-series graphs.

Table 6. Configuration settings for retry duration
Configuration parameter name Configuration parameter required value
Retry duration 60 seconds
Note:
  • Instana does not process CloudWatch data that is backed up to an AWS S3 bucket after the Retry duration period expires.
  • AWS Firehose does not guarantee delivery order after retries, retried data and new data can arrive in a random sequence.

Configure buffer hints settings

AWS Data Firehose buffers CloudWatch metrics in memory based on the configured data size (buffer size) and time interval (buffer interval) before delivering the data to the specified destination. To achieve near-real-time observability data processing in Instana, set the Buffer size and Buffer interval parameter values to low thresholds so that data is sent to the Instana API as quickly as possible.

Table 7. Configuration settings for buffer hints
Configuration parameter name Configuration parameter required value
Buffer size 1 MiB
Buffer interval 60 seconds
Note:
  • With the given buffer hints configuration, the expected latency is within 3 minutes if the selected CloudWatch namespaces have active metric updates.
  • In a CloudWatch metric stream, data is sent every minute and might arrive at the final destination out of order. All configured metrics in the specified namespaces are sent in the metric stream, except metrics with a timestamp that is more than two days old.