Offloading analytics data to AWS S3 exporter

Configure the DataPower Interact Gateway Collector to archive telemetry data to Amazon S3 for long-term retention, backup, and analytics.

1. Identify the namespace where the collector is deployed.

   kubectl get ns

   Sample output:

   NAME                    STATUS   AGE
   cert-manager            Active   5h2m
   datapower-nano-system   Active   4h53m
   default                 Active   5h6m
   envoy-gateway-system    Active   5h5m
   fyre-ci-293280          Active   5h2m
   kube-node-lease         Active   5h6m
   kube-public             Active   5h6m
   kube-system             Active   5h6m
   metallb-system          Active   5h5m

   Note: Lists all namespaces and their deployments, including the DataPower Interact Gateway. In this example, fyre-ci-293280 is the namespace in which the DataPower Interact Gateway is deployed.
2. Create a Kubernetes secret containing the AWS S3 configuration.

   kubectl create secret generic aws-credentials \
     --from-literal=AWS_REGION='ap-south-1' \
     --from-literal=AWS_S3_BUCKET='your-otel-telemetry-bucket' \
     --from-literal=AWS_ROLE_ARN='arn:aws:iam::123456789012:role/OTelS3Role' \
     -n <namespace>

   Important: Replace ap-south-1, your-otel-telemetry-bucket, and arn:aws:iam::123456789012:role/OTelS3Role with your actual AWS region, S3 bucket name, and IAM role ARN.

   Output:

   secret/aws-credentials created

3. Verify that the secret was created successfully.

   kubectl get secret aws-credentials -n <namespace>

4. Identify the collector deployment.

   kubectl get deployment -n <namespace> | grep collector

   Sample output:

   idig-nanogw-nanogw-analytics-collector   1/1     1     1     5h

5. Edit the collector deployment to inject the secret as environment variables.

   kubectl edit deployment <deployment_name> -n <namespace>

   1. Locate the containers section for the analytics-collector container.
   2. Add the following envFrom section under the container definition:

      envFrom:
        - secretRef:
            name: aws-credentials

   3. Save and exit the editor.

      Sample deployment configuration:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: idig-nanogw-nanogw-analytics-collector
        namespace: <namespace>
      spec:
        replicas: 1
        selector:
          matchLabels:
            app.kubernetes.io/name: nanogw-analytics-collector
        template:
          metadata:
            labels:
              app.kubernetes.io/name: nanogw-analytics-collector
          spec:
            containers:
            - name: analytics-collector
              image: your-image:tag
              envFrom:
              - secretRef:
                  name: aws-credentials

6. Identify the collector ConfigMap.

   kubectl get configmap -n <namespace> | grep collector

   Sample output:

   idig-nanogw-collector-config                       1      5h11m

7. Edit the collector ConfigMap to add the AWS S3 exporter configuration.

   kubectl edit configmap <configmap_name> -n <namespace>

   1. Add the following exporter configuration under the exporters section:

      exporters:
        awss3:
          s3uploader:
            region: '${AWS_REGION}'
            s3_bucket: '${AWS_S3_BUCKET}'
            s3_prefix: 'otel-telemetry'
            role_arn: '${AWS_ROLE_ARN}'
          marshaler: otlp_json

   2. Update the service pipelines to include the AWS S3 exporter:

      service:
        pipelines:
          traces/s3:
            receivers: [otlp]
            processors: [batch/apic]
            exporters: [awss3]
      
          logs/s3:
            receivers: [otlp]
            processors: [batch/apic]
            exporters: [awss3]
      
          metrics/s3:
            receivers: [otlp]
            processors: [batch/apic]
            exporters: [awss3]

   3. Save the ConfigMap.

      Sample ConfigMap structure:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: idig-nanogw-collector-config
        namespace: <namespace>
      data:
        config.yaml: |
          receivers:
            ...
      
          processors:
            batch/apic:
              ...
      
          exporters:
            awss3:
              s3uploader:
                region: '${AWS_REGION}'
                s3_bucket: '${AWS_S3_BUCKET}'
                s3_prefix: 'otel-telemetry'
                role_arn: '${AWS_ROLE_ARN}'
              marshaler: otlp_json
      
          service:
            pipelines:
              traces/s3:
                receivers: [otlp]
                processors: [batch/apic]
                exporters: [awss3]
      
              logs/s3:
                receivers: [otlp]
                processors: [batch/apic]
                exporters: [awss3]
      
              metrics/s3:
                receivers: [otlp]
                processors: [batch/apic]
                exporters: [awss3]

8. Restart the collector deployment to apply the changes.

   kubectl rollout restart deployment <deployment_name> -n <namespace>

   Output:

   deployment.apps/idig-nanogw-nanogw-analytics-collector restarted

9. Verify that the collector pods restart successfully.

   kubectl get pods -n <namespace> | grep collector

   Sample output:

   idig-nanogw-nanogw-analytics-collector-7d4f8ccf45-bk468   1/1   Running   0   30s

   The OpenTelemetry Collector is now configured to export telemetry data to Amazon S3. Traces, logs, and metrics will be archived to your configured S3 bucket for long-term storage and analysis.