Configuring an OpenTelemetry collector for API discovery

Configuring an Istio data source

Register the collector in your Istio MeshConfig by running the following command, and adding a discovery provider.

kubectl edit configmap istio -n istio-system

Where istio-system is the namespace where Istio is deployed on your cluster. The default namespace is istio-system.

Update the extensionProviders section with the following information:

    extensionProviders:
      - name: "discoveryOtelCollector"
        opentelemetry:
          service: "management-api-discovery-otel-collector.istio-system.svc.cluster.local"
          port: 5555

For more information about providers, see the Provider Selection section in the Istio documentation at https://istio.io/latest/docs/tasks/observability/telemetry/#provider-selection.

If you deployed Istio by using the IstioOperator, you can configure the information by using the MeshConfig. For more information, see the Istio documentation at https://istio.io/v1.10/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig.

Configuring an NGINX data source

There are multiple ways that you can configure your NGINX data source to support the OpenTelemetry collector, but the following information shows the four main options. For more detailed information about the configuration, see the NGINX readme on github.com.

1. NGINX otel-webserver-module
   To configure the webserver module to send traces to the OpenTelemetry collector, you can update the NGINX deployment to use the OTEL_EXPORTER_OTLP_ENDPOINT. For example:

   env:
   - name: OTEL_EXPORTER_OTLP_ENDPOINT
     value: management-api-discovery-otel-collector.namespace.svc:5555

   Or, you can update the NginxModuleOtelExporterEndpoint in the OpenTelemetry configuration on your NGINX server.
2. NGINX instrumentation module

   Similarly to the webserver module, you can update the NGINX deployment to use the OTEL_EXPORTER_OTLP_ENDPOINT, or you can configure the endpoint in the otel-nginx.toml.

   This module allows different variables to be used in the nginx.conf in order to customize the traces that can be sent to the API discovery service.

3. Ingress-Nginx Controller
   The Ingress-Nginx Controller uses the instrumentation module, so the same configuration options apply. As a minimum, the Kubernetes Ingress-Nginx Controller ConfigMap needs to be configured with the following details to collect the traces from the backend service:

   enable-opentelemetry: "true"
   otlp-collector-host: "management-api-discovery-otel-collector.namespace.svc:5555"

4. NGINX OpenTelemetry module

   You can configure this module to send traces to the OpenTelemetry collector by setting the otel_exporter in the NGINX configuration.

   This module allows different variables to be used in the nginx.conf in order to customize the traces that can be sent to the API discovery service.

• discovery.datasource_type - the datasource type (optional). Valid values are istio, nginx, or left blank. If istio, the required Istio Telemetry CR is installed. Otherwise, Helm ignores the telemetry creation.
• discovery.datasource_name - the datasource name for all of the APIs that are discovered by the collector (optional). If the datasource name is empty, the namespace of the collected API is used.
• discovery.apic_host_domain - the domain name of the API Connect instance where the discovered APIs are being sent. For example, dw-rosa-01.owe.p1.openshiftapps.com.
• discovery.provider_org - the name of the provider organization in API Manager.
• discovery.api_key - the credentials that are required for pushing API files to the API Manager. You can get the API key from the user who has access to the API Manager for posting APIs. For information about how to obtain an API key, see Managing platform REST API keys. The api_key is added to a Kubernetes secret as part of the deployment, and then mounted on the collector deployment pod.
• discovery.processors - the flag to enable or disable the processors. Can be set to enabled or disabled. If set to enabled, the ConfigMap enables the existing batch and memory limiter processor, and will not provide any processors in other cases. If you attempt to send a trace with more than 11000 spans at a time, the collector will reject the payload, and you will be required to turn on the batch processor. The available batch and memory limiter processors are configurable according to the base system requirements. The limit_mib in the memory limiter processor can be increased if you update the resource limits of the collector deployment. Keeping the attributes/ibm-apiconnect, redaction/ibm-apiconnect, filter/ibm-apiconnect, transform/ibm-apiconnect, and resource/ibm-apiconnect processors in the service.pipelines.traces.processors in the same order when enabling processors is required, as there are built-in processors that enhance the performance of the collector. If you don't include these processors, the quality of the collected traffic is degraded, as well as the performance of the collector. For more information, see an example processor ConfigMap template, information about the OpenTelemetry batch processor, and information about the OpenTelemetry memory limiter processor.
• namespace - the namespace where Istio or NGINX is deployed on your cluster. As required by the Istio telemetry integration, the collector pod will also be deployed here.
• images.api_discovery_collector - enables the upgrade of the collector deployment as new versions are released. Note that collectors will require updates in the future to ensure that they remain compatible with the discovery service. Details of any updates are available in the https://github.com/ibm-apiconnect/api-discovery-otel-collector repository.
• logging.log_level - the log level; the default level is info. For debug purposes the level can be increased to debug if needed.
• logging.debugexporter_verbosity - the verbosity level of the debug log; the default value is basic. For detailed information about the exporter, the verbosity level can be increased to detailed. However, it is recommended to keep the verbosity as basic for a collector with a large amount of traffic flowing through it.

• telemetry_namespace - the namespace where the Istio Telemetry CR will be deployed (optional). By default, the values file sets this namespace to istio-system, as the root configuration namespace to provide mesh level collection configuration. However, this setting can be customized based on your collection requirements for the deployed applications that you want to be discovered. For more information, see the Telemetry information at https://istio.io/latest/docs/reference/config/telemetry/.

---
discovery:
  # platform_api_prefix is not required to change for API Connect on Cloud
  datasource_type: istio
  # supported values are istio, nginx
  datasource_name: my-data-source
  platform_api_prefix: platform-api
  apic_host_domain: us-east-a.apiconnect.automation.ibm.com
  provider_org: myOrgName
  api_key: xxxxxx
  namespace: istio-system
  telemetry_namespace: istio-system
  # Enable the processor and edit the processor-configmap to choose values for batch and memory_limiter
  # supported values are enabled and disabled
  processors: disabled
logging:
  log_level: info
  # supported values are json and console
  log_encoder: json
  # supported values are basic and detailed
  debugexporter_verbosity: basic

images:
  api_discovery_collector: ghcr.io/ibm-apiconnect/api-discovery-otel-collector:20240612-173911

To understand more about configuring the OpenTelemetry collector, see the OpenTelemetry collector for API discovery readme on github.com. These API Connect resources include working examples with a test repository.

helm template . | kubectl apply -f -