Self-hosted agent configuration to share data with external systems

Use the following configuration to share traces from the End-to-End Monitoring self-hosted agent to targets such as messaging queues or external Application Performance Monitoring (APM) systems or custom targets being used to store monitoring data.

Pre-requisites

  • Install the self-hosted IBM® Products like Integration Server, Microservices Runtime, or API Gateway for the version of your choice with the latest fixes.
  • For Integration Server V10.15 or later, download the latest JDK 17 compliant fix version.

IBM webMethods Installer based agent setup for hybrid monitoring

To start an installation of IBM webMethods Installer-based Hybrid Monitoring Agent, follow the instructions available Setup that uses IBM webMethods Installer.

OTLP common configuration

Update the following properties according to your requirements in the agent.conf file:

Property Description Sample value
collector.establish_cloud_communication Use this property to enable or disable communication from agent to cloud collector. Set this property to false to enable the OTLP support. Setting this property is mandatory for OTLP support. $ The default value is true.
exporter.establish_external_communication Use this property to establish a connection to an external target system for pushing End-to-End Monitoring traces. Set this property to true to enable the OTLP support. Setting this property is mandatory for OTLP support. ${SW_AGENT_EXTERNAL_ESTABLISH_COMMUNICATION:false} The default value is false.
exporter.default_target Use this property to set your default OTLP target server. ${SW_AGENT_EXTERNAL_TARGET:apm} apm, kafka or custom target
  • APM - Sends traces to any observability platform that supports the OpenTelemetry standard. For example, Honeycomb, Dynatrace, Jeager.
  • Kafka - Sends traces to the Kafka messaging queue. You are required to process the traces from the queue.
  • Custom target - Sends traces to your own user-defined target server.
exporter.target_name Use this property to set the external target system name. Setting this property is optional. ${SW_AGENT_EXTERNAL_TARGET_NAME:<target_name>}
exporter.batch_size Use this property to control the number of records that are sent in a single request to the target server. ${SW_AGENT_EXT_BATCH_SIZE:2000} The default value is 2000.
exporter.healthcheck_interval Use this property to set a target server health check interval in seconds. ${SW_AGENT_EXTERNAL_API_HEALTH_CHECK_INTERVAL:60} The default value is 60.

For custom implementation configurable properties, click Custom implementation configurable properties.

Note: For OTLP transaction details, refer to the logs available at: 
<INSTALL-DIR>\IntegrationServer\instances\default\logs\e2eagentOtlpTraces.log

Develop Anywhere, Deploy Anywhere (DADA) configuration

End-to-End Monitoring supports the capability to transfer Develop Anywhere, Deploy Anywhere transaction traces to OTLP targets. The following mandatory environment variables are required to be assigned when you start the Edge Runtime (ERT) container:

Environment Variable Description Sample value
SW_AGENT_COLLECTOR_ESTABLISH_CLOUD_COMMUNICATION Use this property to enable or disable communication from agent to cloud collector. Set this property to false to enable the OTLP support. Setting this property is mandatory for OTLP support. false
SW_AGENT_EXTERNAL_ESTABLISH_COMMUNICATION Use this property to establish a connection to an external target system for pushing End-to-End Monitoring traces. Set this property to true to enable the OTLP support. Setting this property is mandatory for OTLP support. true
SW_AGENT_EXTERNAL_TARGET Use this property to set the external target system name. Setting this property is optional. The default value is apm.
SW_AGENT_OTEL_ENDPOINT Use this property to update the endpoint URL of the external target APM server. https://<otlp_endpoint>/v1/traces
SW_AGENT_OTEL_HEADERS Use this property to provide key value pairs that contain authorization tokens and required headers that are separated by commas. The header key and value are separated by '#'. The external target system requires valid headers. api-key#value,Content-Type#application/x-protobuf

Sample Docker command



docker run -p 5555:5555 -d -e SAG_IS_CLOUD_REGISTER_URL=https://abc02.stag-int-aws-us.webmethods.io -e SAG_IS_EDGE_CLOUD_ALIAS=EdgeRuntime_E2EM_ERT -e SW_AGENT_OTEL_ENDPOINT=https://<otlp_endpoint>:4318/v1/traces -e SW_AGENT_OTEL_HEADERS=api-key#xxxxxxxxxxxxxxxxxxxxxxxxxFFFFNRAL,Content-Type#application/x-protobuf -e SW_AGENT_EXTERNAL_ESTABLISH_COMMUNICATION=true -e SW_AGENT_COLLECTOR_ESTABLISH_CLOUD_COMMUNICATION=false -e SAG_IS_CLOUD_REGISTER_TOKEN=yyyyyyyyyy --name=E2EM_ERT iregistry.infra.webmethods.io/aim/msr-edge-runtime:11.0.7
Note: The support for OTLP exporter feature in Develop Anywhere, Deploy Anywhere depends on the IBM webMethods Integration release v11.0.7.

OTLP Configuration based on target server

APM-specific configuration for target server

Update the following attributes in the agent configuration, if you need to push End-to-End Monitoring trace data directly to an APM tool like Dynatrace or New Relic.

The following example shows attributes for Honeycomb APM.

Property Description Sample value
exporter.url Use this property to update the endpoint URL of the external target APM server. ${SW_AGENT_OTEL_ENDPOINT:<OTLP_ENDPOINT>}. For Example, https://<otlp_endpoint>/v1/traces
exporter.headers Use this property to provide key value pairs that contain authorization tokens and required headers that are separated by commas. The header key and value are separated by '#'. The external target system requires valid headers. ${SW_AGENT_OTEL_HEADERS:<REQUIRED_HEADERS>} Example: api-key#value,Content-Type#application/x-protobuf
exporter.username Use this property to provide the username of the target server. ${SW_AGENT_EXT_SYSTEM_USER:<username>}
exporter.password Use this property to provide the password of the target server. ${SW_AGENT_EXT_SYSTEM_PASSWORD: <password>}
exporter.support_long_id Use this property to support Trace ID lengths exceeding 16 bytes. Setting this property is optional. ${SW_AGENT_EXTERNAL_SUPPORT_LONG_ID:false} Default value is false. This is the recommended value.
exporter.api_error_codes API error codes for target server health check. Add extra codes according to your requirement. ${SW_AGENT_EXTERNAL_API_ERROR_CODES:502,503,504} The default values are 502,503,504
exporter.resource_attributes Use this property to provide comma-separated key value pairs that contain resource attributes. The attribute key and value are separated by '#'. ${SW_AGENT_OTEL_RESOURCE_ATTRIBUTES:<OTLP_RESOURCE_ATTRIBUTES>} Example: service.name#example_service, service.namespace#example_service_namespace

Supported APM systems

The following APM systems have been verified with End-to-End Monitoring:

APM target server Supported Supported Version Remarks
Honeycomb Yes V10.15 Fix 8
LightStep(Service Now) Yes V10.15 Fix 8
Dynatrace Yes V10.15 Fix 9
New Relic Yes V10.15 Fix 9
Grafana Tempo Yes V10.15 Fix 9
Jeager Yes V10.15 Fix 9
Cisco Appdynamics Yes V10.15 Fix 9
IBM Instana® Yes V10.15 Fix 9
Elastic APM Yes V10.15 Fix 9
Datadog No Planned

KAFKA-specific configuration for target server

Update the following attributes in the agent configuration, if you need to push End-to-End Monitoring trace data to the Kafka messaging queue and later to process this data in a pre-configured Kafka consumer client.

The following example shows attributes for the Kafka server.

Property Description Sample value
exporter.url Use this property to update the endpoint URL of the external target Kafka server. ${SW_AGENT_OTEL_ENDPOINT:<KAFKA_ENDPOINT>}. For Example, https://<host_name>:9091
exporter.headers Use this property to provide key value pairs that contain authorization tokens and required headers that are separated by commas. The header key and value are separated by '#'. ${SW_AGENT_OTEL_HEADERS:<REQUIRED_HEADERS>}. For Example, key#value
exporter.username Use this property to provide the username of the target server. ${SW_AGENT_EXT_SYSTEM_USER:<username>}
exporter.password Use this property to provide the password of the target server. ${SW_AGENT_EXT_SYSTEM_PASSWORD: <password>}
exporter.topic Use this property to provide the topic name defined in Kafka. ${SW_AGENT_EXTERNAL_KAFKA_TOPIC:topic_demo}
exporter.ack Use this property to provide the acknowledgment level. ${SW_AGENT_EXTERNAL_KAFKA_ACK:all} Valid values are 0, 1, and all. The default value is all.
exporter.retries Use this property to define the retries setting that determines the number of times the producer attempts to send a message before marking it as failed. ${SW_AGENT_EXTERNAL_KAFKA_RETRIES:0} The default value is 0.
exporter.client_id Use this property to provide the label that names a particular producer. ${SW_AGENT_EXTERNAL_KAFKA_PRODUCER_ID:OTelTraceProducer} The default value is TraceProducer.
exporter.linger_ms_config Use this property to define the amount of time a Kafka producer waits before sending a batch of messages, in milliseconds ${SW_AGENT_EXTERNAL_KAFKA_MS_CONFIG:0} The default value is 0.
exporter.request_timeout_ms_config Use this property to provide the maximum amount of time the client waits for the response of a request, in milliseconds. If the response is not received before the timeout, the client resends the request or fails the request if the number of retries is exhausted. ${SW_AGENT_EXTERNAL_KAFKA_TIMEOUT_MS_CONFIG:60} The default value is 60.
exporter.send_buffer_config Use this property to provide the size of the TCP send buffer (SO_SNDBUF) to use when sending data. ${SW_AGENT_EXTERNAL_KAFKA_SEND_BUFFER_CONFIG:-1} The default value is -1.
exporter.receive_buffer_config Use this property to provide the size of the TCP receive buffer (SO_RCVBUF) to use when reading data. ${SW_AGENT_EXTERNAL_KAFKA_RECEIVE_BUFFER_CONFIG:-1} The default value is -1.
exporter.compression_type_config Use this property to provide the compression type for all data generated by the producer. Compression consists of full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression. Valid values are: nonegzip snappylz4 zstd ${SW_AGENT_EXTERNAL_KAFKA_COMPRESSION_TYPE:none} The default value is none.
exporter.max_request_size_config Use this property to define the maximum size of a request in bytes. This setting limits the number of record batches that the producer sends in a single request to avoid sending large requests. ${SW_AGENT_EXTERNAL_KAFKA_REQUEST_SIZE:1048576} The default value is 1048576 (1 MB).
Note: Copy the Kafka client library version 3.5.1 or higher in your IS directory and add a full path reference entry to the Kafka library at the end of the file 
<INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf
.

For example:

  • Integration Server and API Gateway: 
    wrapper.java.additional.504=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\custom_jars\kafka-clients-3.5.1.jar;"%JAVA_BOOT_CLASSPATH%"
  • Micro Services Runtime: 
    set JAVA_UHMKAFKA_OPTS=-Xbootclasspath/a:"..\..\..\E2EMonitoring\agent\kafka-clients-3.5.1.jar"
set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHMKAFKA_OPTS%
    at the end of the file <INSTALL-DIR>\IntegrationServer\bin\setenv.bat

Custom implementation

You can implement solutions to share End-to-End Monitoring tracing data to your custom targets like APM tools or custom data stores like Elasticsearch using a Java™-based API provided by End-to-End Monitoring. The reference implementation is available for the following target systems:

  • APM
  • Kafka

For more information on the default implementation, see APM-specific configuration for target server or KAFKA-specific configuration for target server.

However, if you need to implement your own solution by using the End-to-End Monitoring common API, use the following Java docs and sample classes that implement the API.

Prerequisites

The otlpClientPlugin.jar file contains the interfaces and helper classes to implement trace exporter for the plugged-in external target servers.

Use the otlpClientPlugin.jar file in the following directory.
<INSTALL-DIR>\IntegrationServer\instances\default\packages\WmE2EMIntegrationAgent\resources\agent\plugins
Note:
  • Plug-in file path for Microservices Runtime:
    INSTALL_DIR>\IntegrationServer\instances\default\packages\WmE2EMIntegrationAgent\resources\agent\plugins
  • Plug-in file path for API Gateway:
    <INSTALL_DIR>\IntegrationServer\instances\default\packages\WmE2EMAPIAgent\resources\agent\plugins

Custom implementation configurable properties

Property Description Sample value
exporter.default_target Use this property to set the external target system. Valid values are apm and kafka. You can also use a custom target. exporter.default_target=apm
exporter.client_connector_class Use this property to enable the trace export process to refer to specific classes during the processing. Update the full name of the specific class where you have implemented the logic to handle connectivity. Example: ${SW_AGENT_EXTERNAL_CLIENT_CONNECTOR_CLASS:com.softwareag.uhm.agent.exporter.impl.APMTraceExportConnectorImpl}
exporter.client_implementation_class Use this property to enable the trace export process to refer to specific classes during processing. Update the full name of the specific class where you have implemented the logic to process the traces to the external target server. Example: ${SW_AGENT_EXTERNAL_CLIENT_IMPLEMENTATION_CLASS:com.softwareag.uhm.agent.exporter.impl.APMTraceExportClientServiceImpl}
exporter.url Use this property to update the endpoint URL of the external target Kafka server. ${SW_AGENT_OTEL_ENDPOINT:localhost:9092}
exporter.headers Use this property to provide key value pairs containing authorization tokens and required headers that are separated by commas. The header key and value are separated by '#'. Example: ${SW_AGENT_OTEL_HEADERS:<REQUIRED_HEADERS>} key#value
exporter.username Use this property to provide the username of the target server. ${SW_AGENT_EXT_SYSTEM_USER:<username>}
exporter.password Use this property to provide the password of the target server. ${SW_AGENT_EXT_SYSTEM_PASSWORD: <password>}

Custom implementation configuration

  1. Connect all the implemented classes for connectivity and trace exporter to the corresponding properties in the agent.config file. Provide the full name of the connector and client implementation details in the properties exporter.client_connector_class and exporter.client_implementation_class. For example:
    #APM exporter
#com.softwareag.uhm.agent.exporter.impl.APMTraceExportConnectorImpl
#com.softwareag.uhm.agent.exporter.impl.APMTraceExportClientServiceImpl
  2. Add a reference entry of the external client library with full path at the end of the custom_wrapper.conf file at location <INSTALL-DIR>\profiles\IS_default\configuration. For example:
    • Integration Server and API Gateway: 
      wrapper.java.additional.504=-Xbootclasspath/a:..\..\..\IntegrationServer\instances\default\packages\WmE2EMIntegrationAgent\resources\agent\externalClient.jar;"%JAVA_BOOT_CLASSPATH%"
    • Microservices Runtime: 
      set JAVA_UHM_CLI_OPTS=-Xbootclasspath/a:"packages/WmE2EMIntegrationAgent/resources/agent/externalClient.jar"
set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS% %JAVA_UHM_CLI_OPTS%