Troubleshooting .NET tracing on z/OS

Edit online

If tracing is not working as expected, start with general troubleshooting steps before you move to case-specific scenarios.

General troubleshooting

Edit online

Complete the following steps:

  1. Verify prerequisites:

    • Check .NET version compatibility: Ensure that your application runs on .NET Core Runtime 5.0 or later.
    • Verify that the Instana agent is running on the z/OS system.
    • Check agent status in the Instana UI.

  2. Verify environment variables: If tracing is not working, check that the environment variables are:

    • Set correctly.
    • Spelled accurately.
    • Set for the z/OS environment.
    • Valid, accessible by the process or application, and correct.

  3. Verify that the Instana agent endpoint is reachable from the z/OS environment:

    • Use telnet agent_host agent_port (for example, telnet localhost 42699) to test connectivity.

  4. Verify SDK integration:

    • For Instana SDK:

      • Make sure the Instana.Tracing.Core.Sdk package is properly installed.
      • Ensure that custom spans are being created in your code by using CustomSpan.Create().

    • For OpenTelemetry SDK:

      • Make sure all the required OpenTelemetry packages are installed (OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol).
      • Verify that AddOpenTelemetry() is called in your service configuration.
      • Ensure that the OTLP exporter is properly configured to point to the Instana agent (default: http://localhost:4317 for gRPC or http://localhost:4318 for HTTP).
      • Verify that OpenTelemetry plug-ins are enabled in the agent configuration.yaml file.
      • Ensure that custom spans are being created by using ActivitySource.
    Important: Do not use both SDKs simultaneously in the same application as it might lead to duplicate spans, increased overhead, and potential conflicts in trace context propagation.

Case-specific troubleshooting

Edit online

If general troubleshooting doesn't resolve your issue, check out the following troubleshooting scenarios:

Scenario 1: Missing traces or spans

Edit online

Symptom: No traces are being generated, or some services report traces, but others don't.

Troubleshooting steps:

  1. Verify that the .NET version of the application is supported:

    • .NET Core Runtime 5.0 or later
  2. Verify that manual instrumentation is properly implemented by using either Instana SDK or OpenTelemetry SDK.

  3. Check whether the SDK packages are correctly installed:

    • For Instana SDK: Instana.Tracing.Core.Sdk
    • For OpenTelemetry SDK: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol
  4. Verify that custom spans are being created in the application code.
  5. Check Instana agent logs for any connection issues.
  6. If you are using OpenTelemetry SDK, verify that the OTLP exporter endpoint is configured correctly (default: http://localhost:4317 for gRPC).
  7. Check whether the Instana agent has OpenTelemetry plug-ins (com.instana.plugin.opentelemetry) enabled in configuration.yaml.
  8. Collect Instana tracing logs.

  9. Check Instana tracing logs:

    • If logs are empty or no traces are visible in them, spans are not being created properly in the application code.
    • If traces are present in the logs and they contain information that was missing in the UI, the problem occurred either in the backend or agent.

Scenario 2: Application fails to start

Edit online

Symptom: The application fails on startup when Instana SDK or OpenTelemetry SDK is integrated.

Troubleshooting steps:

  1. Verify that the .NET version of the application is supported:

    • .NET Core Runtime 5.0 or later
  2. Verify that the SDK packages are compatible with the .NET runtime version.
  3. Check application logs for any SDK initialization errors.
  4. Verify that the Instana agent endpoint is accessible from the z/OS environment by using telnet agent_host agent_port (for example, telnet localhost 42699).
  5. Collect a crash dump for the application in question if available.

Scenario 3: Agent not detecting .NET application

Edit online

Symptom: The .NET application is not being monitored by the Instana agent and is not present in the UI.

Troubleshooting steps:

  1. Verify that the Instana agent is running on z/OS.
  2. Confirm that the Instana agent has permissions to access the application.

  3. Ensure that .NET Core tracing is enabled in the agent configuration file (*instanaAgentDir*/etc/instana/configuration.yaml):

    com.instana.plugin.netcore:
  tracing:
    enabled: true
    Note: You must restart the Instana agent after changing the configuration file.

  4. If you are using OpenTelemetry SDK, verify that OpenTelemetry plug-ins are enabled:

    com.instana.plugin.opentelemetry:
  grpc:
    enabled: true
  http:
    enabled: true
    Note: You must restart the Instana agent after changing the configuration file.
  5. Verify that the application is creating spans by using the SDK.
  6. Check whether the service name is properly configured in the SDK initialization.

Scenario 4: High CPU or memory usage after enabling SDK-based tracing

Edit online

Symptom: The application shows unusually high CPU or memory usage after SDK-based tracing is enabled.

Troubleshooting steps:

  1. Identify the source:

    • Determine whether the issue is related to the application by checking z/OS system monitoring tools.

  2. If an application memory or CPU spike is present:

    • Check application logs for memory-related exceptions or performance issues.
    • Verify whether the issue is caused by excessive span creation in the application code.
    • Check whether spans are being properly closed or disposed by using using statements or explicit disposal.
    • Review the frequency and volume of custom spans being created.
    • Request application logs and memory dumps for further analysis.

Scenario 5: SDK configuration issues

Edit online

Symptom: Traces are not appearing in Instana UI despite the SDK being integrated.

Troubleshooting steps:

  1. Verify that the SDK is properly initialized in the application startup code.

    • For Instana SDK:

      • Verify that spans are created with proper SpanType (ENTRY, INTERMEDIATE, and EXIT).
      • Ensure that spans are properly disposed by using using statements.

    • For OpenTelemetry SDK:

      • Verify that AddOpenTelemetry() is called in the service configuration.
      • Check whether the OTLP exporter endpoint is correctly configured (default: http://localhost:4317).
      • Verify that ActivitySource is properly registered and used.
      • Ensure that the service name is configured in the resource builder.

  2. Check network connectivity between the application and the Instana agent:

    • Test connectivity by using telnet agent_host agent_port (for example, telnet localhost 42699 for Instana SDK or telnet localhost 4317 for OpenTelemetry SDK with gRPC).
    • Check Instana agent logs for any connection attempts or errors.
    • Review z/OS network configuration and security policies.

  3. Verify whether firewall rules allow communication on the required ports:

    • 42699: Default Instana agent port (for Instana SDK)
    • 4317: gRPC OTLP endpoint (for OpenTelemetry SDK)
    • 4318: HTTP OTLP endpoint (for OpenTelemetry SDK, if you are using HTTP instead of gRPC)

Collecting logs

Edit online

z/OS does not support automatic log collection. Therefore, you must collect logs manually.

To collect logs manually, complete the following steps:

  1. Enable debug logs:

    Set the following environment variables manually at the system level and ensure that paths are valid, accessible by the process or application, and correct:

    INSTANA_NET_LOG_PATH=log_path
INSTANA_NET_LOG_LEVEL=DEBUG
  2. Restart the application.

  3. To collect Instana agent logs, complete the following steps:

    1. Stop the agent:

      instana_install_dir/bin/stop.sh

    2. Remove the logs: Delete all files in the logs directory to start fresh:

      rm -rf instana_install_dir/data/logs/*
    3. Set the log level to Debug: Change the log level severity in the configuration file. For more information, see Setting up the agent configuration file.

    4. Start the agent:

      instana_install_dir/bin/start.sh
    5. Reproduce the issue: Allow the issue to occur and run the agent for approximately 15 minutes.

    6. Collect the logs: Compress the logs directory for submission:

      cd instana_install_dir/data/logs/
zip -r instana_agent_debug_logs.zip *

  4. To collect Instana trace logs, complete the following steps:

    1. Go to agent_dir/etc/instana/com.instana.agent.main.sender.File.cfg.

    2. Add the following lines in the file:

      prefix=instanaTraces
type=traces

      This configuration provides a log file that contains all the traces in agent_dir/data/log with the prefix of instanaTraces.

    3. Restart the Instana agent.
    4. Restart the application.
    5. Allow it to run for approximately 15 minutes.

    6. Verify that traces are being captured:

      • Check the Instana UI by going to Applications > Services and look for your service name.
      • Verify that traces and spans appear with the expected service and endpoint names.

    7. Obtain the logs from the following paths:

      • Path specified by the variable INSTANA_NET_LOG_PATH
      • instana-agent-installation-folder/data/log