Troubleshooting .NET tracing on Windows

Symptoms: Application shows metrics but no traces, some services report, but others don't, or tracing stops after a deployment or update.

Troubleshooting steps:

1. Verify that the .NET version of the application is supported:

   • .NET Core Runtime 5.0 or later
2. Verify that tracing is enabled in the agent configuration.yaml file.
3. Collect trace logs by using Log Collector or by collecting logs manually.

4. Check the Instana tracing logs:

   • If the logs are empty or no traces are visible, the application is not producing spans.
   • If traces are present in the logs but are missing information (for example, host), the problem is with the .NET tracing layer.
   • If traces are present in the logs and contain information that is missing in the UI, the problem occurred either in the Instana backend or the agent.

Symptom: Application fails on startup when the Instana agent is running.

Troubleshooting steps:

1. Verify that the .NET version of the application is supported:

   • .NET Core Runtime 5.0 or later
2. Gather Event Viewer logs by using Log Collector or by collecting logs manually.

3. Check the Instana Event Viewer logs:

   • Determine whether an error or warning appears in the logs. An event contains detailed information, including timestamps and error codes that can help diagnose problems.
   • If an error indicating InstanaPCP.exe exists, it means that InstanaPCP failed. Check the events preceding it and investigate the issue further.
4. Collect a crash dump for the application in question.

Symptom: The .NET application is not being monitored by the Instana agent and is not present in the UI.

Troubleshooting steps:

1. Verify that the Instana agent is running as administrator.
2. Verify that the Instana agent has permissions to the application.

3. Verify that Instana PCP is running:

   • Open Task Manager on Windows and search for InstanaPCP.

4. Collect Instana agent logs by using Log Collector or by collecting logs manually.

   • In the Instana agent logs, the following line should be present:

     com.instana.agent-process-handling - 1.0.35 |  | InstanaPCP.exe passed early window; marked RUNNING

5. Verify that sensors are successfully activated in the agent logs:

   com.instana.sensor-clr - 1.1.141 |  | Activated Sensor

6. Verify that the .NET process is started after the Instana agent.

Symptom: InstanaPCP or the .NET application shows unusually high CPU usage after tracing is enabled.

Troubleshooting steps:

1. Determine whether the issue is related to the application or InstanaPCP by checking the Task Manager.

2. If the issue is with InstanaPCP:

   • Enable debug logs for InstanaPCP. See Collecting logs.
   • Review the logs for recurring messages about missing performance counters or exceptions.

3. If the issue is an application memory spike:

   • Collect Event Logs and .NET logs by using Log Collector or by collecting logs manually.
   • Check Event Logs and .NET logs for memory-related exceptions.
   • Verify whether the issue is caused by Instana instrumentation.
   • Collect a .NET dump file for further analysis. Go to Task Manager, right‑click the process, and create a dump file.

Log Collector is a stand-alone application to collect logs from Instana. You can trigger the Log Collector from the following locations:

• Instana UI
• Console

The Log Collector collects the following logs:

• Application tracing logs
• Instana PCP logs
• CLR debug logs
• Event viewer logs
• Instana agent debug logs
• Instana agent traces logs

Alternatively, you can collect logs manually. Complete the following steps:

1. Enable debug logs:

   1. Add environment variables by using one of the following options:

      Note: Make sure that paths are valid, accessible by the process or application, and correct.

      • Set the following environment variables manually in the system level:

        INSTANA_NET_LOG_PATH: "C:\dc_net_log"
        INSTANA_NET_LOG_LEVEL: DEBUG
        INSTANA_NET_CORE_LOG_LEVEL: DEBUG
        INSTANA_NET_FF_LOG_LEVEL: DEBUG
        INSTANA_NET_PCP_LOG_LEVEL: DEBUG
        INSTANA_TRACER_ENTEREXIT_LOGGING: 1
        INSTANA_EXTENDED_DEBUG: 1

      • Open the command prompt as an administrator and run the following command to add the environment variables:

        setx INSTANA_NET_LOG_PATH "C:\dc_net_log" /M
        setx INSTANA_NET_LOG_LEVEL "DEBUG" /M
        setx INSTANA_NET_CORE_LOG_LEVEL "DEBUG" /M
        setx INSTANA_NET_FF_LOG_LEVEL "DEBUG" /M
        setx INSTANA_NET_PCP_LOG_LEVEL "DEBUG" /M
        setx INSTANA_TRACER_ENTEREXIT_LOGGING "1" /M
        setx INSTANA_EXTENDED_DEBUG "1" /M

   2. Configure the following Windows Registry key:

      CLRDebugLog: "C:\dc_clr_debug_log\clr_"

   3. Restart the application.

2. Collect Instana agent logs:

   1. Stop the agent <instana_install_dir>/bin/stop.sh or if the agent is started as a service, stop the service.

   2. Remove the logs: Delete all files in the log directory to start fresh by running the following command:

      rm -rf <instana_install_dir>/data/log/*

   3. Set the log level to Debug: Change the log level severity in the agent configuration.yaml file. For more information, see Setting up the agent configuration file.
   4. Start the agent <instana_install_dir>/bin/start.sh or start the service.
   5. Reproduce the issue: Allow the issue to occur and run the agent for approximately 15 minutes.

   6. Collect the logs: Compress the logs directory for submission by running the following command:

      cd <instana_install_dir>/data/logs
      zip -r instana_agent_debug_logs.zip *

3. Collect Instana trace logs:

   1. Go to [agent-dir]\etc\instana and open the file com.instana.agent.main.sender.File.cfg.

   2. Add the following lines in the file:

      prefix=instanaTraces
      type=traces

      This configuration provides you with a log file that contains all the traces in [agent-dir]/data/log with the instanaTraces prefix.

   3. Restart the Instana agent.
   4. Allow the application to run for approximately 15 minutes.

   5. Collect logs from the following paths:

      • Path specified by the variable INSTANA_NET_LOG_PATH
      • instana-agent-installation-folder/data/log

4. Collect CLR debug logs:

   1. Open Registry Editor.
   2. Navigate to Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Instana\
   3. Add the registry key CLRDebugLog as a string with value, for example, C:\temp\clr_.

   4. Alternatively, open the command prompt as an administrator and run following command:

      reg add "HKLM\SOFTWARE\Instana" /v CLRDebugLog /t REG_SZ /d "C:\temp\clr_" /f