Troubleshooting .NET Framework tracing on Windows

Symptoms: Application shows metrics but no traces, some services report, but others don't, or tracing stops after a deployment or update.

Troubleshooting steps:

1. Verify that the .NET version of the application is supported:

   • .NET Framework 4.5.2 or later
2. Verify that tracing is enabled in the agent configuration.yaml file.
3. Collect trace logs by using Log Collector or through manual logs collection.

4. Check the Instana tracing logs and interpret findings:

   • If the logs are empty or no traces are visible, the application is not producing spans.
   • If traces are present in the logs but are missing information (for example, host), the problem is with the .NET tracing layer.
   • If traces are present in the logs and contain information that is missing in the UI, the problem occurred either in the Instana backend or the agent.

Symptom: Application crashes on startup when the Instana agent is running.

Troubleshooting steps:

1. Verify that the .NET version of the application is supported:

   • .NET Framework 4.5.2 or later
2. Gather Event Viewer logs by using Log Collector or through manual logs collection.

3. Check the Instana Event Viewer logs:

   • Determine whether an error or warning appears in the logs. An event contains detailed information including timestamps and error codes that can help diagnose problems.
   • If there is an error indicating InstanaPCP.exe, it means that InstanaPCP crashed. Check the events prior to it and investigate the issue further.
4. Collect a crash dump for the application in question.

Symptom: The .NET application is not being monitored by the Instana agent and is not present in the UI.

Troubleshooting steps:

1. Verify that the Instana agent is running as administrator.
2. Verify that the Instana agent has permissions to the application.

3. Verify that Instana PCP is running:

   • Open Task Manager on Windows and search for InstanaPCP.

4. Collect Instana agent logs by using Log Collector or by manual logs collection.

   • In the Instana agent logs, the following line must be present:

     com.instana.agent-process-handling - <version> |  | InstanaPCP.exe passed early window; marked RUNNING

5. Confirm that sensors are successfully activated in the agent logs:

   com.instana.sensor-clr - <version> |  | Activated Sensor

6. Ensure that the .NET process is started after the Instana agent.

Symptom: InstanaPCP or the .NET application shows unusually high CPU usage after tracing is enabled.

Troubleshooting steps:

1. Determine whether the issue is related to the application or InstanaPCP by checking the Task Manager.

2. If the issue is with InstanaPCP:

   • Enable debug logs for InstanaPCP. See Collecting logs.
   • Review the logs for recurring messages about missing performance counters or exceptions.

3. If the issue is an application memory spike:

   • Collect Event Logs and .NET logs by using Log Collector or by collecting logs manually.
   • Check Event Logs and .NET logs for memory-related exceptions.
   • Verify whether the issue is caused by Instana instrumentation.
   • Collect a .NET dump file for further analysis.

Log Collector is a stand-alone application to collect logs from Instana. You can trigger the Log Collector from the following locations:

• Instana UI
• Console

The Log Collector collects the following logs:

• Application tracing logs
• Instana PCP logs
• CLR debug logs
• Event viewer logs
• Instana agent debug logs
• Instana agent traces logs

Alternatively, you can collect logs manually. Complete the following steps:

1. Enable debug logs:

   1. Add environment variables by using one of the following options:

      Note: Make sure that paths are valid, accessible by the process or application, and correct.

      • Set the following environment variables manually in the system level:

        INSTANA_NET_LOG_PATH: "C:\dc_net_log"
        INSTANA_NET_LOG_LEVEL: DEBUG
        INSTANA_NET_FF_LOG_LEVEL: DEBUG
        INSTANA_NET_PCP_LOG_LEVEL: DEBUG
        INSTANA_TRACER_ENTEREXIT_LOGGING: 1
        INSTANA_EXTENDED_DEBUG: 1

      • Open the command prompt as an administrator and run the following command to add the environment variables:

        setx INSTANA_NET_LOG_PATH "C:\dc_net_log" /M
        setx INSTANA_NET_LOG_LEVEL "DEBUG" /M
        setx INSTANA_NET_FF_LOG_LEVEL "DEBUG" /M
        setx INSTANA_NET_PCP_LOG_LEVEL "DEBUG" /M
        setx INSTANA_TRACER_ENTEREXIT_LOGGING "1" /M
        setx INSTANA_EXTENDED_DEBUG "1" /M

   2. Configure the following Windows Registry key:

      CLRDebugLog: "C:\dc_clr_debug_log\clr_"

   3. Restart the application.

2. Collect Instana agent logs:

   1. Stop the agent <instana_install_dir>/bin/stop.sh or if the agent is started as a service, stop the service.

   2. Remove the logs: Delete all files in the log directory to start fresh by running the following command:

      rm -rf <instana_install_dir>/data/log/*

   3. Set log level to Debug: Change the log level severity in the agent configuration.yaml file. For more information, see Setting up the agent configuration file.
   4. Start the agent <instana_install_dir>/bin/start.sh or start the service.
   5. Reproduce the issue: Allow the issue to occur and let the agent run for approximately 15 minutes.

   6. Collect the logs: Compress the logs directory for submission by running the following command:

      cd <instana_install_dir>/data/logs
      zip -r instana_agent_debug_logs.zip *

      For PowerShell:

      Compress-Archive * instana_agent_debug_logs.zip

3. Collect Instana trace logs:

   1. Go to [agent-dir]\etc\instana and open the file com.instana.agent.main.sender.File.cfg.

   2. Add the following lines in the file:

      prefix=instanaTraces
      type=traces

      This configuration provides you with a log file containing all the traces in [agent-dir]/data/log with the prefix of "instanaTraces".

   3. Restart the Instana agent.
   4. Restart the application and run it for approximately 15 minutes.

   5. Collect logs from the following paths:

      • Path specified by the variable INSTANA_NET_LOG_PATH
      • instana-agent-installation-folder/data/log

4. Collect CLR debug logs:

   1. Open Registry Editor.
   2. Go to Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Instana\
   3. Add the registry key CLRDebugLog as a string with value, for example, C:\temp\clr_.

   4. Alternatively, open the command prompt as an administrator and run following command:

      reg add "HKLM\SOFTWARE\Instana" /v CLRDebugLog /t REG_SZ /d "C:\temp\clr_" /f

5. Collect Windows Event Viewer logs:

   1. Open Event Viewer:

      1. Press Win + R.
      2. Enter eventvwr.msc.
      3. Press Enter.

   2. Go to the log that you want to export:

      • Windows Logs: Windows Logs > Application
      • Applications and Services Logs
   3. In the Actions panel, select Save All Events As.
   4. Choose a location and save as .evtx.