Monitoring .NET

Health Signatures

For each sensor, there is a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.

For information about built-in events for the .NET sensor, see the Built-in events reference.

Setting up Tracing for .NET

The tracing component setup is a mostly automatic function of the agent. However, there are is a manual step involved, which we describe in the following sections.

INFO  | com.instana.agent - 1.1.326 | Writing location for instrumentation-advices to registry
INFO  | com.instana.agent - 1.1.326 | Setting up tracing-environment
INFO  | com.instana.agent - 1.1.326 | Registering native profiler for .NET
INFO  | com.instana.agent - 1.1.326 | Agent UUID received:ac:bc:32:ff:fe:90:7d:bb
INFO  | com.instana.agent - 1.1.326 | Registering System.Collections.Immutable.dll in GAC
INFO  | com.instana.agent - 1.1.326 | .NET Tracing is enabled
INFO  | com.instana.agent - 1.1.326 | Registering instrumentation-DLLs in GAC
INFO  | com.instana.agent - 1.1.326 | Registering Instana.ManagedTracing.dll in GAC
INFO  | com.instana.agent - 1.1.326 | Setting environment-variables for .NET tracing
INFO  | com.instana.agent - 1.1.326 | Registry-Key for WAS-Environment does not exist, will create now
INFO  | com.instana.agent - 1.1.326 | Setting environment for WAS
WARN  | com.instana.agent - 1.1.326 | You should consider restarting the service WAS in order to make it use the new profiler. Already running application will not use it until restarted.
INFO  | com.instana.agent - 1.1.326 | Registry-Key for W3SVC-Environment does not exist, will create now
INFO  | com.instana.agent - 1.1.326 | Setting environment for W3SVC
WARN  | com.instana.agent - 1.1.326 | You should consider restarting the service W3SVC in order to make it use the new profiler. Already running application will not use it until restarted.
INFO  | com.instana.agent - 1.1.326 | Setting memoryUsageExcessThreshold set to configured value of 100000000
INFO  | com.instana.agent - 1.1.326 | Self-Monitoring for Windows-Extensions has been started successfully
INFO  | com.instana.agent - 1.1.326 | Setting memoryUsageWarningThreshold set to configured value of 80000000
INFO  | com.instana.agent - 1.1.326 | Starting tracing-session
INFO  | com.instana.agent - 1.1.326 | Windows-Extension REST-Endpoint has been setup successfully at http://localhost:42657
INFO  | com.instana.agent - 1.1.326 | Registering Instana.ManagedTracing.Modules.dll in GAC
INFO  | com.instana.agent - 1.1.326 | Instana ETW-Reader setup and listening

How Tracing Works

Once you enabled tracing by changing the configuration-file, trace-data will be collected for all applications that make use of the supported technologies. Since .NET-Tracing follows the same scheme as all the other tracing-packages, distributed traces in polyglot applications will automatically be collected and displayed in Instana's UI.

When an application with tracing enabled starts up, you might find the startup-time to be increased in comparison to startup without tracing. This is due to the profiler attaching, inspecting the classes contained in the referenced assemblies, and then just-in-time compiling some methods' code for instrumentation. After the instrumentation has been applied, the impact on the runtime performance of your applications is extremely low, as with any other tracing-package that Instana provides.

Disabling Tracing for .NET

When you want to stop tracing of .NET applications for whatever reason, you can simply do so by changing the configuration file back to its initial state, or by just setting the corresponding flag to "false."

configuration.yaml with .NET-Tracing disabled

# CLR Tracing
    com.instana.plugin.clr:
      tracing:
        enabled: false

After changing the config back, you will have to restart the agent for the changes to take effect.

Due to the nature of how the Profiling API works, disabling tracing in the agent will not remove / detach the profiler from processes that have been started with tracing enabled (i.e. your worker-processes). To make sure the profiler is not loaded into the application, you have to restart the host-process.

Enabling Tracing for Non-IIS Hosted Applications

The Instana Agent does not enable tracing for every .NET based application. This is on purpose, since it might produce undesired component behavior like tracing Windows system applications that are not in scope of your monitoring needs.

While IIS and WAS get configured to attach the profiler automatically by setting the appropriate environment-variables, standalone-processes are not. You can enable tracing for these processes by providing the necessary environment-variables by either setting them on the user level (when you use specific technical users which run your applications), or by setting the environment-variables before starting the actual process.

To enable tracing for a specific process you have to provide the following environment-variables with their respective values:

Environment variables needed for tracing

SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={FA8F1DFF-0B62-4F84-887F-ECAC69A65DD3}

You can put these settings into a batch or powershell-script which starts your processes after setting the env-vars. If you are running Windows-Services you can also put these settings into the registry, on a per-service basis, which gives you better control over which services to trace.

Go to HKLM\System\CurrentControlSet\Services\<ServiceName>, add a MULTI_SZ value by the name of Environment (if it doesn't already exist), and then copy the variables COR_ENABLE_PROFILING=1 and COR_PROFILER={FA8F1DFF-0B62-4F84-887F-ECAC69A65DD3} into that value.

To correctly set the contents of a MULTI_SZvalue, put one key-value-pair per line like displayed as follows:

COR_ENABLE_PROFILING=1
COR_PROFILER={FA8F1DFF-0B62-4F84-887F-ECAC69A65DD3}

Hit enter after last line.

Custom Tracing

Instana's .NET Tracing SDK enables to your own custom tracing in situation where our automatic instrumentation does not apply (i.e., you are using components we are not automatically supporting).

Environment variable not defined

Monitoring issue type: clr_env_var_not_defined

One or more environment variables, which are necessary to activate the tracing of .NET application, are not set correctly.

If this happens for applications hosted on IIS, it might be that the iisreset has not been issued at all, or before the agent could register the libraries needed for tracing. As a solution, issue an iisreset from an elevated prompt.

If this does not help, consider restarting the machine, which might be needed depending on the version of Windows you are running and whether there were (un-)installations happening before running the instana agent.

For applications not hosted on IIS, refer to the Enabling Tracing for Non-IIS Hosted Applications section for the list of environment variables and the respective values you need to set for tracing your .NET applications.

Environment variable has an invalid value

Monitoring issue type: clr_env_var_invalid_value

At least one environment variable that is necessary to activate the Instana .NET tracing, does not have the right value. (Depending on which environment variables are missing, you may still see metrics in the Instana dashboards for your .NET application, but definitely not traces.) Please refer to the Enabling Tracing for Non-IIS Hosted Applications section for the list of environment variables and the respective values you need to set for tracing your .NET Core applications with Instana.

Also, it might be the case that the reason for the environment variable to have the wrong value is the presence of another APM tool. (In terms of hooking into the .NET runtime, the mechanisms the various APM use are pretty similar.) If we recognize this situation, you will see it in the message in the Instana user interface. Before Instana can trace your .NET Core application, you will have turn off the other APM agent, at least insofar .NET Core monitoring is concerned; for how to do that, refer to the other APM tool's documentation.

InstanaPCP not running

Monitoring issue type: clr_instana_pcp_not_running

You might find warning messages in your agent's log, stating that InstanaPCP does not seem to be running. This message is triggered whenever the agent tries to interact with InstanaPCP but fails to do so. Usually the agent starts InstanaPCP early in the bootstrapping-process and can then communicate with it. Under the following circumstances, InstanaPCP might not be able to start:

• The agent is not running with sufficient priviliges (i.e. Local System / Administrator)
• The host does not have .NET Framework 4.5 installed
• InstanaPCP restarted itself due to memory overconsumption

In case you see this message repeatedly appear, check for the following:

1. Make sure that your agent runs with a user that has local administrative priviliges. If this is not the case, change the user used to run the agent, or grant it local administrative priviliges. When installing Instana from the installation-package, the service will be run as Local System normally and thus have sufficient priviliges.

2. Check which version of the .NET Framework you have installed on your machine. The Windows-Installer package will automatically install the minimum version InstanaPCP needs, but if you happen to install the agent from a ZIP-archive, you might have to install the framework yourself. If you are unsure whether you have the correct version or not, you can try to run InstanaPCP manually from a command-prompt by locating it under <agent-dir>\data\repo\com\instana\agent-windows-extensions\<latest>\agent-windows-extensions-<latest>. If you do not have the correct framework-version installed, you will see an according prompt.

3. If you see messages in the log that point to over-consumption of memory like these:

   InstanaPCP.exe Private-Bytes is above warning-treshhold for XXX times now
   
   Killing InstanaPCP.exe for excessive memory-usage. Possible memory-leak in native Windows-Libraries.
   

   These messages can point to one of two potential issues:

   1. You are running on a Windows-Machine with an old version of IIS7 having a specific memory-leak. This leak can be patched by installing a Microsoft-Update. This might happen on Windows 2008 machines. Before you install this Microsoft-Update, contact Instana Support to make sure that the memory-leak issue happens on your machine. The support team will then provide you with the link to update your Microsoft package.

   2. You are running a lot of .NET applications on the server and those applications are generating a lot of traces. In this case, you might need to increase the memory limits for InstanaPCP by applying changes to these settings in your configuration.yaml file:

      com.instana.plugin.clr:
        pcp_warn_limit: <MB-treshold before warning>
        pcp_stop_limit: <MB-treshold  to stop / restart the process>
      

InstanaPCP not connected

Monitoring issue type: clr_instana_pcp_not_connected

The host agent cannot connect to the InstanaPCP process, which prevents the retrieval and processing of tracing data and runtime metrics for .NET processes.

This is usually a transient failure, which will resolve itself quickly after the InstanaPCP has finished intializing. If this is not the case, reach out to the Instana support.