Monitoring Windows processes

Edit online

On Windows, the InstanaPCP application starts automatically when the agent starts. The InstanaPCP application performs the following actions:

Sets up the environment for automatic tracing of IIS-hosted applications.
Collects metrics for running processes (such as GC and CPU) and sends them to the agent.
Sends a list of running processes on the Windows host to the agent.
Collects custom performance counter metrics defined in the configuration.yaml file.

Performance counters

Edit online

Instana supports two methods of collecting processes and its metrics from a Windows host.

The first method (default method):

When InstanaPCP starts, it creates a ProcessNameFormat setting with a value of 2 in the Registry. This setting changes the process name format from processname to processname_processid in the following Registry paths:

HKLM:\SYSTEM\CurrentControlSet\Services\.NETFramework\Performance
HKLM:\SYSTEM\CurrentControlSet\Services\PerfProc\Performance

The second method:

This method uses rewritten logic and has no need for changing the process name format. To use this method, set the INSTANA_COLLECT_METRICS_BY_PROCESS_READING environment variable to either true or 1 at the host level. This method provides more flexibility by not changing the process name format and will be the default and only supported option in the future.

Custom performance counters

Edit online

Instana supports monitoring custom performance counters. The custom performance counters are disabled by default. To enable custom performance counters, update the configuration.yaml file as shown in the following example:

com.instana.plugin.perfcounters:
  enabled: true

Performance counters are defined with four different values and names. The performance counters are grouped by category. They can be bound to a specific instance or are host-wide. To uniquely describe a counter, a syntax similar to the following example is used:

[host]\[category]([instance])\[counter name]

instance might be optional depending on the counter. The following counter is an example:

\Processor Information(_Total)\% Processor Time

The attributes of the counter in the example are explained in the following list:

host is empty. The counter is for the current host (localhost).
category is Processor Information.
instance is _Total.
counter name is % Processor Time.

Instana supports only querying localhost counters.

Some categories are only host-wide; instance is not defined for these categories. The following example shows a host-wide category:

\System\Processor Queue Length

After the flag is enabled, define the counters that you want to monitor. Update the configuration as shown in the following example:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: [name of the counters category]
      instance: [instance name]
      counters:
        - [name of the counter]

The following output shows sample values for counters:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: Processor Information
      instance: _Total
      counters:
        - "% Processor Time"

Note: The counter name is enclosed with double quotation mark (") characters because of the special character %_

If you need more counters per category, just add them to the counters array as shown in the following example:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: Memory
      counters:
        - Page Faults/sec
        - Cache Faults/sec
        - Page Reads/sec

Note: The instance is not defined in this example because the Memory category is host-wide.

Note: Querying performance counters is an inherently expensive operation. If you have a lot of counters, it impacts the performance of the agent.

The following example shows multiple categories and counters:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: System
      counters:
        - Processor Queue Length
        - System Calls/sec
    - category: Process
      instance: _Total
      counters:
        - Private bytes
        - Working Set
        - Thread Count
    - category: Memory
      counters:
        - Page Faults/sec
        - Cache Faults/sec
        - Page Reads/sec
    - category: .NET CLR Exceptions
      instance: _Global_
      counters:
        - "# of Exceps Thrown / sec"
        - "# of Pinned Objects"

The following example shows a host dashboard with performance counters:

Figure 1. Example of Performance Counters dashboard

For more assistance, open a support case.

Troubleshooting

Edit online

If you encounter any other issues, contact IBM Support.

InstanaPCP not running

Edit online

Monitoring issue type: clr_instana_pcp_not_running

You might find the warning message InstanaPCP does not seem to be running in your agent's log. This message is displayed when the agent fails to interact with InstanaPCP. The agent starts InstanaPCP early in the bootstrapping process, and then the agent can communicate with InstanaPCP. The agent cannot start InstanaPCP early in the bootstrapping process if any of the following errors occur:

The agent is not running with sufficient privileges (not running as Local System or Administrator).
The .NET Framework version on the host is not supported. Only .NET Framework 4.5.1 and later versions are supported.
InstanaPCP restarted itself due to memory overconsumption.

If this message appears repeatedly, complete the following checks:

Check whether the user that runs the agent has local administrative privileges. If the user doesn't have local administrative privileges, grant local administrative privileges to the user or use a different user account. When you install the Instana agent from the installation package, the service is normally run as Local System that gives sufficient privileges to the service.
Check the .NET Framework version that is installed on your machine. The Windows installer package automatically installs the minimum version of .NET Framework that is required for InstanaPCP. However, if you install the host agent from a ZIP-archive, you might need to install .NET Framework. If you are unsure whether you have the correct version of .NET Framework, go to the \system\com\instana\agent-windows-extensions<latest>\agent-windows-extensions--<latest> directory, and then manually run InstanaPCP as a stand-alone set-up from the command line. If you do not have the correct .NET Framework version installed, a message is displayed.

Check whether you see messages that are related to overconsumption in the log as follows:

InstanaPCP.exe Private-Bytes is above warning-treshhold for XXX times now

Killing InstanaPCP.exe for excessive memory-usage. Possible memory-leak in native Windows-Libraries.

These messages are displayed if any of the following issues occurs:

On the Windows machine where InstanaPCP is running, the IIS version is earlier than 7, and a memory leak occurred. This leak can be patched by installing a Microsoft update. This problem might occur on Windows 2008 machines. Before you install this Microsoft update, contact IBM Support to check whether a memory leak occurred on your machine. The IBM Support team can provide the link to update your Microsoft package.

Many .NET applications are running on the server, and those applications are generating many traces. Increase the memory limits for InstanaPCP by applying changes to the following settings in your agent configuration.yaml file:
```
com.instana.plugin.clr:
  pcp_warn_limit: <MB-treshold before warning>
  pcp_stop_limit: <MB-treshold  to stop / restart the process>
```

InstanaPCP not connected

Edit online

Monitoring issue type: clr_instana_pcp_not_connected

The host agent cannot connect to the InstanaPCP process, which prevents the retrieval and processing of tracing data and runtime metrics for .NET processes.

This issue is usually a transient failure, which resolves itself quickly after the InstanaPCP finishes initializing. If this issue is not resolved, contact IBM Support.