(Wily) Java Agent Performance Tuning Recommendations

Body

The WebSphere L2 Support team handles its fair share of WebSphere Application Server performance analysis. Often it appears as though the Monitoring Agent (that thing that reports whether the system is behaving - or not) is the problem itself. The truth is, the more problems that the agent has to report, the more noise the agent is going to interject into the diagnostic data. It's a vicious cycle.

One of the common agents that are encountered is Wily. Like any performance monitoring tool, Wily can introduce a significant amount of noise into WebSphere Application Server diagnostic data. Wily's performance overhead along with the noisy logging can often skew performance analysis.

In order to eliminate agent noise and false positives, WebSphere Support often requests that clients disable Wily. Disabling the agent is only a temporary measure for testing purposes. Once the diagnostic data is captured, Wily can be re-enabled. However, most clients are reluctant to do disable Wily (even temporarily) since they depend on Wily to monitor the health of their system. So how is a proper analysis to be done?

Although IBM Support is not directly qualified to assist clients in tuning their Wily environment, there exists tuning recommendations that are written by the Wily team. Wily tuning can help reduce Wily noise that will assist IBM Support to positively identify the underlying issue. Proper tuning of the Java™ agent promotes harmony between the application server environment and the Java agent:

The Java Agent Performance Tuning Recommendations document

1. Provides Wily performance tuning recommendations
2. Reminds Wily users that all processes that interact with WebSphere Application Server require tuning for optimal performance

Java Agent Performance Tuning Recommendations

CPU

Disabling the aggregate metrics will help reduce the overall metric load per agent. Metrics show CPU utilization from both JVM and non-JVM processes.

Introscope.agent.disableAggregateCPUUtilization=true

SQL

SQLAgent could potentially generate a large number of unique SQL metrics (aka metric explosion). Reducing the length of SQL statements will help agents to manage the metric load.

Introscope.agent.sqlagent.sql.maxlength=xxx where xxx between 1 and 990

Modify sqlagent.pbd and remove the substring {sql}, which represents the normalized SQL statement. You will only get summary SQL statistics for databases.

Dynamic Instrumentation

Disabling dynamic instrumentation will help reduce overhead on the application caused by the agent. Consider disabling on all production-like agents.

introscope.autoprobe.dynamicinstrument.enabled=false

Remote Dynamic Instrumentation

Disabling remote dynamic instrumentation will help reduce overhead on the application caused by the agent. Consider disabling on all production-like agents.

introscope.agent.remotedynamicinstrumention.enabled=false

Logging

Logging produces I/O overhead for every agent instance. Consider disabling agent logging on production-like agents and leaving the autoprobe for troubleshooting.

introscope.autoprobe.logfile=logs/AutoProbe.log

log4j .appender.logfile=/dev/null

log4j.logger.IntroscopeAgent=OFF

Reduce Memory Overhead

Reducing memory overhead while trading some impact to response times.

introscope.agent.reduceAgentMemoryOverhead=true

The original document here:

Java Agent Performance Tuning Recommendations | CA Technologies Communities
⇒https://communities.ca.com/docs/DOC-231148897

Another Wily tuning document:

CA Technologies Global Webcast Jan 27 2011 Introscope 8x Per... | CA Technologies Communities
⇒https://communities.ca.com/docs/DOC-14500143

title image (modified) credit: (cc) Some rights reserved by geralt