IBM Support

High CPU and I/O utilization in IBM InfoSphere Guardium STAP host

Troubleshooting


Problem

You observe a high CPU and/or I/O usage by the IBM InfoSphere Guardium STAP process.

Symptom

High CPU and/or I/O usage on the database server.

Cause

Some of the common causes are:

    1. An error in the configuration of one of the inspection engines. If there are errors in an inspection engine, the STAP process may be restarting frequently or spending a lot of time trying to reconnect to the inspection engine repeatedly.
    2. The KTAP portion of the STAP is sending connection information along with a confirmation request to the STAP to confirm that the session is the database connection configured in the inspection engine, and this is causing delays.
    3. ORACLE RAC is used, but unix_domain_socket_marker parameter is not set in the STAP configuration file to avoid monitoring potentially large amounts of Oracle RAC traffic.
    4. User ID Chain (UID chain) feature is enabled, ie. parameter hunter_trace=1 in the STAP configuration file. Hunter trace is used for UID chain and can be quite CPU intensive for STAP.
    5. The firewall is enabled (firewall_installed=1). This forces STAP to request verdicts for each new session being observed, which can hurt STAP performance.

Resolving The Problem

  • Review the configuration for all the inspection engines and make sure there are no errors in any of the parameters. For example, make sure the database install directory, executable, ports and any other parameters that are applicable to your inspection engine, are correctly set with no misspellings or wrong values.
  • Set STAP configuration parameter ktap_fast_tcp_verdict to 1 ( ktap_fast_tcp_verdict = 1 in the guard_tap.ini configuration file) and restart the STAP.

    Possible settings:
    ktap_fast_tcp_verdict=0: For tcp connection, ktap will send ioctl to stap to confirm that session is the database connection configured in the inspection engine by checking ports and Ips.

    ktap_fast_tcp_verdict=1: For tcp connection, ktap will not send the request to STAP as long as session's ports are in the range.
  • If using ORACLE RAC, set unix_domain_socket_marker parameter according to the article STAP v8.2 running in Oracle RAC/Linux cluster can potentially cause a server crash below. This will prevent STAP from monitoring potentially large amounts of Oracle RAC traffic.
  • Disable UID Chain feature if not needed by setting hunter_trace=0 and restarting the STAP.
  • Set firewall_installed=0 if SGATE functionality is not needed and restart the STAP.

[{"Product":{"code":"SSMPHH","label":"IBM Security Guardium"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"}],"Version":"9.1;9.0;8.2;8.0.1;8.0;9.5","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
16 June 2018

UID

swg21615502