Performance issue: analyzer queue overflow

Symptoms

Key columns in the Buffer usage monitor report: Analyzer Rate, Analyzer Queue, Flat Log Requests.

You can use Alerting on Analyzer Queue Overflow to help identify symptoms.

Analyzer queue is tracked in the Unit Utilization Level and Deployment Health View. A High status on analyzer queue in these views indicates a likely analyzer queue overflow. You can view the the buffer usage report on the individual collector to confirm.

To alert directly on flat log requests, see Predefined alerts. By default, the alert is set to send to syslog only. Add any receivers that are required. Confirm the alert is active from Setup > Tools and Views > Anomaly Detection.

Causes

There are several reasons for issues with the Analyzer Queue overflowing, but the most common reason is that the sniffer cannot cope with the high rate of traffic that is monitored.

Diagnosing the problem

The analyzer part of the sniffer has a circular buffer. When the queue is full any incoming data is dropped. The amount of dropped data from the last minute is logged in flat log requests. If there was data dropped by the analyzer in the last minute, the flat log requests increase. Increasing flat log requests is the key indicator of analyzer queue overflow. For a healthy sniffer it should not be increasing.

Resolving the problem

In the case of high rate of monitored traffic, you must reduce the amount of traffic that is monitored by the appliance by using one of the following strategies:

  • Introducing rules to filter more traffic. The most effective rule action to achieve filtering is the Ignore S-TAP Session rule because the sessions are ignored by the S-TAP instead of being sent across the network to the appliance.
  • Moving some of the S-TAPs to a less loaded collector
  • S-TAP load balancing. Sometimes, a busy database server alone can overwhelm a collector. In these cases, it might help to load balance the traffic from this database to two or more collectors (for more information, see Linux-UNIX: S-TAP Load Balancing models and configuration guidelines or Windows: S-TAP Load Balancing models and configuration guidelines
  • Consider using a Selective Audit policy. By default, the collector logs all data that is sent to it from S-TAPs or Hardware TAPs. A Selective Audit policy changes this behavior by monitoring only the database traffic that is specified in the policy rules.
  • Adding more collectors to the environment.