IBM Support

QRadar: Netskope Active events can be missed due to a short recurrence value in the log source

Troubleshooting


Problem

When a log source polls for events from the Netskope Active REST API, it is possible to miss some events when the recurrence value. This issue is due to events being created late outside of the polling interval of the API query from QRadar. Short polling intervals can cause events to not be polled as expected by the user.

Symptom

Not all events from Netskope are downloaded to QRadar.

Cause

A delay between the time when an event is created and the time when an event is available in the API can occur, causing some events can be missed when QRadar polls Netskope Active for data. The delay can be anywhere from a few seconds to over 90 minutes in some cases. To alleviate this issue, QRadar Support typically advises users to change the recurrence field from the default one minute value. When events are missed, there is no way to go back and poll the alerts API without collecting duplicate events in QRadar.

Order of operations
In this example, QRadar is polling for events between 10:00 GMT and 10:01 GMT with a recurrence value of 1M. During the polling window, 26 events were collected by QRadar, but two events were inserted after the polling window at 10:03 and not collected.
  1. The QRadar Netskope Active REST API protocol polls the remote Netskope endpoint to query for data by timestamp: 
    [ecs-ec-ingress.ecs-ec-ingress] [NetskopeREASTAPIQuery-ALERTS-10] com.q1labs.semsources.sources.netskopeactiverestapi.api.query.NetskopeActiveRESTAPIQuery: 
    [DEBUG] https://{tenant}.goskope.com/api/v1/alerts?token=**&starttime=1653559200&endtime=1653559260&limit=5000&skip=0
[ecs-ec-ingress.ecs-ec-ingress] [NetskopeREASTAPIQuery-ALERTS-10] com.q1labs.semsources.sources.netskopeactiverestapi.api.query.NetskopeActiveRESTAPIQuery: 
    [DEBUG] Obtained 26 results from query
    Figure 1: The debug log shows the query being run by QRadar to retrieve data for a 1 minute interval.
  2. The Netskope Active log source is configured to poll every minute.
    image-20220526180651-1
    Figure 2: The Recurrence field in the log source is configured by default to poll for new data every minute.
  3. The log source polls for available data. The QRadar protocol attempts to retrieve data with a starttime=1653559200 (10:00 GMT) and endtime=1653559260 (10:01 GMT).
  4. The Netskope Active REST API generates 2 new alert events at 10:03. 
    "_id": "c1723f95fa8711111111111",
"_insertion_epoch_timestamp": 1653559380,
"access_method": "API Connector",
"acked": "false",
"action": "anomaly_detection",
"activity": "Download",
"alert": "yes",
"alert_id": "2f81261442a845231b83c5d99364c4b7",
"alert_name": "Risky Countries",
    Figure 3: The new events are inserted by the API at 10:03 outside of the window where QRadar polled for alert data from the Netskope REST API.

    Results
    The 26 existing events that were available are polled, collected, and parsed by QRadar. However, the two events that were created late are not downloaded because the polling window already occurred. The insertion_epoch_timestamp in the Netskope REST API confirms the events were created after QRadar completed the query for the timeframe.

 

Resolving The Problem

To minimize the number of events not received, increase the value of the Recurrence field in the Netskope Active log source. 
  1. Log in to the QRadar Console as an administrator.
  2. Click the Admin tab.
  3. Start the Log Source Management app.
  4. Select Log Sources.
  5. Select your Netskope Active log source.
  6. Click the Protocol tab.
  7. Click Edit.
  8. Change the value in the Recurrence field to 1H.
    image-20220526183712-2
  9. Click Save.

    Results
    Confirm events between the Netskope Active alerts API and QRadar to determine whether any events were not collected. To compare results, query the Netskope API, then confirm the Log Activity tab displays the same number of events. If you continue to experience issues with missing events, repeat this procedure and extend the Recurrence field to 2H to help reduce the number of missed events. Updating the recurrence value does not eliminate the problem as the closer to the end of the polling window you are means that some events might not be available for download from the API. Extending the recurrence value allows QRadar to poll less frequently and reduces the possibility that an event is generated outside of the end time of the API query.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtEAAQ","label":"Log Activity"}],"ARM Case Number":"TS008371917","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
26 May 2022

UID

ibm16570043

Page Feedback