IBM Support

QRadar: How to properly move a Log Source from one Target Collector to another

How To


Summary

Several outbound protocols use a marker file as a bookmark during event collection. It tells each log source where it last left off while processing events from the last poll. This marker file is stored on the “Target Collector” set within the Overview tab of the log source.

If you have to change Target Collectors for a log source that uses one of these protocols, you need to move the marker file to the new Target Collector. This is so you do not end up with duplicate events in your system. Certain protocols might go too far back to list files found on the destination server or API. This causes the log source to appear to be stopped, or even fill up memory in Ingress to the point it could go Out of Memory (OOM).

Objective

The following protocols use marker files:

  • Akamai kona REST API
  • Amazon AWS S3 REST API
  • Amazon Web Services
  • Ariel REST API
  • IBM Big Fix EDR REST API
  • Blue Coat Web Security Service REST API
  • Box REST API
  • Centrify Redrock REST API
  • VMware vCloud Director
  • Google G Suite Activity Reports Rest API
  • IBM Security Verify Event Service
  • IBM Fiberlink REST API
  • IBM SmartCloud Orchestrator REST API
  • JDBC
  • Sophos Enterprise Console JDBC
  • Log File
  • Microsoft Graph Security API
  • MQ JMS
  • Netskope Active REST API
  • Office 365 Message Trace REST API
  • Office 365 REST API
  • Okta REST API
  • Salesforce REST API
  • SAP Enterprise Threat Detection Alert API
  • SMB Tail
  • Universal Cloud REST API
  • VMWare AppDefense API
  • Windows Defender Advanced Threat Protection Rest API

Steps

Before you begin:

  • This procedure is for moving marker files on Target Collectors within the same QRadar deployment.
  • Some of these protocols require certificates.
  • For any protocol such as Akamai kona REST API that still has the "Automatically Acquire Server Certificate" option present. After moving the marker file, the option Automatically Aquire Server Certificate(s) is set to No. For these types of log sources:
    1. After you disable the log source.
    2. Complete the procedure to move the marker file over to the new Target Collector.
    3. Click the Protocol tab.
    4. Change the option Automatically Aquire Server Certificate(s) to Yes so the certificate can be downloaded to the new Target Collector to resume collection without connectivity errors. 

      image-20230203092538-1
    5. Then, Enable the log source. 
    6. Optional: These certificates are normally stored in a file on disk in the /opt/qradar/conf/trusted_certificates directory. The certificate file can be manually moved to the new target collector or a new certificate can be pulled to the new Target Collector.
       

Obtaining the Provider number (Sensor Protocol Config ID) of the Log Source

Obtaining the Provider number (Sensor Protocol Config ID) of the Log Source

The marker file number can be different that what you see as the ID in the Log Source Management app. The marker file ID is from PostgreSQL, so you need to find it by using the psql command in CLI.

  1. Take the ID you see in the LSM app to the left of the Log Source Name.
    image-20230131115426-1
  2. Use it in the psql command on the CLI: 
    psql -U qradar -x -c “select spconfig from sensordevice where id=#####”
    Example, 
    psql -U qradar -x -c "select spconfig from sensordevice where id=1381;"
spconfig | 768

    Results
    The command returns the value of the Sensor Protocol ID for that log source. 

Some marker files can be named the provider number, such as 1234, a flat file with information inside it. Other times it could be a JSON file with the provider number in the format “#####.json”, others could be called “configId-#####.properties”, but regardless of the actual name, the “provider” number, is in the name of the marker file.

This marker file is stored on the “Target Collector” of the log source under the Overview tab.  You need to use SSH to log in to the Console, then SSH to the Target Collector from there (unless the Console is the Target Collector).  The marker files are stored under /store/ec/ followed by the name of the protocol it uses.

Examples of the names of the marker files


Here is a small list showing what you would see for various outbound protocols:

Log File log source marker file:

/store/ec/sqlite/sessiondata.db_<id>

Example,

/store/ec/sqlite/sessiondata.db_1234

JDBC log source marker file:

/store/ec/jdbc/<id>
Example,
/store/ec/jdbc/22334

Amazon REST API log source marker file:

/store/ec/amazonawsrestapi/<id>
Example,
/store/ec/amazonawsrestapi/2324

Bluecoat WSS log source marker file:

/store/ec/bluecoatwssrestapi/configId-<id>.properties
Example,
/store/ec/bluecoatwssrestapi/configId-12605.properties
Universal Cloud REST API log source marker file:
/store/ec/universalcloudrestapi/<id>/state.json
Example, 
/store/ec/universalcloudrestapi/7308/state.json

If changing the Target Collector, why it is important to move the marker file to that Target Collector.

When a log source polls the vendor device for events, such as those using a REST API, it refers to the last downloaded files or events from the previous polling. It then saves that reference in the marker file. 

If you change the Target Collector on any of these log sources, it starts collection fresh on that host, and there is no marker file present. That means it could start all the way back from the beginning of the list of files available to be pulled. This can result in duplicate events, performance issues on that particular host as it reads the files, and unnecessary EPS license usage.


 

It is simple, but requires just a few extra steps.  Normally, you would:

  • Disable the log source.
  • Change the Target Collector to the new one you want to switch it to.
  • Save the changes.
  • Then, re-enable the log source. 
    Note: Toggling of the log source ensures the thread for that log source stops on the first collector. 

Procedure to move the marker file

  1. Log in to the QRadar UI with admin privileges.
  2. Click the Admin tab.
  3. In the Log Source Management App UI, disable the log source.
    image-20230131081602-2
  4. Use SSH to log in to the Console.
  5. Use SSH to log in to the current Target Collector that log source is assigned to.
  6. Use SCP to move the marker file from /store/ec/<protocol_name>/<provider_#> back to a folder such as /tmp on the Console.
  7. Use SSH to move back to the Console, then use SCP to move that marker file to the same path on the new Target Collector that is collecting data for that log source.
    Note: You might have to use the command mkdir to create the path on the new collector before using SCP to copy it over from the Console. Example, 
    mkdir -p /store/ec/<protocol_name>/
  8. In the Log Source Management App UI, change the Target Collector to the new one you want to assign the log source to.
    image-20230131081740-3
  9. Click Save.
  10. Enable the log source.

    Results
    The updated Target Collector is capturing data from that log source. 

Document Location

Worldwide

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM - SSBQAC"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.2;7.4.3;7.5.0"}]

Document Information

Modified date:
03 February 2023

UID

ibm16856453

Page Feedback