IBM Support

QRadar: "Test failed to start in a timely manner" error in the Log Source Management app

Troubleshooting


Problem

Users can experience the following error when they run the Test function in the Log Source Management app: 'Test failed to start in a timely manner. Please try again or contact support'. This article describes the error and provides troubleshooting steps to resolve the error message.

Symptom

image-20230525102841-1

Cause

Some of causes for this error can be:
  • Outdated RPM protocols
  • ECS-EC-ingress service requires a restart
  • Issues with the marker
     

Environment

This procedure described in thie technical note applies to QRadar SIEM administrators with access to the command line.

Note: If you experience this error on your QRadar on Cloud Console, contact QRadar Support for assistance.

Diagnosing The Problem

Make sure the RPM protocols are up to date. If your protocol fails to run as expected or generates an error, you can compare the RPM version installed in QRadar against the latest version on IBM Fix Central.

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Run the following command to identify what current version is installed. Replace <PROTOCOL_name> with the protocol name you were testing.
    Note: If you are not sure what is the full PROTOCOL name you can try with part of the name until you find the full name.
    rpm -qa | grep -i <PROTOCOL_name>
    Command example:
    # rpm -qa | grep -i AWSRESTAPI
    PROTOCOL-AmazonAWSRESTAPI-7.X-XXXXXXXXXXXXXX.noarch
  3. Go to Fix Central and select Protocol.
    image-20230914093459-1
  4. In the Filter fix details search box, type the protocol name to view the latest version and confirm it matches the version installed on your QRadar Console.
    image-20230914105616-4
  5. If you need to update the RPM in QRadar, see the following documentation for instructions:
    Result
    The administrator confirms that the latest protocol RPM is installed on the Console.

Resolving The Problem

The resolution to this issue depends on its cause. Follow this procedure to determine which steps you must take to fix the error.
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Optional. If the log source is using a different target event collector than the console, SSH to the target event collector.
  3. Run the following command to find any TestTask errors:
    grep ProtocolTestTask /var/log/qradar.log
  4. Confirm whether the following logs are seen:
    [ecs-ec-ingress.ecs-ec-ingress] [SequentialEventDispatcher] at com.q1labs.semsources.sources.base.testing.ProtocolTestTask.<init>(ProtocolTestTask.java:45)
    [ecs-ec-ingress.ecs-ec-ingress] [pool-2-thread-3] at com.q1labs.semsources.sources.base.testing.ProtocolTestTask.runTask(ProtocolTestTask.java:87)

    Results
    Select one of the following procedures based on the output from the logs in step 4.
  • If the initialization and runTask logs are displayed, you can Reload the protocol as described in the next section.
  • If you do not see an initialization and runTask log, then proceed with the section Re-create the marker file.

Reload the protocol

To reload the protocol, you can either restart ingress or reload the protocol manually. Restarting ingress is a simpler procedure but briefly interrupts your event collection.
Restart ingress
This procedure restarts the ingress services and temporarily stops incoming event collection. You must ensure you restart the correct appliance for your log source. Administrators can confirm which appliance owns the log source by reviewing the Target Event Collector field.

Important: This set should be done during a maintenance window, if possible. Restarting the ecs-ec-ingress service temporarily stops event collection while the service restarts for the appliance. Restarting the ecs-ec-ingress service causes graphs to dip to zero events received as it takes 5-10 seconds for the service to restart. Before you restart the ingress, see QRadar: Core services and the impact of restarting services for more information on the impact of restarting the services.

Procedure
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Open an SSH session to the appliance listed as the Target Event Collector that manages your log source.
    Important: The appliance listed as the Target Event Collect is where the protocol tests run. It is important to restart ecs-ec-ingress on the correct appliance that receives the log source events.
  3. Run the following command to restart ingress:
    systemctl restart ecs-ec-ingress
Reload the protocol manually
If you do not want to restart ingress, run the following process to touch the .jar file for the protocol of the specified log source. This procedure resets the connection for that particular protocol:
  1. In the Log Source Management app, note the name of the protocol of the log source for which you get the error message.
  2. SSH into the QRadar console.
  3. To find the required .jar file, access the required directory and list the contents:
    cd /opt/ibm/si/services/ecs-ec-ingress/eventgnosis/lib/q1labs/
    ls
    
  4. If you know the name of the protocol, you can list the .jar files and pipe the output through grep. Example, MSRPC:
    [root@qr750 q1labs]# ls | grep -i rpc
    q1labs_semsources_protocol_windowseventrpc.jar
    q1labs_semsources_protocol_windowseventrpc_ui.jar
    Q1MSRPCTest.jar
    WindowsEventRPC
    [root@qr750 q1labs]#
    The target file would be q1labs_semsources_protocol_windowseventrpc.jar. Note, there might be two files with similar names. The target file is always without "_ui" in the name.
  5. After the necessary .jar file is identified, run the following command:
    touch ./<insert protocol .jar file>
Result
After you complete one of the two provided methods the error no longer appears and you can re-run the testing tool. If you continue to experience issues, contact QRadar support

Re-create the marker file

The marker files are used by some log sources to keep track of the last event ingested so that the events are not ingested twice. Re-creating them can resolve the issue.

Important: The procedure outlined in this section is an advanced procedure. Recreating the marker file can reset the timestamps on certain protocols and can resolve the issue, but might set the marker file to the current time or latest record number. If you complete this procedure on existing log sources that were previously collecting events, it can cause some events to be skipped as you are resetting the last value read time or record id. If you are unsure of completing these steps or are uncomfortable with this procedure, contact QRadar Support for assistance.

Procedure

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Optional. If the log source is using a different target event collector than the console, SSH to the target event collector.
  3. Use the following command to identify the spconfig of the log source. This value is required to identify the marker file of the log source. Replace <Log_Source_Name> with the log source name.
    psql -U qradar -c "select spconfig from sensordevice where devicename = '<Log_Source_Name>'";
    Where <Log_Source_Name>
    
    In this example, the spconfig is 765:
    psql -U qradar -c "select spconfig from sensordevice where devicename = 'Amazon AWS @ hostname'";
     spconfig 
    ----------
           765
    (1 row)
  4. Change directory to the ec folder.
    cd /store/ec/
  5. Use the following command to display the folders on the path. Note the folder with the name of the protocol that is being used by the log source.
    ls -l
  6. Change directory to the protocol folder. Change <protocol_name> to your protocol name:
    cd <protocol_name>
  7. List the files then create a backup of the one with the spconfig number for its name:
    ls -l /store/ec/<protocol_name>/
  8. In order to recreate the marker file, create a backup of the spconfig file.
    cp -p <spconfig> /store/IBM_Support/<spconfig>.bak
  9. Log in to the QRadar Console user interface.
  10. Click the Admin tab.
  11. Click the Log Sources icon.
  12. Select the log source.
  13. Click Disable, then Enable the log source. This will make a new connection of the log source using the new marker file.

    Result
    A new marker file is created, the error no longer appears, and you can re-run the testing tool. If you continue to experience issues, contact QRadar support

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
14 September 2023

UID

ibm16988291