IBM Support

QRadar: How to use DrQ to troubleshoot your deployment

How To


Summary

DrQ is a health check framework that you can use to troubleshoot issues with your QRadar deployment.

Steps

The DrQ script is designed to perform a general health check to help troubleshoot the source of incorrect behaviors such as failed upgrades. DrQ can return false positives such as reporting harmless archived entries in deployment and configuration files as errors. You can follow the recommended remediation instructions the script provides to clear those warnings. If you are not seeing any incorrect behaviors, they do not necessarily indicate issues with your system.

This technical note goes over several common classes of errors and provides more information on how serious they are along with detailed steps to resolve them.

Incorrect file permissions

Incorrect file permissions can make important files inaccessible to system processes. This class of error is important to correct issues that can prevent configuration changes from a deploy or introduce unexpected errors, such as user interface issues.

Example error message

  A Check to make sure deployment files have the correct permissions
     [FAILURE]   
         /store/configservices/host_tokens.masterlist has incorrect file
         permissions. (Found: '-rwxrwxrwx' Expected: '-rw-r--r--')
        [REMEDIATION]   
         Type the following command to correct the file permissions 'sudo chmod
         644 /store/configservices/host_tokens.masterlist' 


These errors can easily be remediated by running the recommended chmod command with root permission.

Missing files

DrQ can report errors when important files are missing on the system. This class of error can commonly effect .conf files or the host_tokens.masterlist and is advisable to correct. See each example for instructions on how to fix the issue by moving undeployed copies of the files to the necessary folder.

Note: To correct this issue, you can complete a Deploy Full Configuration from the Admin tab to ensure the latest files are added to each managed host. However, manually copying .conf files or host_token.master list files is quicker.

Error 1: Configuration file check
DrQ can confirm that configuration files are present in the correct directory.

Example error message
Config Files Checks
  Checks if config files are present and populated correctly, and checks that
  the hostcontext service has been started.     
     [FAILURE]
         lastDeployTime.conf file is missing at
         </store/configservices/deployed/GLOBALSET/lastDeployTime.conf>

Steps to resolve this issue
To resolve this issue, ssh into the affected appliance and run the following command to create a copy of the file in the deployed folder:
cp -p /opt/qradar/conf/lastDeployTime.conf /store/configservices/backup/deployed/GLOBALSET/lastDeployTime.conf


Error 2: Confirming host_tokens.masterlist files
The Console and all managed hosts in the deployment must have matching tokens. DrQ can confirm for administrators when tokens required to receive configuration update from the Console are missing.

Example error message
Deployment verification for host_tokens.masterlist
  Ensure that host_tokens.masterlist is deployed correctly     
     [FAILURE]
         Host token masterlist at
         /store/configservices/deployed/GLOBALSET/host_tokens.masterlist cannot
         be retrieved 
Steps to resolve this issue
To resolve this issue, ssh into the affected appliance and run the following command to create a copy of the file in the deployed folder:
cp -p /opt/qradar/conf/host_tokens.masterlist /store/configservices/deployed/GLOBALSET/host_tokens.masterlist

If you continue to experience issues with this check, you can review the MD5 sum files and ensure that tokens exist in the required directories. For more information, see QRadar: Deploy times out due to missing or mismatched tokens.

Invalid DATANODE_SERVERS property

Another common issue with the deployment.xml is the DATANODE_SERVERS properties. If DrQ reports an error, 'did not find DN property string (x.x.x.x:y) in DATANODE_SERVERS property of eventprocessor component with id Z', then you must remove the property from the deployment.xml. If the error reports 'component with id Z is not correct (expected 'x.x.x.x:y')', then you must edit the IP of that component and change it to the expected IP. See each section for more information on the error and associated fix.

Before you begin
Administrators must confirm that the Console does not have any configuration changes to deploy. All changes must be deployed to ensure hosts have the latest configuration before you attempt to resolve any Data Node errors reported by DrQ.

Error 1: Did not find DN property string (x.x.x.x:y)
This error can display when deployment.xml includes an unexpected entry or the tool might report a pending configuration change from Deploy Changes.



Example error message
Deployment Model DATANODE_SERVERS Property Checks
  Verifies that all event processors have a DATANODE_SERVERS property with the
  correct value
     [FAILURE]
         Did not find DN property string (x.x.x.x:y) in
         DATANODE_SERVERS property of eventprocessor component with id Z.
        [REMEDIATION]
         Ensure all data nodes attached to this event processor are configured
         correctly, then perform a deploy action 

Steps to resolve this issue
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Create back up copies of the deployment.xml
    cp -p /opt/qradar/conf/deployment.xml /store/ibm_support/conf/deployment.xml
    cp -p /store/configservices/deployed/deployment.xml /store/ibm_support/deployed/deployment.xml
  3. Edit the file /opt/qradar/conf/deployment.xml and find the component with the id Z, mentioned in the DrQ error, and remove the component.
  4. Copy the /opt/qradar/conf/deployment.xml to the /store/configservices/deployed/deployment.xml file:
    cp -p /opt/qradar/conf/deployment.xml /store/configservices/deployed/deployment.xml
    
    Results
    Run DrQ again to confirm the error is resolved. This error is often related to a 'Host Tokens Master List Check' error in DrQ.

Error 2: Component with id Z is not correct (expected 'x.x.x.x:y')
DrQ reports this error when the tool believes that IP address of the host in deployment.xml is incorrect.

Example error message

Deployment Model DATANODE_SERVERS Property Checks
  Verifies that all event processors have a DATANODE_SERVERS property with the
  correct value
    [FAILURE] 
          There were 1 or more event processor components in the deployment model 
          with an invalid DATANODE_SERVERS property. 
         [REMEDIATION] 
          Ensure all data nodes are correctly configured, then perform a deploy action.
     [FAILURE]
         DATANODE_SERVERS property (localhost:y) for eventprocessor
         component with id Z is not correct (expected 'x.x.x.x:y').
        [REMEDIATION]
         Ensure all data nodes are correctly configured, then perform a deploy
         action.

Steps to resolve this issue
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Create back up copies of the deployment.xml
    cp -p /opt/qradar/conf/deployment.xml /store/ibm_support/conf/deployment.xml 
    cp -p /store/configservices/deployed/deployment.xml /store/ibm_support/deployed/deployment.xml
    
  3. Open /opt/qradar/conf/deployment.xml and find the component with the id Z, mentioned in the DrQ error. Change "localhost:Y" to the expected value of "x.x.x.x:y". You must read your error to determine these values.
  4. Copy the /opt/qradar/conf/deployment.xml to the /store/configservices/deployed/deployment.xml file:
    cp -p /opt/qradar/conf/deployment.xml /store/configservices/deployed/deployment.xml
    Results
    Run DrQ again to confirm the error is resolved.

Host Tokens Master List Check

This error can occur on it's own or in combination with the "Invalid DATANODE_SERVERS property".

Example error message

Host Tokens Master List Check
  Checks that the hosts listed in host_tokens.masterlist are present in
  deployment.xml, and that all hosts in deployment.xml have a token listed in
  host_tokens.masterlist.
     [FAILURE]
         The host x.x.x.x listed in host_tokens.masterlist does not exist
         in deployment.xml.
        [REMEDIATION]
         Remove the entry for x.x.x.x from
         /store/configservices/host_tokens.masterlist then perform a full
         deploy.


Steps to resolve this issue

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Ensure that the console does not have any pending changes by checking the Admin tab in the QRadar UII. If it does, deploy them and see whether the issue is resolved in DrQ.
  3.  Create a backup copy of the host_tokens.masterlist.
    cp -p /store/configservices/host_tokens.masterlist /store/ibm_support/conf/host_tokens.masterlist
  4. Edit the host_tokens.masterlist to remove the invalid entry. In this example, remove the x.x.x.x line:
    #Wed Jan 04 16:46:00 EST 2023
    CONSOLE_HOSTCONTEXT=XXXXXXXXXXXXXXX
    x.x.x.x=YYYYYYYYYYYYYY
  5. Deploy the changes from the Admin tab.

    Results
    Run DrQ again to confirm the error is resolved.

Errors with the hostcapabilities.xml

DrQ reports an error when it identifies inconsistencies in the hostcapabilities.xml.

Example error message

Host Capabilities Checks
  Checks that hostCapabilities.xml has the correct applianceType and
  activationKey     
     [FAILURE]         
          Error getting appliance type: Unable to retrieve softwareType or         
          applianceType from /opt/qradar/conf/capabilities/hostcapabilities.xml.         
          Check is unable to continue.        
          [REMEDIATION]         
          Verify that /opt/qradar/conf/capabilities/hostcapabilities.xml is         
          valid. 


Steps to resolve this issue

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Ensure that the console does not have any pending changes by checking the Admin tab in the QRadar UII. If it does, deploy them and see whether that resolves the error in DrQ.
  3. Create a back up copy of the hostcapabilities.xml
    cp -p /opt/qradar/conf/capabilities/hostcapabilities.xml /store/ibm_support/conf/hostcapabilities.xml
  4. Open /opt/qradar/conf/capabilities/hostcapabilities.xml in vim or another text editor.
  5. Ensure the applianceType is correct and matches your appliance.
  6. If softwareType is missing, add softwareType="0".
    Example file:
    <?xml version='1.0' encoding='UTF-8' standalone='yes'?>
    <HostCapabilities
    	isConsole="true"
    	IP="x.x.x.x"
    	applianceType="3199"
    	hostName="name"
    	qradarVersion="7.5.0"
    	hardwareSerial="VMware-xx xx xx xx xx xx xx xx-xx xx xx xx xx xx xx xx"
    	activationKey="xxxxxx-xxxxxx-xxxxxx-xxxxxx"
    	managementInterface="ens192"
    	softwareType="0"
    	xmlns="http://www.q1labs.com/products/qradar"
    />
  7. Save your changes.
  8. Log in to the QRadar Console user interface as an administrator.
  9. Click the Admin tab.
  10. Click Deploy Changes.

    Results
    Run DrQ again and it no longer reports the hostcapabilities.xml error.

Errors with deployment.xml

If a fix patch or upgrade failed, and when you run DrQ to diagnose the issue, it reports errors in the deployment.xml, those errors might not be benign and fixing them can solve the issues that might prevent you from upgrading.

Example error message

Verification for Event Collector component properties and their values in dep...
  Ensure that Event Collector components have valid properties and that their
  values are proper in deployment.xml
     [FAILURE]
         ASSET_PROFILER is invalid for Event Collector component ID <X>
        [REMEDIATION]
         ASSET_PROFILER in Event Collector is invalid. The value of Asset
         Profiler is  Update Event Collector's Asset Profiler
         property value with a valid IP and PORT. Asset Profiler IP should be
          or  based on the host id of
         the Event Collector component, if the host id is a console or if it is
         a managed host but not encrypted, then it is . Else it
         should be  The PORT number should be
         based on the LISTEN PORT value of ASSET PROFILER Component.


Steps to resolve this issue

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Ensure that the console does not have any pending changes by checking the Admin tab in QRadar. If the user interface indicates changes are required, click Deploy Changes.
  3. Run the deployment_viewer.py script to get more information on the error:
    /opt/qradar/support/deployment_viewer.py -av
  4. Examine the output.
    Tests to compare the valid components from the deployment.xml to those in the deployed_component table.
    PASSED: All Components from deployment.xml have a valid entry in deployed_component table.
    Testing to see if EC Ingress component exists for all hosts with ECS-EC.
    PASSED: All hosts with an EC component have a valid Ingress component.
    Testing to see if all EC components have a valid EP connection.
    PASSED: All EC components have a valid EP connection
    Testing to see if all valid components have valid connections.
    FAILED: Managed host X with component target Y has a source connection with component Z which doesn't have a valid source host.
    FAILED: Managed host X with component target Y has a source connection with component Z which doesn't have a valid source host.
    Tests to compare the valid components from the deployment.xml to those in the deployed_component table.
    In this error, the source component Z is the issue. This ID might or might not match the IDs that DrQ reported on.
  5. Create copies of the deployment.xml
    cp -p /opt/qradar/conf/deployment.xml /store/ibm_support/conf/deployment.xml
    cp -p /store/configservices/deployed/deployment.xml /store/ibm_support/deployed/deployment.xml
  6. Edit the /opt/qradar/conf/deployment.xml to remove the invalid component. In this example, for ID X, you would remove the following:
        <component hostId="0" changed="false" id="X" instanceName="dataNodeX" version="7.5.0" type="dataNode">
            <bounds width="130" height="90" y="120" x="3380" viewName="semView"/>
            <bounds width="130" height="90" y="120" x="180" viewName="x.x.x.x"/>
            <property activated="false" value="32090" name="DN_LISTEN_PORT"/>
            <property activated="false" value="Y" name="EVENTPROCESSOR"/>
        </component>
  7. Click the Admin tab.
  8. Click Deploy Changes.

    Results
    Run the deployment_viewer.py script and DrQ again to see whether the issue is resolved. Some deployment.xml errors can be benign, so it is important that deployment_viewer.py returns no errors.

Alternate steps for if you believe your deployment.xml is correct
You can compare the deployed version of file to the staged version to ensure there are no undeployed changes that are failing to deploy.
  1. Use SSH to log in to the QRadar Console as the root user.
  2.  Use the cmp command to compare the deployed and staged files.
    cmp /opt/qradar/conf/deployment.xml /store/configservices/deployed/deployment.xml
    
    Result
    When the command returns nothing, the files are the same. If it returns that they are different, click the Admin tab, click Deploy Changes, then run DrQ again. If the files fail to deploy, contact QRadar support.

Intermediate CA file vault-qrd does not exist

DrQ, or Cliniq, returns error pem does not exist:

[FAILURE]
Intermediate CA file
/etc/pki/ca-trust/source/anchors/vault-qrd_ca_int.pem does not exist.
The error "vault-qrd_ca_int.pem does not exist" is benign can be ignored by administrators. QRadar no longer uses vault-qrd_ca_int, so the test run by DrQ is no longer valid. A future update to DrQ is expected to remove this test.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
31 July 2023

UID

ibm17005939