IBM Support

QRadar: Deploy times out due to missing or mismatched tokens

Troubleshooting


Problem

The QRadar console and all managed hosts in your deployment must have matching tokens in host_tokens.masterlist and host.token files to avoid deployment issues.

Symptom

You might find that deployments are timing out for a managed host due to either the managed host or the console token issue. If this is the case, you see similar messages on the console in the /var/log/qradar.log file, indicating that deployment is having problems and ultimately will timeout:
Jul 29 08:04:27 ::ffff:9.55.221.220 [hostcontext.hostcontext] [ConfigChangeObserver Timer[1]] com.q1labs.configservices.hostcontext.exception.HostContextException: Failed to execute url https://127.0.0.1/console/fetchConfig/globalset_list.xml HTTP/1.1 400 Bad Request

Cause

The QRadar Console is responsible for replicating its database and also pushing deployment configuration via the Deploy Changes to all managed hosts in the deployment. Occasionally, one or more hosts might "Time Out" during the Deploy Changes process. 
A common cause for deploy time outs is due to the tokens missing or mismatched between the console and the failing managed host. Hostcontext process uses both  /opt/qradar/conf/host_tokens.masterlist on the console and /opt/qradar/conf/host.token on the managed host for communication between the systems. Additionally, a matching token session with the managed host token id has to be present on the console /store/sessions folder as a file. This token is what is used for authentication, and the contents of the file are a serialized java object that allows the user to login.  

Diagnosing The Problem

The first step in diagnosing a possible token issue is to run the validate_deployment.sh script:
[root@hostname-console ~]# /opt/qradar/support/validate_deployment.sh
The two output lines that we are interested in when looking for token issues are the following (if these lines are missing from the output, auto-updates are not enabled, and the script was not updated to include these checks. To manually apply the auto-update bundle, see Technote 2003034 - QRadar: How to Manually Install the QRadar Weekly Auto Update Bundle):
GOOD: All hosts have a valid masterlist
GOOD: All hosts have a valid token in their masterlist
If you see a BAD line output, continue troubleshooting deployment issues by checking for the existence and matching of the tokens, and the existence and matching of the session files as follows:
  1. On the Console, check the token for the failing managed host from the /opt/qradar/conf/host.token file, to compare it with the token on the managed host as described in Step 2  of the "Resolving The Problem" section (the token for the managed host timing out is the one after the "=" symbol for the managed host with IP of 10.10.10.116).
     
    [root@hostname-console ~]# cat /opt/qradar/conf/host_tokens.masterlist
    CONSOLE_HOSTCONTEXT=c9da5690-2d95-4b97-80c2-dbc2aa69be71
    10.10.10.116=55fe8879-f67a-422f-9dec-d8061e62dab0
    

    Note: Another useful command to see if all the host_tokens.masterlist files are present and match on a console is for example (where the quote symbols are forward quotes):

    [root@hostname-console ~]# md5sum `locate host_tokens.masterlist`
    e1adbd00d84c9972a93c0e17cedb9ae9  /opt/qradar/conf/host_tokens.masterlist
    e1adbd00d84c9972a93c0e17cedb9ae9  /store/configservices/host_tokens.masterlist
    e1adbd00d84c9972a93c0e17cedb9ae9  /store/configservices/backup/deployed/GLOBALSET/host_tokens.masterlist
    e1adbd00d84c9972a93c0e17cedb9ae9  /store/configservices/backup/deployed/LOCALSET/host_tokens.masterlist
    e1adbd00d84c9972a93c0e17cedb9ae9  /store/configservices/deployed/GLOBALSET/host_tokens.masterlist
    e1adbd00d84c9972a93c0e17cedb9ae9  /store/configservices/deployed/LOCALSET/host_tokens.masterlist
    

  2. On the managed host (ssh to managed host IP) check if the token in /opt/qradar/conf/host.token matches the one in the console /opt/qradar/confhost_tokens.masterlist file:
     
    [root@hostname-managed_host ~]# cat /opt/qradar/conf/host.token
    55fe8879-f67a-422f-9dec-d8061e62dab0[root@hostname-console ~]# 


    Important: If the token is "correct" when you cat it, the command prompt should show up right at the end of the host.token string, i.e., there is no carriage return on the end of the file. Otherwise, it would cause the deploy to fail. For example, this would be a BAD file entry (using cat would show the extra carriage return and using vi would not):

    [root@hostname-managed_host ~]# cat /opt/qradar/conf/host.token 
    55fe8879-f67a-422f-9dec-d8061e62dab0
    [root@hostname-console ~]#
    


    Note: The same useful command to see if all the host.token files are present, and match can be run on the managed host. For example (where the quote symbols are forward quotes), the output is similar to this, and you will see just two host.token files:

    [root@hostname-managed_host ~]# md5sum `locate host.token`
    829268b46f16e4c4ca3bcf7d53e05aae  /opt/qradar/conf/host.token
    829268b46f16e4c4ca3bcf7d53e05aae  /store/configservices/deployed/LOCALSET/host.token
    
    
  3. Lastly, make sure the host.token for the host that is failing, has a corresponding file in /store/sessions on the console using the token string as a filename:

    [root@hostname-console ~]# ls -lah /store/sessions/ | grep 55fe8879
    -rw-r--r--  1 nobody nobody  449 Aug 18  2018 55fe8879-f67a-422f-9dec-d8061e62dab0


    Important: The session token in /store/sessions has to be owned by "nobody nobody". If that is not the case, permissions have to be modified via chown command (replace the token id with the one present on your system):

    [root@hostname-console ~]# chown -R nobody:nobody 55fe8879-f67a-422f-9dec-d8061e62dab0

If there is an issue with any of the tokens or token files, follow the instructions in the "Resolving The Problem" section to remediate the situation.

Resolving The Problem

The tokens have to match between the console's /opt/qradar/conf/host_tokens.masterlist and managed host's /opt/qradar/conf/host.token files.

If they differ, instead of just copying the console token to the managed host, use the token present on the console /store/sessions folder since that token cannot be recreated just by creating a new token file via the touch command. This leads to three options:

  1. If the current /store/sessions token is the one from the managed host, then update the console /opt/qradar/conf/host_tokens.masterlist file to match it.

  2. If the current /store/sessions token is the one from the console, /opt/qradar/conf/host_tokens.masterlist, then update the managed host /opt/qradar/conf/host.token to match the console entry for that managed host.

  3. If you see both session tokens (from the console and from the managed host) in the /store/sessions folder, then copy the one from the console host_tokens.masterlist to the managed host /opt/qradar/conf/host.token file.

Instructions on how to update the host.token file:

To update the host.token file and not insert a carriage return, use the echo -n command (no carriage return added) to insert the token into the file. First backup the host.token file via:

cp /opt/qradar/conf/host.token /opt/qradar/conf/host.token.backup

Then:

echo -n "host.token.string-from-console-host.token_masterlist-for.this.host" /opt/qradar/conf/host.token

Example:

[root@hostname-managed_host ~]# echo -n 55fe8879-f67a-422f-9dec-d8061e62dab0 > /opt/qradar/conf/host.token
 

What to do if the /store/sessions file is missing:

If the tokens between the managed host and the console match but it is not present in the /store/sessions folder, then that file has to be restored from a backup config since it cannot be recreated by simply running the touch file command.

The other option if the /store/sessions token is missing on the console for the managed host system and there is no config backup, is to unregister and reregister the managed host from the Console, that would generate a new token for the managed host and a new session token.

Where do you find more information?
  QRadar Documentation QRadar Forums QRadar Knowledge QRadar Software QRadar Notifications QRadar Support QRadar YouTube

Document Location

Worldwide

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":""}]

Document Information

Modified date:
31 March 2020

UID

ibm10961320