IBM Support

QRadar: Deploy times out due to missing or mismatched tokens

Troubleshooting


Problem

The QRadar Console is responsible for replicating its database and also pushing deployment configuration to all managed hosts in the deployment. Occasionally, one or more hosts might timeout during the Deploy Changes process. The Console and all managed hosts in the deployment must have matching tokens in /opt/qradar/conf/host_tokens.masterlist and /opt/qradar/conf/host.token files to avoid deploying changes communication issues.

Symptom

After an administrator attempts to deploy a change, the Console or the managed hosts timeout and display messages similar to the following in /var/log/qradar.log. Note the IP address of the appliance that timed out from the user interface or the logs. 

Host token issues can be reported with the following log messages:
  • Unable to retrieve authentication token for RPC call
  • Host token invalid. Unable to download database updates
  • Unable to decrypt the host token from: /opt/qradar/conf/host.token
  • Failed Read Host Token File: host.token
Example on 7.5.0+ Console
[ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [INFO] [NOT:0000006000][X.X.X.X/- -] [-/- -]Following message suppressed 39 times in 300000 milliseconds
[ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [ERROR] [NOT:0000003000][X.X.X.X/- -] [-/- -]Unable to retrieve authentication token for RPC call
[ConfigChangeObserver Timer[1]] com.q1labs.frameworks.crypto.DecryptException: com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
[ConfigChangeObserver Timer[1]]    at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:56)
[ConfigChangeObserver Timer[1]]    at com.q1labs.core.shared.jsonrpc.RPC.readAuthenticationToken(RPC.java:291)
[ConfigChangeObserver Timer[1]]    at com.q1labs.core.shared.jsonrpc.RPC.executeMethodWithTimeout(RPC.java:213)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.getActionRequest(ConfigChangeObserver.java:426)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.timeExpired(ConfigChangeObserver.java:401)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$ConfigChangeObserverTask.run(ConfigChangeObserver.java:662)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[ConfigChangeObserver Timer[1]]    at java.lang.Thread.run(Thread.java:822)
[ConfigChangeObserver Timer[1]] Caused by: 
[ConfigChangeObserver Timer[1]] com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:390)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:373)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.Crypto.decrypt(Crypto.java:73)
[ConfigChangeObserver Timer[1]]    at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:53)
[ConfigChangeObserver Timer[1]]    ... 12 more
[ConfigChangeObserver Timer[1]] Caused by: 
[ConfigChangeObserver Timer[1]] javax.crypto.BadPaddingException: Given final block not properly padded
[ConfigChangeObserver Timer[1]]    at com.ibm.crypto.provider.AbstractBufferingCipher.a(Unknown Source)
[ConfigChangeObserver Timer[1]]    at com.ibm.crypto.provider.AbstractBufferingCipher.engineDoFinal(Unknown Source)
[ConfigChangeObserver Timer[1]]    at javax.crypto.Cipher.doFinal(Unknown Source)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:387)
[ConfigChangeObserver Timer[1]]    ... 15 more
Example on 7.5.0+ managed host
managed-host.local systemd[1]: Starting hostcontext daemon...
managed-host.local systemd[1]: Started hostcontext daemon.
managed-host.local python[12887]: detected unhandled Python exception in '/opt/qradar/lib/python/qradar/mks.py'
managed-host.local replication[12496]: Host token invalid. Unable to download database updates.
managed-host.local hostcontext[11107]: com.q1labs.hostcontext.lifecycle.LifeCycleException: Unable to reset running lock
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5349)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.start0(HostContext.java:733)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.access$700(HostContext.java:98)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext$5.run(HostContext.java:915)
managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: unable to release running lock, future actions will not run until this lock is released
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1726)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5337)
managed-host.local hostcontext[11107]: ... 3 more
managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: Unable to determine if backup already running
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:556)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1722)
managed-host.local hostcontext[11107]: ... 4 more
managed-host.local hostcontext[11107]: Caused by: java.lang.Exception: Tomcat is not running. Unable to update a backup running lock (key:BACKUP_RUNNING_105, jsonObject:null)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.core.BackupUtils.setBackupRunningLock(BackupUtils.java:2221)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:552)
managed-host.local hostcontext[11107]: ... 5 more
managed-host.local systemd[1]: hostcontext.service: main process exited, code=exited, status=1/FAILURE
managed-host.local systemd[1]: Unit hostcontext.service entered failed state.
managed-host.local systemd[1]: hostcontext.service failed.
Example on 7.4.3 and older on the Console or managed host:
[hostcontext.hostcontext] [ConfigChangeObserver Timer[1]] com.q1labs.configservices.hostcontext.exception.HostContextException: Failed to execute url https://127.0.0.1/console/fetchConfig/globalset_list.xml HTTP/1.1 400 Bad Request

[hostcontext.hostcontext] [main] java.lang.Exception: Unable to decrypt the host token from: /opt/qradar/conf/host.token

[hostcontext.hostcontext] [main] com.q1labs.configservices.common.ConfigServicesException: Failed Read Host Token File: host.token

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.0;7.4.1;7.4.2;7.4.3;7.5.0"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
21 July 2022

UID

ibm10961320

Page Feedback