Known issues with the Probe for Message Bus
This section explains some known issues with the Probe for Message Bus.
Recommendation when using SSL
If the probe
is running on Java 6 and encounters an SSLHandshakeException
with
a fatal alert handshake failure message using TLSv1 , TLSv1.1, or
TLSv1.2 protocol, you should upgrade to Java 7. To use TLSv1.2 SecurityProtocol,
Java 7 is required.
You have the following options:
- Either install OMNIbus V8.1 (because it comes with JRE 7)
- or, install JRE 7 separately
Upgrading the common TransportModule library
After
upgrading the TransportModule
, the probe can fail
to start due to a ClassNotFoundException
being thrown
during startup. This is possibly due to the probe missing a dependency
libraries in its CLASSPATH
.
TransportModule
version
12 contains the following new files to set the CLASSPATH
used:
- TransportModule.env (for UNIX operating systems)
- TransportModule.bat (for Windows operating systems)
You can configure the probe to call these
files and include
the TRANSPORT_CLASSPATH
variable in the probe's CLASSPATH
.
Update
the probe’s environment file or batch file with the following
lines to replace old variables which defines the path to the libraries
used by the TransportModule and make sure the TRANSPORT_CLASSPATH
is
used.
# Load TransportModule and dependency JARS
. $OMNIHOME/java/jars/TransportModule.env
CLASSPATH_SETTING=${TRANSFORMER_JAR}:${TRANSPORT_CLASSPATH}:${JACKSON_JAR};
REM Set TRANSPORT_CLASSPATH as specified in TransportModule
call %OMNIHOME%\java\jars\TransportModule.bat
set MESSAGEBUS_CLASSPATH=%TRANSFORMER_CLASSPATH%;%TRANSPORT_CLASSPATH%;
%JACKSON_CLASSPATH%
Incorrect setting for the keyStorePassword property in the HTTP transport properties
If an incorrect password is set using the keyStorePassword property in the HTTP transport properties file, the probe will fail to load the Keystore during initialization. This will make the probe hang, instead of shutting down. To shutdown the probe, kill the probe process manually. The following log message is printed when this error occurs:
2017-02-24T05:06:56: Debug: D-JPR-000-000: com.ibm.tivoli.oidk.ProbeImpl.connect
EXITING
2017-02-24T05:06:56: Information: I-JPR-000-000: [HttpParser]:
Max http payload size: 2097152
2017-02-24T05:06:56: Error: E-JPR-000-000: Failed to create server listening socket
'null:5490'. [java.net.SocketException: java.security.NoSuchAlgorithmException:
Error constructing implementation (algorithm: Default, provider:
IBMJSSE2, class: com.ibm.jsse2.ec)]
2017-02-24T05:06:56: Error: E-JPR-000-000: Fail to subscribe Transport module
to the interface
2017-02-24T05:06:56: Error: E-JPR-000-000: Failed to connect; ProbeException:
Fail to subscribe Transport module to the interface;
TransportSubscribeException:
Failed to start all HTTP Server ports.
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.ProbeException:
Fail to subscribe Transport module to the interface
at com.ibm.tivoli.oidk.ProbeImpl.subscribe(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.
connect(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.
connectAndRun(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.run(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.oidk.Probe.start(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.oidk.Probe.main(Unknown Source)
Caused by: com.ibm.tivoli.netcool.integrations.transportmodule.
TransportSubscribeException:
Failed to start all HTTP Server ports.
at com.ibm.tivoli.netcool.integrations.transportmodule.HttpTransport.
subscribe(Unknown Source)
... 6 more
2017-02-24T05:06:56: Information: I-UNK-000-000: Probewatch: Unable to get events.
Failed to connect; ProbeException:
Fail to subscribe Transport module to the interface;
TransportSubscribeException: Failed to start all HTTP Server ports.
2017-02-24T05:06:56: Debug: D-UNK-000-000: Rules file processing took 18 usec.
2017-02-24T05:06:56: Debug: D-UNK-000-000: Flushing events to object servers
2017-02-24T05:06:56: Debug: D-UNK-000-000: 1 buffered alerts
2017-02-24T05:06:56: Debug: D-UNK-000-000: Flushing events to object servers
2017-02-24T05:06:56: Debug: D-UNK-000-000: 0 buffered alerts
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.framework.
ProbeRunner.connect EXITING
2017-02-24T05:06:56: Information: I-JPR-000-000: DISCONNECT 'Unable to connect'
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.framework.
ProbeRunner.resetForRetry ENTERING
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.framework.
ProbeRunner.haltScheduledTasks ENTERING
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.framework.
ProbeRunner.haltScheduledTasks EXITING
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.netcool.omnibus.probe.framework.
ProbeRunner.disconnectProbe ENTERING
2017-02-24T05:06:56: Debug: D-JPR-000-000:
com.ibm.tivoli.oidk.ProbeImpl.disconnect
ENTERING
Exception in thread "MessageSenderThread" java.lang.NullPointerException
at com.ibm.tivoli.netcool.integrations.transportmodule.http.HttpServer.
isRunning(Unknown Source)
at com.ibm.tivoli.netcool.integrations.transportmodule.HttpTransport.
httpServerIsAlive(Unknown Source)
at com.ibm.tivoli.netcool.integrations.transportmodule.HttpTransport.
isConnected(Unknown Source)
at com.ibm.tivoli.oidk.ProbeImpl.disconnect(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.
disconnectProbe(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.
resetForRetry(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.
messageReceived(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.services.impl.
SimpleMessageService$SenderThread.
send(Unknown Source)
at com.ibm.tivoli.netcool.omnibus.probe.services.impl.
SimpleMessageService$SenderThread.
run(Unknown Source)
Recommended Action: Ensure the correct keystore password is set when securing the HTTP transport.
Probe fails to start when using a JSON parser configuration file as a TransformerFile due to an incompatible Java version
Java 7 is required by the supporting libraries used to load the JSON parser configuration file. If a lower Java version is used, the probe will fail to initialize when loading the parser configuration file and will print the following error message:
Exception in
thread "ProbeRunner" java.lang.UnsupportedClassVersionError: JVMCFRE003
bad major version; class=com/fasterxml/jackson/databind/ObjectMapper,
offset=6
Consider upgrading to Netcool/OMNIbus V8.1 which includes JRE 7, or install JRE 7 separately.
When integrating with iDirect Pulse
Disconnection during a clean start
In the first connection to iDirect (clean start), the probe creates a WebSocket connection with the /api/1.0/dde/alarm?start_time__gte=0 URI to query historical alarms and continue listening for new alarms or updates to existing alarms. However, the URI time window maybe too large, which may cause the server to disconnect from the probe.
Recommended action: Use a smaller time window, or configure the probe to enable Retry so that it reconnects using the WebSocket Persistent URI /api/1.0/dde/alarm?start_time__gte=++ProbeDisconnectTime++. You must set the DataBackupFile’ property for the probe to record the last disconnection time in the backup file.
During beta testing, the following query windows were tested and the results are as below:
Query window | Connection status |
---|---|
Past 1 day |
OK |
Past 1 week |
OK |
Past 1 month |
OK |
Past 1 year |
Probe disconnected |
No events received through WebSocket while the Pulse server still initializing after a restart
If the Pulse server
is restarted and is still initializing when the probe connects, the
server might accept the WebSocket connection but then send an event
with error code 500
(Internal Server Error) and then
send no further alarms or a close connection request to the probe.
The probe continues listening but is unaware that the server is not
sending any alarms.
Recommended action: Increase the value set by the RetryInterval property to give sufficient time for the server to be ready before attempting to connect. The probe only retries the connection if it has successfully established a WebSocket connection before the server was restarted.
Configure the probe to disconnect and shutdown due to a period of inactivity by setting the Inactivity property to a time (in seconds) greater than zero to shut down the probe. The probe will then need to be restarted. You can configure a Process Agent to manage the probe process and restart it automatically if it is down.
No new updates received after a period upon successful WebSocket connection
If the server stops sending updates but does not disconnect the probe, the probe session may have ended on the server but the server did not request to close the connection.
Recommended Action: Enable the following set of properties in the WebSocket transport to send a periodic HTTP request as a keep-alive mechanism and inform the server that the probe is still listening. The following configuration is a suggestion and should be changed to use the correct URI if necessary. The subscribeRefreshInterval must be configured to a period before the probe session ends.
subscribeRefreshURI=/api/1.0/config/element/user?limit=1
subscribeRefreshMethod=GET
subscribeRefreshContent=
subscribeRefreshInterval=30
Recommendation
when using WebSocketTransport or WebhookTransport
with autoReconnect=ON
If HeartbeatInterval is set to too a low number, AutoReconnect may not arrive at the maximum count.
For example:
The attempts of autoReconnect are run by an exponentially increased interval. For a count of 5 attempts (1s, 2s, 4s, 8s, 16s) a total of 31s is required to complete 5 attempts.
If the HeartbeatInterval probe property is configured to 10s and the probe detects that the transport is not in an active connection state, it may shutdown earlier, after the 3rd try, leaving the 4th and 5th tries un-attempted.
Recommend
Action: When using WebSocketTransport or WebhookTransport with autoReconnect=ON
,
set the HeatbeatInterval to a value greater than
31s (one minute is recommended).
Recommendation
when using WebSocketTransport and WebhookTransport
with autoReconnect=ON
and the httpHeaders transport
property
The httpHeaders transport
property is used for all outgoing HTTP
messages and
is capable of accepting the HTTP header token substituted from the
probe properties file or from tokens retrieved from an EMS at runtime.
However, httpHeaders tokens
may be set to a null
value during the early phase
of probe initialization before tokens from the EMS are retrieved,
which may risk request failure.
Recommend Action: When in use, set the probe properties as tokens using httpHeaders.
When in use, set dynamically retrieved tokens from EMS with the header properties specifically designed for its use case to delay the use of the following external token names:
loginRequestHeaders
loginRefreshHeaders
logoutRequestHeaders
resyncRequestHeaders
subscribeRequestHeaders
subscribeRefreshHeaders
Monitoring updates from ZooKeeper
Updates to topics or brokers are monitored by the ZooKeeper's topic/broker watch function when used.
Updates to topics or brokers are not displayed via ProbeWatch messages. They can be found in the probe log.
TLS handshake issue in Message Bus 8.0 when connecting to a server which only accepts TLSv1.2 Security Protocol using Webhook or Websocket transport
The Message Bus Probe Webhook and WebSocket transports have an HTTP client component which is used to make REST API calls to a remote target system. This component has an OAuth2.0 Module to request an access token from servers using the OAuth2.0 standard.
Known Issue Symptom
When configured to request an access token from a remote server which only accepts the TLS v1.2
protocol, the probe will throw a TransportAuthorizeException
error due to a TLS
handshake failure. This is due to that the OAuth2.0 module in the HTTP client component of the
transport starts the SSL handshake with a lower TLS version which is rejected by the server.
Resolution
Configure the probe's Webhook or Websocket transport to use loginRequest properties, instead of tokenEndpointURI to create a HTTP request to request an access token. For example, to create the HTTP request below, use the settings:
POST /token HTTP/1.1
Host: server.example.com
Authorization:Basic VXNlckZvckJhc2ljQXV0aGVudGljYXRpb246UGFzc3dvcmRGb3JCYXNpY0F1dGhlbnRpY2F0aW9u
Content-Type: application/x-www-form-urlencoded
grant_type=password&username=johndoe&password=A3ddj3w
Configure the probe properties file:
Host: 'server.example.com'
Port: 443
Username : 'UserForBasicAuthentication'
Password: 'PasswordForBasicAuthentication'
EnableSSL: 'true'
## Keystore file should contain the target server certificate imported.
KeyStore : '/home/keystore.jks'
KeyStorePassword: 'TheKeystorePassword'
Configure the transport properties file:
httpVersion=1.1
loginRequestURI=/token
loginRequestMethod=POST
loginRequestHeaders=
Authorization=Basic ++Username++:++Password++,Content-Type=application/x-www-form-urlencoded
loginRequestContent=grant_type=password&username=johndoe&password=A3ddj3w
# For TLSv1.2 enabled server
securityProtocol=TLSv1.2
Configuring stronger SSL/TLS key exchange
In an environment where a stronger SSL/TLS server supports key exchanges is required, you can use the jdk.tls.ephemeralDHKeySize property to set the appropriate ephemeral DH key size. For example, if you need to have key exchanges with at least 112 bits of security which translates to a minimum key size of 2048 bits for Diffie Hellman and RSA key exchanges, the following can be set:
Edit $OMNIHOME/probes/java/nco_p_message_bus.env.
At the bottom of the file, add the following line:
NCO_JPROBE_JAVA_FLAGS=-Djdk.tls.ephemeralDHKeySize=2048 $NCO_JPROBE_JAVA_FLAGS
Additional troubleshooting topics
For additional troubleshooting topics for issues on common libraries such as Non-native, see https://www.ibm.com/support/knowledgecenter/SSSHTQ_int/omnibus/probes/all_probes/wip/reference/troubleshoot_probe_mtupacketissue.html.