If a DSI cluster is unstable try changing the heartbeat policy
Diagnosis and treatment of a heartbeat failure in a DSI cluster
If a Decision Server Insights cluster stops processing events or stops communicating with clients, add the -Xverbosegclog JVM parameter to the jvm.options
file in your runtime servers. You can then check in the logs to see if the garbage collection (GC) pause times are longer than 30 seconds. If the GC times are longer than 30 seconds, update the configuration files to include properties to change the heartbeat policy and the XIO timeouts to match the heartbeat value.
How to change the heartbeat policy
- First of all, create a file named
catalogServer.properties
. - Add the following properties to the file (
catalogServer.properties
):heartBeatFrequencyLevel=10 xioTimeout=90 xioReadTimeout=90 xioWriteTimeout=90
- Copy and paste the file into the root folder ($DSI_HOME/runtime/wlp/usr/servers/catalog_name) of each catalog server.
- In the
server.xml
file of each catalog server, edit the xsServer element to add the file. The following example uses the file namedcatalogServer.properties
.
- Create a file named
runtimeServer.properties
. - Add the following properties to the file (
runtimeServer.properties
):xioTimeout=90 xioReadTimeout=90 xioWriteTimeout=90
- Copy and paste the file
runtimeServer.properties
into the root folder of each runtime server. - Add a reference to the properties file in the xsServer element of the
server.xml
files.
- Create a file named
client.properties
, and add the line: xioRequestTimeout=90000. - Copy and paste the file
client.properties
into the root folder of each runtime server. - Declare the file in the
jvm.options
file in each runtime server: -Dobjectgrid.client.props=client.properties - Create a new properties file named
objectGridClient.properties
, and add the following properties:xioRequestTimeout=90000 xioTimeout=90 xioReadTimeout=90 xioWriteTimeout=90
- Copy and paste the file
objectGridClient.properties
into the root folder of each connectivity (inbound and outbound) server. - Declare the file in the
jvm.options
file for each connectivity server: -Dobjectgrid.client.props=objectGridClient.properties - Finally, restart the catalogs, runtime servers, and connectivity servers in this order after making these changes.
Leave a Reply