webMethods Knowlegebase : When first TSA is PASSIVE, failover to second ACTIVE TSA slows down Cache Manager initialization (1768653)

Problem

Customer operates six cache managers under IS 9.6 with IS_9.6_Core_Fix6 and SCG_9.6_0_TerracottaFix1.

Cache managers are pointing to a pair of ACTIVE-PASSIVE Terracotta 3.7.10 TSA servers:

url="WMTC-QA01:9510,WMTC-QA02:9510"

The name of the first TSA server in tc-config.xml is

<server host="xxx-qa01" name="TC01_1">

and is intended as ACTIVE in normal operation.

The name of the second TSA server in tc-config.xml is

<server host="xxx-qa02" name="TC02_1">

and is intended as PASSIVE in normal operation.

When first TSA server xxxQA01:9510 is PASSIVE -

there is very slow IS startup and delays of total 6-7 minutes in cache managers initialization.

In this case of first TSA being PASSIVE, failover to xxx QA02:9510 occurs for every cache manager. This failover (going to the next one) can be seen in wrapper.log:

INFO | jvm 1 | 2015/08/04 15:26:06 | 2015-08-04 15:26:06,018 WARN - We couldn't load configuration data from the server at 'WMTC-QA01:9510'; this error is temporary, so this source will be retried later if configuration can't be loaded elsewhere. Skipping this source and going to the next one. (Error: Connection timed out: connect.)

INFO | jvm 1 | 2015/08/04 15:26:07 | 2015-08-04 15:26:07,059 INFO - Successfully loaded base configuration from server at 'WMTC-QA02:9510'

We can see the long initialization of cache managers in IS server.log, by looking at time stamps emitted by IS trace for component 0017. The total delay accumulated by cache managers initialization is 6-7 minutes.