Troubleshooting ClusterID issues in a Sterling B2B Integrator cluster

Body

When implementing a multiple node cluster or adding a new node to an existing cluster,

often issues arise where different ClusterIDs are generated, meaning that the nodes cannot see each other and load distribution fails.



This Blog aims to provide a few hints and tips on how to troubleshoot those kind of issues.



How is the clusterID generated?



By default the ClusterID is generated automatically based on four parameters of the database connection that can be found in the jdbc.properties file:



Driver + URL + Catalog + UserID



Example from jdbc.properties:



<vendor>Pool.driver=oracle.jdbc.OracleDriver

<vendor>Pool.url=jdbc:oracle:thin:@ldap://DatabaseHost:389/Z10TEST3,cn=OracleContext,dc=test

<vendor>Pool.catalog=Z10TEST3

<vendor>Pool.user=testUser



Note that some of those properties can be generated based on information from the sandbox.cfg using following parameters:



<vendor>_JDBC_URL (per example: ORACLE_JDBC_URL)

DB_DATA

ORA_DATA, DB2_DATA or MSSQL_DATA (Depending on your database)

DB_USER

ORA_USER, DB2_USER or MSSQL_USER (depending on the database)



Also you need to take into account that jdbc.properties entries can be overwritten by entries in customer_overrides.properties.



The ClusterID by default will be encrypted and will look something like:



ClusterID:00000001- 5c- 74 -5a 5-9 61 65 57 76 31 6d 35 73 57 72 45 76 ztzyaewv1m5swrev00000011 79 64 78 4b 75 41 64 4b 66 51 51 3d 0d 0a ydxkuadkftt.





How to check if the ClusterID is the issue?



1. Check the noapp logs, you might see error invalid clusterID errors, see example below.



ALL 000000000000 GLOBAL_SCOPE java.lang.Exception: 000110020188 WORKFLOW.QUEUE.ERR_Node_Exception1 [ClusterScheduler] received invalid ClusterID:00000001- 49- 33 -30 8-5 55 72 64 73 7a 6e 6d 64 6e 65 35 7a b30ewrdsznmdne5z 00000011 77 50 58 6c 30 31 4f 2f 41 4f 59 3d 0d 0a wtcl01o aol..



2. You can use the queueWatcher to View Cluster Multicast information on each node, this should give you the detailed cluster information of all three nodes.

If you do not see all the information of all three nodes when connecting to one of the nodes queueWarcher, you know you have a problem.



The queueWatcher can be accessed using a URL like: http://host:baseport/queueWatch



Note: If you cannot access the URL, make sure that the parameter queueWatcher is set to true in the noapp.properties_platform_ifcresources_ext, you can also set it on your customer_overrides.properties.



noapp.properties_platform_ifcresources_ext: queueWatcher=true

Customer_overrides.properties: noapp.queueWatcher=true

3. You can compare the clusterID info from all three nodes and see if they are different since the clusterID should be the same on all nodes of the same cluster.



4. You can use the dashboard UI by going to:

Operations => System => Cluster => Node Status



This will list all nodes, click on the node name to obtain more info about that node, this will open a new windows with the info for that particular node, scroll to the bottom of that page and click on the info button for Multicast Info.

See below an example of the Cluster multicast information from a node that has a different clusterID then the other nodes of the cluster, notice that although in the dashboard UI you can see all three nodes listed, in the queue watcher multicast information on node 3, only node 3 information is listed.



Cluster Multicast info from queue Watcher:







Dashboard UI cluster node info:

How to correct the issue?



1.Verify that JDBC driver, URL, Catalog and UserID for the jdbc connection matches on all three properties files, jdbc.properties, customer_overrides.properties and sandbox.cfg and make the necessary changes if required.



2.If you still have different clusterIDs after adjusting the all required properties entries, you can set the parameter encryptClusterID to false in the noapp.properties_platform_ifcresources_ext on all nodes of the cluster, this will generate the clusterID in clear text and you will be able to compare them and identify which parameter is causing the issue.



See below examples of the, by default, encrypted clusterID and one unencrypted after setting the encryptClusterID to false.



Encrypted ClusterID:

ClusterID:00000001- 5c- 74 -5a 5-9 61 65 57 76 31 6d 35 73 57 72 45 76 ztzyaewv1m5swrev00000011 79 64 78 4b 75 41 64 4b 66 51 51 3d 0d 0a ydxkuadkftt.



Unencrypted ClusterID:

ClusterID:oracle.jdbc.OracleDriverjdbc:oracle:thin:@ldap:/DatabaseHost:389/Z10TEST3,cn=OracleContext,dc=testZ10TEST3testUser



Note: Please note that if you make any changes in the sandbox.cfg, you will need to stop the node and run setupfiles before restarting the node, any other changes in the customer_overrides.properties or directly in the jdbc.properties only require a restart for them to take effect.







Please don't hesitate to leave any comments if you think that there is additional info to be added to the blog or open a new PMR with support if you need further assistance.