Multicast ports in node to node communications

Cluster nodes are configured to communicate with each other using JGroups, an open source toolkit that provides flexibility for protocol configuration.

JGroups provides rich open management features, along with multiple protocol support. JGroups supports multicast (UDP) and TCP-based communication protocols.

You can choose between UDP and TCP during configuration. It is strongly recommended to use TCP communication, but the decision to use UDP or TCP depends on several factors, including the company's security policy (some disable UDP). For more information on UDP and TCP, see the JBoss.org Community Documentation at http://www.jgroups.org/manual/html/user-advanced.html.

When JGroups is configured to use multicast (UDP), all cluster nodes communicate with each other on a specific IP address and port. The configuration of multicast ports is based on the installation base port. All clusters that are on the same subnet that is configured on the same base port send multicasting messages on the same multicast IP address and port.

To avoid this situation, each cluster on the same subnet needs to be configured on different base ports. Install your clusters on different port ranges or on different network segments with multicast forwarding restricted, so that they do not interfere with each other. The default multicast address is 239.255.166.17. This address is configurable, with a port range of 10 ports, starting with the multicast base port for the instance.

All nodes that are participating in the same cluster must be installed on the same multicast base port (the multicastBasePort property in the noapp.properties_platform_ifcresources_ext.in file). This value is usually computed from the system base (non-multicast) port, but can be configured separately in the noapp.properties_platform_ifcresources_ext.in file, to allow different nodes in a cluster to be installed at different (non-multicast) port ranges. Also, all the nodes in the cluster must be installed in the same subnet.

For node to node communications, the properties are defined in the jgroups_cluster.properties file. The following attributes are used to define communications:
  • property_string - default value is UDP.
  • distribution_property_string - default value is TCP. This attribute must never be set to UDP.

If you want to change the communication for cluster multicast from the UDP protocol to TCP, you need to change the value of the property_string property in the jgroups_cluster.properties.in file (after you back up the file). Then, run the setupfiles command. You can change this value right after the installation or after you start running the cluster. If you change the file after you start the cluster, you need to stop all nodes of the cluster, change the value on each node, and then restart your cluster.

For more information, see JGroup properties.

To change the communication for cluster multicast from the UDP protocol to TCP, use the following value for the property_string property in the jgroups_cluster.properties.in file:

property_string=TCP(start_port=any_available_port_number): 
TCPPING (initial_hosts=this_instance_host_ip[start_port_number],
theothernode_instance_host_ip[theothernode_start_port_number];port_range=2;
timeout=5000;num_initial_members=3;up_thread=true;down_thread=true):
VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(down_thread=true;up_thread=true;
gc_lag=100;retransmit_timeout=3000):pbcast.GMS(join_timeout=5000;
join_retry_timeout=2000;shun=false;print_local_addr=true;
down_thread=true;up_thread=true)

In addition, if you are using TCP for both property_string and distribution_property_string, the initial_hosts list for TCPPING should contain all the hosts in the cluster.

If you are using the FIFO Routing Adapter, the use of UDP protocol for the jgroups property lock.protocolStack with cluster environments using the same base ports in the same subnet can sometimes cause the FIFO Routing Adapter not to start correctly and stay stuck in starting status.

To correct this issue the jgroups property, lock.protocolStack needs to be updated to TCP.

Use the following value for the property_string property in the jgroups_cluster.properties.in file or the customer_overrides.properties file with the jgroups_cluster prefix replacing HOST_ADDR and MULTICAST_NODE_PORT3 in the string with values from sandbox.cfg:

lock.protocolStack=TCP(bind_addr=HOST_ADDR;bind_port=MULTICAST_NODE_PORT3):TCPPING(initial_hosts=HOST_ADDR[MULTICAST_NODE_PORT3];port_range=0;timeout=5000;num_initial_members=2):MERGE2(min_interval=3000;
max_interval=5000):FD_ALL(interval=5000;timeout=20000):FD(timeout=5000;max_tries=48;level=ERROR):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(retransmit_timeout=100,200,300,600,1200,2400,4800;
discard_delivered_msgs=true):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=20000;max_bytes=0):pbcast.GMS(print_local_addr=true;join_timeout=5000)

For more information about UDP, TCP, and JGroups communications, see the Sterling B2B Integrator Clustering documentation.