Troubleshooting
Problem
Three grant new realm servers (qaum01, qaum02, and qaum03) are configured in a Universal Messaging (UM) cluster, but the UM cluster is not able to form.
In nirvana.log from reach realm server, this message "Waiting for cluster to achieve quorum" is repeatedly thrown and the realm servers are never able to form the cluster.
Symptom
Three realm servers are configured in cluster, but they're not able to form the cluster.
Cause
On the system where qaum01 is running, it resolves its hostname with the correct IP address. However, on the systems where qaum02 and qaum03 are running, they resolve their hostnames with the local IP address 127.0.0.1 instead of the actual IP addresses.
Due to the incorrect hostname resolution, qaum01 cannot communicate with qaum02 and qaum03, and vice versa. This prevents the UM nodes from forming a quorum, and cluster is not able to form.
Environment
Universal Messaging version 11.1
Three realm servers are running on three different systems with the below hostnames:
qaum01: host1.abc.local
qaum02: host2.abc.local
qaum03: host3.abc.local
Diagnosing The Problem
After analyzing the nirvana.log (locating at this directory:
1) On the system where qaum01 is running on, it's able to resolve its hostname with the correct IP address 172.xxx.yyy.zzz as seeing below in the nirvana.log of qaum01:
----
[Thu Sep 11 15:50:24.169 UTC 2025] [ServerStarterThread] Realm Validation: Resolved hostname host1.abc.local to host1.abc.local/172.xxx.yyy.zzz for realm qaum01
----
However, on the system where qaum02 is running, it's not able to resolve its hostname with the correct IP address. It's currently resolved its hostname with the local IP address 127.0.0.1 as seeing below in the nirvana.log of qaum02:
---
[Thu Sep 11 15:50:36.864 UTC 2025] [ServerStarterThread] Realm Validation: Resolved hostname host2.abc.local to host2.abc.local/127.0.0.1 for realm qaum02
---
And, the same issue for qaum03 as seeing below in the nirvana.log of qaum03
---
[Thu Sep 11 15:51:49.243 UTC 2025] [ServerStarterThread] Realm Validation: Resolved hostname host3.abc.local to host3.abc.local/127.0.0.1 for realm qaum03
---
2) Due to the above, qaum01 is not able to communicate with qaum02 and qaum03 and vice versa. This is the root cause why the realm servers are not able to form the quorum since they're not able to communicate to each other due to hostnames for qaum02 and qaum03 were resolved to the local IP address instead of the actual IP addresses. Below is the error that is seeing in the qaum01 nirvana.log:
[Thu Sep 11 15:54:16.128 UTC 2025] [Inter Realm Connection Pool:0] java.io.IOException: java.net.UnknownHostException: host2.abc.local
java.io.IOException: java.net.UnknownHostException: host2.abc.local
Resolving The Problem
Check the /etc/hosts file on the systems where qaum02 and qaum03 are running and compare it with the /etc/hosts file on qaum01 to ensure the actual IP addresses for qaum02 and qaum03 are configured correctly.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
19 December 2025
UID
ibm17250575