ClearCase MultiSite and DNS   

We had recently a problem at a remote site that suddenly failed to ship anything to other sites. The symptoms, as well as IBM Rational support, indicated a network problem. We could have spent weeks in a very expensive wild goose chase had we not found contact with a person who had experienced the same problem before.

So, stick this to your checklist!

When ClearCase MultiSite prepares to ship packets to a destination VOB server, it first tries to send the packets directly. If the destination is unreachable, or if the host name is not even resolvable, ClearCase looks up the shipping server that is the next hop for this destination. However, if ClearCase fails to get response from host name resolution, or the response comes too slowly, the shipping attempt fails even when ClearCase has already resolved the next hop server name and initiated the contact. The failure looks like a network failure, because ClearCase already had made the first contact with the net hop.

One might be tempted to think that a shipping server only ever needs to know about the hosts it actually communicates with - and these can be defined locally. However, DNS has to be properly configured for ClearCase to get a response to its attempts to resolve the non-resolvable destination host names of the remote VOB servers. If there is even one DNS server in your /etc/resolv.conf that does not respond, or responds too slowly, ClearCase fails to ship, and the failure looks nothing like a DNS failure. There is nothing in the error messages that would help you and point you to check the DNS configuration. It is unusually hard one to diagnose.

So, even if you define the resolvable host names locally, keep DNS configured properly for the non-resolvable ones.

Oh, boy. I don't know where to begin.

> DNS is needed to pull the multisite license(every
> clearcase license go through dns first)

I cannot fathom what you mean by that. First of all, DNS is not equipped to "pull through" any kind of licenses. Secondly, shipping functionality does not need any license at all - at least, on Linux/UNIX, it doesn't, because it runs as root, and root does not need a license. We have a shipping server that is completely disconnected from our ClearCase license server by firewalls, and it works fine.

> and not for the shipping.

ClearCase does not directly call DNS. It calls a system service that performs the host name resolution, and that system service calls DNS whenever it encounters a host name that is not defined locally. When we use shipping servers to reach remote VOB servers that are not directly reachable, those VOB server host names are non-resolvable, and that is what ClearCase finds out when the system service responds to ClearCase. If DNS is configured right, that is. If it is not configured properly, and ClearCase does not receive the information that the host is non-resolvable, then shipping fails.

> It always good if the two host communicate via ALBD
> then DNS is not needed unless this two are physically
> connected.

What is that supposed to mean? ClearCase servers always communicate via albd, regardless of DNS.

> we could have done this multitool lsshipinfo -host A from
> site B to talk to abld of A.if this work then i think DNS
> is not needed at all.

Aha, I see. You "think". So your reply is all imagination, not based on real experience.

Well, my posting is based on not only my own experience, but also on the responses we have received from Rational support over the years - and not only support, but engineering too.

In the first instance when we had this problem, and by our own means learned that it had something to do with DNS, Rational engineering confirmed that MultiSite always attempts to resolve the destination VOB server's name first, and only then looks up the next-hop shipping server as defined in the shipping.conf file. And they told us (this we could not have imagined) that although MultiSite manages to resolve the next-hop shipping server's name, it still fails in the shipping attempt, if the look-up for the destination VOB server's name times out. This is not what "I think", this is what Rational engineering actually told us.