We have heard that a “tens of kilometres” limit applies to the distance between the two sides of a Geographically Dispersed pureScale Cluster (GDPC). Buy why?
This is based on a physical limitation i.e. the speed of light in glass (fibre) which is about 5 µs / km. From this we can calculate a round-trip times from member to CF as follows at these distances:
3km = 30 µs
10km = 100 µs
50 km = 500 µs
100 km = 100 µs (or 1 ms)
300km = 3000 µs (or 3 ms)
This will have a significant effect on the performance of the cluster, especially when we start to get into tens of kilometres. The “normal” times for RDMA actions are of the order of 15 µs so to those we need to add this latency for the distance. Compared to a normal pureScale cluster (all in one location) an RDMA action will be slower at a distance as follows.
3km = 3 times slower
10k = 8 times slower
50k = 33 times slower
100k = 66 times slower
300k = 200 times slower
µs = microseconds or 10-6 seconds
ms = milliseconds or 10-3 seconds