What's The Latency, Kenneth?
MartinPacker 11000094DH Comments (2) Visits (8729)
OA37826 really is the gift that keeps on giving: I got really nosy about Coupling Facility links when it came out  , though most customers didn’t get the added benefits of CFLEVEL 18 for a while.
This post is about a customer installation which pointed out another benefit of the instrumentation. 
Here’s a simplified version of their Parallel Sysplex environment:
They have 3 routings between two data centres - and 6 links from each CEC to the CF image in the other data centre. Structure duplexing is not used - as the customer is using external (to this sysplex) coupling facilities.
According to the SMF 74 Subtype 4 data the signalling latency from MVSA to CFB is 161μs (x2), 172μs (x2), and 176μs (x2). You can see the three routes on the diagram.
MVSB shows the same signalling latencies to CFA - which is to be expected.
You’ll notice I’ve used latency - which is what 74–4 gives you. A good rule of thumb is each 10μs of latency translates into 1 kilometer of distance. 
I was supplied with the customer’s own diagram and it shows slightly different distances. The discrepancies between the two sets of estimates are not accounted for by any inaccuracy in that formula. I say that because one of the customer’s path distance estimates is substantially lower than my minimum, one if substantially higher, and the third about the same. 
It could be a matter of the vendor being inaccurate, though not by much (and life isn’t usually that simple). If the discrepancy was massive compared to this you might begin to suspect “fibre suitcases” left in the route. In any case for once SMF can give you a view of distance.
The “local” latency is 1μs, which is the same as I’ve seen in previous cases. The latency value is an integer number of microseconds and the minimum value is 1 for a supported link type. It means “very short link indeed”.
Both the high values (161 - 176 μs) and the low value (1μs) are consistent with the Adapter types - HCA3-O LR (1X) in the former case and HCA3-O (12X) in the latter. Talking of which, the physical adapters are reported in the Channel Path Data Section (as mentioned in System zEC12 CFLEVEL 18 RMF Instrumentation Improvements ), alongside the latency. So we can see which links / CHPIDs / PCHIDs / ports etc use which routes. 
In this customer case there is nothing to recommend.
I simply observe the three-route solution, which is patently sensible.
Impact On My Reporting
In my tabular report that documents the paths between z/OS systems and coupling facilities I had one row per z/OS-to-CF pairing. It had the range of latencies for that pairing. In the customer example it said “161 - 176”.
That was useful as it alerted me to the possibility (I hadn’t considered before) of multiple latencies and hence multiple routes. But it told me I could do better:
Now, for each link I list the latency separately - if there is any variation. So, “tourist information” perhaps but I can discuss with a customer their use of alternate routes between sites. 
I consider this a nice little piece of (easy to code) extra information.
Note that the Coupling Facility does not select paths based on distance/latency. And that latency values are static in all the sets of data I’ve seen. These two facts are mutually consistent.
Also signalling latency is not the same as request service time. It might be interesting to compare latency to service time to try to understand the non-CPU component of service time. But expect Async requests to weaken the correlation - as requests can be expected to be delayed sometimes for reasons unrelated to signalling links.
And finally all this applies equally to links between coupling facilities for structure duplexing.
Anyhow look up the data in your favourite performance reporting tool and try it. You’ll like it!
And wasn’t I naïve when I wrote “Call me nosey  but I really want this - as I like to figure out whether machines are close together or in different data centres.”