Sometimes we notice that an ISL is actually a bottleneck in a fabric. Not a congestion bottleneck where the throughput demand is just too high for the ISL's bandwidth. This one could be solved by putting another cable between the both switches. But if you have a latency bottleneck your ISL won't be running at the maximum of the bandwidth. The contrary is the case: it lacks the buffer credits to ensure a proper utilization. If you see a latency bottleneck on an ISL it's often back pressure from a slow drain device attached to the adjacent switch. But every now and then I get a case where it's just the ISL. Sometimes in one direction, sometimes in both. Even with lengths were you don't think about using long distance settings at all.
But in the past we did exactly that!
When we encountered a situation like that, the first step was always to get rid of everything that reduces buffer credits for the real traffic flows, like an active QoS setting without having QoS zones. If the problem was still there the only way to give it more buffers was to configure a long distance mode then. We solved performance bottlenecks on ISLs by setting up a let's say 50m ISL in a 10km long distance mode (LE). I described this also 2 years ago in the article How to NOT connect an SVC in a core-edge Brocade fabric. While this indeed gives you more buffers, it comes with a drawback.
Long distance and Virtual Channels
On normal ISLs we have Virtual Channels. They work in a way that the buffer credit management of the ISL is logically partitioned into 8 channels. When we talk about normal Class 3 open systems traffic it's used this way:
|2||Class 3 data|
|3||Class 3 data|
|4||Class 3 data|
|5||Class 3 data|
VC 0 is used for inter-switch communication. For example if a new zoning is distributed to all switches. The VCs 6 and 7 are not really of interest most of the time. We have to focus on VCs 2, 3, 4, and 5. (Mind the Oxford comma!) If you have a slow drain device that is reached using Virtual Channel 2 in your fabric, then at least the traffic of the other 3 data VCs is unaffected. With a long distance mode like LE you lose that advantage.
Buffer distribution on Virtual Channels on a normal ISL:
Buffer distribution on Virtual Channels on a LE configured ISL:
While you have more buffers in total now, only the first data VC has them assigned. There is no partition of data traffic anymore and the result is the risk of Head of Line Blocking (HoLB). A latency bottleneck (for example due to back pressure from a slow drain device) will always impact ALL the user data going over that ISL! That's a high price for those additional buffers.
With FabricOS v7.2x Brocade introduced a new command:
It allows you to assign a freely configurable number of credits between 5 and 40 for that ISL. You might ask:
But LE mode gives me 80 on 16Gbps!?!
Yes, but look at the distribution:
Not the whole data part of the link will share the 40 buffers. Each data VC gets its own 40 buffers and they are still handled independently! No Head of Line Blocking! And remember: This is not meant for long distance connection and it still comes for free! It works on 8G switches, too, as long as they are running at least v 7.2x.
To give 40 buffers to each data VC on an ISL at port 1 you would enter:
portcfgeportcredits --enable 1 40
With the --disable parameter you switch back to normal mode and with --show you can see the current configuration of a port.
And please keep an eye on the number of remaining buffers in portbuffershow :o)
So from now on if you need just some more buffers on your ISLs to keep everything running smoothly: