Troubleshooting link errors

Fault isolation

Assume for these procedures that the observed errors originate from link errors and are not the result of configuration issues, network overload or network equipment failures.

Whenever CRC errors are discovered on a particular link, it is easy to jump to the conclusion that the link is causing the network issue. This might not be the case. Since CRC errors are just symptoms of a link issue, we need to trace the propagated error to where it originated.

Figure 1 shows a simplified network involving a server, a switch, and a storage device. In this example, assume that the server experienced an error at port 1. This observable error can potentially originate from links 1, 2, 3 or 4 and/or SFP 1, 2, 3 or 4.

To determine the original failing link, the observable CRC error needs to be tracked back to the first occurrence of the CRC error. By following this process, it is discovered in this example that CRC errors observed in link 4 were propagated from link 3, which in turn originated from link 2.

Dust, dirt, or other contaminants

One of the most common optical link problems is caused by dust, dirt, or oil in the connectors and fiber. 8 and 16 Gbps links are more prone to such issues while lower link data rates, such as 1, 2, or 4 Gbps may be unaffected.

Once the failing port has been identified by following the above fault isolation process, the receive power of the transceiver sitting in that port needs to be determined. An abnormally low receive power usually means that the physical link is dirty.

The receive power can be checked by querying the SFP diagnostics data via the command line interface. This information will provide a rough gauge whether the receive power is abnormally lower than the minimum receive specification of the transceiver. It is also prudent to compare this receive power with those of neighboring transceivers.

Best practices for minimizing link loss

The "link margin" or the "power budget" of the link is a measure of signal power gain or loss expressed in decibels (dB). Maintaining a healthy link budget is critical to establishing a reliable and stable network.

Attenuation on LWL connections

In the datacenter environment, there may be 8 Gbps or 4 Gbps LWL transceivers that are connected to 2 Gbps LWL transceivers using single-mode fiber over short distances. Such connections need to be optically engineered because there is a possibility that the transmit power of the 8 Gbps / 4 Gbps LWL transceivers may saturate the receiver of 2 Gbps LWL transceivers and cause CRC errors.

It is important to check the maximum receive power of the 2 Gbps LWL transceiver in the manufacturer's datasheet.

Without taking into account connector and fiber losses, the transmit power of 8 Gbps / 4 Gbps LWL transceivers may be observed to overdrive the -3dB maximum receive power of 2 Gbps LWL transceivers. If the maximum receive power of the 2 Gbps LWL transceiver is 0dB, a 4 Gbps LWL transceiver with a -1dB transmit power will not overdrive the 2 Gbps transceiver. Although some 8 Gbps LWL transceivers may reduce their transmit power to 4 Gbps levels, this may still overdrive a 2 Gbps LWL transceiver.

Best Practice for LWL connections – Optically engineer a long-distance connection

It is always a good practice to optically engineer a long-distance connection. Most link issues caused by SFP incompatibility can be solved either by use of 4 Gbps LWL transceivers or use of 8 Gbps LWL transceivers that employ rate select.

2G LWL SFP maximum receive power

The IBM SAN b-type 8 Gbps and 16 Gbps directors and switches use the latest high bandwidth Fibre Channel technology and auto-negotiate to 16 Gbps, 8 Gbps, 4 Gbps, or 2 Gbps based on the link data rate capability of the attached transceiver. Negotiation to 1 Gbps is not supported. Since 8 and 16 Gbps equipment is more sensitive, the existing cable plant may require additional attention to cable care after upgrading from 2 or 4 Gbps fibre. In a few cases, the Tx power of the switch can be higher than the connected equipment. In nearly all of those cases, there is enough attenuation in the existing cable plant so as not to require additional attenuation. The common 2 Gbps SFP Rx maximum power levels are listed in Table 2 and can be used as a quick method to relieve concerns of over saturation.