Network Setup - MTU size
This test shows the impact on transaction throughput when changing the Maximum Transmission Unit size.
The Maximum Transmission Unit (MTU) size is the maximum packet size in bytes that a network interface can send over a network device. This size includes the protocol information, such as the TCP and IP headers. This size does not contain the Ethernet header. This study is concerned with the MTU settings that can be set in the configuration files in SLES, for example, in /etc/sysconfig/network/ifcfg-qeth-bus-ccw-0.0.0xxx for each device. The effective MTU size is also negotiated during the connection creation with the other end point, and the MTU can be reduced to the capacity of that device.
When the MTU is smaller than the packet size, the data must be divided into MTU sized packets. On the receiving side the packets must be reassembled, thus creating overhead that scales with the quantity of packages required. When the MTU is only slightly too small, there can also be wasted bandwidth. For example, when the MTU is slightly smaller than the average packet size the secondary packet can be almost empty.
An MTU size bigger than the packet size should have no impact on the network bandwidth, if all intermediate nodes on the path to the final destination support the MTU size, because only the real data are sent. But the number of packages processed in parallel is calculated with the TPC/IP windows size and the MTU size. When the window size stays constant and the MTU size is increased, this number is decreased, which might lead to a performance degradation, when the smaller MTU size would be appropriate as well.
Therefore, the ideal is to find the MTU size that fits to the package size used by the application and is supported by all intermediate nodes on the path to the final destination.
This study used as MTU sizes 1492 or 8192. For Jumbo frames, an MTU size of 8992 would be also possible, but 8192 is considered to be more memory efficiently because it fits exactly in two 4 KB pages, while 8992 would need three. With HiperSockets on IBM® System z®, the MTU size is implicitly defined by the MFS (maximum frame size) set on the IOCDS parameter. The MTU size for HiperSockets with an MFS size of 16 KB is 8192 bytes.
With Oracle RAC, the public interfaces are used to communicate with the clients for transactions and the private network is used as the interconnect between the nodes for messages and cache blocks.
The interface for the Oracle interconnect exchanges cached database pages between nodes, therefore we expect large packets, where a large MTU size might improve performance of the entire cluster.

Observation
The baseline example that these tests are measured against are with public LAN Ethernet interface set at 8192 and the private interface which was HiperSockets set at 8192. The first variation shows the impact of modifying the 1 Gb OSA interconnection to 1492, resulting in an improvement of 4.1%. The second example shows the private interface being set with MTU scaled up to 16384, with the result being a decrease in transactional throughput. The average packages size for send or received was 1290 - 1330 bytes in all scenarios.
Conclusion
It seems that either Oracle RAC or the Oracle client use as maximum package size 1330 bytes for all TCP/IP connections. Increasing the MTU size without increasing the TCP/IP window size leads to the effect that fewer packages are held in the TCPIP stack, which explains why the best throughput is seen with the small MTU sizes (1492 for the LAN and 8192 for the Interconnect).