AIX or VIOS Start-up Memory Problems

I have been working with a couple of customers that have a newly setup AIX Logical Partition (LPAR) that fails to start due to not having enough memory. The common factor is that the LPAR for AIX (or VIOS) includes a good number of what I call "high speed" adapters like 10 Gbps Ethernet, or 4 Gbps or 8 Gbps Fibre Channel adapters. They tend to have multiple adapters of each type for redundancy and throughput. For example, trying to start the LPAR with 1 or 2 GB of memory but it turned out they need 4 to 5 GB to get it powered on and running.

Information

I did not find much official information about the memory requirements for these adapters and have had to call in favours to find out the little facts I have. I think the problem is that it is complicated. There are many adapter types, the memory foot print depends on may things like negotiated line speeds, the number of active ports, numbers of remote connections and protocols actually used. Also the device drivers seem to be very pessimistic when AIX boots and grab the maximum (just in case) but then relax a bit and free up memory resources once AIX is up and running.

It Hurts even at Slow Speeds

Even if you are using high speed adapters mostly for lower latency or forward thinking and planning for the future infrastructure and not for high throughput in the short term - you still hit these start up problems. For example, you have multiple 8Gb Fibre Channel adapters but know you are never going to use anything like this much bandwidth and may even be connecting to a 2Gb SAN for now.

Ball Park Figures

From my collection of experts in the Labs and services,  I keep getting estimates between 150 and 256 MB per adapter port absolute minimum. Of course, many adapters have multiple ports but note, often you can't concurrently drive every port at the maximum theoretical bandwidth limit.

Contemplation

1. So I was thinking ... at 10 Giga-bits per second on an Ethernet adapter (roughly 1 Giga-Bytes/s), with a 256MB buffer, it will fill the buffer in just a quarter of a second!!! That is assuming your application can cope with such data rates and you have the CPU capacity to drive the I/O this hard. In that quarter of a second, it will accept the incoming packets and then try to get the far end to stop sending more. A buffer overrun means retransmission and the performance will dramatically be effected.  This tells me that 256MB or better yet double that, is reasonable starting point for these high speed adapters.
2. It is also odd thinking about Mega-bytes these days - for some reason we bung in 4 GB in to every Windows Laptop without thinking and then start talking MBs when we get to Servers with potentially 100's of users attached!!
3. Starving your high speed adapters of memory is like putting a 3 litre engine on a go-kart. It might seem like a good idea at the start but you know it is not really a good move and something will break pretty quickly.
   

Conclusion

• My current Safe Rule of Thumb is 2 GB plus 512 MB per high speed adapter - this is memory for the LPAR or VIOS with the physical adapter(s). This is NOT an IBM statement but my "best guess" given the information available.
  
• Example, with four 10 Gbps Ethernet and four 8 Gbps Fibre Channel = 2GB + 8 * 0.5 GB = 6 GB.
• You may get away with a bit less but the "Failure to Launch" AIX or VIOS is extremely annoying and then you might have to remove 0.25 GB from a dozen other LPARs to get this LPAR running - not a good use of any ones time.
• Virtual I/O Server client LPARs are a completely different case, as they are not dealing with the external "real world" and the Hypervisor provides rapid bandwidth throttling and the VIOS provide the buffering.

Hope this helps, any one got an alternative Rule of Thumb, thanks Nigel Griffiths



Identificações:  10gb 8gb fc vios memory ethernet high firbe-channel speed adapters aix