Over my career as an AIX administrator I have run in to problems with the "Maximum Virtual Adapters" LPAR profile setting many times. This setting controls the highest virtual slot number that you can use on an VIO or AIX LPAR. It is not dynamically changeable and requires a reboot to modify.
By default when you create a VIO server the Maximum Virtual Adapters is set to only 20. Any time you are creating a VIO server I recommend setting this setting much, much higher. The higher you set this to the more memory overhead, so don't go crazy. Determine how many slots it will take to support the number of LPAR's you plan on having on the system, and then at least double that number to come up with what you want to set this to. Also @nixysug pointed out that if you have the maximum virtual adapters set to above 1,000 you might have issues with live partition mobility (for more info see http://nixys.fr/blog/?p=214
Here is the default:
Lets suppose you have a dual VIO system that was setup with the default Maximum adapters set to 20. You are using either Virtual SCSI or Virtual Fibre Channel (doesn't really matter which one as the slots work basically the same way). You have come up with a slot numbering convention where slots 10 and under are used for Virtual Ethernet. For the VSCSI/VFC slots, your convention is to start at slot 11 and have 2 slots per AIX LPAR. One slot will go to VIO1 and the other slot to VIO2. You decide to have the odd number slots go to VIO1 and the even number slots go to VIO2. This is a very common slot numbering convention and is often recommended/used in IBM documentation like Redbooks.
Here is a diagram of how the slots are laid out in this example:
This works fine until you get an urgent request to add an additional LPAR to the system. You go to add the "p6_aix5" LPAR and decide to use slot 19 to VIO1 and slot 20 to VIO2. When you go to add the slot 20 to VIO2 you'll get a message like this:
The system will not allow you to create any slots equal to or greater than whatever the maximum virtual adapters setting is set to. So if you have a maximum virtual adapters to the default, 20, then you will only be able to use slots numbered 19 or below.
As previously mentioned the only way to change this maximum virtual adapters setting is to change the VIO server profile, shut the VIO server all the way down, and then re-activate it. Depending on your environment having a downtime on the VIO server might be a big deal, even if you have dual VIO servers.
If you have a urgent request to build a new LPAR, here are your options at this point:
- Change the profile for both VIO servers and change the maximum virtual adapters setting to be higher, and then shutdown each VIO server one at a time and re-activate them. This is the best option.
- We originally said our convention was to use slots less than 10 for Virtual Ethernet. If you aren't using all of these for Virtual Ethernet you can steal some of these and use them for the new partition. For example, if they were available you could just use slots 9 and 10 and still follow the rest of your slot number convention (odd slot to VIO1, even slot to VIO2).
- Get creative and totally go against your slot number convention. - Keep reading if you are interested in this option.
I have found that there is a lot of confusion out there about virtual slot numbers on POWER servers. For the first several years I worked on these servers I had a lot of incorrect assumptions about how the slots worked.
A lot of people are under the incorrect impression that the client and server adapter numbers need to match. For example, some people think that if you create an adapter with slot 11 on the LPAR that the slot on the VIO server also needs to be slot 11. This is totally incorrect, and you can in fact have VSCSI client slot 11 connect to VSCSI VIO server slot 16 with no problems at all.
Another common misconception is that the slot numbers are "global" across the entire system and must be unique across the entire system. In fact, it is possible to setup multiple LPAR's that all use the same virtual slot number. For example, you could have 100 LPAR's and have every one of them use virtual slot numbers "4" and "5" for their virtual SCSI client adapters. Here is an example of a slot numbering convention where this is done (notice each LPAR uses slot 4 and 5 for the client adapters):
Back to our original issue where we are trying to create a new LPAR and are getting "The maximum number of virtual slots must be greater than the highest slot number" error message.
Let's suppose we want to build 2 additional LPAR's and not have a VIO downtime.
One method we could use to build these 2 new LPAR's right away and avoid a VIO downtime is to break our original slot numbering convention. For the other LPAR's we had used slots "11" and "13" on VIO1 and slots "12" and "14" on VIO2. Because these slot numbers do not need to be globally unique across the system there is nothing from stopping us from using slots "11" and "13" on VIO2 and slots "12" and "14" on VIO1 (flip-flopped from how the other LPAR's had been setup).
So the solution is to build the 2 new LPAR's like this:
p6_aix5 client slot 11 mapped to VIO2 slot 11.
p6_aix5 client slot 12 mapped to VIO1 slot 12.
p6_aix6 client slot 13 mapped to VIO2 slot 13.
p6_aix6 client slot 14 mapped to VIO1 slot 14.
Below is a diagram of how this setup would look. Note that slots 11 are used by p6_aix5 and p6vio2. Slots 11 are also used by p6_aix1 and p6vio1; but as discussed this isn't a problem.
So if you ever run in to the this error message relating to the maximum virtual adapter setting... Don't panic. If you can't reboot the VIO server to change it you still have options. You might just need to get creative and think outside the box and be willing to break/change your slot numbering convention.
A tool that might help you out with understanding and validating your Virtual Slot configuration is the "pslot" tool. This is a Perl program I wrote that will visualize and validate your virtual slots. It was used to create the diagrams in this article. It is free and open source, and can be downloaded from http://pslot.sourceforge.net/