Are you backing up your AIX systems
over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your
backup server and clients on the same physical POWER system? You are most
likely backing up over Virtual Ethernet to another AIX LPAR that is running
your enterprise backup software, such as TSM or Legato Networker for example.
And you probably have a dedicated private virtual network (and adapters) on
both the clients and the server to handle the traffic for the nightly backups.
The next question is, have you tuned your Virtual Ethernet adapters?
There are several tips available for
tuning your Virtual Ethernet adapters for better performance on AIX. These tips
include changing settings such as MTU size, TCP window sizes, enabling
largesend, etc. I highly recommend the following blog posts from Anthony English
and Nigel Griffiths on this subject:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en
OK, so you got everything humming
along nicely, your backups are flying over the virtual network (across the POWER
hypervisor) and everybody is happy. After a period of time, you notice that the
backups have started to “slow down”. They are taking longer to finish. The
overall throughput of a backup drops. Some backups start in the evening around
9pm and are still running the next morning at 7am! In some cases you need to
kill the backups or even reboot the backup server LPAR for things to return to
normal.
“What is going on!?” You cry.
Well, there are a number of reasons
why this could be happening. For example, your shared processor pool may be
overwhelmed during the backup window. As we know, Virtual Ethernet adapters require
CPU to do their work. If the CPU pool is running low on available CPU
resources, this could contribute to the problem. And of course there could be
tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or
there may be issues with other pieces of the infrastructure, like network and
SAN switches, adapters, etc. Perhaps there’s an issue with the applications
and/or databases on the AIX systems? They often have their own mechanisms/tools
for backing up their data to your enterprise backup software. Is the backup
server sized to cope with the load i.e. CPU, memory, disk layout and I/O,
sufficient tape drives, disk storage pools, etc?
So assuming you’ve checked all of
the above (and more), then perhaps you’ve hit a problem that I encountered
recently. In my particular case, backups “over the hypervisor” were slowing
down, without any discernible cause. Initially the backups would be “very fast”
but after a month or so, things would start to slow down dramatically.
We noticed that there were very
large (and increasing) values for “Packets
Dropped”, “Hypervisor Send/Receive
Failures” and “No Resource Errors”
in the output from the netstat –v
command.
ETHERNET
STATISTICS (ent1) :
Device Type:
Virtual I/O Ethernet Adapter (l-lan)
Hardware
Address: 41:ba:13:e7:25:0b
Elapsed
Time: 42 days 4 hours 3 minutes 34 seconds
Transmit
Statistics: Receive
Statistics:
-------------------- -------------------
Packets:
5978589961 Packets: 26139832411
Bytes:
779465989202 Bytes:
711051516630458
Interrupts:
0 Interrupts: 6804561727
Transmit
Errors: 0 Receive Errors: 0
Packets
Dropped: 0 Packets Dropped: 86012309
...
Max
Collision Errors: 0 No Resource Errors:
46113807
...
Hypervisor
Send Failures: 0
Receiver Failures: 0
Send Errors: 0
Hypervisor Receive Failures: 46113807
After some discussion with IBM AIX
support, we discovered that would should increase some of the buffer sizes for
our Virtual Ethernet adapter (the entX device). This would alleviate the no
resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed
that the Medium, Large and Huge buffers had
all reached their maximum values in the past.
...
Receive Information
Receive Buffers
Buffer Type Tiny Small
Medium Large Huge
Min Buffers 512 512
128 24 24
Max Buffers 2048 2048 256 64
64
Allocated 513 535
148 28 64
Registered 512 510
127 24 13
History
Max Allocated 576 951
256 64 64
Lowest Registered 502
502 64
12 11
...
The advice from IBM
support was to increase these buffers using the chdev command (they also advised that we should reboot for the
changes to take effect):
#
chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96
-a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P
#
shutdown -Fr
Since implementing this
tuning change (to the adapter on the backup server), we have not had a repeat
of the problem. We will continue to monitor the performance and I’ll be sure to
let everyone know if we have further issues.
95
