Chris's AIX Blog

Just say you change the queue_depth on a hdisk with chdev –P. This updates the devices ODM information only, not its running configuration. The new value will take effect next time I reboot the system. So now I have a different queue_depth in the ODM compared to the devices current running config (in the kernel).

What if I forget that I’ve made this change to the ODM and forget to reboot the system for many months? Someone complains of an I/O performance issue....I check the queue_depths and find they appear to be set appropriately but I still see disk queue full conditions on my hdisks. But have I rebooted since changing the values?

How do I know if the ODM matches the devices running configuration?

For example, I start with a queue_depth of 3, which is confirmed by looking at lsattr (ODM) and kdb (running config) output:

# lsattr -El hdisk6 -a queue_depth

queue_depth 3 Queue DEPTH True

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;                                                   < In Hex.

Now I change the queue_depth using chdev –P i.e. only updating the ODM.

# chdev -l hdisk6 -a queue_depth=256 -P

hdisk6 changed

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

kdb reports that the disks running configuration still has a queue_depth of 3.

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;

Now if I varyoff the VG and change the disk queue_depth, both lsattr (ODM) and kdb (the running config) show the same value:

# umount /test

# varyoffvg testvg

# chdev -l hdisk6 -a queue_depth=256

hdisk6 changed

# varyonvg testvg

# mount /test

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x100;                                    < In Hex = Dec 256.

# echo "ibase=16 ; 100" | bc

256

This is one way of checking you’ve rebooted since you changed your queue_depth attributes. I’ve tried this on AIX 6.1 and 7.1 only.

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.kdb/doc/kdb/kdb_pdf.pdf

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

        0514-018 The values specified for the following attributes

                 are not valid:

     max_transfer     Maximum TRANSFER Size

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth     256                              Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

1. Remove the cd0 device from the LPAR i.e.

   # rmdev –dl cd0

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

   $ rmvdev –vtd vtopt0

3. Shutdown the LPAR.

4. Reboot the VIOS.

5. Restart the LPAR.

6. Change the max_transfer value for the hdisk on the LPAR.

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 0                                Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x100000                         Maximum TRANSFER Size      True

pvid            00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth     3                                Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

The following explanation was given by IBM support:

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

Here are some questions I received recently regarding VLAN tagging on the VIO server. My answers are shown in green.

“Hi Chris,

Q: I’m trying to understand when, where and why there would be the need to use ‘mkvdev –vlan (etc.) on the VIOS, and I’m wondering whether you would be able to clarify this for me, please.

Is it necessary to add the VLAN tag devices to the SEA, or is it suffice to just have them defined within the Virtual Ethernet itself which is part of the SEA?”

A: It is suffice to simply define the VLAN ids assigned to the Virtual Ethernet adapters associated with the SEA.

“Q: For completeness, on the rare occasions I have done this, I have added the VLAN’s to the Virtual Ethernet and also as VLAN devices on the VIOS (mkvdev –vlan etc.)”

A:  mkvdev –vlan is not necessary, unless the VIOS needs to communicate with hosts on different VLANs i.e. you need an IP address on the VIOS for each VLAN. This does not mean the SEA will bridge this VLAN traffic for VIOCs.

“Q: The reason I started thinking of this is, is because one of our customers wants to add new VLAN’s to their SEA, but they’re not running Power7 hardware. Therefore, the online method would be to add a new Virtual Adapter which contains the new VLAN ID’s to the VIOS using DLPAR, then use chdev –dev (etc.) on the SEA to include the new Virtual Ethernet.”

A: Agreed. The “IBM PowerVM Virtualization Managing and Monitoring” Redbook states: “If your system doesn’t support dynamic VLAN modifications and you are modifying the VLAN list of a virtual Ethernet adapter that is configured in a SEA with ha_mode enabled, the HMC will not allow you to reconfigure the list of VLANs on that interface. You will need to add an additional virtual Ethernet adapter and modify the virt_adapters list of the SEA, or modify the profile of both Virtual I/O Servers and re-activate both Virtual I/O Servers at the same time.”

“Q: From the phone call I had, it would appear that the VLAN tags are included on the Virtual Ethernet device, but have not been added to the SEA by running mkvdev –vlan (etc. ) on the VIOS’s.  This leads me to assume that the ‘mkvdev –vlan’ is only required if there is a requirement to access the VIOS itself from a particular VLAN. Am I right, or is there something I’m not understanding? I’m unable to find documentation that explains the answer. Do you happen to know?”

A: That is also my understanding (based on my experience). On page 483 of the “IBM PowerVM Virtualization Introduction and Configuration” Redbook , it states: “The addition of VLAN interfaces to the SEA adapter is only necessary if the VIO Server itself needs to communicate on these VLANs”.

“Q: Hi Chris,

We are trying to associate a new entX Virtual Ethernet Trunk Device to an existing SEA. The new device must be configured for VLAN tagging. The existing virtual Ethernet adapter that (is already associated with the SEA) is not configured for VLAN tagging. This device will remain associated to the SEA and continue to pass untagged packets to the already configured network.

Ultimately  the configuration we want would be two entX devices associated with the existing SEA. One entX device is configured for notagged packets and the other entX device is configured for tagging.

How do we configured this?”

A: What you are trying to do should work:

Existing setup:

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

Proposed setup:

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

VIOC2 boot1 PVID 22 ---> VIOS1 ent6 PVID 92*, VID 22, 802.1Q   ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=22 (NEW)

VIOC2 boot2 PVID 23 ---> VIOS2 ent6 PVID 93*,  VID 23, 802.1Q  ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=23  (NEW)

*The PVID can be any unused/throw away number. The new ent6 adapter would be configured similar to this:

Then you would run this command to associate the new ent6 with the existing SEA, ent5:

$ chdev -dev ent5 -attr virt_adapters=ent3,ent6

Have a look at the following sections of this Redbook:

3.2.3 Virtual I/O Server: Dynamically add or modify an existing adapter such that the required VLAN is listed in the 802.1q Additional VLAN fields (on POWER7 only).

3.2.2 Hardware Management Console: Dynamic VLAN modification in the GUI (on POWER7 only).

http://www.redbooks.ibm.com/redpieces/abstracts/sg247590.html?Open

This document (by Anthony English) is also a very useful document relating to VLANs, SEAs and VIOS:

http://www.ibm.com/developerworks/aix/library/au-managevlans/index.html?ca=drs-

Reply: “hmm ok I see what you are saying, I will give it a go and tell you how it turns out...thanks. ok finally got around to testing using a VIOS at DR site. Created new virtual adapter PVID 55 and VID 888 (ent9) then added it to the existing SEA as shown below:

accounting    enabled   Enable per-client accounting of network statistics                 True
ctl_chan      ent6      Control Channel adapter for SEA failover                           True
gvrp          no        Enable GARP VLAN Registration Protocol (GVRP)                      True
ha_mode       auto      High Availability Mode                                             True
jumbo_frames  no        Enable Gigabit Ethernet Jumbo Frames                               True
large_receive no        Enable receive TCP segment aggregation                             True
largesend     0         Enable Hardware Transmit TCP Resegmentation                        True
lldpsvc       no        Enable IEEE 802.1qbg services                                      True
netaddr       0         Address to ping                                                    True
pvid          40        PVID to use for the SEA device                                     True
pvid_adapter  ent4      Default virtual adapter to use for non-VLAN-tagged packets         True
qos_mode      disabled  N/A                                                                True
real_adapter  ent2      Physical adapter associated with the SEA                           True
thread        1         Thread mode enabled (1) or disabled (0)                            True
virt_adapters ent4,ent9 List of virtual adapters associated with the SEA (comma separated) True

VLAN Ids :
Invalid VLAN ID Packets: 0
Port VLAN ID:    40
VLAN Tag IDs:  None
Invalid VLAN ID Packets: 0
Port VLAN ID:    55
VLAN Tag IDs:   888
Invalid VLAN ID Packets: 0
Port VLAN ID:    91
VLAN Tag IDs:  None

“So looks ok. Thanks for your help”.

This entry is similar in theme to one of my previous posts about verifying your hdisk queue_depth settings with kdb. This time we want to check if an attribute for a Virtual FC (VFC) adapter has been modified and whether or not AIX has been restarted since the change. The attribute I’m interested in is num_cmd_elems. This value is often changed from its default settings, in AIX environments, to improve I/O performance on SAN attached storage.

From kdb you can identify the VFC adapters configured on an AIX system using the vfcs subcommand. Not only does this tell you what adapters you have, but it also identifies the VIOS each adapter is connected to and the corresponding vfchost adapter. Nice!

(0)> vfcs

NAME      ADDRESS             STATE   HOST    HOST_ADAP  OPENED NUM_ACTIVE

fcs0      0xF1000A00103D4000  0x0008  vio1    vfchost10  0x01   0x0000

fcs1      0xF1000A00103D6000  0x0008  vio2    vfchost10  0x01   0x0000

You can view the current (running) configuration of a VFC adapter using the kdb vfcs subcommand and the name of the VFC adapter, for example fcs1:

(0)> vfcs fcs1

adapter: fcs0   partition_num: 0x14     partition_name: aixlpar1

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

state: 0x8  opened: 0x1  flags: 0x1

host_name: vio1       host_device: vfchost10

maxCmds: FA     maxDMALength: 100000    SCSIid: 6F1E9A

portName: C050760399100168      nodeName: C050760399100168      nport_id:

6F1E9A

cmd_q.base_addr: 0xF1000A00102D0000     asyncq.base_addr: 0xF1000A0019761000

crq[0]: 0x8001000000000000      crq[1]: 0x8013960

ctl_head: 0x0   ctl_tail: 0x0

cmd_head: 0x0   cmd_tail: 0x0

pool_head: 0xF1000A001975BD08   pool_tail: 0xF1000A0019759D88

head_active: 0x0        tail_active: 0x0

cancel_head: 0xF1000A0019758090 cancel_tail: 0xF1000A0019758E58

num_cmds_allowed: 0xFA  num_cmds_active: 0x0    num_cmds_pending: 0x0

head_flush_q: 0x0       tail_flush_q: 0x0       cmd_flush_q: 0x0

no_elem: 0x0  no_dma: 0x0  no_sglist: 0x0  bad_mad: 0x0  qfull: 0x0

h_dropped: 0x0  link_down_cnt: 0x0  num_frames: 0x0  tx_lock: 0x0

rcv_lock: 0x0   ctl_lock: 0x0   lock: 0xFFFFFFFFFFFFFFFF

ctl_event: 0xFFFFFFFFFFFFFFFF   open_event: 0xFFFFFFFFFFFFFFFF

reset_cmd: 0x0  log_cmd: 0x0

link_cmd: 0x0   ctl_cmd->cmd_state: 0x0

inp_reqs: 0x12594       out_reqs: 0x1A7FD4      ctrl_reqs: 0x44

inp_bytes: 0x14973F90  out_bytes: 0xEC14EC18  npiv_logout_sent: 0x0

npiv_logout_rcvd: 0x0  act_hi_water: 0x10  pend_hi_water: 0x1

Using the output from this command we can determine the current (running) value for a number of VFC attributes, including num_cmd_elems.

So I start with an adapter with a num_cmd_elems value of 200. Both the lsattr command and kdb report 200 (C8 in hex) for num_cmd_elems.

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 200 Maximum Number of COMMAND Elements True

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

I change num_cmd_elems to 400 with chdev –P (remember, the –P flag only updates the AIX ODM, and not the running configuration of the device in the AIX kernel. You must either reboot for this change to take effect or offline & online the device).

# chdev -l fcs1 -a num_cmd_elems=400 -P

fcs1 changed

Now the lsattr command reports num_cmd_elems is set to 400 in the ODM.

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

But kdb still has num_cmd_elems set to C8 (200).

Are you backing up your AIX systems over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your backup server and clients on the same physical POWER system? You are most likely backing up over Virtual Ethernet to another AIX LPAR that is running your enterprise backup software, such as TSM or Legato Networker for example. And you probably have a dedicated private virtual network (and adapters) on both the clients and the server to handle the traffic for the nightly backups. The next question is, have you tuned your Virtual Ethernet adapters?

There are several tips available for tuning your Virtual Ethernet adapters for better performance on AIX. These tips include changing settings such as MTU size, TCP window sizes, enabling largesend, etc. I highly recommend the following blog posts from Anthony English and Nigel Griffiths on this subject:

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en

OK, so you got everything humming along nicely, your backups are flying over the virtual network (across the POWER hypervisor) and everybody is happy. After a period of time, you notice that the backups have started to “slow down”. They are taking longer to finish. The overall throughput of a backup drops. Some backups start in the evening around 9pm and are still running the next morning at 7am! In some cases you need to kill the backups or even reboot the backup server LPAR for things to return to normal.

“What is going on!?” You cry.

Well, there are a number of reasons why this could be happening. For example, your shared processor pool may be overwhelmed during the backup window. As we know, Virtual Ethernet adapters require CPU to do their work. If the CPU pool is running low on available CPU resources, this could contribute to the problem. And of course there could be tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or there may be issues with other pieces of the infrastructure, like network and SAN switches, adapters, etc. Perhaps there’s an issue with the applications and/or databases on the AIX systems? They often have their own mechanisms/tools for backing up their data to your enterprise backup software. Is the backup server sized to cope with the load i.e. CPU, memory, disk layout and I/O, sufficient tape drives, disk storage pools, etc?

So assuming you’ve checked all of the above (and more), then perhaps you’ve hit a problem that I encountered recently. In my particular case, backups “over the hypervisor” were slowing down, without any discernible cause. Initially the backups would be “very fast” but after a month or so, things would start to slow down dramatically.

We noticed that there were very large (and increasing) values for “Packets Dropped”, “Hypervisor Send/Receive Failures” and “No Resource Errors” in the output from the netstat –v command.                              

ETHERNET STATISTICS (ent1) :                                           

Device Type: Virtual I/O Ethernet Adapter (l-lan)                      

Hardware Address: 41:ba:13:e7:25:0b                                    

Elapsed Time: 42 days 4 hours 3 minutes 34 seconds                    

Transmit Statistics:       Receive Statistics:                         

--------------------       -------------------                         

Packets: 5978589961        Packets: 26139832411                        

Bytes: 779465989202        Bytes: 711051516630458                      

Interrupts: 0              Interrupts: 6804561727                      

Transmit Errors: 0         Receive Errors: 0                           

Packets Dropped: 0         Packets Dropped: 86012309               

...                                                                                             

Max Collision Errors: 0    No Resource Errors: 46113807           

...                                                                       

Hypervisor Send Failures: 0                                            

  Receiver Failures: 0                                                  

  Send Errors: 0                                                       

Hypervisor Receive Failures: 46113807                             

After some discussion with IBM AIX support, we discovered that would should increase some of the buffer sizes for our Virtual Ethernet adapter (the entX device). This would alleviate the no resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed that the Medium, Large and Huge buffers had all reached their maximum values in the past.                                                           

...

Receive Information                                                    

  Receive Buffers                                                      

    Buffer Type              Tiny    Small   Medium    Large     Huge  

    Min Buffers               512      512      128       24       24  

    Max Buffers              2048     2048      256       64       64  

    Allocated                 513      535      148       28       64  

    Registered                512      510      127       24       13  

    History                                                            

      Max Allocated           576      951      256       64       64  

      Lowest Registered       502      502       64       12       11  

...

The advice from IBM support was to increase these buffers using the chdev command (they also advised that we should reboot for the changes to take effect): 

# chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96 -a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P             

# shutdown -Fr

Since implementing this tuning change (to the adapter on the backup server), we have not had a repeat of the problem. We will continue to monitor the performance and I’ll be sure to let everyone know if we have further issues.