• 4 replies
  • Latest Post - ‏2013-04-15T17:27:47Z by OneSkyWalker
6 Posts

Pinned topic ndisk64 performance puzzlement

‏2013-04-05T16:06:20Z |
I have an LPAR configured uncapped with two virtual CPUs running in Shared-SMT-4 mode on a 9179-MHB. I have an 8 GB file created with 'dd if=/dev/zero of=benchtest.out bs=1M count=8192'. The hdisk underlying benchtest.out has queue_depth = 64.

When I run:

ndisk64 -C -R -r 70 -q -t 90 -b 64K -M 64 -f benchtest.out

I expect to see 64 active I/O requests at any given time to the hdisk underlying benchtest.out. That is, with zero wait time, each child process should keep one I/O in flight all the time (except for the brief period between the end of one I/O request and the start of another). But I see only 2 active I/O requests typically. Please see the iostat.lDRT.out file in the attached ndisk64.test.tar archive. Focussing on the interval ending at 17:48:17 in the iostat.lDRT.out file, the avg sqsz of 2 (actually a sample at the end of the interval, not an average in the interval) is confirmed by the duration of all I/O operations = rps*avgrserv+wps*avgwserv = 541.4*4.7+246.5*1.4 = 2889.68 milliseconds per second in the interval - that is, an average of 2.88968 I/O operations are active at any given instant in the interval.

CPU resource isn't a problem. Please see the vmstat.Iwt.out file in the attached ndisk64.test.tar archive.

Lock contention should not be a problem, since the ndisk64 -C flag is specified.

The ndisk64 command is spawning 64 children. Please see the attached proctree.out file in the attached ndisk64.test.tar archive.

So why can't the 64 children drive a higher I/O rate?
Updated on 2013-04-06T05:17:46Z at 2013-04-06T05:17:46Z by OneSkyWalker
  • nagger
    1738 Posts

    Re: ndisk64 performance puzzlement

    Interesting question.
    There is no details of your actual disks in the question.

    I retried this test against my LUN on a V7000 over vSCSI - single disk as AIX sees it.

    I get 1000+ tps on iostat output:
    Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn hdisk2           0.0       0.0       0.0          0         0 hdisk0          98.0     89672.1     1426.5      27776     12016 hdisk3           0.0       0.0       0.0          0         0 hdisk1           0.0       0.0       0.0          0         0 usbms0           0.0       0.0       0.0          0         0

    Although you can queue up 64 I/O requests with this queue depth and 64 processes it clearly can't do in practice.
    A single real physical disk can do only about 200 I/O per second.
    Clever disk subsystems can cache and read RAID4 stripes etc. and so go faster but they can fill the write cache.

    Also the -q option is flushing disk updates to disk which can slow things down.

    Just because the ndisk command is doing 64KB reads and writes the disk I/O can be different.
    My ndisk is reporting:
    $ ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Command: ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Synchronous Disk test (regular read/write) No. of processes = 64 I/O type         = Random Block size       = 65536 Read-WriteRatio: 70:30 = read mostly Sync type: none  = just close the file File open() mode = Concurrent I/O mode - O_CIO Number of files  = 1 File size to be determined from the file(s) (raw devices will fail) Run time         = 20 seconds Snooze %         = 0 percent ----> Running test with block Size=65536 (64KB) ................................................................ Determining the size of file:benchtest.out File benchtest.out size 8589934592 bytes   Proc - <-----Disk IO----> | <-----Throughput------> RunTime Num -     TOTAL   IO/sec |    MB/sec       KB/sec  Seconds 1 -       320     15.9 |      1.00      1018.93  20.10 2 -       334     16.7 |      1.04      1068.36  20.01 3 -       311     15.5 |      0.97       989.77  20.11 4 -       299     14.9 |      0.93       955.71  20.02 ... 60 -       313     15.6 |      0.97       997.90  20.07 61 -       314     15.7 |      0.98      1002.99  20.04 62 -       312     15.5 |      0.97       994.37  20.08 63 -       333     16.6 |      1.04      1065.50  20.00 64 -       296     14.8 |      0.92       944.20  20.06 TOTALS     20427   1018.5 |     63.66     65183.41   With -q that drops to Num -     TOTAL   IO/sec |    MB/sec       KB/sec  Seconds ... TOTALS     11551    577.0 |     36.07     36930.96

    About half.

    I am using nstress version 7.5 - the version is output in the help info, when you use no options.

    Does this help?
    thanks, Nigel Griffiths
  • OneSkyWalker
    6 Posts

    Re: ndisk64 performance puzzlement

    After posting this question, I discovered that the following command behaves as expected:

    ndisk64 -C -R -r 100 -q -t 90 -b 64K -M 64 -f benchtest.out

    That is, if all I/O requests are reads, I see 63 active I/O requests at any given time (queue avg sqsz) to the hdisk underlying benchtest.out.

    The issue seems to be that a write I/O request to a file opened with the CIO option blocks the start of any other I/O requests (read or write) until the write I/O request completes. That is not the behavior a reader of the Improving Database Performance With AIX Concurrent I/O white paper (downloadable from would expect. The white paper says, "Under Concurrent I/O, multiple threads can simultaneously perform reads and writes on a shared file."

    I have opened PMR 66591,49R,000 to see what the IBM Support Center has to say about the way Concurrent I/O is handling write I/O requests on AIX V6.1 TL8 SP1.

    Please note that according to the notice posted at it might not be possible to update this forum entry again until Feb 15, so stay tuned.
  • OneSkyWalker
    6 Posts

    Re: ndisk64 performance puzzlement

    Oops, stay tuned until Apr 15, not Feb 15.
  • OneSkyWalker
    6 Posts

    Re: ndisk64 performance puzzlement


    I  opened PMR 66591,49R,000 and the IBM Support Center explained that the -q option on ndisk64 causes the file to be locked for the duration of each write I/O request even if the file is opened with the CIO option (ndisk64 -C option specified and/or filesystem mounted with cio option).

    To mimic behavior of Oracle (and many other applications which use CIO), it is best to omit the -q option when specifying -C and/or using CIO.