Topic
  • 4 replies
  • Latest Post - ‏2013-04-15T17:27:47Z by OneSkyWalker
OneSkyWalker
OneSkyWalker
4 Posts

Pinned topic ndisk64 performance puzzlement

‏2013-04-05T16:06:20Z |
I have an LPAR configured uncapped with two virtual CPUs running in Shared-SMT-4 mode on a 9179-MHB. I have an 8 GB file created with 'dd if=/dev/zero of=benchtest.out bs=1M count=8192'. The hdisk underlying benchtest.out has queue_depth = 64.

When I run:

ndisk64 -C -R -r 70 -q -t 90 -b 64K -M 64 -f benchtest.out

I expect to see 64 active I/O requests at any given time to the hdisk underlying benchtest.out. That is, with zero wait time, each child process should keep one I/O in flight all the time (except for the brief period between the end of one I/O request and the start of another). But I see only 2 active I/O requests typically. Please see the iostat.lDRT.out file in the attached ndisk64.test.tar archive. Focussing on the interval ending at 17:48:17 in the iostat.lDRT.out file, the avg sqsz of 2 (actually a sample at the end of the interval, not an average in the interval) is confirmed by the duration of all I/O operations = rps*avgrserv+wps*avgwserv = 541.4*4.7+246.5*1.4 = 2889.68 milliseconds per second in the interval - that is, an average of 2.88968 I/O operations are active at any given instant in the interval.

CPU resource isn't a problem. Please see the vmstat.Iwt.out file in the attached ndisk64.test.tar archive.

Lock contention should not be a problem, since the ndisk64 -C flag is specified.

The ndisk64 command is spawning 64 children. Please see the attached proctree.out file in the attached ndisk64.test.tar archive.

So why can't the 64 children drive a higher I/O rate?
Updated on 2013-04-06T05:17:46Z at 2013-04-06T05:17:46Z by OneSkyWalker
  • nagger
    nagger
    1640 Posts

    Re: ndisk64 performance puzzlement

    ‏2013-04-05T17:10:41Z  
    Interesting question.
    There is no details of your actual disks in the question.

    I retried this test against my LUN on a V7000 over vSCSI - single disk as AIX sees it.

    I get 1000+ tps on iostat output:
    
    Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn hdisk2           0.0       0.0       0.0          0         0 hdisk0          98.0     89672.1     1426.5      27776     12016 hdisk3           0.0       0.0       0.0          0         0 hdisk1           0.0       0.0       0.0          0         0 usbms0           0.0       0.0       0.0          0         0
    


    Although you can queue up 64 I/O requests with this queue depth and 64 processes it clearly can't do in practice.
    A single real physical disk can do only about 200 I/O per second.
    Clever disk subsystems can cache and read RAID4 stripes etc. and so go faster but they can fill the write cache.

    Also the -q option is flushing disk updates to disk which can slow things down.

    Just because the ndisk command is doing 64KB reads and writes the disk I/O can be different.
    My ndisk is reporting:
    
    $ ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Command: ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Synchronous Disk test (regular read/write) No. of processes = 64 I/O type         = Random Block size       = 65536 Read-WriteRatio: 70:30 = read mostly Sync type: none  = just close the file File open() mode = Concurrent I/O mode - O_CIO Number of files  = 1 File size to be determined from the file(s) (raw devices will fail) Run time         = 20 seconds Snooze %         = 0 percent ----> Running test with block Size=65536 (64KB) ................................................................ Determining the size of file:benchtest.out File benchtest.out size 8589934592 bytes   Proc - <-----Disk IO----> | <-----Throughput------> RunTime Num -     TOTAL   IO/sec |    MB/sec       KB/sec  Seconds 1 -       320     15.9 |      1.00      1018.93  20.10 2 -       334     16.7 |      1.04      1068.36  20.01 3 -       311     15.5 |      0.97       989.77  20.11 4 -       299     14.9 |      0.93       955.71  20.02 ... 60 -       313     15.6 |      0.97       997.90  20.07 61 -       314     15.7 |      0.98      1002.99  20.04 62 -       312     15.5 |      0.97       994.37  20.08 63 -       333     16.6 |      1.04      1065.50  20.00 64 -       296     14.8 |      0.92       944.20  20.06 TOTALS     20427   1018.5 |     63.66     65183.41   With -q that drops to Num -     TOTAL   IO/sec |    MB/sec       KB/sec  Seconds ... TOTALS     11551    577.0 |     36.07     36930.96
    


    About half.

    I am using nstress version 7.5 - the version is output in the help info, when you use no options.

    Does this help?
    thanks, Nigel Griffiths
  • OneSkyWalker
    OneSkyWalker
    4 Posts

    Re: ndisk64 performance puzzlement

    ‏2013-04-06T05:15:08Z  
    After posting this question, I discovered that the following command behaves as expected:

    ndisk64 -C -R -r 100 -q -t 90 -b 64K -M 64 -f benchtest.out

    That is, if all I/O requests are reads, I see 63 active I/O requests at any given time (queue avg sqsz) to the hdisk underlying benchtest.out.

    The issue seems to be that a write I/O request to a file opened with the CIO option blocks the start of any other I/O requests (read or write) until the write I/O request completes. That is not the behavior a reader of the Improving Database Performance With AIX Concurrent I/O white paper (downloadable from http://www.ibm.com/systems/power/software/aix/whitepapers/db_perf_aix.html) would expect. The white paper says, "Under Concurrent I/O, multiple threads can simultaneously perform reads and writes on a shared file."

    I have opened PMR 66591,49R,000 to see what the IBM Support Center has to say about the way Concurrent I/O is handling write I/O requests on AIX V6.1 TL8 SP1.

    Please note that according to the notice posted at https://www.ibm.com/developerworks/forums/index.html it might not be possible to update this forum entry again until Feb 15, so stay tuned.
  • OneSkyWalker
    OneSkyWalker
    4 Posts

    Re: ndisk64 performance puzzlement

    ‏2013-04-06T05:17:46Z  
    Oops, stay tuned until Apr 15, not Feb 15.
  • OneSkyWalker
    OneSkyWalker
    4 Posts

    Re: ndisk64 performance puzzlement

    ‏2013-04-15T17:27:47Z  

    I  opened PMR 66591,49R,000 and the IBM Support Center explained that the -q option on ndisk64 causes the file to be locked for the duration of each write I/O request even if the file is opened with the CIO option (ndisk64 -C option specified and/or filesystem mounted with cio option).

    To mimic behavior of Oracle (and many other applications which use CIO), it is best to omit the -q option when specifying -C and/or using CIO.