Pinned topic ndisk64 performance puzzlement
When I run:
ndisk64 -C -R -r 70 -q -t 90 -b 64K -M 64 -f benchtest.out
I expect to see 64 active I/O requests at any given time to the hdisk underlying benchtest.out. That is, with zero wait time, each child process should keep one I/O in flight all the time (except for the brief period between the end of one I/O request and the start of another). But I see only 2 active I/O requests typically. Please see the iostat.lDRT.out file in the attached ndisk64.test.tar archive. Focussing on the interval ending at 17:48:17 in the iostat.lDRT.out file, the avg sqsz of 2 (actually a sample at the end of the interval, not an average in the interval) is confirmed by the duration of all I/O operations = rps*avgrserv+wps*avgwserv = 541.4*4.7+246.5*1.4 = 2889.68 milliseconds per second in the interval - that is, an average of 2.88968 I/O operations are active at any given instant in the interval.
CPU resource isn't a problem. Please see the vmstat.Iwt.out file in the attached ndisk64.test.tar archive.
Lock contention should not be a problem, since the ndisk64 -C flag is specified.
The ndisk64 command is spawning 64 children. Please see the attached proctree.out file in the attached ndisk64.test.tar archive.
So why can't the 64 children drive a higher I/O rate?
nagger 100000MRSJ1631 PostsACCEPTED ANSWER
Re: ndisk64 performance puzzlement2013-04-05T17:10:41Z in response to OneSkyWalkerInteresting question.
There is no details of your actual disks in the question.
I retried this test against my LUN on a V7000 over vSCSI - single disk as AIX sees it.
I get 1000+ tps on iostat output:
Disks: % tm_act Kbps tps Kb_read Kb_wrtn hdisk2 0.0 0.0 0.0 0 0 hdisk0 98.0 89672.1 1426.5 27776 12016 hdisk3 0.0 0.0 0.0 0 0 hdisk1 0.0 0.0 0.0 0 0 usbms0 0.0 0.0 0.0 0 0
Although you can queue up 64 I/O requests with this queue depth and 64 processes it clearly can't do in practice.
A single real physical disk can do only about 200 I/O per second.
Clever disk subsystems can cache and read RAID4 stripes etc. and so go faster but they can fill the write cache.
Also the -q option is flushing disk updates to disk which can slow things down.
Just because the ndisk command is doing 64KB reads and writes the disk I/O can be different.
My ndisk is reporting:
$ ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Command: ./ndisk64_75 -C -R -r 70 -t20 -b 64K -M 64 -f benchtest.out Synchronous Disk test (regular read/write) No. of processes = 64 I/O type = Random Block size = 65536 Read-WriteRatio: 70:30 = read mostly Sync type: none = just close the file File open() mode = Concurrent I/O mode - O_CIO Number of files = 1 File size to be determined from the file(s) (raw devices will fail) Run time = 20 seconds Snooze % = 0 percent ----> Running test with block Size=65536 (64KB) ................................................................ Determining the size of file:benchtest.out File benchtest.out size 8589934592 bytes Proc - <-----Disk IO----> | <-----Throughput------> RunTime Num - TOTAL IO/sec | MB/sec KB/sec Seconds 1 - 320 15.9 | 1.00 1018.93 20.10 2 - 334 16.7 | 1.04 1068.36 20.01 3 - 311 15.5 | 0.97 989.77 20.11 4 - 299 14.9 | 0.93 955.71 20.02 ... 60 - 313 15.6 | 0.97 997.90 20.07 61 - 314 15.7 | 0.98 1002.99 20.04 62 - 312 15.5 | 0.97 994.37 20.08 63 - 333 16.6 | 1.04 1065.50 20.00 64 - 296 14.8 | 0.92 944.20 20.06 TOTALS 20427 1018.5 | 63.66 65183.41 With -q that drops to Num - TOTAL IO/sec | MB/sec KB/sec Seconds ... TOTALS 11551 577.0 | 36.07 36930.96
I am using nstress version 7.5 - the version is output in the help info, when you use no options.
Does this help?
thanks, Nigel Griffiths
Re: ndisk64 performance puzzlement2013-04-06T05:15:08Z in response to OneSkyWalkerAfter posting this question, I discovered that the following command behaves as expected:
ndisk64 -C -R -r 100 -q -t 90 -b 64K -M 64 -f benchtest.out
That is, if all I/O requests are reads, I see 63 active I/O requests at any given time (queue avg sqsz) to the hdisk underlying benchtest.out.
The issue seems to be that a write I/O request to a file opened with the CIO option blocks the start of any other I/O requests (read or write) until the write I/O request completes. That is not the behavior a reader of the Improving Database Performance With AIX Concurrent I/O white paper (downloadable from http://www.ibm.com/systems/power/software/aix/whitepapers/db_perf_aix.html) would expect. The white paper says, "Under Concurrent I/O, multiple threads can simultaneously perform reads and writes on a shared file."
I have opened PMR 66591,49R,000 to see what the IBM Support Center has to say about the way Concurrent I/O is handling write I/O requests on AIX V6.1 TL8 SP1.
Please note that according to the notice posted at https://www.ibm.com/developerworks/forums/index.html it might not be possible to update this forum entry again until Feb 15, so stay tuned.
Re: ndisk64 performance puzzlement2013-04-15T17:27:47Z in response to OneSkyWalker
I opened PMR 66591,49R,000 and the IBM Support Center explained that the -q option on ndisk64 causes the file to be locked for the duration of each write I/O request even if the file is opened with the CIO option (ndisk64 -C option specified and/or filesystem mounted with cio option).
To mimic behavior of Oracle (and many other applications which use CIO), it is best to omit the -q option when specifying -C and/or using CIO.