Topic
2 replies Latest Post - ‏2012-05-16T17:26:46Z by Steve_ATS
UNIX-OS
UNIX-OS
4 Posts
ACCEPTED ANSWER

Pinned topic Some Random time disk IO response time is too high (max serv > 7 S)

‏2012-05-06T02:57:58Z |
Hi,

We have application using binary data database. Database files stored in Aix 6.1 JFS2 file system, most of the time application running find. LUNs service time and %tm_act looks good.
Some random time disk max service time goes too high (>7 seconds) and disk buys goes upto 99% it last about a minutes then coming to normal status. From iostat didn’t see large number of IOs going to these LUNs. File system spread across all the LUNs using LVM max spread option. During this time IO stops on all the LUNs on this file system

Is this service time issue looks like storage issue? Or application can cause this kind of problem?
Is Inode locking cause this kind of issue?


Disks:                     xfers                                read                                write                                  queue                    time -------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- --------- %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv hdisk38 12.4 306.5K  52.1  75.4K 231.1K  13.3   7.9    0.3   29.4     0    0  38.8   0.7    0.4   16.7     0    0   0.0    0.0    0.1    0.0   0.0   0.0  12:21:03 hdisk38         12.7 308.9K  52.4  83.2K 225.7K  14.4   7.7    0.3   38.5     0    0  38.0   0.7    0.4   14.5     0    0   0.0    0.0    0.1    0.0   0.0   0.0  12:21:33 hdisk38         65.3 140.0K  24.1  40.6K  99.4K   7.0 102.8    0.3   10.2S    0    0  17.1  58.4    0.4   10.1S    0    0   0.0    0.0    0.1    0.0   2.0   0.0  12:22:03 hdisk38         23.3   3.3K   0.6   1.2K   2.0K   0.2   8.8    6.0   11.7     0    0   0.3   0.6    0.5    0.8     0    0   3.2S   0.0   55.1S   0.0   0.0   0.0  12:23:03 hdisk38          6.5 166.4K  27.3  50.4K 116.1K   8.1   7.4    0.3   22.5     0    0  19.2   0.7    0.4   12.5     0    0   0.0    0.0    0.1    0.0   0.0   0.0  12:23:33 hdisk38         17.3 376.5K  64.2 116.8K 259.7K  20.2   7.8    0.3   35.6     0    0  44.0   0.8    0.4   14.4     0    0   0.0    0.0    0.1    0.0   0.0   0.0  12:24:03 hdisk38         17.6 423.2K  71.7 116.1K 307.1K  20.0   7.9    0.3   39.1     0    0  51.6   0.8    0.4   14.2     0    0   0.0    0.0    0.1    0.0   0.0   0.0  12:24:33
Updated on 2012-05-16T17:26:46Z at 2012-05-16T17:26:46Z by Steve_ATS
  • Steve_ATS
    Steve_ATS
    38 Posts
    ACCEPTED ANSWER

    Re: Some Random time disk IO response time is too high (max serv > 7 S)

    ‏2012-05-16T17:25:31Z  in response to UNIX-OS
    I would run iostat at a higher frequency, even down to 1 second and then see if these spikes occur. Then I'd run some 10 second filemon traces to see if they report the same thing at the filesystem and/or physical layer. filemon in later AIX 6.1 levels has the ability to trace processes (if you can narrow it down to that) or generate a sequential report of all IOs. See abbreviated, detailed and collated modes. They're in AIX 6.1, but you may have to look at the AIX 7.1 pubs (run filemon -h to see the usage).

    If they show up in filemon, that tells you it isn't some iostat bug.
    • Steve_ATS
      Steve_ATS
      38 Posts
      ACCEPTED ANSWER

      Re: Some Random time disk IO response time is too high (max serv > 7 S)

      ‏2012-05-16T17:26:46Z  in response to Steve_ATS
      Also, check your error report to see if any hdisk or subsystem errors are being generated.