Topic
  • 3 replies
  • Latest Post - ‏2013-11-20T11:35:37Z by XF07_Harald_Dunkel
XF07_Harald_Dunkel
XF07_Harald_Dunkel
2 Posts

Pinned topic JFS2 horrible slow

‏2013-11-07T15:06:32Z |

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

 

  • Wouter Liefting
    Wouter Liefting
    129 Posts

    Re: JFS2 horrible slow

    ‏2013-11-11T14:19:21Z  

    Can you post the mount options (output of mount command or the /etc/filesystems stanza) and the attributes of the filesystem/LV/disk (lsattr -El command)?

    Also, if this is an LPAR, some LPAR settings like cpu entitlement?

    What does topas and/or nmon tell you while you are running the tarball extract? 100% CPU time, 100% disk time?

    Do you have any disk errors reported in errpt?

  • GarlandJoseph
    GarlandJoseph
    167 Posts

    Re: JFS2 horrible slow

    ‏2013-11-11T22:49:49Z  

    Can you post the mount options (output of mount command or the /etc/filesystems stanza) and the attributes of the filesystem/LV/disk (lsattr -El command)?

    Also, if this is an LPAR, some LPAR settings like cpu entitlement?

    What does topas and/or nmon tell you while you are running the tarball extract? 100% CPU time, 100% disk time?

    Do you have any disk errors reported in errpt?

    Keep it simple...I would look at i/o before cpu performance.   Is the joural  file on the same disk?  What exactly do you mean by "logging turned on"  What does lsvg -p and lsvg -l show?  You should use iostat to drill down to the i/o statistics.   Set up a test program and copy i/o statistics before and after logging enabled.

    Updated on 2013-11-11T22:50:31Z at 2013-11-11T22:50:31Z by GarlandJoseph
  • XF07_Harald_Dunkel
    XF07_Harald_Dunkel
    2 Posts

    Re: JFS2 horrible slow

    ‏2013-11-20T11:35:37Z  

    Keep it simple...I would look at i/o before cpu performance.   Is the joural  file on the same disk?  What exactly do you mean by "logging turned on"  What does lsvg -p and lsvg -l show?  You should use iostat to drill down to the i/o statistics.   Set up a test program and copy i/o statistics before and after logging enabled.

    Sorry for the delay. My test host was not available for tests.

    To answer the questions:

    - the journal is on the same disk (hdisk1), using a dedicated partition for logging.

    - "logging turned off" means "log = NULL" in the appropriate record in /etc/filesystems.

    - lsvg:

    # lsvg -p localvg
    localvg:
    PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
    hdisk1            active            558         37          00..00..00..00..37
    # lsvg -l localvg
    localvg:
    LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
    loglv00             jfs2log    1       1       1    open/syncd    N/A
    fslv01              jfs2       512     512     1    open/syncd    /export
    fslv02              jfs2       8       8       1    open/syncd    /sample
     


    - iostat (1sec interval) showed me this for logging disabled:

    Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
    :
    :
    hdisk1          98.0     976.0     167.0          0       976
    hdisk1         100.0     1108.0     169.0          0      1108
    hdisk1          99.0     1168.0     171.0          0      1168
    hdisk1         100.0     2232.0     194.0          0      2232
    hdisk1         100.0     1384.0     178.0          0      1384
    hdisk1         100.0     1136.0     166.0          0      1136
    hdisk1          99.0     1680.0     183.0          0      1680
    hdisk1          98.0     1288.0     172.0          0      1288
    hdisk1          99.0     1408.0     198.0          0      1408
    hdisk1          98.0     1188.0     167.0          0      1188
    hdisk1         100.0     1264.0     168.0          0      1264
    hdisk1         100.0     1460.0     173.0          0      1460
    hdisk1          99.0     976.0     165.0          0       976
    hdisk1         100.0     920.0     167.0          0       920
    hdisk1         100.0     1180.0     170.0          0      1180
    hdisk1          92.0     2368.0     194.0          0      2368
    hdisk1         108.0     1716.0     180.0          0      1716
    hdisk1         100.0     2072.0     191.0          0      2072
    :
    :


    With logging enabled I got this:

    Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
    :
    :
    hdisk1         100.0     828.0      96.0          0       828
    hdisk1         100.0     720.0     104.0          0       720
    hdisk1         100.0     848.0      89.0          0       848
    hdisk1          98.0     576.0      94.0          0       576
    hdisk1          99.0     852.0      90.0          0       852
    hdisk1         100.0     1292.0      98.0          0      1292
    hdisk1          99.0     1068.0     100.0          0      1068
    hdisk1         100.0     988.0     101.0          0       988
    hdisk1          99.0     1224.0      99.0          0      1224
    hdisk1          99.0     1228.0      98.0          0      1228
    hdisk1          99.0     720.0     100.0          0       720
    hdisk1          99.0     636.0      89.0          0       636
    hdisk1          99.0     1568.0     142.0          0      1568
    hdisk1          97.0     916.0     102.0          0       916
    hdisk1         100.0     708.0      99.0          0       708
    :
    :

    In both cases I extracted the Linux source tar file, i.e. about 40000 files in about 2500 directories. Without logging this took "just" about 2 minutes. With logging enabled it took more than 25 minutes (including sync).

     

    Updated on 2013-11-20T12:37:27Z at 2013-11-20T12:37:27Z by XF07_Harald_Dunkel