Topic
  • 5 replies
  • Latest Post - ‏2013-02-22T17:45:16Z by SystemAdmin
B2T8_Scott_Stewart
B2T8_Scott_Stewart
2 Posts

Pinned topic Performance Issues w/ IDS 10.00 Running On VMWare Guest

‏2012-05-23T22:03:19Z |
Hello!

We have been attempting to troubleshoot an performance issue with IDS instance hosted on VMWare (ESXi v4.1) guests. The guest OS is Red Hat Enterprise Linux, versions 4.9 and 5.8 (64-bit). During ETL operations, particularly exports from the database, we are seeing as much as a 2x increase in time required to complete the operation, as compared with the original physical hosts from which these instances were migrated. This occurs with both versions of Red Hat and two different versions of IDS ( 10.00 and 11.70.FC4EX). We've tried multiple memory and core configurations, RAID types, etc. to no avail. During test runs, monitoring at the IDS level shows a 3 sec latency per transaction in some cases, while the OS and VMWare report little if any I/O activity or latency.

Write performance from a strictly OS perspective using tools such as iozone and dd show performance in the 380 - 620 MB/sec range which is faster than the original physical hosts, yet IDS doesn't seem to be able to take advantage of the increase in capacity. We have cases open with VMWare and IBM, but thus far, nothing has budged the needle.

I know that there are quite a few variables involved in an virtualized environment and over the past couple of months, I feel like we've tried every combination in the book. Other VM-based DBMS environments, such as Oracle, do not seem to be experiencing these issues in our organization. Before we abandon the effort to move IDS into a virtualized configuration, I'd like to see if anyone else has experience with mid-sized (500 GB - 1 TB) Informix instances in VMWare.

Thanks in advance for any information that you can provide!
Scott
Updated on 2013-02-22T17:45:16Z at 2013-02-22T17:45:16Z by SystemAdmin
  • JJ@IBM
    JJ@IBM
    18 Posts

    Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest

    ‏2012-05-24T07:17:48Z  
    Hi,

    What device type are you using at the VM level? i.e. Files, Block or Raw

    Under 11.70.FC4, have you got direct I/O set?

    Are you using KAIO or AIO?

    When using dd, what block size did you use?

    How did you split up the device(s)? i.e. Under LVM or direct

    JJ

    "You are what you is"
  • andreasl
    andreasl
    372 Posts

    Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest

    ‏2012-05-24T09:18:02Z  
    Hi,

    "Write performance ... faster than the original physical hosts" sounds suspicious and could mean some sort of caching involved - something you should consider very carefully (with any (vendor's) database activity). A database write to disk needs to be 100% reliable, if data consistency is of any relevance this might be at least as important as speed.

    If Informix write performance isn't as fast this simply might mean it is smart enough (or configured well enough) to circumvent such caching and write directly to disk, so what you're sensing simply might be "physical disk speed" (whatever "physical disks" are here).

    JJ's questions regarding KAIO and/or direct i/o would shed some light on this (well, DIRECT_IO shouldn't be available yet in v10 afaik).
    There also is a conv=direct option to dd for writing through during a dd test.

    HTH,
    Andreas
  • B2T8_Scott_Stewart
    B2T8_Scott_Stewart
    2 Posts

    Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest

    ‏2012-05-24T13:49:35Z  
    • JJ@IBM
    • ‏2012-05-24T07:17:48Z
    Hi,

    What device type are you using at the VM level? i.e. Files, Block or Raw

    Under 11.70.FC4, have you got direct I/O set?

    Are you using KAIO or AIO?

    When using dd, what block size did you use?

    How did you split up the device(s)? i.e. Under LVM or direct

    JJ

    "You are what you is"
    Hi, JJ

    We've tried just about every type of structure for our extract output with no measurable difference in performance:

    • ext3 filesystem
    • ext3 filesystem with noatime parameter
    • Direct I/O
    • raw slice

    Disks were managed by Linux native LVM in all cases.

    As far the dd command is concerned, I've tried a number of different block sizes (2k/4k/8k) with the following syntax:

    time bash -c "dd if=/dev/zero of=/quickstore/ddtest bs=2k count=500000; sync; sync"

    The database in question uses a mix of 2k and 8k pagesizes (most are 2k).
    Thanks!
    Scott
  • BlinddogMS
    BlinddogMS
    10 Posts

    Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest

    ‏2012-06-06T14:15:54Z  
    Hi, JJ

    We've tried just about every type of structure for our extract output with no measurable difference in performance:

    • ext3 filesystem
    • ext3 filesystem with noatime parameter
    • Direct I/O
    • raw slice

    Disks were managed by Linux native LVM in all cases.

    As far the dd command is concerned, I've tried a number of different block sizes (2k/4k/8k) with the following syntax:

    time bash -c "dd if=/dev/zero of=/quickstore/ddtest bs=2k count=500000; sync; sync"

    The database in question uses a mix of 2k and 8k pagesizes (most are 2k).
    Thanks!
    Scott
    Hi.
    Have you seen this material, entitled:
    "Best Practices for Running Informix in VMWare vSphere Virtual Environments" ?
    Check it out, maybe it give you some great tips...

    https://www.ibm.com/developerworks/wikis/display/ifmxchatwithlab/Home

    Regards.
  • SystemAdmin
    SystemAdmin
    1143 Posts

    Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest

    ‏2013-02-22T17:45:16Z  
    Hi, JJ

    We've tried just about every type of structure for our extract output with no measurable difference in performance:

    • ext3 filesystem
    • ext3 filesystem with noatime parameter
    • Direct I/O
    • raw slice

    Disks were managed by Linux native LVM in all cases.

    As far the dd command is concerned, I've tried a number of different block sizes (2k/4k/8k) with the following syntax:

    time bash -c "dd if=/dev/zero of=/quickstore/ddtest bs=2k count=500000; sync; sync"

    The database in question uses a mix of 2k and 8k pagesizes (most are 2k).
    Thanks!
    Scott
    Scott,

    I'm curious if you were able to resolve the performance issue w/IDS 10 & 11 running on a VMware guest?

    We seem to be running into a similar issue, although it only occurs with our backups.

    Thanks,

    John