We have been attempting to troubleshoot an performance issue with IDS instance hosted on VMWare (ESXi v4.1) guests. The guest OS is Red Hat Enterprise Linux, versions 4.9 and 5.8 (64-bit). During ETL operations, particularly exports from the database, we are seeing as much as a 2x increase in time required to complete the operation, as compared with the original physical hosts from which these instances were migrated. This occurs with both versions of Red Hat and two different versions of IDS ( 10.00 and 11.70.FC4EX). We've tried multiple memory and core configurations, RAID types, etc. to no avail. During test runs, monitoring at the IDS level shows a 3 sec latency per transaction in some cases, while the OS and VMWare report little if any I/O activity or latency.
Write performance from a strictly OS perspective using tools such as iozone and dd show performance in the 380 - 620 MB/sec range which is faster than the original physical hosts, yet IDS doesn't seem to be able to take advantage of the increase in capacity. We have cases open with VMWare and IBM, but thus far, nothing has budged the needle.
I know that there are quite a few variables involved in an virtualized environment and over the past couple of months, I feel like we've tried every combination in the book. Other VM-based DBMS environments, such as Oracle, do not seem to be experiencing these issues in our organization. Before we abandon the effort to move IDS into a virtualized configuration, I'd like to see if anyone else has experience with mid-sized (500 GB - 1 TB) Informix instances in VMWare.
Thanks in advance for any information that you can provide!
This topic has been locked.
5 replies Latest Post - 2013-02-22T17:45:16Z by SystemAdmin
Pinned topic Performance Issues w/ IDS 10.00 Running On VMWare Guest
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2013-02-22T17:45:16Z at 2013-02-22T17:45:16Z by SystemAdmin
JJ@IBM 2000004YBW18 PostsACCEPTED ANSWER
Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest2012-05-24T07:17:48Z in response to B2T8_Scott_StewartHi,
What device type are you using at the VM level? i.e. Files, Block or Raw
Under 11.70.FC4, have you got direct I/O set?
Are you using KAIO or AIO?
When using dd, what block size did you use?
How did you split up the device(s)? i.e. Under LVM or direct
"You are what you is"
B2T8_Scott_Stewart 270003B2T82 PostsACCEPTED ANSWER
Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest2012-05-24T13:49:35Z in response to JJ@IBMHi, JJ
We've tried just about every type of structure for our extract output with no measurable difference in performance:
- ext3 filesystem
- ext3 filesystem with noatime parameter
- Direct I/O
- raw slice
Disks were managed by Linux native LVM in all cases.
As far the dd command is concerned, I've tried a number of different block sizes (2k/4k/8k) with the following syntax:
time bash -c "dd if=/dev/zero of=/quickstore/ddtest bs=2k count=500000; sync; sync"
The database in question uses a mix of 2k and 8k pagesizes (most are 2k).
BlinddogMS 110000BDJS10 PostsACCEPTED ANSWER
Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest2012-06-06T14:15:54Z in response to B2T8_Scott_StewartHi.
Have you seen this material, entitled:
"Best Practices for Running Informix in VMWare vSphere Virtual Environments" ?
Check it out, maybe it give you some great tips...
SystemAdmin 110000D4XK1143 PostsACCEPTED ANSWER
Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest2013-02-22T17:45:16Z in response to B2T8_Scott_StewartScott,
I'm curious if you were able to resolve the performance issue w/IDS 10 & 11 running on a VMware guest?
We seem to be running into a similar issue, although it only occurs with our backups.
andreasl 100000CHVU369 PostsACCEPTED ANSWER
Re: Performance Issues w/ IDS 10.00 Running On VMWare Guest2012-05-24T09:18:02Z in response to B2T8_Scott_StewartHi,
"Write performance ... faster than the original physical hosts" sounds suspicious and could mean some sort of caching involved - something you should consider very carefully (with any (vendor's) database activity). A database write to disk needs to be 100% reliable, if data consistency is of any relevance this might be at least as important as speed.
If Informix write performance isn't as fast this simply might mean it is smart enough (or configured well enough) to circumvent such caching and write directly to disk, so what you're sensing simply might be "physical disk speed" (whatever "physical disks" are here).
JJ's questions regarding KAIO and/or direct i/o would shed some light on this (well, DIRECT_IO shouldn't be available yet in v10 afaik).
There also is a conv=direct option to dd for writing through during a dd test.