IDS license revenue grew by double digits in the first half of 2006
Administrating and Developing with Informix
From archive: August 2006 X
The Q2 numbers are in and Informix Dynamic Server experienced another large increase in revenue to build on the growth of Q1. As usual the details are not released publicly but I am officially allowed to say, and I quote..
IDS license revenue grew by double digits in the first half of 2006
Following the recent IBM disclosure concerning the IDS security fixes, David Litchfield of NGSS announced the vulnerabilities at Black Hat 2006 this week.
At the risk of sounding biased, here are a couple of quotes from the SearchSecurity.com write-up that to me illustrate the differences in the approach of IBM and certain other database companies when it comes to fixing security vulnerabilities:
"The good news, Litchfield said, is that IBM has already addressed the flaws in versions 7.31.xD9, 9.40.xC8, or 10.00.xC4. Unlike his often strained exchanges with Oracle, Litchfield said IBM has been responsive."
"While the Informix problems have been addressed, Litchfield said they point to a larger issue: Database flaws are pervasive throughout the industry. He again used Oracle as an example, noting how the database giant has fixed more than 100 serious flaws but has yet to address another 400-plus vulnerabilities, which is the estimated number of unpatched flaws according to his work and that of other researchers."
A key lesson that NGSS often repeats is to make use of the existing security features available in your data server and operating system to restrict unauthorized access and hence minimize the opportunity for malicious use. For IDS look out for Jonathan Leffler's Paranoid DBA presentation and note the new Security chapter in the IDS 10.0 Administrators Guide.[Read More]
There are some useful "getting started" articles about installing IDS on Windows at Jean Georges Perrin's website: IDS on Windows Series. Every step of the installation includes a picture so it's worth a look for anyone wanting a little more graphic detail than the IDS Installation Guide for Windows. Note these articles have been linked from IIUG website since June.[Read More]
The classes and presentations are being finalised for the IBM Information on Demand conference at Anaheim on October 15-20. Here's the flash:
The main Informix related topics can be viewed here: Data Servers - Informix, also scheduled are some pre-conference technical education courses on Sunday 14th, including a one day Informix Dynamic Server Problem Determination Class. I'm scheduled to talk about "Rapid Development using Java Server Pages (JSP) and IDS" on the Wednesday, though I hope I get to attend the rest of the conference. Entertainment will be provided by Wayne Brady and Gladys Knight! among others.
Here's a link to the conference blogs.[Read More]
Until Informix Dynamic Server 10.0 the number of MSC (or Miscellaneous) Virtual Processors was limited to one. With IDS 10.0 multiple MSC VPs can be configured using the VPCLASS onconfig parameter (e.g. VPCLASS msc,num=2), or added dynamically using onmode -p 1 msc.
What does that mean and why would I want to use it?
The MSC VP is a type of AIO VP, executing operating system calls on behalf of other threads. While the AIO VP mainly takes care of filesystem I/O calls the MSC VP executes some network and authentication calls like getpwnam(), gethostbyname(), gethostbyaddr() etc. (It also maintains some statistics like VP CPU usage.) The MSC VP gets used at client connection time for the operating system part of user validation, and in most cases one is sufficient.
Get to the point
In some cases, for example when a connection pool creates a large number of client connections at once, and/or when a PAM (Pluggable Authentication Module) implementation causes the operating system part of user authentication to take longer, the MSC VP can either be very busy serving the new connections or waiting for the OS to respond. When this happens new client connections can take longer to service and users will see a longer delay when connecting for the first time. Situations like this might benefit from having an extra MSC VP. Running onstats like onstat -g iov, onstat -g glo and onstat -g stk <tid> can help show what the MSC VP is doing but keep in mind there are other possible causes when new connections experience delays (such as operating system and network problems) so some troubleshooting (and tech support assistance) would generally be required, but at least dynamically adding a VP is an easy test to make if you suspect this might help.[Read More]
Suppose you find your IDS instance hung? There are users waiting and you need to run some diagnostics quickly.. what information do you collect? Hopefully you'll never be in that situation thanks to IDS's legendary uptime but it's good to be prepared..
A new Technote is available which includes a (UNIX) script to gather information that will be useful to Technical Support: What to collect if a IBM Informix IDS server gets hung. Give it a try and see what you think. You can enter feedback directly, or put it here and I'll forward it to the author.[Read More]
gbowerman 100000B5T0 2,249 Views
The Resolution Team is dead, long live the Resolution Engineering and Development (or RED) Team. Apparantly there was another completely different team with a similar name somewhere in the IBMosphere, so to avoid confusion we underwent nomenclature metamorphosis. The new name is slightly more descriptive of what RT, I mean the RED team, actually does.[Read More]
Following IBM's Informix security announcement on July 31st and David Litchfield's corresponding Informix vulnerability disclosures at the Black Hat conference, news reports have been slowly popping up. Some, like this: searchsecurity.techtarget.com have been fairly accurate. Others have reported the vulnerabilities but omitted to mention that IBM has actually fixed them all, like this one: upi.com - I tried submitting a comment to that page pointing this out last week, but the page still says "No Comments Exist".
Yesterday the SANS Institute Internet Storm Center blog Handler's Diary covered the story. The article concludes with:
So given the facts above, are you asking the right questions of your vendors? How certain are you thatyour favorite software vendor is writing secure code? Do you have the ability to change software packages if you find that a product has been found to have basic programming errors? And can your organization afford to let known holes live unpatched for 1.5 years?
These are all good questions to ask, but give a misleading impression of how IBM handled the situation. Here's how IBM worked with NGSS from an inside perspective...
In 2004 David Litchfield at NGSS contacted IBM to advise that he was planning to look at Informix Dynamic Server and asked if we could provide him with some software. This enquiry was welcomed by IBM as it was known that NGSS is responsible in giving vendors time to fix problems discovered before announcing them, and gave us the opportunity to discover and resolve problems in a controlled manner, the end result being a much more secure product, so a recent copy of IDS was sent to NGSS for testing.
We were aware of press reports about Oracle having yet to address hundreds of security problems discoverd by NGSS so set about putting together an international team of advanced support, development engineers and IDS architects to address any reported problems as quickly as possible as well as look for unreported problems in these areas of the code. Beginning in January 2005 problem reports started coming in and around 20 individual problems were reported to us in total and each one was immediately assigned an owner to reproduce the problem, log a defect and start fixing it. Each fix was peer reviewed, development reviewed and put through QA. Some problems were determined to be duplicates of known problems which were already fixed, others were determined to be test case errors, some fixes resolved multiple problems. For the purposes of IBM's announcement these problems were categorized into 14 individual fixes, which started appearing in IDS fix-packs around May 2005. Throughout this process NGSS and IBM actively co-operated in understanding and reproducing the problems.
So if the problems were fixed and in IDS fix packs a year ago why did IBM only recently announce them? From the beginning the announcement was planned to coincide with David Litchfield's announcement to make sure as many customers as possible already had the fixes in place before they were published (this is particularly important for older code branches like 7.31 which has an annual fix-pack release cycle). David informed us a few weeks ago that he would be publicising them at Black Hat 2006 so the IBM announcement was made the day before. The main goal for IBM throughout was to learn from the mistakes of other database companies and make sure that the fix for every problem was in generally available versions of IDS before it was made public. This goal was achieved.[Read More]
The latest DB2 magazine has a PHP and IDS tutorial that makes a great introduction to using PHP with Informix. Compare this with the earlier developerWorks articles: A step-by-step how-to guide to install, configure, and test a Linux, Apache, Informix, and PHP server and A step-by-step how-to guide to install, configure, and test a Windows, Apache, Informix, and PHP server.
On the subject of new IDS articles developerWorks recently added a guide to making use of the powerful Archecker utility: Perform point-in-time table-level restore in Informix Dynamic Server.[Read More]
The IDS User-Defined Routines and Data Types Developer's Guide hasa Complex Number example data type which creates acomplex number row type and defines overloaded plus and divide operators for them to support adding two complex numbers and dividing a complex number by an int:
CREATE ROW TYPE complex(r FLOAT, i FLOAT);
Once these operators are supported functions like sum() and avg()start working for complex numbers. This made me wonder about building a more complete treatment of complex numbers into the server, and how much can be donewithout resorting to external functions. How about the product of two complexnumbers which is defined as:
Following the same logic as the documented example, these, along with the modulus could be defined as:
CREATE FUNCTION times(c1 complex, c2 complex) RETURNING complex; RETURN row((c1.r * c2.r) - (c1.i * c2.i), (c2.r * c1.i) + (c1.r * c2.i))::complex;END FUNCTION;
Let's put it to the test. Multiplying i*i should result in -1.
CREATE TABLE t1(col1 complex);insert into t1 values(ROW(0,1)::complex);select col1*col1 from t1;...(expression) ROW(-1.00000000000,0.00)1 row(s) retrieved.
So far so good.
Now let's see if these functions can be used as the basis for more advanced operations such as transformations in Wessel's complex plane. I'll pick theparticularly scenic subset of the complex plane known as the Mandelbrot set, which is built by repeated iterations of the transformation
Here's a function written in Stored Procedure Language that takes a point on thecomplex plane as an argument and returns 0 if it is a member of the Mandelbrotset, or the number of iterations used to determine that it is not a member ofthe set, which could then be charted as a colour by a client application.
CREATE FUNCTION mandelbrot(c complex) RETURNING int;DEFINE iter, max_iter, result int;DEFINE z complex;LET max_iter = 999;LET z = ROW(0,0)::complex;FOR iter = 1 TO max_iter LET z = (z*z) + c; IF modulus(z) > 2 THEN EXIT FOR; END IF;END FOR;IF iter > max_iter THEN LET result = 0;ELSE LET result = iter;END IF;RETURN result;END FUNCTION;
Let's test a couple of coordinates:
execute function mandelbrot(ROW(0,1)::complex);execute function mandelbrot(ROW(1,1)::complex);...(expression) 01 row(s) retrieved.There you have it, the Mandelbrot set constructed entirely from SQL & SPL! I'm not sure ifI'll ever get around to writing a client to call this function and display the results for a set of coordinates.
Update 8/25/06: I got around to it here.
In case you're wondering what possible use there could ever possibly be for constructing a Mandelbrot set using SQL.. well, probably none, it's just a thought experiment to push the IDS built-in mathematics capabilities and see where they go...[Read More]
The term DLL hell is usually invoked with reference to conflicts between DLL versions, missing DLLs and multiple copies of DLLs. In the 14th century the poet Dante portrayed metaphysical hell as having multiple layers or circles, and in my opinion the analogy readily extends to DLL hell.
What concerns me lately I'll call the fifth circle - the swamp-like water of process address space, where wrathful DLLs fight over load addresses and slothful DLLs lie gurgling beneath the surface, fragmenting contiguous space.
A process running on 32-bit Windows has 2GB of address space by default. Into this space it needs to fit any operating system and application DLLs it loads, as well as any shared memory segments it attaches to. A DLL can have a default load address set at link time, and the operating system DLLs are usually set to load in the top 256 Mb of process address space, above 0x70000000. Incidently, a great tool to view DLL load addresses and process address space on Windows is Process Explorer from sysinternals.com.
Applications which attach to IDS shared memory segments, such as oninit, onstat, onbar need a contiguous free block of address space as large as the segment they are attaching to. So if onstat connects to the Resident segment and onstat -g seg shows it to be 1GB in size (due to the number of buffers configured for example), the onstat address space will need a 1GB contiguous gap where no DLLs are loaded. The first place it will try and attach the resident segment is the value of the onconfig parameter SHMBASE which is where oninit attaches it. The default SHMBASE value on Windows is 0xC000000.
The problem starts when when a DLL has a base address somewhere in the middle of the process address space. This fragments the address space and reduces the maximum size of shared memory segment that could be attached. If base address is not set at link time, DLLs have a default address of 0x10000000. A DLL loaded there would certainly cause a problem as it's only 64 Mb higher than default SHMBASE. If a process loads multiple DLLs which have a default load address of 0x10000000, one will be loaded there and the rest will be dynamically rebased to wherever the OS sees fit.
There are currently two defects for XBSA DLLs loaded by onbar which are set to the default load address and hence onbar returns errors in larger IDS shared memory configurations:
IC50382 ISM DEFAULT LOAD ADDRESS PREVENTS ONBAR FROM WORKING WITH IDS INSTANCES WITH LARGE SHARED MEMORY CONFIGURATIONS
IC50204 TDPI CAUSING ONBAR ERROR CODE -43399 WITH RESIDENT SHARED MEMORY SEGMENT BEYOND CERTAIN SIZE ON WINDOWS - this one only relates to the XBSA DLLs supplied with Tivoli Storage Manager
CA Storage Manager XBSA DLLs also have this problem - in the past we've had to rebase their DLLs to make them play nicely with onbar.
These defects are currently open, though Technical Support can work around the problem for you by rebasing the DLLs manually using the Windows Platform SDK rebase tool.
What can be more frustrating is when an operating system DLL has a load address outside of the recommended system DLL range of 0x70000000 to 0x7FFFFFFF. Windows currently has several bad DLLs which can cause problems for IDS:
873453 The base memory address setting of the Samlib.dll file in Windows Server 2003 may interfere with programs that require a large shared memory setting
894472 Third-party programs that require lots of memory do not run in Windows XP Service Pack 2
912570 Programs that require lots of memory may not run after you install Windows Server 2003 Service Pack 1
913409 A program that allocates a large block of contiguous memory may not start or may intermittently fail in Windows Server 2003
924054 Programs that request lots of contiguous memory may fail after you install security update 921883 (MS06-040) on a Windows Server 2003 Service Pack 1-based computer or a Windows XP Professional x64 Edition-based computer
If you are experiencing problems with onbar or other IDS utilities on Windows which go away when the number of buffers is reduced you may be in this particular circle of DLL hell. Depending on the problem the solution could be to request a patch from Microsoft, or seek help rebasing XBSA DLLs from tech support. Now would be a good time to familiarize yourself with Process Explorer to assist with the troubleshooting. A good way to determine the base address of a DLL is to use the Windows Platform SDK utility dumpbin.exe. E.g. to see the base address of xpsp2res.dll type:
Some additional process address space can be opened up by setting the /3GB boot.ini switch - this will provide an extra 1GB of address space above 0x80000000 for any process which is built with the IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header (and IDS binaries are built with this).
One glimmer of light on the horizon is that when x86_64 IDS is available for for Windows 64-bit, this problem should largely be a thing of the past. Shared memory segments larger than 2GB will be available, and we can start ascending the terraces of DLL Purgatory - ok I probably took the analogy too far that time.
Before anyone asks when the x86_64 Windows port of IDS will be ready, that has not been finalized yet; all I can say is that it is in progress.[Read More]
Just for fun (and QA) I ran some complex plane coordinates through my mandelbrot() stored procedure and displayed the output using the java.awt.Graphics package. Here's the result..
To save coding I took an example program from the Java Developer's Alamanac 1.4 and replaced the code inside the loop that generates a pixel in the Mandelbrot set image with:
String sqlstr = "execute function mandelbrot(ROW(" + p[c] + "," + q + ")::complex)";rs = stmt.executeQuery(sqlstr);rs.next();pixels[pIx++] = (byte)(rs.getInt(1) % 16);
Perversely I implemented the most inefficient method possible - generating and selecting each pixel individually over the network, which if nothing else provided ample time to monitor the stored procedure stack with onstat -g ses.
To utilize the power of IDS in a practical scenario the coordinate and generated data would be stored in a table. For a real example of an Informix powered scientific project, check out Barrodale Computing Services R&D - the Grid Slicer Demo is fun (here's a PDF write-up: http://www.barrodale.com/docs/ibm_grid_writeup.pdf).[Read More]
Got something to say? Say it at IDUG North American Conference 2007 and get free admission. The deadline for abstracts is September 1st 2006, so if you were thinking about submitting a presentation now is the time to visit the website. The 2007 conference is on May 6-10 in San Jose California, home of the San Jose Sharks and the safest big city in America. Here are the links:
Online Submission Form
And of course the conference already has its own blog.[Read More]
Having seen various iterations of this guide over the years, it's good to see the latest version available on developerWorks: Oracle to IBM Informix Dynamic Server porting guide. The guide draws on the experiences of consultants doing real migrations, development engineers, with collaboration and feedback from various other teams. This version runs to 90 pages and is up to date with Oracle 10g release 2, IDS 10.0 and recent Migration Toolkit enhancements.
The porting guide is continuously refined by feedback and experience and as noted in the introduction:
This document is intended to be a living document. Missing items, better ways of implementing Oraclefunctionality in Informix other than what is documented here, examples to better illustrate a point, andother suggestions should be sent to Sam Marino so they may be included with future revisions.If your attention span/available time doesn't run to 90 pages there's a good developerWorks introductory article to new MTK features here: Migrating to IBM database servers gets easier with the latest MTK release.[Read More]