If you're into the "Interactive entertainment industry" be sure to check out Rajesh Nair's new blog Virtual Worlds, MMOGs, Serious data - Informix Technology for Persistent Worlds.
I am somewhat ignorant when it comes to virtual worlds and massive multi-player online gaming (unless you count Webkinz, where I play a mean game of Quizzy's Word Challenge), but it is clear there is vast potential for data management in this segment.
I am hoping Rajesh will teach me more about this area when I next bump into him at a conference. In the mean time his blog is a good start, especially the white paper it links to Informix Dynamic Server - Fun and Games: The Data Server for Interactive Entertainment. Another link that caught my eye was a press release from February: Icarus Studios switches from mySQL to Informix - that highlights some of the strengths and potential Informix has in this market.[Read More]
Administrating and Developing with Informix
I've been putting IDS 11.50.xC2 though its paces this week and it has been a pleasure to see a few long standing usability issues addressed, particularly on Windows. For everyone who uses IDS on Windows I strongly recommend upgrading to 11.50.xC2. In this post I'll mention a few of the new features in xC2 that make it worthwhile. To see the full list of changes in xC2 take a look at the Release notes for UNIX and Release notes for Windows. Update 9/13/08: See also Fernando's Blog.
Multiple Installations of IDS on the Same Windows machine
When the IDS installer switched to using Installshield as its framework in version 11.10 this implementation caused a slight problem. Only one copy of each major version of IDS could be installed, so you couldn't for example have a separate development and production installation of 11.10 on your machine. At least you could still create multiple instances via the Instance Manager.
In IDS 11.50.xC1 support for multiple installations was reintroduced via an undocumented "-multiple" command line argument to the setup.exe program. Every time you ran "setup.exe -multiple" a new installation would be created in the path of your choice.
In IDS 11.50.xC2 the multiple installation support is once again the default behavior. Any time you run setup.exe, if there is an existing installation of IDS on the machine, you will be prompted to create a new instance. If you want to run setup.exe to maintain an existing installation, you can use the new -path or -instnum arguments to identify a specific installation.
Note for backward compatibility the "-multiple" argument is still supported, though no longer required.
The ISM GUI Program is back
If you're very old you might remember earlier versions of IDS 7.x included a graphical interface for the Informix Storage Manager called ISM.exe that allowed you to create and view storage devices, monitor storage manager operations and other tasks. In some ways it provided equivalent functionality to the UNIX ism_watch program. For obscure reasons, shrouded in mystery, the ISM GUI was dropped from the distribution some time in 7.3x and never seen again. Until 11.50.xC2 when it has been resurrected and rebuilt by popular demand.
The Configuration Wizard is improved
The configuration wizard was a new install feature introduced in IDS 11.50.xC1 thanks in part to input we received from usability sessions. For the first time you could have the install program detect your hardware configuration (CPU settings, disk size and speed), ask you some questions about the expected number of decision support and OLTP users and create an optimized onconfig file for you. Internal TPCC benchmarks showed that performance from a Config Wizard versus a manually tuned configuration file was very close and significantly better than the default configuration based on onconfig.std (incidently onconfig.std was also improved in 11.50 - with better inline documentation for parameters and more realistic values).
One usability issue with the Config Wizard on Windows in xC1 was that the onconfig file it would generate would have some UNIX specific values for parameters such as for Java stored procedures (JVPJAVAVM etc) which would mean extra work configuring those features on Windows. In xC2 the config wizard was redesigned to layer its onconfig enhancements on top of the onconfig.std file for each platform, resulting in an optimized onconfig file tailored to the platform it was installed on.
Banishing DLL Hell with SHMNOACCESS
Anyone who likes to use large shared memory configurations on 32-bit Windows has to think about where DLL's are loaded in memory by processes such as oninit.exe, onmode.exe, onstat.exe and onbar.exe. Any process that attaches to a shared memory segment needs a matching contiguous range of address space, and any DLL loaded by that process potentially fragments the address space. I have ranted about the DLL hell issue on previous occasions (part 1, part 2, part 3, part 4).
Even with all Microsoft and IBM fixes, it was still possible to get into a situation where the oninit process creates a shared memory segment that overlaps with where another process that needs to attach to that segment has loaded a DLL, resulting in a "Unable to attach to shared memory" error. Enter the SHMNOACCESS onconfig parameter. You can use this to exclude any regions of address space where you do not want oninit to create a shared memory segment. As of 11.50.xC2 it works on all types of segment (resident, virtual etc). This was the final tool needed to avoid 32-bit DLL problems in large memory configurations. As an anonymous tech support engineer (Tim Steele) said, "the Windows witch is dead!".
Debug libraries are included on Windows
This is one of those features most customers do not need to care about, but makes tech support and development engineers who need to debug optimized code very happy. The .PDB debug libraries for the IDS executable processes are now included in the install media (but not installed). Using these it is possible to see a valid stack if you attach a debugger to an oninit process or analyze a crash dump. This can dramatically reduce the time to find the root cause of a problem.
Better Windows Server 2008 Support
In most cases IDS 11.50 works fine on Windows Server 2008, but in FC1 (64-bit) there was one class of Intel CPU (Intel64) that the installer wouldn't recognize. This and a few other Win2K8 issues have been fixed in xC2. One fix is still to come - getting the finderr utility working again without needing to download Winhelp. There are plans in place to fix this soon.
Why IDS on Windows anyway?
I talk about IDS on Windows a lot. Being the Windows architect might be a factor, but there are reasons why the Windows port of IDS is important. Windows is one of our top 5 production platforms along with Linux, Solaris, HP and AIX (in no particular order). Windows is also often the platform people see IDS on for the first time, simply because the Developer Edition is freely downloadable on the web and a lot of people are using Windows when they download it. We also provide it on the IDS Discovery DVD we give away at conferences. That's why I'm very happy to see a release of IDS that has significant improvements for Windows.
If you deploy IDS on non-Windows platforms, there's a reasonable chance you have a laptop running Windows. IDS has a small footprint so you can have the Developer Edition running for free on your laptop without much overhead. There it's good for learning/training, development and demonstrations. As well as IDS on my XP OS, I also have it running via VMWare on Ubuntu, SLES, XP (again) and Windows Server 2003 all on the same machine. Running multiple instances of IDS on multiple operating systems at the same time does not put much strain on the machine.
Mac OS X Installer improvements
11.50.FC1 saw our first Mac OS X port, and with FC2 the installer has been improved. Now it can automatically make any required adjustments to kernel parameters, and it creates a terminal icon for easier access to the demo instance. A lot of development work is going into to making further installer improvements for Mac OS.
Update 9/9/08: Check this developerWorks article: Demystifying the Informix Database bundle (IDS and CSDK) installation on Mac OS X for more detail.
IDS Developer Edition is now upgraded every fixpack
For IDS 11.10 there was only one version of the free Developer Edition provided, 11.10.xC1. I had issues with this, because any defects found in xC1 were not going to get fixed for people developing with and evaluating IDS for the first time. Now with IDS 11.50 the Developer Edition is upgraded every fixpack so you get the latest and greatest. Get your free 11.50.xC2 IDS Developer Edition for the platform of your choice here.
Enterprise Replication Hardening (all platforms)
One other area I'll talk about is ER. There has been a big focus on improving the resilience of ER, making it more efficient with memory and better able to handle low memory situations, and generally work better in an embedded environment. Among improvements in xC2 are:
There are various other new features, such as improvements to OAT and SQL enhancements, as well as numerous defect fixes. 11.50.xC2 is one of those releases where you really want to take a look at the release notes to see what you are getting and what you can make use of.[Read More]
There is a trend within IBM to move to using DVD as the standard software install media.
There has been talk about doing this within Informix before, partly because we're close to the maximum CD size on some platforms (where more than one copy of Java Runtime needs to be on the CD for example).
One advantage moving to DVD would provide is the ability to bundle additional components, for example Data Server Driver, which provides the Comment Client API's to connect to IDS via the DRDA protocol. Currently if you want components like this you would have to download them separately.
Once concern that has been raised is could this lead to bloating of IDS? E.g. if we're not so concerned about space might the install footprint go up? We would certainly have to be very careful about what is installed by default if we were to add components to the bundle.
Would anyone be inconvenienced if the IDS install media was no longer available on CD? If you see a problem with that please leave a comment or contact me. I am also curious as to what percentage of people download all their products electronically.[Read More]
This first beta platform for IDS 11.50 was Mac OS X, and for the last week Informix Dynamic Server has been enjoying the top spot in the Macintosh Products Guide
This coincided with the publication on Friday of a new developerWorks article by Akmal B. Chaudhri: Install IBM Informix Dynamic Server on Apple Mac OS X which includes an install demo along with the PDF transcript.
To download IDS 11.50 on the Mac, including the free Developer Edition go to http://www.ibm.com/developerworks/downloads/im/ids/learn.html.
And once you've installed IDS you may as well set up OAT. Leo Chan recently wrote an IDS Experts post on Setting Up OpenAdmin Tool (OAT) on Mac OS.
What have your experiences of installing IDS on the Mac been like? What have we done right and what can be improved? I have finally re-enabled comments as the memories of the last spam attack begin to fade, so please let me know what you think.[Read More]
The IDS 11 web page carries a very significant announcement today (well, yesterday) to coincide with this week's MacWorld Expo...The Big Cats Unite! Announcing IDS "Cheetah 2" for the Mac OS X Leopard platform - the first available Cheetah 2 beta release is now available to download for Mac OS.
Here is some of the press coverage:IBM's IDS to Support Mac Platform - Internet News.
IBM Informix Dynamic Server to Deliver Support for Mac OS X - CNN Money
There has been some speculation on the newsgroups as to what IBM's motivation for this is. Bernie Spang is quoted in the announcement as saying:
Expect to see more announcements in the coming weeks concerning Cheetah 2 features, and open beta availability for other platforms. In the mean time, if you have Mac OS X 10.5, give the first Cheetah 2 Beta a test and enjoy IDS on a Mac. Oh, and let us know what breaks. Well it is a beta, what do you expect :-)
Update: 1/17/08 - I'm adding a comment made by Brad Patterson to the main post as it is worth knowing before you download:
The files are x86_64 (64-bit) only and will not run on "older" Intel Macs.
myMac:bin informix$ file oninit
oninit: Mach-O 64-bit executable x86_64
I'll have to wait until I upgrade my laptop to Core 2 Duo to run this -- hopefully by then we'll be on a GA release!
Thanks Brad for testing this and sharing your discovery.[Read More]
The IDS Developer Edition is now downloadable from the Informix Trials and Demos site. Click on Informix Dynamic Server Enterprise Edition Trials. It's a fully-featured free download of IDS 11.10.xC1D with no time-limit.
So far the following platforms are there:
Note the file sizes are currently listed in KB instead of MB.
Here is a reminder of the Developer Edition specific limits:
Pay particular attention to the session limit if you are developing applications which use a connection pool that tries to set a higher number of sessions.[Read More]
Back in September when I announced my move to Development to work on installation, Windows and common libraries, I glibly invited the massed ranks of blog readers (both of you) to send feedback:
I am interested in any feedback and suggestions you might have regarding the IDS installation process (on any platform) - does it meet your needs?
Gerardo Santana took me at my word and left some thought provoking comments:
About the installation process, last time I checked I had to unpack a tar file into $INFORMIXDIR before running the installer. I fail to see the reason for that requirement, and it bugs me because it makes me give /usr more space than it needs. I want to start the installation process from anywhere I want (my home for instance).
And why should it use Java anyways? It would be nicer that IBM could offer packages for the package managers of the platforms it supports (RPM is not enough). As if the Informix installation was so complicated anyways (a bunch of files dropped in a single directory). Or am I missing something?
I made a vague commitment to reply to these comments, and since then more than one person has asked me when I'm going to get around to it. Admittedly it has taken rather a long time to reply. In part because I usually tend to write blog posts in a hurry (can you tell?) and this was something that deserved a considered response. After all, Gerardo makes an excellent point - Install should be simple.
Firstly, there is more than copying a bunch of files to a directory, setting file permissions and linking libraries. For a UNIX DBA who knows the operating system and has everything set up, this might be sufficient, but there are various additional requirements that clients, partners, and other IBM teams ask for, including (in no particular order):
For these reasons it becomes more cost-effective and reliable to choose an install framework that has support for many of the above features built-in, can support multiple platforms, and is straightforward to extend and customize. Currently Installshield is the install framework used by IDS on Windows, and Installshield Multi-Platform (ISMP) is used on UNIX.
One price that comes with an Installshield framework is a Java Runtime dependency. A Java Runtime Environment is bundled in the install media so for most users this is transparent. Java makes a good choice to write a multi-platform install framework - it has been around for a while and many of the platform specific issues go away.
That is not to say there are not other choices, both Java and non-Java. We continue to evaluate install frameworks to identify which has the best combination of cost-effectiveness, usability, reliability and platform support. We are looking at both home-grown solutions and 3rd party install frameworks used by other IBM Software Group development teams. We are also actively looking into how we can provide simpler options for embedding IDS and deploying instances to multiple machines.
Regarding wanting to start the install process from anywhere. That sounds like a reasonable request and if the demand is there it aught to be fixable.
Thanks for the comments Gerardo, and please leave more. Thanks also PJ in the install team for providing insight into current ISMP functionality.
Part 1 of this article looked at a standard case of the popular Could not initialize security subsystem error which can occur when running command line IDS utilities on Windows.
Part 2 looks at a more obscure case of this error. Here's an email I received last week:
I installed a Cygwin environment and a telnet server on our Windows 64-bit machine.I can now access the machine and log-in as informix to a nice UNIX-like bash shell. However I would like to access the informix server from this shell. When I have defined INFORMIXDIR/INFORMIXSERVER etc. However I always end up with ERROR could not initialize security subsystem.Since this is usually a problem accessing the Informix part of the registry, my first thought was that this had something to do with Cygwin being 32-bit and not being able to access the 64-bit registry keys under HKEY_LOCAL_MACHINE\Software\Informix. With hindsight this couldn't be the problem, since it would be the 64-bit IDS utilities which access the registry keys, not Cygwin.
Strangely the problem only occurred when connecting via the telnet server. A Cygwin shell executed via Remote Desktop worked fine.
To investigate further, I used the Sysinternals utility Regmon to monitor which registry keys were accessed when the onstat utility was executed via a telnet session. This showed one problem - the Informix environment variables were being set in a "DOS format" environment shell, which added an extra linefeed character at the end of the INFORMIXSERVER environment value. This showed up as a mysterious block at the end of the Registry access string in Regmon.
Once the environment variables were fixed, onstat would still not run in the telnet session. It failed with:
GLS initialization failed, error -23101
A set of instructions for installing OpenAdmin Tool for IDS in a Linux XAMPP environment written by Erika von Bargen was added to the IDS Experts blog over the weekend. Now anyone wishing to install OAT with XAMPP can refer to:Read More]
Today I was caught using the non-existent word "shutdowning", which reminded me that mental exercise is important for keeping ones brain young looking and stopping it from, er, shutdowning.
What better way to exercise the gray matter than by writing your own web quiz game?
20 Questions is a simple quiz game written in ZK and small enough to be embedded in a web frame. It is based on question-answer pairs such as world capitals, US state capitals, English county towns and chemical elements. The main point of the quiz is to improve working memory.
The question-answer pairs are loaded into a database, and randomly shuffled by the application before challenging the user, who can choose 5, 10, 15 or 20 question quizzes, depending on how much work they are avoiding. One user complained that the questions are not multiple choice... kids these days, the idea is to test memory rather than guessing ability.
For the database the questionable design decision of having a separate table for each quiz was made. It would have been neater and better normalized to have a single quiz table with an extra quiz id column which references a lookup table for quiz name. That would allow the application to be data independent and dynamically build a list of quizzes based on the rows in the lookup table.
The main reason a separate table for each quiz was chosen for this example is that for a small number of quizzes it makes for a smaller and simpler web application. Should I ever add a new quiz I will stop prevaricating and normalize the design moving forward.
With excuses out of the way, here is the schema for the quiz database with its four quiz tables. This article assumes Informix Dynamic Server as the data server of choice, naturally, though there is nothing data server specific about the schema.
create database quiz;create table elements ( idx serial not null , question varchar(30), answer varchar(30), primary key (idx) );
database quiz;load from 'count_towns.unl' insert into county_towns;load from 'elements.unl' insert into elements;load from 'state_capitals.unl' insert into state_capitals;load from 'world_capitals.unl' insert into world_capitals;update statistics;Presentation layer
To get started with ZK refer to the Quick Start guide on the documentation page. The visual part of the quiz, written to $TOMCAT_HOME/webapps/quiz/index.zul is as follows:
<window title="20 Questions" width="400px" border="normal" onOK="processAnswer(answer.value);"> <zscript src="quiz.jul"/> <vbox> <label id="selectLabel" value="Please select a game.."/> <hbox> <listbox id="gameList" rows="1" mold="select"> <listitem value="elements" label="Elements"/> <listitem value="county_towns" label="English County Towns"/> <listitem value="state_capitals" label="State Capitals" selected="true"/> <listitem value="world_capitals" label="World Capitals"/> </listbox> <listbox id="numq" rows="1" mold="select"> <listitem value="5" label="5"/> <listitem value="10" label="10"/> <listitem value="15" label="15"/> <listitem value="20" label="20" selected="true"/> </listbox> <button id="newGame" label="New Game" width="90px" height="24px" onClick= "newGame(gameList.selectedItem.value,numq.selectedItem.value);"/> </hbox> <hbox> <label id="question" value="" visible="false"/> <textbox id="answer" value="" visible="false"/> <button id="subButton" label="Submit" width="90px" height="24px" visible="false" onClick="processAnswer(answer.value);"/> </hbox> </vbox></window>Note that the main window has an onOK handler so the enter key can be pressed instead of selecting the Submit button with the mouse.
The presentation layer calls two functions, newGame() and processAnswer(). The Java code to implement those functions - what I'll call the service layer - is in a file called quiz.jul and is listed below.
Note that the JDBC database connection was implemented by adding a connection pool called jdbc/quiz to the Tomcat server.xml file. (For an example of setting up a connection pool in Tomcat see sections 4 and 7 of this post.)
// quiz.jul - service layer for quiz application
To obviate the need to use the mouse while playing, in addition to the onOK handler in the presentation layer, the focus shifts back to the answer text box after each question, and to the New Game button at the end of a game by calls to the focus() method.
Despite my reservations about the unedifying qualities of multiple choice it wouldn't be too much work to extend this program (and db schema) by adding support for radio button multiple choice answers. Suggestions for new quizzes (or mistakes in the existing ones) are always welcome.[Read More]
The Geospatial Technology journal Directions Magazine has just published a Q&A with IBM's Director of Data Servers Bernie Spang: Informix Spatial Data Technology: Update and Positioning. The article discusses the Web Feature Service Datablade module, the Spatial and Geodetic blades. Also mentioned is the R-Tree index design contrast between Oracle's and Informix's spatial offerings.
My favourite recent example of Informix datablade solutions (including spatial) at work is the Impire AG live action sports content.[Read More]
To be in England in the Summer time. A kestrel hovers. A dragonfly skims over water meadows rich with clover. The liquid call of the nuthatch. The subtle beauty of the spotted orchid. These and other fond memories are evoked as I gaze through my hotel room window and contemplate the scenes of pastoral harmony before me..
This week's Cheetah training for Tech Support takes place in London and Sydney. In London we're starting with Continuous Availability presentations. Here's a photo Ed Raven took of Holger Kirstein enlightening us with the Mach 11 feature
Second time around Cheetah continuous availability features are becoming much clearer. Today I had fun setting up Shared Disk Secondaries on Windows. A fairly straightforward process using Instance Manager to create a new instance, setting ROOTSPACE to match the primary instance, and adding the SDS parameters to the primary and secondary onconfig files.[Read More]
For the past 6 months I've been tinkering with a chatbot that provides an instant messaging interface to our product defect database via SameTime. Development consists of sporadically adding commands in my spare time to meet various user requirements like fuzzy search, subscribing to nightly bug reports, reporting defects by owner, submitter, severity, etc. One of these needs is to expand coverage to other databases used by tech support, for example those used to track customer calls and internal technical discussions.
Some of the databases are difficult for mere mortals to obtain direct access to, and can only be publicly accessed by logging on to the official internal web interface. These restrictions led me to experiment with Jakarta Commons HttpClient as a means to retrieve and process information from secure websites. HttpClient is straightforward to use and has all the functionality required for this purpose. Once the basic infrastructure was in place it was easy to add chatbot commands that retrieve information from any websites. One of my first experiments was to add a "whatis" command to the bot in order to provide an interface to Wikipedia.
The whatis program
The whatis program is a simple command line example to demonstrate using HttpClient to look up Wikipedia articles and return the information as text. It retrieves the raw HTML from the website and does a little processing to extract the main text of the article and filter out the HTML. To build and run it you need the following Jakarta Commons libraries in your CLASSPATH: HttpClient, logging, codec.
// wiki.java - a simple Apache Jakarta Commons HttpClient demo to// download articles from Wikipedia.org
This example calls getResponseBodyAsString to get the requested web page as a string. It would be more efficient and predictable to write your program around the getResponseBodyAsStream method, or to limit the downloaded size.[Read More]
I was CC'd on an interesting VP Private Memory Cache Q&A that helps explain this feature a little further. VP Private Memory Cache was a performance feature announced in IDS 10.00.xC6. There was a slight hiccup due to the default value being uninitialized in that version, which should not distract from the usefulness of this feature..
1. Where is private cache allocated from? in the resident or virtual segment?
2. How is the VP private cache used? From my understanding:
When a VP needs to read some data pages from disk, it will search the VP cache first to check whether there is enough space, if it cannot find the space, it malloc the memory from the buffers.
So I feel now Informix has two level's of cache, the first is vp private cache and the second is buffers. But when one vp needs to access a data page which lies in another vp's private cache, what does it do?
3. After the data page was read into the vp private cache, it maybe updated or deleted, when is the data page is being flushed to disk to mark it clean and accommodate the new data page? in checkpoint?
A.The vp private memory cache is not used for buffers in the buffer pool, or for any pages read from disk. I think that's where the confusion is starting. When a thread needs to allocate memory from its own session pool, for example, that's when this VP-private cache comes in. Think of all the memory that threads allocate from pools like the 'global' pool or the 'rsam' pool or their own session pool (e.g. pool name '125'). It's that memory that goes into the VP-private cache when it's freed.
Here's the big picture. Before we had this VP-private cache feature, every VP would fight every other VP for the same memory in a particular shared memory segment. The memory in that segment had to be protected by a latch. So when Thread 1 on VP 1 needed a block of memory from Segment 1, it first acquired the latch, then took the memory, then released the latch. Meanwhile if Thread 2 on VP 2 needed memory from the same segment it would have to wait for the latch to be released in order to get a block of memory from Segment 1. Typically these threads need these blocks of memory for their session-private pools. Again, this is not related to the buffer pool that contains pages from disk.
You can imagine that in a high-stress environment with a lot of VPs and a lot of threads the latch on Segment 1 would become a performance bottleneck.
The solution we chose was to allow each VP to build its own private cache of memory blocks as blocks were freed. In other words, the first time a memory block was allocated from a segment, it would be allocated the same way it always has been. But if that memory block was freed, where ordinarily it would go back to the segment, now it remains allocated but is tracked by the freeing VP as part of its private cache. The next time a thread on that same VP needs a memory block it does not need to acquire any latch to get it. It simply takes the block from that VP's private cache. We know that no other thread will try to allocate memory from that same cache simultaneously, because only one thread can run on a VP at a time.
The size of an individual VP private cache is limited by the VP_MEMORY_CACHE_KB configuration parameter. In other words if you set VP_MEMORY_CACHE_KB to 1000, no VP-private memory cache in the server can exceed 1000 KB (1 mb) in size. Calculating the maximum amount of memory that an instance can allocate toward all VP-private caches is a matter of multiplying the value of VP_MEMORY_CACHE_KB times the number of VPs.
If you set VP_MEMORY_CACHE_KB to 0, the feature is turned off.
The minimum non-zero value for VP_MEMORY_CACHE_KB is 800, I believe.
Thanks JC.[Read More]
Update 6/12/07: The article has been updated and is back online!
Want a summary of IDS 11.10 features in a developerWorks article, with handy links to the documentation for each feature? IBM Software Engineers Inge Halilovic, Mohan Saraswatipura, Radhika Gadde and Priyambada Behera have published such an article: What's new in IDS Cheetah (IDS 11)? It makes for a good digest of the major Cheetah features - useful to scan through if you think you know all the major enhancements, or have missed the hype and want to know what Cheetah is all about.[Read More]