In the last few blog entries, I've been talking about TimeSeries. This time, I'd like to diverge a little for a change. Still there is a tie to TimeSeries
About a year ago, I went to a E&U conference. As you may know, Informix is making a push in this industry due to the advantages that TimeSeries can provide to this industry. In one of the sessions I attended, the presentor mentioned in passig the "Did you know?" video on youtube. Just the context when it was mentioned made me pay attention. I took a note and decided to look it up later. Last time I checked, it had had over 14 million viewing!
"Did you know?" starts with a global view of the world ("If you are 1 in a million in China, there are 1300 people just like you") and continues to talk about the evolution of the impact of technologies on our lives and its impact in the future.
Some other highlights:
- the top 10 in-demand job in 2010...did not exist in 2004
- (dated!) 200 million register users in MySpace would represent the 5th largest country in the world (What about facebook?!?)
- It too the radio 38 years to reach an audience of 50 million people, it toook television 13 year and facebook 2 years
- It is estimated that 4 exabytes of unique information will be generated this year. This is more than the previous 5000 years
Like it says in the video, we are living in exponential times.
Take a look at it, it's only 5 minutes of your time: http://www.youtube.com/watch?v=cL9Wu2kWwSY
The other day I put out a SPL function that converts a DATETIME into a Unix timestamp (number of seconds since Jan 1, 1970). I needed this function as a compatibility function. In the blog entry, I should have called it UNIX_TIMESTAMP. It manes sense to have the reverse function that takes a timestamp and returns a datetime year to second. Here it is:
CREATE FUNCTION from_unixtime(secs int)
RETURNING datetime year to second
WITH (NOT VARIANT)
DEFINE ival interval hour to second;
DEFINE dt datetime year to second;
DEFINE days, hh, mi, ss integer;
LET ss = MOD(secs, 60);
LET secs = secs / 60;
LET mi = MOD(secs, 60);
LET secs = secs / 60;
LET hh = MOD(secs, 24);
LET days = secs / 24;
LET ival = (hh || ':' || mi || ':' || ss)::interval hour to second;
LET dt = EXTEND((MDY(1, 1, 1970) + days), year to second) + ival;
This was the last day of the conference with a 35 sessions. I was surprised to see how many people attended the presentations until the end. I see this as a big endorsement of the value provided by these presentations.
On my part I delivered one presentation first thing in the morning and another one starting at 2:10 pm. Despite that, my session was well attended.
Overall a very successful conference that was well worth attending.
In case you missed the announcement this week, IBM announced the availability of the Informix warehouse as of March 5, 2009. Here is a quote from the announcement this morning:
From Kevin Brown, lead architect for IBM and Jim Kobielus from Forrester Research:
"This can save weeks of effort into just a few hours," Brown said. "In addition, customers often did without information because of the cost of effort to get the information. The lost opportunity cost savings is harder to quantify, but can be significant once they use their warehouse platform for smarter decision-making."
Take a look at the press release at: http://www-03.ibm.com/press/us/en/pressrelease/26840.wss
See also the Informix Warehouse page on the ibm site:
I recently met with a consultant friend of mine. He told me that he would like to have a new feature in the database server: a blocking select.
This feature would block on a SELECT until a new row is available. The SELECT statement could also have a condition (WHERE clause). If no rows are returned, the SELECT blocks until it could return at least one row. This way, an application could be event-driven without having to constantly check if there was anything new in the database. You can imagine the overhead of checking if there is something new every few seconds. A blocking SELECT could possibly reduce the pooling overhead and be more responsive depending on what the pooling interval is.
IDS does not have a blocking SELECT and as far as I know, nobody else has it either. I can think of two ways to achieve this:
- Use triggers on the table.
This could be an INSERT and an update trigger. The trigger would test the condition and then execute a user-defined routine (UDR) to send a signal to a process that would tell the process to read the table.
This approach could work when the table does not receive too many INSERTs and UPDATEs. The UDR could potentially find information from a file or from a configuration parameter stored in a table. It could also keep that data in memory to avoid getting the parameter for each trigger execution.
- Use the Change Data Capture API (CDC)
CDC provides an SQL interface to the database logs. You can configure your "log snooping" for a specific table. Each time a new log record is created, the function would return. The application then needs to look at the type of record and decide what to do. This approach should be very efficient and add virtually no overhead to the database server.
Another approach would be to use the InfoSphere Change Data Capture product to replicated specific records on a message queue and have the application subscribe to these messages.
Maybe a blocking SELECT would still be a neat feature to have but with all the different ways we can provide event-driven processing, maybe it is not needed after all.
The day started with a keynote presentation by Dr. Anant Jhingran on "Cloud computing, databases and the role of IDS". He was assisted by our own Guy Bowerman. That was quite a good start to another great day of learning and networking.
There were 35 sessions covering subjects including Gillani Fourgen case tool, Genero report writer, IDS tasks and sensors, performance tuning, backups, trouble shooting, encryption expert, and index enhancements. Quite a range of subjects and that's not the half of it!
I had interesting conversations with some partners. One of them mentioned how the AGS Server Studio product transformed someone that knew nothing of databases into a database administrator in no time flat. Looks like the ease of use of IDS with the ease of use of AGS is an unbeatable combination. I also had a discussion about collecting and sharing the information about sensors worldwide to monitor the health of the planet. Talk about a stimulating conversation.
There is one more day of this! I don't know how much more I can take :-).
There is so much going on!
As you surely know, we've been doing a closed beta of the next version of Informix. We have received a lot of great feedback and we keep on working on this release.
We still can't talk about it. It is just a matter of time before we can do so stay tuned.
On other fronts, I am working on a follow up to my Application development short book. I've received a lot of positive feedback on this book and I am excited about continuing on the subject. When will it be ready? I'm hoping sometime this year.
Finally, do you realize that we are barely more than a month away from the Information on Demand (IOD) conference? I hope to see you there.
There's a book I read many years ago called "Controlling Software Projects" by Tom DeMarco. The first chapter starts with: "You can't control what you can't measure". This statement has popped up in my head regularly, especially since the IOD conference last October. While I was there, I had a long conversation with a customer that seems to be measuring everything and the results show.
IDS provides the ability to collect a lot of measurements. With the latest IDS version, we have even more capabilities than ever before. If we add the Open Admin Tool for IDS to the mix, monitoring has never been easier. Of course, monitoring and measuring are two different things. You need to collect the data provided in the monitoring and compare it over time.
Here's an example of measurement: Each time new features are added to the IDS code line, performance testing is done. The results are compared to previous performance runs. If the performance declines, it triggers an investigation to fix the performance issue. The IDS development team always aims at improving IDS performance.
So, we know IDS is always improving. We know that we can collect a lot of data on the IDS execution and tune the engine to get the most out of it. Is that enough?
My answer is NO. As data access experts, we need to get involved in the application side and monitor and measure what is being done there. How much data are they pulling out? what are they doing with it? There can be cases where implementing a stored procedure could greatly reduce data movement and yield both additional performance and scalability. This is just one example of what can be done.
You can't control what you can't measure. Don't limit yourself to the database server. By lending your expertise to a larger team, you can make a significant difference in your work environment.
Lately, I've heard a lot about cloud computing. It starts with fast networks and technologies such as Mashups and REST interface. A mashup allows you to take information and processing coming from different sources and put them together to provide a new application and new insights based on the data manipulation.
REST (REpresentational State Transfer) is a way to access data, possibly through a web service, using the HTTP protocol (Web server protocol). This makes it simpler than using the SOA SOAP interface.
Now, providers like Google and Yahoo! provide email and storage services. I even hear that, with the help of virtualization, some companies are looking at providing a "web stack" that would provide the ability to run company businesses remotely. That sounds like a new twist to service-bureaux of olds.
There are several issues around this starting with response time and the complexity to provide what companies really want but it seems something that could take off. There are a lot of talks about web applications so why not a web IT infrastructure? That has a potential benefit of centralizing all the administrative needs.
I think this is something worth discussing. What do you think? Post comments to my blog to let me know.
Once again, another full day. There were Informix sessions on embeddability, virtualization/cloud computing, security, and zero-downtime upgrade. We also heard a great presentation on database tuning from Rick Rabe and Tom Girsch from Hilton Hotels.
Great sessions altogether. Now on to Thursday.
Since I've been on a common driver kick lately, might as well keep on going...
There was a chat with the lab on Feb 25th that talked about the common Java JDBC driver (referred as the JCC driver): Top 10 reasons to consider IBM Data Server Driver for JDBC and SQLJ for IDS
You can use the JCC driver with IDS when connecting using the DRDA protocol. Some of the benefits include:
- Better integration with WebSphere
- Ability to use the capabilities PureQuery
- Better tracing and debugging
- Full IDS clustering support
- Superior performance over the Informix JDBC driver
All this is significant:
- PureQuery can increase the performance of SQL statements by analyzing the usage and make changes transparently from the application. For example, it can detect the use of the same statement with different literals and convert that under the cover into a prepared statement.
- Full IDS clustering support includes working with the connection manager to automatically and transparently connect to an alternate server when the primary fails.
- Superior performance: It provides a 5% to 10% performance boost over the Informix JDBC driver.
If you are using Java, maybe it is time to start looking into the JCC driver. You can download it from the IBM site at (10MB):
For more information on this chat with the lab:
Here's Where you can find information on this chat with the lab:
Second day of class in Strasbourg. It started well: I covered IDS extensibility and the students went through the lab without more serious problems than misspelling and keyboard issues.
Life was good until we got to Data Studio..
I went through the presentation quickly since all the students said they were familiar with eclipse. We then moved on to the lab. Fifteen minutes into the lab, most students were still waiting for Data Studio initialization to complete. For some, it took much longer. All that to say that the lab machines were under-configured in memory. I should have mentioned that the lab machines needed at least 1GB of memory and not only 512MB.
Despite the memory problems, most student were able to get through the lab. I suspect that it will be impossible to do the lab on web services since we also need to start WAS CE in addition to Data Studio.
We continued with a discussion on OO and databases followed by a review of web environments with more emphasis on application servers. These went quicker since, as expected, students covered the OO approach in detail during their years at the university. this was a good thing since we were behind in my original schedule.
the day ended with a presentation on WAS CE followed with the lab where they were able to see an application that runs in the application server and accesses Informix. Another decent day, now on to the last one.
Here's another one:
This whitepaper summarizes a comparison study in which the total cost of ownership of IBM Informix is shown to be 31+ percent less than Microsoft SQL Server. This is of great interest to mid-size businesses that face considerable challenges in their IT environments to reduce costs while gaining competitive advantage.
You can download the report here: http://ht.ly/2Ln0d
There were 2 keynote addresses, 25 sessions, 5 bird-of-a-feather sessions, usability labs and demos in the exhibit hall. Then the day closed with an Hawaiian Luau sponsored by Gillani. A packed day to say the least.
All this was topped with the announcement of IDS 11.50.xC4 that includes a new storage optimization feature and the Informix warehouse bundle. Things are moving fast with Informix!
The storage optimization includes compression, repack the spaces that is saved and shrink the dbspace to free the space. This new feature could save 30% to 50% or even more in some cases.
The Informix warehouse bundle include a too that allows you to define your warehouse and define the process of extract, load and transform. There is a chat with the lab schedule for Wednesday, April 29th at 8:30 AM Pacific, 10:30 AM Central, 11:30 AM Eastern, 4:30 PM London, 5:30 PM Paris that will provide more information on the subject.
I am currently in Belgium, Brussels with Terri Gerber. Last Tuesday, we had a successful meeting with an Informix customer. After the meeting, in late afternoon, Patrick Billens took us around to show us the sites of Brussels including among other things the royal palace and the "grande place". Little did I know that soon, Terri would almost cause an international incident with the digital equipment she was carrying.
It happened soon after we saw the king's working palace. The flag at the top of the palace indicated that the king was currently there working. We turned the corner after the palace and Patrick pointed to another building with an American Flag in front of it. Terri took out her digital camera and quickly took a picture. Within seconds, two Belgium police officer were in hot pursuit and quickly caught up with us. Before all h**l broke loose, Patrick intervened, avoiding the worse. I could only imagine what could have happened: Two officers tackling the red-jacket wearing culprit followed by a struggle to get to the digital camera.
Terri did not surrender her camera. Instead, she wrestled with its ease of use and deleted the offending picture in front of the officers. Satisfied with the action taken, we were free to go on our way.
At this time, Terri is on her way back home to Massachusett. Is the picture really deleted? I'm sure it would be possible to un-delete it. Hopefully this won't be a concern for the department of homeland security. If that becomes an issue, I'm sure Terri would be willing to share information with homeland security on how much Informix could help them make the country safer.