Modified by keshavamurthy
Applications generally look for NoSQL database for one of the following reasons:
Flexible schema or schema on read. Data or its type cannot be predetermined. In Informix NoSQL, it's accomplished via JSON and BSON type, added natively into the database. Just like LVARCHAR. Not only you won't know the column names (in this case KEY-VALUE pair, you won't know their type as well). It's not just you're able to store, you need to index and query as well.
Scaling out for increase data capacity, lower latency and performance.
New type of data modelling like GRAPH.
While NoSQL generally means Not Only SQL, if you attend any of the NoSQL conferences, meetups, people more often mean
SQL. Recently, Michael Stonebraker made a point in SE Radio (http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/) that most popular NoSQL databases like MongoDB and Cassandra have query languages very similar to SQL -- they're Not Yet SQL. While Hadoop when thru a phase of customer query languages, majority of vendors have SQL like interface to Hadoop.
In case of Informix, we started with SQL. Added flexible schema with the native support of JSON, MongoDB compatible query language (in fact, Informix NoSQL customers have to use the driver for MongoDB) and scale out via range and hash based sharding.
Once we added that, the question was, how would enterprises with TBs of data in relational form and hundreds of applications running on this data exploit the innovations in the API and the technology advantages.
When it comes to data and its infrastructure, evolutions seems to work much better than revolution.
Replicating the data from one form to another just won't work in most cases due to cost and consistency issues. So, we've added support for MongoDB APIs to access SQL data (relational data), features (joins, stored procedures, views) and enabled SQL to access JSON data.
Tomorrow, TUES Dec 17th 11.30 EST, John miller will be talking on these features. Rigister Now.
Deep Dive Continues: Hybrid storage & application.
Register at buff.ly/18sT065
IBM IOD Conference started with big bang today morning. Tomorrow, Atul Gawande
will be the general session's guest speaker
at IBM IOD conference 2010. He has long list of accomplishments, but did you know he also contributed to Informix v11.70? Here's how.
When we were starting v11.70 work, I saw his interview with Charlie Rose
where he was talking about simple checklists saving lives in hospitals by reducing mistakes in routine things like proper installation of IV lines
. He had written an article on it in New Yorker
, was discussing results of world wide study of results using 90-second checklist for surgery. Here's a quote from the article: If
a new drug were as effective at saving lives as Peter Pronovost’s
checklist, there would be a nationwide marketing campaign urging doctors
to use it. Later in 2009, He wrote the book , The Checklist Manifesto: How to Get Things Right.
Within Informix, we've had checklist for a long time(e.g. what all the code to make when we do parser changes), but didn't have a feature wide checklist. In analyzing customer bugs for the last few years, I had observed, even
though we try to cover all cases between unit, functional, integration,
stress and interoperability testing, we do miss some cases and
combination. We would depend on the requirements, design and test specs and their review to catch all scenarios. At the start of Informix v11.70 development, I created a simple checklist, that started with something like this.
We added all the statements from ALLOCATE to UNLOCK, all major features which would bring new scenarios (MACH11, Enterprise replication, Administrative API, etc), all the tools, and other features planned for v11.70. Development, QA analyzed these from requirements (both externally visible and engineering requirement), design issues that may crop up in that combination, what relevant tests are necessary for that scenario.
Every checklist analysis yielded new scenarios to consider, decide what should be the requirement, do design to include scenario and test in house. We saw improvements internally in scenarios we caught and addressed before we released the product. Hopefully you've tried Informix v11.70 by now and hope you see the improvements to the product and quality. Checklist is not an alternative to good requirement collection or good design, but it helps doing things right.
The Doloitte technology trends 2011
report point to new in-memory technologies like SAP HANA changing the business intelligence is done.
This trend bodes well for IBM Informix warehouse accelerator
(IWA) we recently released. SAP HANA and IWA both try to exploit in-memory systems to accelerate query performance and reduce administration+tuning at the same time. These used to be inversely correlated, but with 3rd generation database technology, we're trying to get both together! It's like talking and browsing together on the phone! :-)
I saw one of their videos indicating queries on 700m row fact tables returned in 5-7 seconds... IWA has similar numbers with queries on more than billion row fact tables returning 2-4 seconds
. IWA is transparent to Informix applications, runs in virtualized environment, on the cloud, etc -- are plus points for IWA. It'll be interesting to see if someone spends time to do a full blown comparison.
The improvements in hardware configurations are aiding this trent as well.
Recently, IBM announced new versions of eX5 systems.
New MAX5 expansion capabilities include:
• Eight-socket x3850 X5: up to 192 DIMM slots (6 TB)
• Four-socket x3850 X5: up to 96 DIMM slots (3 TB)
• Two-socket x3690 X5: up to 64 DIMM slots (2 TB)
• Two-socket HX5: up to 32 DIMM slots (512 GB)
Having these large memory systems enable more VMs and enable improved CPU usage.
These systems bring down the overall cost and improve the cpu utilization for existing software.
New generation of software like IWA automatically maximize the CPU usage and larger memory systems enable larger data warehouse.
The eX5 tagline is:Maximize your memory, minimize cost, simplify deployment.
This -- and maximize performance -- could very well have been tag line for IWA as well :-)
Check out the article on MERGE statement in Informix.
To enable to XPS customers to move to Informix and enhance productivity for Informix warehouse customers, we implemented MERGE statement in 11.50.xC5 and significantly enhanced its performance in 11.50.xC6. We had multiple customers doing both functional and performance testing during early validation program. The team that developed this feature has written a nice article with fully working examples (simply cut and paste into your stores_demo database).
The standard on MERGE standard stipulates we update a row ONLY once and not end up updating a newly inserted row within the same statement. In gist, manipulate a row only once in the statement, following the rules of set operations. We invented a new mechanism to handle the Halloween problem
in MACH11 environment and have filed a patent on it. Below is the overview.
[Re-posted due to blog issues]
Informix 11.70.FC5 has three of significant features for Informix warehouse accelerator (IWA).
1. Supporting Informix host running on Solaris/Intel. IWA will still run on Linux/Intel on a separate hardware.
2. Partition based refresh from Informix to IWA
3. Deploying and using IWA in various HA cluster configurations (Primary, SDS, HDR secondary, RSS, etc).
The few slides below give you a straight forward explanation of the two main use cases.
fyi: You can download the power point version with animation at: http://slidesha.re/KnfTYK
I've added commentary for each slide below.
is the IWA architecture and steps to deploy and use IWA till
11.70.FC4.. IWA takes a snapshot of the data from Informix and runs
queries on that data. The transactions and loads happening on Informix
won't immediately change the data on IWA. To do that, you've to refresh
the data either using the studio or command line tool. When you
refresh in this method, you'll reload the entire data. Time to
completely refresh is directly correlated to size of your data set.
Informix 11.70.FC5 alleviates this issue by allowing you to ONLY refresh the changed partitions of the fact table.
Slide 2: The
Sales table is the fact table in this mart. After you've setup the
datamart in IWA, your load jobs will load data into one of the
partitions or add new paritions into the fact table. In this case, you
simply do the following:
1. execute dropPartMart() procedure for each of previously existing partitions that were modified.
2. execute loadPartMart() procedure for each of modified and new partitions.
do have to know or keep track of which of the partitions in your fact
table were modified since the last refresh and which of the new
partitions are added.
Time cyclic data management enables you to limit the amount of data you
keep in your data marts. If your business requires a 3 week window, at
the start of every week, you roll off (detach) the oldest week and
attach the data from the latest week.
This slide is better seen with animation in power point. In step 2,
execute the dropPartMart() to drop the IWA data for the partition you're
about to detach and then detach the partition. After you attach the
new partition, issue the loadPartMart() to refresh the data to IWA.
When you have large data marts, using the partition based refresh
will enable you to refresh the data quickly and resume your analysis.
Because you can refresh the data quickly, you can choose to refresh the
data often to enable analysis with latest data.
This is a step towards eventual goal of near/real time analysis.
Couple of years ago, at IOD, Nestle CIO said, enterprises are moving from "reporting model" to "tooling model".
What does that mean? Executives are no longer satisfied with simply using canned reports and taking decisions on it.
They need to have drill down and interact with the data to get the right data to make right decision in real time.
While this trend has been going on for some years, it took incredible investment from CIO and the organization to get
all the pieces together and working together.That's where Pronto Software comes in.
They have built an incredible award winning ERP application suite
Pronto has complete ERP solution with Informix server and have tightly integrated Cognos BI into the same solution.
From asset management to analytics, distribution to data intelligence, financials to forecasting -- It's all in there.
In this solution, not only they run the business, they help analysts have a conversation with the real-data in real time.
This helps SMB and enterprise, "find their moment"!
At IOD Conference in Las wegas, Wednesday, Oct 24th. 2.30 to 3.30PM, Chad Gates, Senior Development Manager at Pronto
will be talking about this solution and how they've integrated the stack. Should be very interesting.
Look up for session 4202: Pronto Xi, Informix and IBM Cognos 10 – Designing integrated ERP and real-time BI for Mid-Market:
fyi: Pronto also has the demo at the Expo floor.
IBM Informix group (as well as many other groups within IBM) have been collaborating with Intel to create and improve products.
I wrote earlier about the benchmark we did earlier with Nehalem and Westmere
. Recently, Intel gave us a machine with 8 socket Nehalem (64 cores) and 2TB RAM.... It's been a great to do some performance measurements on this.. More on this as we progress.
At IBM Information on Demand conference at Las Vegas next week, Intel is going to exhibit their latest processors, servers based on those. Tran Jantz from Intel and I will be presenting Intel-Informix talk on Thursday, 11.30, focused on the technologies within Intel and how Informix Warehouse Accelerator is exploiting each of those technologies. To entice you to explore all Intel booths and Intel-IBM talks at IOD, Intel will have a raffle Intel 160GB SSDs -- one at the talk and another at expo. Amazon prize for this is $280.. pretty good deal!
We'll have continuous demo of Informix Warehouse Accelerator. There will also be continuous demo of OAT, Genero, Flex Grid, Smart Data, at the demo room.
Count the number of times the basketballs are passed in this video.
Did you see the Gorilla? Apparently, 46% people focused on counting passes
failed to observe the Gorilla walking around
. With so much focus on BI (Business Intelligence),
predictive analysis, we could use a tool to predict, catch disruptive forces on our business or to create one ourselves.
What's the gorilla in database or RDBMS business now? Map-reduce model? Hadoop, HadoopDB?
says progress of civilization can be tracked via amount and speed of information processing.
By that definition, Informix v11.70 is progress -- making it much easier load and maintain even more data, easier than before, faster than fast.
Fred has written a nice blog
about some of the some of the features and even more come soon.
I plan to write on details/examples/nuances of individual features underneath.
To start with here are the list of all big and small features for warehousing.
For those who've followed features in 11.50 fixpack will recognize features such as compression, external tables and MERGE in earlier fixpacks.
In the last couple of years, we've focused on building upon a strong partitioning, parallel query processing foundation within Informix to offer
differentiating features to make Informix warehousing as easy as Informix transaction processing.
Of course, apart from warehousing, we've had number of other features in different domains. Informix 11.70 Infocenter
has all the details/etc.
I'll mention two features I'll visit in future blogs...
- Stored procedure debugger. You can debug SPL stored procedure from IBM Optim Data Studio environment and Microsoft Visual Studio .NET environment. For the longest time, our friends at Server Studio provided the SPL debugger. This new debugger is based on new API. See Optim data studio setup and Visual Studio setup details in our manuals.
- We've had support for direct Informix to Websphere MQ interaction since 10.00.xC3. We have enhanced this to support interaction with Websphere MQ anywhere on the network. we've also enhanced the MQ support to inquire if a message has arrived on your queue. You can use this function and schedule DBScheduler jobs to execute stored procedures on message reception. This is way, you can trigger execution of some business logic automatically when you receive a message.
I'll be at IBM Information On Demand Conference at Las Vegas, 24th thru 28th of October.
I have couple of sessions describing the Panther fragmentation and query processing features.
I'll also demo these in EXPO. Will post the timings/etc soon.
One more thing...Dr. Guy M. Lohman
Manager of Disruptive Information Management Architectures in the Advanced Information Management Dept. at IBM Research Division`s Almaden Research Center (ARC) in San Jose, CA, will be talking on some of the latest research cutting edge technology for complex query processing at the session on Tuesday, October 26th, 4.30pm.
Hope to see you there.
Lester Knutsen did extensive testing of of IWA before the release and gave us very valuable input.
Now Lester and his team have created a new benchmark+demo.
Look forward to see you at the webcast.
Announcement from Lester Knutsen:
We are hosting a Webcast to demo the Informix Warehouse Accelerator on February 28th at at 2:00pm EST. I have been using it for over a year and I am continually shocked at how fast it is. In one set of benchmarks it ran 9 hours of queries in 14 minutes. Please join Mike Walker, Art Kagel, and me for a webcast that will demonstrate our current benchmarks with this exciting new database technology. We will demonstrate ad-hoc queries on a bookstore database with 250 million customers and over 400 million records in the fact table.
Informix Warehouse Accelerator Demo Webcast
1. Benchmarks - Fast Performance Demo
2. How to Setup the Accelerator
3. How the Accelerator Works
4. Smart Mart Demo - How to Automatically Build a Data Mart
Topic: Informix Warehouse Accelerator Demo
Date: Tuesday, February 28, 2012
Time: 2:00 pm, Eastern Standard Time (New York, GMT-05:00)
For more Information:http://advancedatatools.com/Informix/IWAWebcasts.html
To register for this meeting
1. Go to https://advancedatatools.webex.com/advancedatatools/j.php?ED=180784857&RG=1&UID=1054423202&RT=MiMxMQ%3D%3D
2. Register for the meeting.
After the Watson Jeopardy contest, I saw couple of very good presentations at ibmwatson.com, reading up on the work in the book Final Jeopardy.
I'm half way thru the book... will review the book in another blog.
Yesterday, I got to know Dr. David Ferrucci will be speaking at IIUG Conference in Lenexa this May.
This is pretty exciting.
Here is the announcement.
Dr. Ferrucci is the lead researcher and Principal Investigator for the
Watson/Jeopardy! project. He has been a Research Staff Member at IBM's T.J.
Watson's Research Center since 1995 where he heads up the Semantic Analysis and
Integration department. Dr. Ferrucci focuses on technologies for automatically
discovering valuable knowledge in natural language content and using it to
enable better decision making.
In 2007, Dr. Ferrucci accepted the Jeopardy! challenge. Dr. Ferrucci was
tasked to create a computer system that can rival human champions at the game of
Jeopardy!. Introducing Watson, IBM's computer powered by DeepQA technology.
Watson appeared on three Jeopardy! episodes, in February 2011, where man vs
machine was put to the test. Watson was able to answer questions posed in
natural language to defeat two past Jeopardy! all time winning
In addition to Dr. Ferrucci, distinguished keynote speakers at the 2011 IIUG
Conference include: Dr. Arvind Krishna, Jerry Keesee and Rob Thomas.
To celebrate the CHANGING of The Guard with Anita McKeithen from IBM taking over the CHAT WITH THE LABS, the IIUG is going to give away ONE full conference pass INCLUDING a Full Day of Tutorials at its annual conference, more than a $1000 value to one lucky attendee of the first CHAT with the LABS that Anita is hosting, this Thursday April 21, 2011. For details about the IIUG conference May 15 - 18, 2011 in Overland Park (KC area), KS, USA visit http://www.iiug.org/conf
And anyone on the call can get a FREE day of Sunday tutorials included with their registration if they are on the call.... For details, join the call and we will tell you how to get this VERY limited one time offer. (New Registrations Only)
The Call Details
April 21, 2011
11:30 AM - 1:00 PM Eastern (GMT-05:00)
10:30 AM Central
8:30 AM Pacific
16:30 hrs London
17:30 hrs Frankfurt, Paris
10:00 PM India
IBM Informix Genero
Speakers: Jerry Keesee (Director of Informix worldwide, IBM), Shawn Moe(IBM Technical Product Manager)
IBM Informix Genero is a new product that we are happy to be able to offer in partnership with Four Js. Informix Genero offers an exciting set of functionality for the modernization of Informix 4GL applications. In this Chat with the Labs session, we will discuss the business aspects of the new Informix Genero offering, including packaging, sales, and technical support. Informix Genero is offered in both development and runtime editions. We will discuss some of the new language constructs, platform support, and other functionality that Informix Genero offers to Informix 4GL developers. We'll discuss the Informix Genero architecture, some approaches for converting from Informix 4GL to Informix Genero, and some options for extending your applications with the new Informix Genero user interface functionality. We'll wrap it up with a discussion on Informix Genero web services and how they extend the new Informix 4GL web services support.
Jerry Keesee, Director of the Informix Lab will introduce the call and Shawn Moe, IBM Technical Product Manager will be our technical speaker.
Presentation material will be available at http://www.ibm.com/informix/labchats prior to the event.
To register and receive instructions for attending this webinar, point your browser to:https://events.webdialogs.com/portal/wipevents/register.php?id=b9c49ccae3&l=en-US
At 2010 IIUG conference, when Jack Parker saw the preview of Informix warehouse accelerator, he offered his Amex card to buy the product. Two problems: I couldn't sell since the product was not released. I presume his Amex card has some credit limits :-) I suspect, after that incident, Jack started working on his performance tuning in the old route and produced nice article on Informix Performance tuning (part 1).
Note to Jack -- Good news is Informix Ultimate Warehouse Edition with Informix Warehouse Accelerator is now GA. You can buy it any time, even with your Amex.
Nita Dembla developed smarter, faster update statistics for Informix 11.70 and has a good article on it. If you're on 11.70, you should automatically be experiencing the the faster update statistics. With little bit of time, you can make it way faster!. Check out the article.
Earlier this year, we collaborated with Intel measure performance and scaling of IWA on Intel's latest processors: Nehalem and Westmere. The detailed report was published as article in IBM Data Management Magaine.
Apart from the details of the technology, the core message is, IWA exploits the latest features like SIMD, larger on-chip cache in Intel processors. As Intel innovates, IWA customers benefit directly with better performance and lower TCO.Here is a snippet from the article.