The general session started with an example of context computing and an interview with Captain Phillips.
All that was pretty exciting but what stole the show is the announcement of the partnership
between IBM and Twitter for analytics.
Then I went on my way to attend Streams sessions talking about use cases.
The first one i attended is about a partner, Voci, that has a appliance that converts audio to text.
In addition, it adds additional metadata such as the type of voice, accent, sentiment.
This solution can be augmented with InfoSphere Streams and BigInsight to take actions in real-time.
The next session was a panel of expert on geospatial analytics.
In the afternoon, I attended a session on the features of the new Streams beta that was announced last Friday.
You can find more information at http://ibm.co/streamsdev.
I followed with a session on context computing used to counter fraud. I finished my day
with a panel of users.
The conference is winding down with the last day tomorrow.
Another full day.
It started at 7:00 with a breakfast meeting and was followed by a conference call.
I then went to the conference bookstore for a book signing activity and moved on to a customer lunch.
As I mentioned in other blog entries, my new book is now out, at least at the conference:
"The Power of Now: Real-Time Analytics and IBM InfoSphere Streams"
My afternoon was taken by a Streams and text analytics lab.
I went back to the conference floor and had interesting conversations with many technical people
from different world regions. The conference sure provides great opportunities.
I'll be able to catch up on some Streams sessions Tomorrow. I can't wait to hear about some customer/partners stories
Also, I heard through the grapevine that there my be a big announcement at the general session.
I'll make sure not to miss that either.
After walking by 3 different Starbucks, I arrived at the conference breakfast hall.
I thought I would have a quiet breakfast by myself when I saw Bruce Brown, a big data partner expert.
Soon after, I was sitting others joined us: They were long time InfoSphere Streams experts. That was a great opportunity to talk shop and exchange information.
Then it was time to attend the general session that started at 8:15.
The session started with Jake Porway and Jeff Jonas talking about context computing.
The session was so packed with information that it is impossible to summarized properly.
Lets just say that Bob Picciano talked about three imperatives:
Data is the new natural resource, basis for business advantage
Systems of engagements
Multiple speakers expanded on these themes.
I particularly likes the line: "Geospatial data will become analytics superfood".
There were many interesting sessions to choose from but because of multiple engagements, I only attended
the Joy Global session where they described the real-time analytics they while monitoring mining equipment.
There was so much, if you are not at the conference, you may want to look for InsightGo to be able to attend some general sessions remotely.
Now it's time to move on to Tuesday!
The event went as planned at the Mandalay Bay convention center with presentation on:
Internet of things
Informix gateways and Informix capabilities for the internet of things
IBM Internet of Things foundation
Real-time analytics with Streams in the context of an internet of things architecture
Many people attended and were engaged in the presentations. Overall a success.
The Insight conference officially started with the opening reception.
We are getting ready for a great week of learning and networking.
We're up and going.
The conference is still being setup but there are events happening this Saturday.
This morning I was participating in the "Big Data and Analytics EdCon". This is part of an education session for faculties
offered under the IBM Academic Initiative. This was a hands on session introducing InfoSphere Streams and it was full!
All sorts of other sessions are taking place in other areas of the Mandalay Bay convention center.
Tomorrow, I'll be part of the "Internet of Things Deep Dive" as I mentioned in my previous blog entry.
The deep dive goes from 11:00am until 5:30. There is still time to register for it:
If you are already in Las Vegas for the Insight conference, this would be a good use of your time.
Finally, Sunday evening, the Insight conference officially starts with the Solution EXPO Grand Opening Reception
starting at 6:00pm.
I'll post comments on the conference daily so, stay tuned!
We are barely more than two weeks away from the Insight conference.
As I mentioned in my previous blog, lots of interesting sessions on Streams. Still there is more.
As you know, Streams is excellent at providing real-time analytics. It can be used with other
products to provide a solution in many domains. One of them is the Internet of Things (IoT).
It happens that I'll be participating in an IoT deep dive on Sunday October 26.
I'll be joining the main speakers:
Michael Curry, Vice President, WebSphere Product Management, IBM.
Jerry Keesee,Director, Real-Time Context Computing, IBM.
Jeff Jonas, IBM fellow and chief scientist, context computing
The technical section is divided in three parts:
Kevin brown talking about sensors and gateways
Peter Crocket telling us about the IBM IoT Foundation
Jacques Roy covering data-in-motion with Streams
You can register for the event at: http://insight-deep-dive.eventbrite.com
Don't forget to come see me at Insight in my sessions and labs as well as a book signing
session on Tuesday October 28 at the Insight Conference book store between 9:30 and 10:30.
The book is: "The Power of Now: Real-Time Analytics and IBM InfoSphere Streams"
See you in Vegas!
Ok, this is probably not news to you but there is information you should know.
The Insight conference, formerly known as Information on Demand (IOD), is going on Oct 26-30.
This is only 35 days from now! There is a lot of good content. Fro me, it starts on Sunday with an IoT deep dive call/meeting.
From there, I'll go to the demo ped to spend my evening. Please come visit
For the week, I am particularly interested in the Streams sessions such as:
Just to name a few. I am involved in a few sessions:
LCI-4252A: Hands-on lab "Streams and text analytics" on Tuesday afternoon (2:00pm)
LCI-5454A: Hands-on lab "The Internet of Things and Geospatial Analytics Powered by InfoSphere Streams", on Thursday morning (10:00am)
IIS-7096A : Expert Exchange "How to Harness the Internet of Things"
The other exciting part for me is that I am coming out with a new book:
"The Power of Now: Real-Time Analytics and IBM InfoSphere Streams"
I am doing a book signing on Tuesday between 9:30 and 10:30.
The Insight conference provides many excellent learning opportunities on many subjects including Cloud, mobile/Social, security, analytics, and more.
It is also a great opportunity to network with experts from IBM, partners, and other customers.
I'm looking forward to see many of you there at the Mandalay Bay in Las Vegas.
For more information on the conference, please go to the following web site:
When we talk about processing data in real time, it is easy to just write a program and be done with it.
The problems start piling up when we add analytics and volume.
A program is easy to write when it can process records sequentially. Once you reach the limit of this sequential processing, you start adding complexity that may represent the bulk of your work: You start by using multi-threading and eventually you need to also go to multi-processing to take advantage of multiple machines. It is much easier to use a framework to reduce those issues.
Still, a framework may give you the ability to distribute your processing but how easy is it to do? Now you want proper tools to assemble the many operations that you want to link together. Then, you also need to have the tools to easily identify bottlenecks so you can parallelize you operations. What about all the standard operations you would expect to be able to do?
This is where a platform comes in. It gives you the foundation for distributed processing but also gives you pre-built capabilities to interact with the outside world (files, message queues, databases, and so on) and also analytics so you don't have to reinvent the wheel.
For a more complete discussion on the subject, take a look at my two articles on the IBM Datamag site: part 1 and part 2.
InfoSphere Streams is starting to engage the open-source community to provide additional capabilities to its real-time analytics platform.
This is still very early in the process and we can assume we'll see evolve quickly. That may also be a way to consolidate
the offering of the most popular open-source toolkits currently available on the Streams Exchange.
One of the projects is under the name resourceManagers.
The current available resource manager that is available to support Streams is Yarn!
Learn more about what is available for Streams on GitHub by looking at the newest page from the InfoSphere Streams playbook:
Streams on GitHub.
Anyone remembers this cartoon? I think the first time I saw it was in the '80s. Still, it keeps coming back.
This used to apply to IT requests. It can also be applied to all sort of things, including how quickly you want to go from data to actionable information.
In Today's world, it seems that we need to get insights now. This is one reason for the rise of the interest in "data in motion".
Real-time analytics apply in many industries including medical, telecommunication, and security. You can find additional examples in the
following article: Big Data in Motion Where? Everywhere.
There is a special need in processing machine data. The data can be generated at such a rate that we need machines to analyze all that data.
You can find more information on machine data examples in the ebook: The Rise of Machine Data: Are You Prepared.
Data in motion processing is here to stay. It is a great approach to solve many business problems. Of course, this approach does not work in a vacuum.
It is a great complement to new and established systems based on data at rest. Here, I mean systems that use data repositories such as operational
data stores, data warehouses, Hadoop (BigInsights) and other NoSQL repositories.
The IBM solution for data in motion is InfoSphere Streams. You can download a free copy of the software to learn about it.
It is called the InfoSphere Streams Quickstart Edition. Visit the streamsdev site to download a copy of it and access an introductory lab (under Docs).
Do you know about IBM Data Magazine? It is the regular newsletter based on ibmdatamag.com that many people receive in their inbox
every few weeks (or is it weekly?).
This online magazine contains articles related to: Big Data and Warehousing, Databases, Information Strategy, Integration and governance.
There are multiple regular columnists and I am now one of them. I am covering Data in motion in a monthly column.
My first article got published on January 31st and is titled: "Getting the big data ball rolling".
You can find it at: http://ibmdatamag.com/2014/01/getting-the-big-data-ball-rolling/
I have put together a plan for a series of articles. When it gets more in depth, I will complement the articles with
my blog entries. I will also continue to cover other subjects and likely more technical subjects in this blog.
Hopefully this will get me to write a blog entry a bit more regularly than I've done lately.
Until next time...
I have to say, these are busy times!
With TimeSeries PoC and multiple activities around Streams, time flies by quickly.
It's been a while since I updated the InfoSphere Streams Playbook. This was overdue. There are new videos, training material and capabilities that were not reflected in the playbook. Here's what I updated:
In this section, I updated the databases supported and support for MQTT
There is now a link that should provide the complete lists of available videos dynamically. Also, I cleaned up the tutorials and added a brand new series of tutorials.
Video use cases
Some new youtube videos that show interesting use of Streams
With the end of the year so close, we can expect everyone to prepare for the new year. Looks like 2014 will be another fun year!
The other day I ran across an article on Infoworld.com: Cloudera pitches Hadoop for everything. Really?”
Of course, the article starts by mentioning the expression about hammers and nails. This is an old story and it appears that it is getting ready to repeat itself. Like it’s been said: “those who forget the past are doomed to repeat it”.
Hadoop has been the biggest star of the big data story. I have to say that it is revolutionizing data processing and for good reasons. Many seem to point to the use of cheap clusters based on commodity hardware. I personally prefer to attribute it to the large amount of data that has different requirements from traditional data processing.
The traditional data processing needs are still there and still growing. Getting rid of “silos” of data has proven extremely difficult. It also relies on getting rid of years of investments and re-writing many proven applications.
Instead of trying to fit everything into Hadoop, it is much better to have an overall strategy that takes into accounts the different needs of different data sets and make sure the overall architecture accommodates exchange of information between all of them.
Cloudera want to become the “enterprise data hub” powered by Hadoop. Like the article mentions, “Hadoop i still seen on all sides as a bucket of parts..”. Maybe it is a bit early to talk about an enterprise data hub based on Hadoop.
Of course, if all you have is a hammer, everything looks like nail
There is now a new resource for Streams: https://www.ibmdw.net/streamsdev/
The Streamsdev site includes articles, blog entries, videos, and intro labs. You can also get to the download the latest quickstart edition of Streams from there. This way, you can download either the product or a vmware image with it and do the lab at your leisure.
This site is put together by developers for developers. Still, if you are new to InfoSphere Streams, you can find something there for you too. Just go to the getting started section under "Docs".
Since the IBM Information on Demand (IOD) conference starts this weekend, you can also find information on the activities (labs, presentation) on Streams during the conference. You can see the next few acticities on the mainpage or a more complete calendar under events.
This site is evolving. You should go look at it at least once a week to see what's new.
Hopefully many of you are going to the IOD conference next week. Enjoy the conference and learn a lot!
Last week, on October 22, IBM announced a new version of InfoSphere Streams: version 3.2.
This follows version 3.1 that was announced on May 21.
The new version includes some nice improvements such as remote development, Rest API for data access, and improved toolkits.
Over the next few blog entries, I'll go into more details on these features. In the meantime, you can find information on
InfoSphere Streams 3.2 at:
If you are interested in trying Streams, IBM provides the quick start edition that you can download as native product or
as a VMWare image. you can download it at:
Of course, you may need more information on how to use Streams. You can start by browsing through the InfoSphere Strreams Playbook at:
If you have questions, don't hesitate to drop me a note or comment on my blog entries.
Until next time!