Hat's off to the development team, for another on time (actually, a day early) delivery of a planned release. This fixpack release adds in not only a number of defect fixes, but adds support for Red Hat Enterprise Linux V6, CentOS from Red Hat and also PowerLinux. This gives customers an extra option for hardware, use of IBM Power hardware running Red Hat Linux.
The new version is available on our FixCentral site for download. It's a full replacement for prior versions, and uses the License Certificate File from the V2.0 version which can be obtained via Passport Advantage sites for customers.
And on another note, had an interesting conversation with Wayne Eckerson at the Smarter Analytics event last week, who wrote about Streams on his blog. We'd talked about Streams and BigInsights, and he asked if Streams was better than Hadoop. It's not really better or worse, just different. But there are some applications that would certainly be better off if you run them on Streams. A key question is, do you need to persist the data? Do you need to save data for historic purposes, and make analytic comparisons over long periods of time? There are many use cases where people really don't want to save the data, only capture meta data. Google was a case point we discussed - if all they wanted to do was create indicies of which pages point to other pages to understand which pages have the most linked pages, and create counts of key words on pages, they could do all this in realtime on Streams. It can even use map reduce style application pattern to run in parallel on many machines. It's not MapReduce a la Hadoop, but it's the programming concept to map the data onto different machines, and then reduce to the indicies you want.
Best of luck to everyone as we close out the first quarter!
Roger_Rea 2000005TUR Tags:  streams rtap advertising realtime analytic processing 1 Comment 3,633 Views
In case you missed it, in October Arvind Sathi published Customer Experience Analytics. Thisflashbook describes the key to real-time adaptive customer relationships. Many Streams use cases are mentioned in the first chapter to describe how in many industries, real time information is analyzed to help make recommendations for next best actions and marketing activities. An entire chapter is devoted to Stream Computing and gives a short overview of InfoSphere Streams. This makes a great giveaway for customers and prospects, especially in Marketing departments that are working to better understand customers and improve interactions with customers.
Nice article on What is Big Data. Raises the question of Streaming Processing vs Complex Event Processing, and predicts that the term CEP will pass away in favor of Streaming Processing. While I might use Stream Processing instead of Streaming, or better yet, Real Time Analytic Processing (RTAP) it shows that the CEP vendors haven't captured the mindshare in the industry. Streams is at the forefront of a new computing paradigm. With over double the number of new customers in 2011 as in 2010, Streams had great growth last year, and is posed for even greater growth in 2012, with many new use cases, references and assets available to accelerate growth.
in case you missed this, the Webcast has been posted to the Streams DeveloperWorks Wiki Reference Materials section of the Streams DW Wiki.
Announcing IBM InfoSphere Streams v220.127.116.11
IBM InfoSphere Streams, Version 2.0.0 Fixpack 3 is now available from the IBM Support FixCentral. In addition to several fixes, some exciting new capabilities were added:
To learn more about these capabilities, IBM will host a Developers Conference Webcast, Wednesday December 7, 2011 from 1:00PM to 5:00PM U.S. Eastern Standard Time.
Please join us on the web at LotusLive Meetings:
1. Conference ID: 2058889
2. Link to the web conference:
· General Access: http://www.webdialogs.com/join/?schedid=2058889
· IBM employees: https://lli.ibm.com/meeting/join/?schedid=2058889
Audio in listen only mode will be available on the Webcast.
For ability to listen and ask questions during Q&A periods the following conference information is available:
We look forward to sharing details about these new capabilities.
Roger ReaIBM InfoSphere Streams Product Manager
A broad team of experts, shepherded and corralled by by Sandra Tucker has completed the total re-write of the Streams RedBook - 456 pages worth! Overview, terms and concepts, sample applications, best deployment practices, advanced programming techniques and integration are all covered to the new Streams Processing Language and InfoSphere Streams v2.0.
Printed versions will be available at IOD, with several of the authors available to sign and chat about Streams. Please, come join us between 11:30 and 1:30 Pacific Time on Wednesday October 26 at the IOD Bookstore: Mandalay Bay Convention Center, Bayside Foyer, Level 1!
Recently, I went looking to get an update on University of Maryland Baltimore County's efforts to use Streams to build a wildfire monitoring application. It was the subject of a press release for Streams in the fall of 2009. Turns out they completed the application in the fall of 2010, but I hadn't learned of it until just recently.
They describe their project on the web, show how Streams is used to analyze image data and predict how smoke from wildfires will disburse. They've done analysis using historic data sets, and have some really cool Google Earth images. They have even posted the Streams source code to Google. It involved some rather complex modeling and analytics, using the Local Ensemble Transform Kalman Filter (LETKF) as data assimilation algorithm.
So, another cool application for Streams, and reference story!
Hi ... new article posted today on DeveloperWorks, Integrate MATLAB code into InfoSphere Streams.
Matlab®, from MathWorks, is a high-level language and interactive environment that enables you to perform computationally intensive tasks faster than with traditional programming languages such as C, C++, and Fortran.
According to Wikipedia, Matlab is short for matrix laboratory and by 2004 had over 1 million users. The users perform all kinds of numeric computing functions.
These mathematical models and computations can now be easily incorporated into a Streams application, to promote re-use. When combined with Streaming analytics from Streams, a combined Streams/Matlab application has ability to process huge volumes of data at high velocity by distributing the applications across one or more nodes of the Streams runtime.
If you're a Matlab user, and need to scale out applications that use Streaming data, you should certainly investigate using Streams.
If you're a Streams user and want to do matrix math (common in Finance, Engineering, Scientific and Healthcare) you might want to consider Matlab!
Hi ... I'm haven't been very active here, but perhaps it's time to turn over a new leaf. Perhaps a new approach - I've added a calendar entry to take 30 minutes a week and add a short new topic.
In the last year, we've grown significantly in terms of number of people educated about Streams, customers, partners and universities with interesting applications and proof points. Telecommunications has been especially promising new industry, with 2 customers in production and another moving to production this month.
On Tuesday April 12th, we'll have an update on Streams capabilities.
I'll host two sessions to give you a a reasonable chance to attend in a convenient time zone (which might be Wednesday morning for you).
Tuesday April 12, 2011
12:00 noon to 1:00 PM Eastern Daylight Time
10:30 PM to 11:30 PM Eastern Daylight Time
1. Conference ID: 2058889
2. Link to the web conference:
For IBM employees:
People outside IBM:
If a Country has an AT&T Direct Number, the audio conference requires two-stage dialing. First,
dial the AT&T Direct Number. Second, dial the Toll-Free Dial-In Number.
Country Access Type AT&T Direct Number Dial-In Number Global Dialing Comment
Angola Toll-Free 808-000-011 888-426-6840
Argentina Toll-Free 0800-222-1512
Australia Caller Paid 0-2-80318490
Australia Toll-Free 1-800-85-4950
Austria Caller Paid 0-1-2530601
Austria Toll-Free 0800-07-0079
Bahamas Toll-Free 1-800-872-2881 888-426-6840
Belgium Caller Paid 0-2-8946410
Belgium Toll-Free 0800-3-9022
Brazil Toll-Free 0800-890-0288 888-426-6840 FROM BRAZIL
Brazil Toll-Free 0-800-888-8288 888-426-6840 BRAZIL OTHER
Brunei Darussalam Toll-Free 800-1111 888-426-6840
Bulgaria Toll-Free 00800-117-4514
Canada Toll-Free 888-426-6840
Cayman Islands Toll-Free 1-800-225-5288 888-426-6840
Chile Toll-Free 1230-020-0719
China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS
China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS
Colombia Toll-Free 01-800-5-1-81588
Costa Rica Toll-Free 0-800-011-4114 888-426-6840
Croatia Toll-Free 0800-777-954
Cyprus Toll-Free 8009-1373
Czech Republic Caller Paid 2-39016353
Czech Republic Toll-Free 800-143-484
Denmark Caller Paid 32711870
Denmark Toll-Free 80-717000
Dominican Republic Toll-Free 1-888-426-6840
Ecuador Toll-Free 1-999-119 888-426-6840
Egypt Toll-Free 2510-0200 888-426-6840 FROM CAIRO
Egypt Toll-Free 02-2510-0200 888-426-6840 OUTSIDE OF CAIRO
Finland Caller Paid 0-9-72519565
Finland Toll-Free 0800-9-18357
France Caller Paid 0-170911580
France Toll-Free 0800-94-0558
Germany Caller Paid 0-69-2443-2290
Germany Toll-Free 0800-000-1018
Ghana Toll-Free 0191 888-426-6840
Greece Toll-Free 00-800-11-006-7783
Hong Kong Caller Paid 30713844
Hong Kong Toll-Free 800-90-5535
Hungary Caller Paid 0-1-7789195
Hungary Toll-Free 06-800-19-306
Iceland Toll-Free 800-9823
India Toll-Free 000-117 888-426-6840
Indonesia Toll-Free 001-803-1-006-7783
Ireland Caller Paid 0-1-5264424
Ireland Toll-Free 1-800-943-427
Israel Toll-Free 1-809-417-783
Italy Caller Paid 0-2-00621263
Italy Toll-Free 800-975100
Jamaica Toll-Free 1-800-872-2881 888-426-6840
Japan Caller Paid 0-3-64042596
Japan Toll-Free 00531-11-0086 JAPAN KDD USERS
Japan Toll-Free 0066-33-830321 JAPAN C&W USERS
Japan Toll-Free 0034-800-900377 JAPAN NTT USERS
Kazakhstan Toll-Free 8^800-121-4321 888-426-6840
Korea (south) Caller Paid 0-2-348-31399
Korea (south) Toll-Free 00798-1-1-006-7783
Latvia Toll-Free 8000-3906
Lithuania Toll-Free 8-800-31299
Luxembourg Caller Paid 24871280
Luxembourg Toll-Free 800-2-7910
Macedonia Toll-Free 0-8000-4288 888-426-6840
Malaysia Caller Paid 03-6207-4471
Malaysia Toll-Free 1-800-81-5412
Mexico Toll-Free 001-888-426-6840
Monaco Toll-Free 800-93771
Morocco Toll-Free 00-211-0011 888-426-6840
Neth. Antilles Toll-Free 1-888-426-6840
Netherlands Caller Paid 0-20-7996800
Netherlands Toll-Free 0-800-363-6036
New Zealand Caller Paid 09-912-7548
New Zealand Toll-Free 0800-442905
Norway Caller Paid 23162149
Norway Toll-Free 800-16771
Pakistan Toll-Free 00-800-01-001 888-426-6840
Panama Toll-Free 001-888-426-6840
Peru Toll-Free 0800-77-942
Philippines Toll-Free 1-800-1-111-0558
Poland Toll-Free 0-0-800-111-1929
Portugal Toll-Free 800-7-80880
Romania Toll-Free 0-800-896910
Russia Toll-Free 8^495-363-2400 888-426-6840 RUSSIA OTHER
Russia Toll-Free 363-2400 888-426-6840 MOSCOW & ST PETERSBURG
Russia Toll-Free 8^10-800-110-1011 888-426-6840 OR RUSSIA OTHER
Senegal Toll-Free 800-103-072 888-426-6840
Singapore Caller Paid 66221594
Singapore Toll-Free 800-110-1842
Slovak Republic Toll-Free 0-800-000-101 888-426-6840
Slovenia Toll-Free 0800-80768
South Africa Toll-Free 0800-983-687
Spain Caller Paid 9-1-7878580
Spain Toll-Free 900-8-01334
Spain Toll-Free 900-8-01334
Sweden Toll-Free 0200-12-5807
Switzerland Caller Paid 0-43-2101240
Switzerland Toll-Free 0800-89-4800
Taiwan Caller Paid 0-2-77457238
Taiwan Toll-Free 00801-10-4334
Thailand Toll-Free 001-800-11-006-7783
Trinidad & Tobago Toll-Free 1-800-872-2881 888-426-6840
Turkey Toll-Free 0811-288-0001 888-426-6840
Ukraine Toll-Free 8^100-11 888-426-6840
United Arab Emirates Toll-Free 0-800-121 888-426-6840
United Arab Emirates Toll-Free 0-800-151 888-426-6840 U.S. MILITARY BASE
United Kingdom Caller Paid 0-20-30596451
United Kingdom Toll-Free 0800-368-0638
Uruguay Toll-Free 000-411-006-7783
USA Caller Paid 215-861-6239
USA Toll-Free 888-426-6840
Uzbekistan Toll-Free 8^641-744-0010 888-426-6840
Venezuela Toll-Free 0800-102-9535
Vietnam Toll-Free 1-201-0288 888-426-6840
Hope to hear from you next week!
The telco case study was presented today at The Data Warehouse Institute conference in Las Vegas, Nevada. The audience supported the premise that their data warehouses and data marts rarely provide results within hours of data being generated. Usually it's daily or weekly reports.
So for organzations that can benefit from notification of anomalous behaviour in seconds, the new Streams Mining Toolkit can deliver the results that can help their business.
Customers stopped by the IBM Booth at the showcase to ask about extreme low latency and also about Predictive Modelling and use of Predictive Model Markup Language.
One analst quickly grokked the significance of Streams - your challenge, he explained looking at the torrents of data slide, is to explain to the market why you actually implement a parallel archcitecture and how that is different from related technologies like CEP that merely do some functions in parallel. A truly parallel architecture like Streams offers delivers a more scalable runtime to meet the most demanding applications.
The telco pilot presented in the Case Study hits a peak rate of processing about 500,000 Call Detail Records per second on only 15 blades. Using the new InfiniBand support in this release, the options trading application was re-done at rates up to about 12.5 million messages per second.
The new release of Streams announced Tuesday looks to deliver significant new function to customers and expand the use cases for Streams to anyone that is doing data mining using PMML models, and can benefit from getting results in seconds instead of in days.
Roger Rea, IBM
I'm in beautiful, snowy Stockholm this week where KTH (Royal Technical University) and the Swedish Road Administration held a seminar on Smarter Traffic for smart cities. Not only is it great to be here, but it's hilarious to see my name throughout the city. rea means 'sale' in Swedish, so my name is posted in stores everywhere!
Stockholm has long been a leader in providing information to their citizens on traffic options, and implemented with IBM a traffic congestion system. Taking pictures of license plates to charge for entering the core city during certain hours has limited traffic congestion and reduced pollution.
Professor Haris has called this a wonderfully elegant system, but says it's essentially a batch billing system. His vision is to collect data from taxis, city trucks, traffic light loop induction sensors, weather, subway and more to provide a truly intelligent transportation system.
Last fall, Prof Haris and his team with IBM demonstrated collecting data from GPS devices, stored in data files and played back in realtime, then mapped to google maps to show average speeds on various road segments. With this success, the Stockholm Road Administration opened up their taxi cab information to KTH. Since mid December, they have been monitoring nearly 1500 taxis with GPS information and source destination information to provide realtime speed data and estimated arrival time information. Additional data sources will also be forthcoming.
One eventual goal is to have a cell phone applications so people could text a destination, and have the application reply with options - 34 minutes via taxi and 22 minutes via subway. Thus, in realtime, changing behaviour to better balance the city traffic and transportation.
The seminar was in part to thank IBM for a Shared University Research grant where in addition to delivering InfoSphere Streams software, IBM provided 10 blade servers to continue the Transportation education and research at KTH