I recently conducted a webinar on “Analytics on the Go” with Arun Kannan. Details as follows
Learn how the Real Time Video Analytics solution improves banking transaction interactions by recognizing customers at the time of premise entry. By analyzing advertisements and videos in conjunction with structured data, the solution presents a 360 degree view of each customer, enabling staff to present relevant offers and promotions and convert cross-sell and up-sell opportunities. Additionally, it uses pattern recognition to identify repetitive patterns and stationary objects, further strengthening on-premise surveillance and security.
Download this presentation from https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=96960515-2ea1-4391-8170-b0515d08e4da#fullpageWidgetId=Wd0e64f75f750_4dc0_8b39_95bda936bafe&file=3983e40f-3d35-4b31-9210-9ab516eb23ff
IBM IM Champion 2013
Modified by Gleeson
Big Data analytics for IT 'outage avoidance', join the Open Beta May 14th
Today (May 14th) IBM launches an Open Beta for 'Tivoli Analytics for Service Performance', or TASP; a new behavioral learning Big Data analytic capability that helps IT operations teams avoid outages.
Detect problems before they become business or service impacting
TASP is a state-of-the-art analytic solution for detecting emerging problems before they impact services or interrupt business. Built on a foundation of the industry’s most powerful and scalable streaming analytic engine InfoSphere Streams, it processes performance and metric data in real-time from the existing IT monitoring and performance management solutions. Applying advanced analytics technologies which mathematically discover complex relationships between metrics and learns the normal operational behavior of IT and network environments. Based on this learned behavior, it is able to detect anomalies indicative of faults and provides early problem detection, prior to a service disruption.
- Maximizes early detection of service and application issues and spots problems while they are emerging to avoid business impacting service disruptions and outages.
- Learns the normal operational behavior of dynamic infrastructures, such as a cloud, and identifies problems before you know to look for them—catches problems the first time they happen.
- Analyses performance and monitoring data across silos, domains and vendors. Provides a single analytic solution for complete heterogeneous monitoring infrastructures.
We invite you to try the solution for yourself with the Open Beta. You do not need to be an IBM customer – just sign up for a 'My IBM account' here (upper right corner).
After you have created an account you can find the download and instructions here.
Part of IBM's Cloud & Smarter Infrastructure analytics family.
Driven by the dynamic nature of cloud, and the explosion of structured, unstructured, network and metric data, the analytics suite provides a range of analytics for IT application and operations management functions that span workload optimization, performance management, outage avoidance, diagnostics and insight.
The family offers a wide range of analytics techniques ranging from behavioral learning, to pattern identification based on best practices, to capacity planning expert systems fed on usage data and budget requirements. Data can be aggregated from structured and unstructured content, metrics, and events for rapid analysis.
The combination of these techniques provides insight, advice and actions to Application Owners, Operations, Subject Matter Experts, Engineers and Planners, that address business challenges that were previously beyond the reach of traditional tools and techniques.
Unmatched strength: IBM Big Data platform and Research.
IBM Big Data
The Cloud & Smarter Infrastructure analytic family is underpinned by the industry’s most powerful, scalable Big Data platforms.
But Big data is more than simply a matter of size; it is an opportunity to find insights in new and emerging types of data and content, to make your business more agile, and to answer questions that were previously considered beyond your reach. Until now, there was no practical way to harvest this opportunity. Today, IBM’s platform for big data uses state of the art technologies including patented advanced analytics to open the door to a world of possibilities.
Focused on innovations for more than a century, the organization famous for beating Chess Masters and Jeopardy contestants, and more recently, creating the worlds smallest movies from atoms. IBM has topped the annual patent recipient list for 20 consecutive years. From 1993-2012, IBM inventors received nearly 67,000 U.S. patents, and in 2012 alone, received a record 6,478 patents
These strengths and vision show IBM’s long-term, strategic commitment to innovation and demonstrate the patience to allow scientific discovery to find its way into the solving problems currently beyond existing tools and techniques.
Hat's off to the development team, for another on time (actually, a day early) delivery of a planned release. This fixpack release adds in not only a number of defect fixes, but adds support for Red Hat Enterprise Linux V6, CentOS from Red Hat and also PowerLinux. This gives customers an extra option for hardware, use of IBM Power hardware running Red Hat Linux.
The new version is available on our FixCentral site for download. It's a full replacement for prior versions, and uses the License Certificate File from the V2.0 version which can be obtained via Passport Advantage sites for customers.
And on another note, had an interesting conversation with Wayne Eckerson at the Smarter Analytics event last week, who wrote about Streams on his blog
. We'd talked about Streams and BigInsights, and he asked if Streams was better than Hadoop. It's not really better or worse, just different. But there are some applications that would certainly be better off if you run them on Streams. A key question is, do you need to persist the data? Do you need to save data for historic purposes, and make analytic comparisons over long periods of time? There are many use cases where people really don't want to save the data, only capture meta data. Google was a case point we discussed - if all they wanted to do was create indicies of which pages point to other pages to understand which pages have the most linked pages, and create counts of key words on pages, they could do all this in realtime on Streams. It can even use map reduce style application pattern to run in parallel on many machines. It's not MapReduce a la Hadoop, but it's the programming concept to map the data onto different machines, and then reduce to the indicies you want.
Best of luck to everyone as we close out the first quarter!
In case you missed it, in October Arvind Sathi published Customer Experience Analytics
. Thisflashbook describes the key to real-time adaptive customer relationships. Many Streams use cases are mentioned in the first chapter to describe how in many industries, real time information is analyzed to help make recommendations for next best actions and marketing activities. An entire chapter is devoted to Stream Computing and gives a short overview of InfoSphere Streams. This makes a great giveaway for customers and prospects, especially in Marketing departments that are working to better understand customers and improve interactions with customers.
Nice article on What is Big Data
. Raises the question of Streaming Processing vs Complex Event Processing, and predicts that the term CEP will pass away in favor of Streaming Processing. While I might use Stream Processing instead of Streaming, or better yet, Real Time Analytic Processing (RTAP) it shows that the CEP vendors haven't captured the mindshare in the industry. Streams is at the forefront of a new computing paradigm. With over double the number of new customers in 2011 as in 2010, Streams had great growth last year, and is posed for even greater growth in 2012, with many new use cases, references and assets available to accelerate growth.
in case you missed this, the Webcast has been posted to the Streams DeveloperWorks Wiki Reference Materials section of the Streams DW Wiki.
Announcing IBM InfoSphere Streams v220.127.116.11
IBM InfoSphere Streams, Version 2.0.0 Fixpack 3 is now available from
Support FixCentral. In addition to several
fixes, some exciting new capabilities were added:
- Big data platform
integration: The new HDFS (Hadoop Distributed File System) adapter will allow
Streams to deliver data to/from InfoSphere BigInsights at very high rates.
Streams will also now include the text analytics engine used by
BigInsights. This is in line with IBM’s vision to build the industry’s
most comprehensive, enterprise-ready platform for big data.
- Performance boosting
upgrades: The DB2 Parallel Writer significantly increases the volume of data
that can be simultaneously written to the database. Additionally, the Streams Processing
Language has been enhanced to give more application control over analytic
- More robust security: The certificate-based non-password login for Streams
console provides greater security for the most sensitive applications and
is a requirement for many government agencies.
- Flexibility & ease of
use: The feature pack will
also include more granular log file management capabilities, support for
multiple network interfaces, and tooling to automate definition of
To learn more about these
capabilities, IBM will host a Developers Conference Webcast, Wednesday December
7, 2011 from 1:00PM to 5:00PM U.S. Eastern Standard Time.
Please join us on the web at
1. Conference ID: 2058889
2. Link to the web conference:
General Access: http://www.webdialogs.com/join/?schedid=2058889
IBM employees: https://lli.ibm.com/meeting/join/?schedid=2058889
Audio in listen only mode will be
available on the Webcast.
For ability to listen and ask questions
during Q&A periods the following conference information is available:
We look forward to sharing details about
these new capabilities.
IBM InfoSphere Streams Product Manager
A broad team of experts, shepherded and corralled by by Sandra Tucker has completed the total re-write of the Streams RedBook
- 456 pages worth! Overview, terms and concepts, sample applications, best deployment practices, advanced programming techniques and integration are all covered to the new Streams Processing Language and InfoSphere Streams v2.0.
Printed versions will be available at IOD
, with several of the authors available to sign and chat about Streams. Please, come join us between 11:30 and 1:30 Pacific Time on Wednesday October 26 at the IOD Bookstore: Mandalay Bay Convention Center, Bayside Foyer, Level 1!
Reviewing a presentation done at last years Information on Demand conference by Centerpoint Energy
about Streams, they commented that to improve velocity of analysis, they considered increasing capacity of their data warehouse. But commented that a bigger warehouse meant bigger hardware.
Which sparks the question, Does Big Data Require Big Hardware? Emphatically, NO!
Earlier this week, I reviewed use cases of Hadoop
on the apache.org site, and it was rather surprising to see some of the huge clusters of hardware used to perform analytics. But in many of these use cases, sentiment analysis of blogs and tweets, index building of web sites, etc there is no need to save the data longer than what's needed to create the meta data. These use cases are much more suitable to real time analytic processing (RTAP) - analyze the data in memory, create meta data, and discard the original data. In the case of web pages, blogs, etc the data is all persisted elsewhere anyway - there's no need for a company to create a second copy of it (incurring the cost of managing all the data). One paper
cited the cost to manage data in a warehouse at between $500,000 and $1,000,000 per terabyte of storage. Wikipedia
cited $10,000 to $150,000 per terabyte for initial purchase of a warehouse, with 80% of the TCO from monitoring and tuning. Not to mention backups, disaster recovery, etc. If the initial costs are only 10% of the TCO, then again we see ranges of $100K to $1.5M per terabyte.
But doesn't a warehouse appliance or hadoop lower these costs? Well, yes, but ... one article said a Netezza appliance would all purchase of a terabyte of storage for $2,500 and that hadoop was merely $250 per terabyte. But again, if the TCO is 10x that number to manage, back up, restore you're up to $25K to $250K.
Streams with real time analytic processing (RTAP) allows you to analyze and get the results you need without saving the data, eliminating the TCO of managing the data over it's lifetime.
And we have more and more customers who are finding that Streams blazing speed means reduced hardware - 10x reduction in the number of blades for one customer application. Another company comparing log analysis and 39 different benchmark tests found that on the same hardware, Streams could handle at least 10x the number of events per second as other complex event processing systems. Early work to port a set of applications is indicating a 17x throughput improvement.
Does Big Data Require Big Hardware? NO..... by handling 10x the volume on the same hardware and by eliminating long term data management costs, a small number of x86 nodes can analyze your big data more effectively.
The question you must ask is, must the data be saved for the future? If not, then look to Streams to save money and improve operational efficiency.
Recently, I went looking to get an update on University of Maryland Baltimore County's efforts to use Streams to build a wildfire monitoring application. It was the subject of a press release for Streams
in the fall of 2009. Turns out they completed the application in the fall of 2010, but I hadn't learned of it until just recently.
They describe their project
on the web, show how Streams is used
to analyze image data and predict how smoke from wildfires will disburse. They've done analysis using historic data sets, and have some really cool Google Earth images. They have even posted the Streams source code
to Google. It involved some rather complex modeling and analytics, using the Local Ensemble Transform Kalman Filter (LETKF) as data assimilation algorithm.
So, another cool application for Streams, and reference story!
Hi ... new article posted today on DeveloperWorks, Integrate MATLAB code into InfoSphere Streams.Matlab®
, from MathWorks, is a high-level language and interactive environment that enables you to
perform computationally intensive tasks faster than with traditional
programming languages such as C, C++, and Fortran.
According to Wikipedia, Matlab is short for matrix laboratory and by 2004 had over 1 million users. The users perform all kinds of numeric computing functions.
These mathematical models and computations can now be easily incorporated into a Streams application, to promote re-use. When combined with Streaming analytics from Streams, a combined Streams/Matlab application has ability to process huge volumes of data at high velocity by distributing the applications across one or more nodes of the Streams runtime.
If you're a Matlab user, and need to scale out applications that use Streaming data, you should certainly investigate using Streams.
If you're a Streams user and want to do matrix math (common in Finance, Engineering, Scientific and Healthcare) you might want to consider Matlab!
Hi ... I'm haven't been very active here, but perhaps it's time to turn over a new leaf. Perhaps a new approach - I've added a calendar entry to take 30 minutes a week and add a short new topic.
In the last year, we've grown significantly in terms of number of people educated about Streams, customers, partners and universities with interesting applications and proof points. Telecommunications has been especially promising new industry, with 2 customers in production and another moving to production this month.
On Tuesday April 12th, we'll have an update on Streams capabilities.
I'll host two sessions to give you a a reasonable chance to attend in a convenient time zone (which might be Wednesday morning for you).
Tuesday April 12, 2011
12:00 noon to 1:00 PM Eastern Daylight Time
10:30 PM to 11:30 PM Eastern Daylight Time
1. Conference ID: 2058889
2. Link to the web conference:
For IBM employees:
People outside IBM:
If a Country has an AT&T Direct Number, the audio conference requires two-stage dialing. First,
dial the AT&T Direct Number. Second, dial the Toll-Free Dial-In Number.
Country Access Type AT&T Direct Number Dial-In Number Global Dialing Comment
Angola Toll-Free 808-000-011 888-426-6840
Argentina Toll-Free 0800-222-1512
Australia Caller Paid 0-2-80318490
Australia Toll-Free 1-800-85-4950
Austria Caller Paid 0-1-2530601
Austria Toll-Free 0800-07-0079
Bahamas Toll-Free 1-800-872-2881 888-426-6840
Belgium Caller Paid 0-2-8946410
Belgium Toll-Free 0800-3-9022
Brazil Toll-Free 0800-890-0288 888-426-6840 FROM BRAZIL
Brazil Toll-Free 0-800-888-8288 888-426-6840 BRAZIL OTHER
Brunei Darussalam Toll-Free 800-1111 888-426-6840
Bulgaria Toll-Free 00800-117-4514
Canada Toll-Free 888-426-6840
Cayman Islands Toll-Free 1-800-225-5288 888-426-6840
Chile Toll-Free 1230-020-0719
China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS
China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS
Colombia Toll-Free 01-800-5-1-81588
Costa Rica Toll-Free 0-800-011-4114 888-426-6840
Croatia Toll-Free 0800-777-954
Cyprus Toll-Free 8009-1373
Czech Republic Caller Paid 2-39016353
Czech Republic Toll-Free 800-143-484
Denmark Caller Paid 32711870
Denmark Toll-Free 80-717000
Dominican Republic Toll-Free 1-888-426-6840
Ecuador Toll-Free 1-999-119 888-426-6840
Egypt Toll-Free 2510-0200 888-426-6840 FROM CAIRO
Egypt Toll-Free 02-2510-0200 888-426-6840 OUTSIDE OF CAIRO
Finland Caller Paid 0-9-72519565
Finland Toll-Free 0800-9-18357
France Caller Paid 0-170911580
France Toll-Free 0800-94-0558
Germany Caller Paid 0-69-2443-2290
Germany Toll-Free 0800-000-1018
Ghana Toll-Free 0191 888-426-6840
Greece Toll-Free 00-800-11-006-7783
Hong Kong Caller Paid 30713844
Hong Kong Toll-Free 800-90-5535
Hungary Caller Paid 0-1-7789195
Hungary Toll-Free 06-800-19-306
Iceland Toll-Free 800-9823
India Toll-Free 000-117 888-426-6840
Indonesia Toll-Free 001-803-1-006-7783
Ireland Caller Paid 0-1-5264424
Ireland Toll-Free 1-800-943-427
Israel Toll-Free 1-809-417-783
Italy Caller Paid 0-2-00621263
Italy Toll-Free 800-975100
Jamaica Toll-Free 1-800-872-2881 888-426-6840
Japan Caller Paid 0-3-64042596
Japan Toll-Free 00531-11-0086 JAPAN KDD USERS
Japan Toll-Free 0066-33-830321 JAPAN C&W USERS
Japan Toll-Free 0034-800-900377 JAPAN NTT USERS
Kazakhstan Toll-Free 8^800-121-4321 888-426-6840
Korea (south) Caller Paid 0-2-348-31399
Korea (south) Toll-Free 00798-1-1-006-7783
Latvia Toll-Free 8000-3906
Lithuania Toll-Free 8-800-31299
Luxembourg Caller Paid 24871280
Luxembourg Toll-Free 800-2-7910
Macedonia Toll-Free 0-8000-4288 888-426-6840
Malaysia Caller Paid 03-6207-4471
Malaysia Toll-Free 1-800-81-5412
Mexico Toll-Free 001-888-426-6840
Monaco Toll-Free 800-93771
Morocco Toll-Free 00-211-0011 888-426-6840
Neth. Antilles Toll-Free 1-888-426-6840
Netherlands Caller Paid 0-20-7996800
Netherlands Toll-Free 0-800-363-6036
New Zealand Caller Paid 09-912-7548
New Zealand Toll-Free 0800-442905
Norway Caller Paid 23162149
Norway Toll-Free 800-16771
Pakistan Toll-Free 00-800-01-001 888-426-6840
Panama Toll-Free 001-888-426-6840
Peru Toll-Free 0800-77-942
Philippines Toll-Free 1-800-1-111-0558
Poland Toll-Free 0-0-800-111-1929
Portugal Toll-Free 800-7-80880
Romania Toll-Free 0-800-896910
Russia Toll-Free 8^495-363-2400 888-426-6840 RUSSIA OTHER
Russia Toll-Free 363-2400 888-426-6840 MOSCOW & ST PETERSBURG
Russia Toll-Free 8^10-800-110-1011 888-426-6840 OR RUSSIA OTHER
Senegal Toll-Free 800-103-072 888-426-6840
Singapore Caller Paid 66221594
Singapore Toll-Free 800-110-1842
Slovak Republic Toll-Free 0-800-000-101 888-426-6840
Slovenia Toll-Free 0800-80768
South Africa Toll-Free 0800-983-687
Spain Caller Paid 9-1-7878580
Spain Toll-Free 900-8-01334
Spain Toll-Free 900-8-01334
Sweden Toll-Free 0200-12-5807
Switzerland Caller Paid 0-43-2101240
Switzerland Toll-Free 0800-89-4800
Taiwan Caller Paid 0-2-77457238
Taiwan Toll-Free 00801-10-4334
Thailand Toll-Free 001-800-11-006-7783
Trinidad & Tobago Toll-Free 1-800-872-2881 888-426-6840
Turkey Toll-Free 0811-288-0001 888-426-6840
Ukraine Toll-Free 8^100-11 888-426-6840
United Arab Emirates Toll-Free 0-800-121 888-426-6840
United Arab Emirates Toll-Free 0-800-151 888-426-6840 U.S. MILITARY BASE
United Kingdom Caller Paid 0-20-30596451
United Kingdom Toll-Free 0800-368-0638
Uruguay Toll-Free 000-411-006-7783
USA Caller Paid 215-861-6239
USA Toll-Free 888-426-6840
Uzbekistan Toll-Free 8^641-744-0010 888-426-6840
Venezuela Toll-Free 0800-102-9535
Vietnam Toll-Free 1-201-0288 888-426-6840
Hope to hear from you next week!
The telco case study was presented today at The Data Warehouse Institute conference in Las Vegas, Nevada. The audience supported the premise that their data warehouses and data marts rarely provide results within hours of data being generated. Usually it's daily or weekly reports.
So for organzations that can benefit from notification of anomalous behaviour in seconds, the new Streams Mining Toolkit can deliver the results that can help their business.
Customers stopped by the IBM Booth at the showcase to ask about extreme low latency and also about Predictive Modelling and use of Predictive Model Markup Language.
One analst quickly grokked the significance of Streams - your challenge, he explained looking at the torrents of data slide, is to explain to the market why you actually implement a parallel archcitecture and how that is different from related technologies like CEP that merely do some functions in parallel. A truly parallel architecture like Streams offers delivers a more scalable runtime to meet the most demanding applications.
The telco pilot presented in the Case Study hits a peak rate of processing about 500,000 Call Detail Records per second on only 15 blades. Using the new InfiniBand support in this release, the options trading application was re-done at rates up to about 12.5 million messages per second.
The new release of Streams announced Tuesday looks to deliver significant new function to customers and expand the use cases for Streams to anyone that is doing data mining using PMML models, and can benefit from getting results in seconds instead of in days.
Roger Rea, IBM
I'm in beautiful, snowy Stockholm this week where KTH (Royal Technical University) and the Swedish Road Administration held a seminar on Smarter Traffic for smart cities. Not only is it great to be here, but it's hilarious to see my name throughout the city. rea means 'sale' in Swedish, so my name is posted in stores everywhere!
Stockholm has long been a leader in providing information to their citizens on traffic options, and implemented with IBM a traffic congestion system. Taking pictures of license plates to charge for entering the core city during certain hours has limited traffic congestion and reduced pollution.
Professor Haris has called this a wonderfully elegant system, but says it's essentially a batch billing system. His vision is to collect data from taxis, city trucks, traffic light loop induction sensors, weather, subway and more to provide a truly intelligent transportation system.
Last fall, Prof Haris and his team with IBM demonstrated collecting data from GPS devices, stored in data files and played back in realtime, then mapped to google maps to show average speeds on various road segments. With this success, the Stockholm Road Administration opened up their taxi cab information to KTH. Since mid December, they have been monitoring nearly 1500 taxis with GPS information and source destination information to provide realtime speed data and estimated arrival time information. Additional data sources will also be forthcoming.
One eventual goal is to have a cell phone applications so people could text a destination, and have the application reply with options - 34 minutes via taxi and 22 minutes via subway. Thus, in realtime, changing behaviour to better balance the city traffic and transportation.
The seminar was in part to thank IBM for a Shared University Research grant where in addition to delivering InfoSphere Streams software, IBM provided 10 blade servers to continue the Transportation education and research at KTH