IBM Big Data

with Tags: analytics

IBM Big Data - Where do I start?

mrmadira Tags: hadoop analytics big insights starting data up text jaql bigindex bigsheets 19,757 Views

BigInsights 1.3 released. What's new? Enhancement to Performance, Manageability, Consumability and Integration capabilities. Where do you start?

If you are new to BigData concepts you can start with this
1. http://www.ibm.com/bigdata - Quick introduction to Big Data. Reading time - 5 minutes
2. http://www-01.ibm.com/software/data/bigdata/enterprise.html - Give you an overview of the two products in IBM Big Data - InfoSphere Streams and InfoSphere BigInsights - Reading time 10 minutes
3. http://bigdatauniversity.com - This contains an excellent Certification Course for Hadoop Fundamentals - and has a good coverage on the open source foundational components such as Hadoop, MapReduce concepts, Pig, Hive, Flume JAQL etc. There are videos, hands-on downloadable VM, lab exercises, reading material etc. The bible of Hadoop and MapReduce reference pdf book is available for download. If your expertise so far has been one line summary of each of the technolgies mentioned above, you will need to spend about 3 to 4 days to cover this course, reading time + exercises. There's a test that you can appear for at the end of the course and yes, you get a certificate if you clear it. Reading up and clearing certification time 4 to 5 days

Videos on Big Data

1. http://www.youtube.com/user/ibmbigdata - Has around 14 demo videos including the following. 1) Twitter Sentiment Analysis 2) Email analytics 3) Telcom - CDRs and Call drop use case 4) Big Sheets 5) Text Analytics tutorials
2. http://www.youtube.com/watch?v=ibPcPUwkVE4 - Here's another awesome introductions to BigData. Each sentence in each slide explained.

Head-On into BigInsights

1. http://www-01.ibm.com/software/data/infosphere/biginsights/ - Get datasheets, webinar on how you can leverage cloud, VM with BigInsights and Karmasphere studio for download.
2. http://www.ibm.com/developerworks/data/library/techarticle/dm-1110biginsightsintro/ - Quick introduction to the architecture of IBM Big Data with a preview of what you can expect when you get deeper, Though it is based on 1.2, it still is a good starting point. Also explains the difference between the basic edition and enterprise edition. Reading time - 1 hour

A little deeper into BigInsights
1. http://publib.boulder.ibm.com/infocenter/bigins/v1r3/index.jsp - Get more deep into installation, functions and features of each of the components of IBM BigInsights - BigSheets, BigIndex, JAQL, Text Analytics etc
2. http://www-01.ibm.com/software/data/infosphere/biginsights/basic.html - Get your hands dirty - download the Basic Edition to try it out.

Advanced Topics on Big Insights
1. http://www.ibm.com/developerworks/cloud/library/cl-cloudhadoop/ - Cloud and Big Insights - Tutorial on how IBM Smart Cloud can be used to deploy BigInsights Reading time - 1 hour

Get Connected
1. http://www.ibm.com/developerworks/wiki/biginsights - Wiki
2. http://www.ibm.com/developerworks/forums/forum.jspa?forumID=2409 - Forum
3. https://www.facebook.com/#!/IBMbigdata - Facebook
4. https://twitter.com/#!/IBMbigdata - Twitter

We'll keep editing this page as I get more starting points in these categaroies. Want to see more info here? Comment on this blog and let me know.

Building Extractors using InfoSphere™ BigInsights Text Analytics in Eclipse

NisanthSimon Tags: biginsights analytics analysis aql infosphere text big articles data hadoop 2 Comments 11,565 Views

This blog provides an overview on

1) Text Analytics System
2) How to setup the Text Analytics System in Eclipse.
3) How to build the extractors to extract structured information from unstructured or semistructured text.
4) How to run and cross verify the extracted results in Eclipse.

5) How to extract the AOG from the Eclipse project.

Pre-Requsite

Install IBM InfoSphere BigInsights v1.2
Eclipse Version 3.4 or 3.6 is required

1) Text Analytics system

Many enterprises maintain large amount of unstructured data like emails, logs, call-center records, wikis, blogs etc. There is an increasing need for understanding this unstructured data and provide better insight. The Text Analytics system provides scalability and ease-of-use in developing extractors that extract structured information from unstructured data. Our information extraction system is built around Annotation Query Language (AQL), a declarative rule language with a familiar SQL-like syntax.

Refer http://publib.boulder.ibm.com/infocenter/bigins/v1r2/index.jsp?topic=%2Fcom.ibm.swg.im.infosphere.biginsights.doc%2Fdoc%2Fbiginsights_aqlref_con_aql-overview.html for various constructs available in AQL.

2) How to setup the Text Analytics System in Eclipse
a) Start BigInsight and open BigInsight console using http://localhost:8080/BigInsights/console/NodeAdministration.jsp

b) Click on Eclipse Tool Tab on top-right side.

c) Open Eclipse and Install new software and provide the above plug-in repository URL.

3) How to build the extractors to extract structured information from unstructured or semistructured text.

Follow below steps to build annotators using Eclipse.

a) Open the Eclipse workspace. Go to File --> New --> others --> BigInsights --> BigInsights Project and create a new project.

b) To create an Annotator file. Right click on the project created and new --> BigInsights --> AQL file. Provide an AQL file name. You can write AQL in it.

4) How to run and cross verify the extracted results in Eclipse

a) Open the Project properties and update the main AQL file. You can add Dictionaries, UDF and other dependent jars in data path.

b) To execute the AQL, right click on Main AQL file and open run configuration. Update the Input collection field.

c) You can see the extracted results under Annotation Explorer. Here the column Span Attribute Value will have the extracted text and column Span Attribute Name refers to View Name.Attribute name.

There are various options to filter the results. Here I am filtering the results only for view NameCandidate.

You can see the provenance, by double clicking the 'Explain' in NameCandiate tab. Provenance provides backtracking on the views to understand which which views are responsible for the extracted result.

To cross-verify with the Input Corpus, click on the fileName in Input Document column under Annotation Explorer.Here, select the attributes and the value for that attribute will be highlighted in the input corpus.

5) How to extract the AOG from the Eclipse project.

AOG is the compiled plan of the extractor. You can extract the AOG by right clicking the project and Export --> Text Analytics.

Now you generted the AOG. You can deploy the AOG in BigInsight and test for larger Corpus. I will be covering - How to run the AOG from JAQL in my next blog.

Harnessing Real Time Predictive Analytics - Part 1

MohanDani Tags: time stream processing analytics big ibm streams real and spss infosphere data 5,594 Views

We speak about Big Data and need for Real Time Analytics etc. Can we really infer and make decisions if we dont have the knowledge to interpret data?, Speed to respond without content is a lost opportunity and nothing else. Hence,Harnessing knowledge which is empirical and making instant decisions which is accurate was WISHFUL thinking in the past. Streams and SPSS products come together to render this solution seamlessly.

IBM SPSS Modeler provides a state of the art environment for understanding data and producing predictive models. InfoSphere Streams provides a scalable high performance environment for real time analysis of data in motion including traditional structured or semi-structured data to unstructured data types. Some applications have a need for deep analytics against stored information and low latency high volume real time use of those analytics to provide scoring.

Lets Take a look at some of the Actors in this story:

Data Analyst – a modeling expert that knows how to use the IBM SPSS Modeler tools to build and publish predictive models.

Streams Application Developer – an InfoSphere Streams developer responsible for building applications.

Lets hear the Plot of the Movie along with Actors: The problem is in a retail scenario of Online shopping, A user browses through various products and chooses a few products into his basket to check out. For example: If a person purchases Tennis Racket we can present him with various types of tennis ball and highlight the training ball,also if we knew that this person is a championship player and consumes high quality tennis balls. that can be offered to him rather than training balls. This leveraging of past known transaction pattern analysis and predictive next step solves this problem efficiently.

We will discuss the Actors roles in part of the future movie. More to come in Part -2 where we see actors working their roles where a discussion ensues about when to apply what kind of algorithm and how to use it in real time.Some of the hilarious failures from these learning models and how they eventually mature the sales process in the Online retail store called "FREEFORALL" Retail Chain.

Feed for Blog Entries ▼

Blogs

IBM Big Data

▼ Tags

▼ Similar Blogs

developerWork...

Thoughts from...

IBM Redbooks ...

Keep on Learn...

Blue Blog

▼ Similar Ideation Blogs

IBM PureData-...

IBM Tunisia H...

Open Ideas

IBM Forms Exp...

▼ Archive

▼ Blog Authors

IBM Big Data

IBM Big Data - Where do I start?

Building Extractors using InfoSphere™ BigInsights Text Analytics in Eclipse

Harnessing Real Time Predictive Analytics - Part 1