Q6 What types of insight are orgs getting from mobile generated bigdata?
craigmullins A6: Proximity and buying patterns can be used to target potential customersNatasha_D_G Agreed esp when u can connect w/ historical RT @BigDataAlex: So many problems require real-time analytics as an initial layer.
jeffreyfkelly A6 sentiment, engagement patterns, geo-locations patterns
cristianmolaro A6: by analyzing big amounts of data, users are discovering cause - effect correlations that they were not aware of
Betharonoff @TheSocialPitt From Shakespeare to McD's! BigData covers the gamut
craigmullins A6: With GPS/sensors in mobile devices, when u walk past your favorite coffee shop your device can alert u to the daily special
cristianmolaro: 90% of data unstructured... 10% of data structured; this 10% matters the most
mjcavaretta Much value in unstructured to structured.. MT @cristianmolaro: 90% of data unstructured... 10% of data structured; this 10% matters the most
TheSocialPitt A6 Or is it deriving meaning from that 90%? @cristianmolaro
BigDataAlex I think it is deriving meaning for the 90% - whether it is structured or unstructured.
BigDataAlex A6:they are getting a wealth of location information, proximity, distance, rate, clustering polygons-the where of consumerism.
A6: Many businesses are gleaning social & customer sentiment,
customer feedback, location data, etc from mobile generated bigdata
craigmullins A6: When offering up mobile bigdata it is important not to get overwhelmed by the volume of the data that is available
cristianmolaro A6: Bigdata sitting in a enormous hard-disk worst nothing if you cannot get insight from it
cristianmolaro A6: a practical example: how weather conditions in Brussels can impact the car accident rate 40 KM south of the city?
BigDataAlex A6:Combine mobility and immersive visualization and you have created a new experience layer.
cristianmolaro A6: today's technology makes possible what we call bigdata... but the question is how do you prepare to get advantage of it
furrier A6: the big issue for mobile generated data is learning about user experience patterns
Natasha_D_G Becomes a junk yard RT @cristianmolaro: A6: bigdata in enormous hard-disk worth nothing w/o insight from it
craigmullins A6: Avoid push technologies that inundate users with a ton of unwanted data
craigmullins A6: Instead use tools that provide filtering to enable users to get only the information they need to make business decisions
furrier A6: IBM gets the notion of the "learning machine". It's true here with mobile data-loop in the data & loop back value
jeffreyfkelly A6 can glean insight on buying patterns - mobile v. online v. brick-and-mortar
Empirix A6 M2M insights can be discovered that can help insurance orgs and in customizing the user experience
AllanKoivo A6: or increasing % that matters
TheSocialPitt Ex: http://t.co/AZTwWUpjMQ RT @craigmullins: A6: W/ GPS/sensors when u walk past fave coffee shop device can alert u special
Natasha_D_G Gold mine 4 #CX personalization RT @furrier: A6: big issue 4 mobile generated data = learning abt #UX patterns
jameskobielus A6: deep machine-data analytics insights on geo-localization, sentiment, behavior, and other signals sourced from mobiles
cristianmolaro A6: geo-localization data correlated with behavior patterns adds a complete new dimension to the way we approach data
furrier @jameskobielus right on geo is key
Empirix A6 you can also learn how apps are used & how to improve service delivery
jbondre @Natasha_D_G @craigmullins True, but it is intrusive. Marketing on demand is better, then marketing as a intrusion.
jameskobielus A6: mobile-sourced bigdata fleshes out the "720-degree customer view" (external behavior + internal experiences)
TheSocialPitt A6 #Watson is using data from mobile devices to help doctors diagnose, treat patients better, faster.
Natasha_D_G @jbondre @craigmullins Not intrusive if you request it in settings though
cristianmolaro A6: you carry a small almost-supercomputer with a gps in your pocket every day... think about the potential...
jbondre @Natasha_D_G @craigmullins PPL looking for coffee will be way more receptive to the msg. Pushing Msgs is like Minority report.
motohero Nods in agreement "@Empirix: A6 you can also learn how apps are used & how to improve service delivery
Natasha_D_G Data can tell if they were doing a look up RT @jbondre: @craigmullins PPL looking for coffee = more receptive to the msg.
IBMbigdata Push v pull RT @jbondre PPL looking for coffee will be way more receptive to the msg. Pushing Msgs is like Minority report.
craigmullins @jbondre True, I don't want my smartphone buzzing when I pass every store... just my faves
cristianmolaro A6: one of the biggest challenges that mobile bigdata comes with is organizations realizing the opportunities behind
furrier @edd Dumbill lays it out http://t.co/33GrnKlXLk great watch
jbondre @Natasha_D_G @craigmullins good idea for 1, turns into good idea for everyone, becomes noise. bigdatamgmt CX is about the consumers wants
motohero @IBMbigdata @cristianmolaro privacy.concerns :-)
Natasha_D_G Done right can be successful MT @craigmullins: @jbondre True, I don't want my smartphone at every store... just my faves
jameskobielus A6: mobile-gen data + data sourced from all other channels = fodder 4 dynamic multi-channel experience optimization
Betharonoff Can u get metadata on how Bigdata is used? Whether users like push v. pull, intrusion v. on-demand in yr app?
AllanKoivo Was thinking same thing RT @IBMbigdata Push v pull RT @jbondre PPL looking for coffee will be way more receptive to the msg.
jeffreyfkelly More here http://t.co/XdwvGburDV RT @TheSocialPitt A6 Watson using data from #mobile devices to help diagnose, treat patients
motohero gotta feed, cheeeeeze! thanks for the great insight folks
mjcavaretta Value in internal analytics. MT @motohero: Nods in agreement @Empirix: A6 learn how apps are used to improve service delivery
furrier Economist editor Ken Cukier @kncukier has amazing important book Big Data Revolution http://t.co/m2OXdmeCsb
cristianmolaro Actually mobile bigdata is a mass grid of interconnected devices... do you internet mobile?
jameskobielus A6: wearable & implanted mobile devices will deliver unparalleled insights into wellness, health, & experience
TheSocialPitt Lots of talk about sandwiches, cheese, coffee on #bigdatamgmt. The downside of Twitterchat over lunch time.
craigmullins But I want certain vendors to shout for me to come to them!
craigmullins Certain = opt-in (or some other verification mechanism)
Natasha_D_G Hilarious! RT @craigmullins: But I want certain vendors to shout for me to come to them!
jbondre @craigmullins 4sqr model is nice. Oh, you're near X, did you think about trying Y? Only at interaction is there a suggestion
jeffreyfkelly yes, based on my (your) interests RT @craigmullins: But I want certain vendors to shout for me to come to them!
craigmullins Is the smartphone constantly buzzing the "rubber biscuit" (since we already mentioned the wish sandwich)
TheSocialPitt Bow bow bow! RT @craigmullins: Is smartphone constantly buzzing the "rubber biscuit" (since we mentioned the wish sandwich)
craigmullins I won't sacrifice battery life for auto check in on 4Square!
jbondre @craigmullins In a year, when phones are more efficient, and batteries better, this type of location push is very feasible.
craigmullins @jbondre I still wish 4Square was more automatic. RFID to auto check in?
jbondre @craigmullins Totally possible, RFID not needed. Look up Sonar. The issue is constant GPS pinging drains battery.
CrystaAnderson @craigmullins @jbondre Glad 4Square not automatic - want to retain more #privacy control
craigmullins @CrystaAnderson Makes me wonder where you've been! ;-)
AllanKoivo & customer backlash @craigmullins Need some form of opt-in or we run up against privacy concerns, no?
CrystaAnderson Amen! RT @craigmullins: I won't sacrifice battery life for auto check in on 4Square!
jbondre @TheSocialPitt Agreed. I use 4sqr, but I do not push the feed to my FB or Twitter unless I have something to say.
craigmullins @AllanKoivo backlash if the customer even knows about it
craigmullins A6 Need some form of opt-in or we run up against privacy concerns, no?
jbondre @Natasha_D_G @craigmullins An opt-in might work. Still, its better to be there for the consumer, not shout to come to you.
jameskobielus A6: vehicle-sourced mobile data insights will help traffic planners dynamically optimize world transportation grids
TheSocialPitt Already happening in EU RT @jameskobielus: vehicle-sourced mobile data insight will help traffic planners optimize transportation
Last one! Q7 When should you tap into smartphones, mobile clients as sources for bigdata apps?
Natasha_D_G A7: Yesterday was too late to tap into smartphones & mobile clients as sources for #bigdata!!
jeffreyfkelly A7 whenever such data would add value to analytics and resulting insights
BigDataAlex A7:Mobility will drive the Internet of Things, connecting us to billions of sensors measuring our world.
jeffreyfkelly A7 mobile devices are akin to tracking devices in your pocket - back to the privacy issue, not everyone realizes this
BigDataAlex A7: What to extract, what do I pursue, how does this improve our lives? If I have a million sensors can I predict the weather?
cristianmolaro A7: insurance companies could correlate geo-localization data with weather and traffic conditions to draw car accident patterns
BigDataAlex A7: what do we measure, why, and how do we extract the metadata from the edge and make it actionable, valuable?
Always. Smartphones becoming the most ubiquitous, valuable source of
ambient, geo, sentiment, & experience data; traffic conditions
Empirix A7 As mobile devices continue to proliferate it becomes a critical necessity to tap into the resulting bigdata available
Empirix A7 At some point the not so distant future using BigData from mobile devices will become a need to have not nice to have
cristianmolaro: A7: anyone with a mobile device is almost a bigdata walking sensor today
jeffreyfkelly i'd take out the word "almost"
TheSocialPitt Indeed. Human machines
craigmullins Nice image
Natasha_D_G Nicely put
Another great tweetchat on the hashtag #bigdatamgmt! Here are the results:625 tweets, 157 users, 153,849 reach, 1,849,624 impressions
This is the recap of the Mobile Data tweetchat for questions 4 and 5. See also Recap of Tweetchat: "Mobile Data: Taking Your Big Data On the Road" - Part 1
Q4 Does mobile big data require different retention policies?
BigDataAlex A4:Retention for mobility is a key factor: do we really need to remember every location we have been - It links back to privacy
craigmullins A4: Data is retained for internal + external reasons: internal because it is needed for biz – external because laws demand it.
A4: You may choose to retain more data for mobile bigdatamgmt but be
careful because data once retained is discoverable during court trials
BigDataAlex A4: The question is who is doing the retaining of the data....
jeffreyfkelly A4 mobile data growing exponentially, so how much is retained and for how long gets tricky
craigmullins A4 Bigdata does not necessarily change retention reqmts but it complicates the issue
BigDataAlex A4: Mobile data is the new oil field.
Natasha_D_G Useless unless we mine & leverage it correctly RT @BigDataAlex: A4: Mobile data is the new oil field.
cristianmolaro A4: maybe not: retention should be a business variable not a access device matter
craigmullins A4: There are over 150 different regulations (at local, state, national, and international levels) that impact data retention.
furrier A4: yes it does - to save or not to save that is the question legal issue
IBMbigdata Whether tis nobler to save or not... RT @furrier: A4: yes it does - to save or not to save that is the question #legalissue
cristianmolaro A4: is not the amount of data or the older that it is that matters the most, but what you can get from it... insight!
craigmullins A4 #bigdata can be more costly to retain for long periods simply due to its massive volume
Dmattcarter can hadoop help? RT @craigmullins: A4 #bigdata can be more costly to retain for long periods simply due to its massive volume
BigDataAlex A4: Some folks save it all - some say save what you end - Big Data is about figuring out the difference.
Empirix A4 Different groups within an org might need the data for longer, to make informed decisions based on cust history
jameskobielus A4: Mobile doesn't impact retention policies at server--but should @ client. Keep sensitive info on device limited
jeffreyfkelly #BigData Shakespeare RT @furrier: A4: yes it does - #mobile data: to save or not to save that is the question
IBMbigdata RT @Empirix: A5 Your biz model & regs define retention policies- becomes more of a ? of do you have the right tech to hold it
TheSocialPitt Quoting Shakespeare on #bigdatamgmt chat! Well done!
cristianmolaro A4: there is a trend to increase data retention
Empirix @cristianmolaro Also need the right tools, questions, and people to get you there
furrier As navigation promotion here are all the videos from IOD http://t.co/tugNzS4MGE #bigdata
zacharyjeans Cold storage data solutions like @ironmountain will have more and more play in big data management
AllanKoivo A4: Key is HOW the data is retained. Policies always need to be dynamic to reflect new technologies
craigmullins @Dmattcarter Not sure how/if Hadoop would decrease storage costs?
cristianmolaro A4: not all bigdata have to be saved for later... think about live digital video recording... it leaves just for a short while
motohero Drill ready? Mine is! "@BigDataAlex: A4: Mobile data is the new oil field.
craigmullins A4 Over large spans of time technology becomes the problem, not the solution, for data retention
Natasha_D_G How so? RT @craigmullins: A4 Over large spans of time technology becomes the problem, not the solution, for data retention
craigmullins A4 Consider retaining data back in the 1960s. Could've been on punched cards. Can you even read them today?
BigDataAlex A4:is there enough power to run or water to cool the data we retain?
jeffreyfkelly A4 yes @cristianmolaro, w #Hadoop, storage constraints lifted - but retention policy still an issue and, while cheap, not FREE
cristianmolaro and what about being able to read today's bigdata in 20 years from now?
Betharonoff @AllanKoivo They've started testing digital data stored in DNA as coding system
motohero @craigmullins @Dmattcarter IBM is the oldest masters of storage, maybe they can provide insight about what they are creating
craigmullins A4 The longer you retain data the more likely the tech used to retain it has become obsolete
furrier A4: this is a good video to watch on compliance data issues http://t.co/crGpu821Sv
jameskobielus A4 Mobile #bigdata makes imperative 2 have enterprise rights mgmt policies 2 prevent multi-device over-retain/leak
Q5 Which type of #bigdata platform (#Hadoop, Streams, etc.) is best for #mobile clients?
A5 Your business model & regs define your retention policies - it
becomes more of a question of do you have the right tech to hold it
BigDataAlex A5 Streams!
A5: this has too many dimensions: all areas are affected - no one
technology wins; ease of ingest & extraction are key to this
IBMbigdata Should have known Alex would vote for InfoSphere Streams
cristianmolaro A5: any platform that would allow users to get information from bigdata
jeffreyfkelly A5 real-time key to mobile BigData - must get insights to mobile workers at the right time to take action
BigDataAlex A5: Big Telcos are trying to figure how to parse 700,000 calls per second and deal with dropped call management and service.
craigmullins A5: I’d look for customizable reports, dashboards, + graphs that users can adapt to their mobile preferences + needs
craigmullins A5: I think the presentation + interface on the mobile device is more important than Hadoop, etc.
BigDataAlex A5: Streaming analytics allows customers to keep up with the volume and interoperate with Hadoop for non-real-time analytics.
jameskobielus A5: No particular back-end bigdata platform preferred 4 mobile. Need front-end mobile access infrastructure agnostic to all
furrier @craigmullins I totally agree on the easy of use I would add reduce the steps to get extraction + insights
craigmullins A5: Maybe innovative interaction support (gesture, voice, etc.) too
jeffreyfkelly A5 definitely streaming, CEP-style tech to allow automated actions - must take action while customer still engaged
cristianmolaro A5: sometimes bigdata users are like people having a solution looking for a problem to solve...
AllanKoivo A5: not all mobile clients are the same- it depends on the objective of the client. There is no one size fits all solution
Natasha_D_G Usability, scalability all key RT @furrier: @craigmullins ease of use I would add reduce the steps to get extraction + insights
motohero A5 nosql on the bottom, webtech (pig, R) on top
furrier A5: many uses cases hadoop is great but what does real time mean?
Empirix A5 Predictive Analytics capabilities are important for keeping a competitive edge
craigmullins Like most new tech, marketers co-opt it! MT @cristianmolaro: A5: Bigdata users are ... looking for a problem to solve...
cristianmolaro A5: the one able to provide fast access to organization's insight with easy of operation...
BigDataAlex A5: Streams means real-time which means millions in the Telco space, esp., in Eurozone.
mjcavaretta BigData opportunities across the entire space of high-volume, high-value data generation.
Empirix @cristianmolaro Yes - you need the right questions first or there's no direction or focus
Natasha_D_G Should start w/ biz prob RT @cristianmolaro: A5: sometimes bigdata users =people having solution looking 4 problem to solve
IBMbigdata Is "real time" changing? RT @furrier: A5: many uses cases hadoop is great but what does real time mean?
furrier @motohero not all nosql but mostly yes SQL is a easy way to extract ontop of semi structured data- agree on above db layer
jeffreyfkelly how about in-time? RT @furrier: A5: many uses cases hadoop is great but what does real time mean?
cristianmolaro A5: Bigdata is everywhere... in any organization... you just have to discover how to leverage it
BigDataAlex Real-time at network line speed - 1, 10, and 100 GigE speeds - analytics in process, in flow - compute memory.
jameskobielus A5: In-memory #bigdata clients/servers with back-end streaming best for real-time mobile
motohero @furrier real-time is the here and now model of customer(for exmpl) that a decision can be derived from
Empirix Need the right tools to extract meaning @cristianmolaro: A5: bigdata is everywhere in an org, have to discover how to leverage it
AllanKoivo & other industries as well RT @BigDataAlex: A5: #Streams = real-time which means millions in the Telco space
motohero @furrier my vision is that SQL could be the cheese in the bigdatamgmt sandwich - that middle layer that can be exploited also
jameskobielus A5: Need lower-latency in-motion bigdata platforms closer to mobile client, batch & "data-at-rest" further
BigDataAlex So many problems require real-time analytics as an initial distillation process or layer.
furrier @motohero awesome! except some are making a "wish sandwich" - customer wish they had some "meat" :-)
jeffreyfkelly suddenly I'm hungry for lunch
TheSocialPitt Super-size me RT @motohero: @furrier my vision is SQL cld be cheese in bigdatamgmt sandwich - that middle layer that can be exploited also
jameskobielus A5: The mobile bigdata environment should have SQL-query-virtualization front-end to simplify access
Another great tweetchat on the hashtag #bigdatamgmt! Here are the results:625 tweets, 157 users, 153,849 reach, 1,849,624 impressions
And.. we expect traffic for the rest of the day, so these stats may increase!
again, we had an active group of experts leading the discussion and
were very pleased at the responses from other attendees. Here is a
summary of the discussion:
Q1 Which enterprise initiatives require mobile access to bigdata?
craigmullins A1: Marketing and sales initiatives are ripe areas for mobile bigdata support
craigmullins A1: But they are not the only ones: retail, healthcare, finance, manufacturing, etc. etc.
BigDataAlex A1:advertising, marketing, field services, law enforcement, land management, infrastructure management, all require mobility
cristianmolaro A1: almost any enterprise initiative could benefit from bigdata in a "small" mobile package...
craigmullins A1: @cristianmolaro ...not just from but also to the portable bigdata consumer
susvis A1 - Customer support seems like a must have on mobile
Natasha_D_G A1: Sales & service depts. need customer insight at right time, at right place which may be on road
jeffreyfkelly A1 don't forget social & mobile closely linked - consumer creating data via social apps on mobile devices - worlds collide!
TheSocialPitt A1 Some automakers are doing interesting things with mobile bigdata
cristianmolaro A1: the challenge is to get useful information from a "portable" bigdata mobile source of information
jameskobielus A1: Any enterprise decision automation initiative supporting mobile employees with analytic-powered apps bigdata
craigmullins A1: Avoiding a siloed implementation of mobile bigdata + cloud is important
jeffreyfkelly A1 failure to take mobile data into account in marketing, you're missing possibly most important data related to consumer
furrier consumer hyperscale ex: Facebook =10.8% of all time spent on the Internet; 150 million+ desktop visitors, 97 million mobile
Natasha_D_G A1: Smart orgs empower their mobile customer facing teams w/ data access bigdata
craigmullins: A1: If you think about it, mobile bigdata + cloud are all part of the same problem (…and solution)
BigDataAlex A1: Yes, CraigMullins - see massive convergence between big data, mobility and cloud - could not agree enough.
furrier A1: Most IT pros we talk to call their projects "Mobile Desktop" as hyperscale come to IT mobile desktop will be 50/50 usage
BigDataAlex A1: combine mobility with location analytics - where you are is changing how we buy, how we work, and how we play
jeffreyfkelly A1 marketing for sure - consumers are mobile and so must marketing campaigns
jameskobielus @IBMbigdata A1: any app that aggregates & analyzes "ambient" data from smartphones & uses it to optimize mobile experience
BigDataAlex A1: the tablet will replace the PC in short order....
IBMbigdata Bold prediction? RT @BigDataAlex: A1: the tablet will replace the PC in short order....
Natasha_D_G I believe it! RT @BigDataAlex: A1: the tablet will replace the PC in short order....
AllanKoivo Agree, laptop sales will be impacted as well. RT @BigDataAlex: A1: the tablet will replace the PC in short order....
furrier A1: the mobile desktop is very hot; meaning moving desktop functionality to smartphone -software is key to this
craigmullins Outsell? Yes. Replace? Not so fast. --> @BigDataAlex: A1: the tablet will replace the PC in short order....
BigDataAlex Ok - outsell. But I will argue replacement is not far off for those under 20.
BigDataAlex A1:real estate is huge
furrier VMware is talking Big Fast Data in today's Pivotal Initiative spin out; IBM is already there; future is "learning data" imo
cristianmolaro A1: I can bring more computation power in my mobile device today than what was used for getting the man to the moon
zacharyjeans A1: The hydrogen bomb was created with roughly 5K of computing power.
doug_laney A1. Mobile = comparatively restricted platform for info delivery, but entirely new source of data collection.
TheSocialPitt @doug_laney Mobile is definitely extremely valuable as a bigdata source. You have any reports on this?
IBMbigdata MT @doug_laney: A1. Gartner: Mobile *access* to BigData a lot less interesting or valuable than as a source of bigdata.
Natasha_D_G Say more? RT @doug_laney: A1. Gartner believes Mobile *access* to BigData a lot less valuable than as a source of bigdata.
Q2 Which users, developers, stakeholders require mobile access to bigdata?
Natasha_D_G A2: In an always connected world of instant, we’re quickly moving to ALL needing mobile access to bigdata
BigDataAlex A2: Everyone is working toward access to mobile data - people want their data, all the time, everywhere, all the time.
craigmullins A2: There are 3.2 billion mobile subscribers today, which is almost half of the world's population.
Natasha_D_G Everyone! RT @craigmullins A2 Folks travel, work remotely, or need flexibility can most benefit from mobile access to bigdata
furrier A2: issue with mobile enterprise is security access with security top issue
Texiwill @furrier I think it is more about proving identity of the mobile user not the security access of the mobile device.
Natasha_D_G #CXO tribe in full force! RT @Empirix: A2 All who want to optimize the programs they are working need access to bigdata
cristianmolaro A2: who could not benefit from mobile access to bigdata? That could be easier to answer...
IBMbigdata Who wouldn't? RT @cristianmolaro: A2: who could not benefit from mobile access to bigdata? That could be easier to answer...
BigDataAlex A2: Big Data is being driven in large part by the explosion of mobile content and location information
Cristianmolaro A2: mobile + bigdata + global seem to go together very well nowadays
BigDataAlex A2: Verizon predicts that data will replace voice by 2015 in terms of data pipe traffic
Empirix A2 All who want to optimize the programs they are working need access to bigdata - and tools to help them utilize the data
jameskobielus A2: anybody who requires mobile access to decision support/automation powered by massive multi-structured data sets
jeffreyfkelly A2 mobile access to BigData doesn't need to be a full functional analytics but insights based on situation, location
Natasha_D_G Relevance RT @jeffreyfkelly: A2 mobile access to BigData doesn't need fully functional analytics but insights based on situation
IBMbigdata Must define RT @jeffreyfkelly: A2 mobile access to BigData doesn’t need full analytics but insights based on situation, ltn
jameskobielus A2: any data scientist or SME developing, collaboration, or tweaking statistical models from wherever
Q3 What security privacy risks of providing mobile access to big data apps?
craigmullins A3: Any time data flows over a network security + privacy must be managed
Natasha_D_G Indeed RT @craigmullins: A3: Any time data flows over a network security + privacy must be managed
furrier if hyperscale comes fast then mobile will displace faster the pacing item is enterprise hyperscale- scale out
Natasha_D_G A3: Orgs must be cognizant that mobile data access subject to same security risks of standard access
cristianmolaro A3: security is a major concern for many organizations. The technology to provide secure mobile access is already there
BigData as source creates new data governance challenges - trust models
for data gen'd from mobile devices & people doing so.
BigDataAlex A3: Mobility security and privacy are huge - as we become more transparent we have to extend technical and ethical guidelines.
Natasha_D_G A3: Compliance nightmare > mobile access adds dimension of stolen/lost device & all org data in wrong hands
TheSocialPitt A3 I pity the IT guys dealing with BYOD!
Empirix Agree - but it's a reality that must be faced @TheSocialPitt: A3 I pity the IT guys dealing with #BYOD!
TheSocialPitt Absolutely. BYOD is here to stay, thankfully RT @empirix: Agree - but it's a reality that must be faced
cristianmolaro A3: mobile can be a security exposure if you do not take care of protecting what matters: your data!
zacharyjeans RememberBigData as we know it today, in #BigDataMgmt, will seem like small data, in a decade.
jeffreyfkelly @zacharyjeans very true - BigData is relative, need flexible platforms that scale with the times
TheSocialPitt That long?! ;-) RT @zacharyjeans: Remember BigData as we know it today, in #BigDataMgmt, will seem like small data, in a decade
gargaurav_ @IBMbigdata RT @zacharyjeans: Remember BigData as we know it today, in BigDataMgmt, will seem like small data, in a decade.
AllanKoivo A3: Mobile faces the same security issues all internet operations face
TheSocialPitt True - just more and more...mobile RT @allankoivo: A3: Mobile faces the same security issues all internet operations face
jameskobielus A3: same risks as with any mobile data access: theft, loss, eavedropping...but MORE data is at risk with bigdata
BigDataAlex A3: One of the biggest challenges we have is security on mobile devices and mobile social network-who is tired of being hacked?
A3 There are lots & many have yet to be discovered - you must use
your best judgment when it comes to protecting sensitive info
cristianmolaro A3: is not the amount of data what matters, but the information that it carries or you can get from it
jeffreyfkelly @ted_friedman for sure - Mobile bigdata has BIG (pun intended) implications for data governance and security
Marcio_Saito Mobile: No fixed location, devices more accessible, more use of untrusted networks
cristianmolaro A3: some studies shows that most of the security breaches come from INSIDE organizations... mobile is not a risk per se
AllanKoivo All connected ops are a risk @cristianmolaro: A3: some studies show breaches come from INSIDE #mobile is not a risk
Natasha_D_G Good point RT @cristianmolaro: A3: some studies shows that most of the security breaches come from INSIDE org: Mobile is not a risk
jeffreyfkelly A3 @jameskobielus more data at risk and potentially more sensitive data
craigmullins A3 @cristianmolaro Mobile increases the risk by exposing the data over a network though, increasing external risk
craigmullins A3 @cristianmolaro So even if much risk is internal, mobile increases the risk
Natasha_D_G Adds a new layer RT @craigmullins: A3 @cristianmolaro So even if much risk is internal, mobile increases the risk
BigDataAlex A3: how do we anticipate the distributed cyber-attack and mobile devices - DDOS through mobility will be huge.
furrier A3: identity & authentication are key to mobile security - same old issue with new architecture & new software models
zacharyjeans The biggest risk to opening BigData is the 'Question'. BigData is meaningless w/ out good questions.
jameskobielus A3: privacy risks can be considerable, considering sheer comprehensiveness of personally identiable info in bigdata
IBMbigdata Great points about security concerns with #bigdatamgmt - BYOD, risk from within orgs, etc
cristianmolaro A3: mobile access to bigdata can be a security exposure, but also an opportunity for massive distributed data processing!
jeffreyfkelly A3 security, privacy risks are similar to traditional data and mobile, but stakes are higher w BigData
furrier A3: some MobileANGLE "65% of firms will invest in mobile; only 7% will invest in mobile device mgt for security/policy
TheSocialPitt Whoa! RT @furrier: A3: "65% of firms will invest in mobile; only 7% will invest in mobile device mgt for security/policy
NeerajT4 RT @padma8376 - a3) Bigdata apps may see a tad bit more scrutiny given the data they can access. But are we there?
The GigaOM Structure:Data Conference is taking place next week in New York City: March 19-21.
very own Paul Zikopoulos will be speaking at the conference... so make
sure you add that to your agenda. Oceanic Suite 1:05 pm, Wednesday
talks are both entertaining and informative. Another draw is that Paul
will be handing out and signing copies of his book "Harness the Power
of Big Data."
For more details about the conference, see GigaOM Structure: Data Conference.
More about Paul’s talk:
Date: March 20
Time: 1:05 PM
Room: Oceanic Suite
Venue: Pier Sixty at The Chelsea Piers (map) New York, NY 10011
has worked with hundreds of clients to identify the highest impact big
data analytics use cases. Learn from the author of “Harnessing the Power
of Big Data” about these use cases and the technologies needed to turn
big data into a competitive advantage.
Speaker: Paul Zikopoulos. Director, Information Management WW Technical Professionals, IBM
Social media connections for the conference:
tweet about it (@gigaom / #dataconf)
Our next tweetchat is scheduled for Wednesday, March 13 at noon ET. Here are the details: IBM Tweetchat – Mobile Data: Taking Your Big Data On the Road - March 13.
Use the hashtag: #bigdatamgmt.
You don’t have to be on twitter to “listen” to the conversation.
Simple Google the hashtag. If you are on twitter, here is a tip from
organizer Wikibon: As for the Twitter client, the easiest things to use
is tweetchat.com. Just log on using your twitter handle, enter the
hashtag #bigdatamgmt and it will set up a easy-to-use room. You'll be
able to see the real-time stream of tweets and it automatically adds the
hashtag to each tweet.
help you prepare for the tweetchat, we’ve pulled together a few
articles and webinars that you may wish to explore before taking part in
The major highlights include:
- 6 MB footprint, in-memory database to fit on mobile devices
- Full-featured relational DB with standard SQL API, procedures, triggers
- Fast and reliable access to enterprise data offline
- Enterprise level data security
- Persistent data storage and Automatic recovery
- Transactional storage also during connection loss
- Built-in replication capabilities allowing synchronization with IBM databases
- Flexible options such as partitioning data or creating views to customize data for each device or user
addition to this reading list, we’ve pulled together a working list of
questions to help stimulate discussion and to get you thinking about
what you may wish to ask during the chat. The ones below came from IBM
expert James Kobielus, who will be truly embracing the "mobile" concept
of the tweetchat, participating from 35,000 feet. The goal is to ask 5-6
questions during the chat itself, and the may include some of the
Q1 Which enterprise business initiatives require mobile access to big data?
Q2 Which users, developers, and other stakeholders require mobile access to big data?
Q3 Which type of big data platform (Hadoop, stream computing, etc.) is best suited to serving mobile clients?
Q4 When should you tap into smartphones and other mobile clients as data sources for big data applications?
Q5 What are the security & privacy risks of providing mobile access to big data applications?
Q6 When if ever will #mobile devices become development platforms for big data applications?
Q7 Is there a killer app for mobile access to big data?
We hope you are able to join us on Wednesday. Follow my blog for a replay of the tweetchat once it has finished.
What are you sacrificing for the promise of big data?
of today's companies are making trade-offs in areas of security and
governance to leverage the promise of big data. Can you have both?
have been several responses to this question. First off, I’d like to
point you to the tweetchat that took place on Wednesday, February 27.
Several experts, including Jeff Kelly of Wikibon discussed the topic:
Garbage In/Garbage Out and data governance’s role in the big data era.
This was a very exciting event and all who participated said they had
fun and are looking forward to the next tweetchat. I had to split the
recap into two blog entries:
plan to write a third blog entry from the information gathered by
pulling out the list of related articles that were mentioned during the
Articles and blog entries related to this theme.
Managing Database Change
by Craig Mullins
looked at the many aspects of managing change, including 9 requirements
for successful change. He also shared his DBA perspective as the
“custodian of database changes.”
3 Big Data Issues: Security, Governance and Archiving
by Dave Beulke
notes, “While the new kid database Hadoop may be getting a lot of
press, it needs more capabilities to keep up with the mature platforms
and databases for data security, governance, archiving and temporal
Productionizing your Big Data: A Checklist of Key Considerations
by James Kobielus
your big data investment production ready? James shares five key
considerations to help ensure your big data investment can function as a
reliable business asset.
Must You Sacrifice Privacy for Big Data?
by Larry Dubov
organizations move towards big data, privacy concerns become very real.
Larry traces the history of privacy and explains why good governance
can help reduce the need to compromise on privacy.
IBM InfoSphere: A Platform for Big Data Governance and Process Data Governance
by Sunil Soares
new flashbook provides an in-depth look at several aspects of governing
your big data, with several case studies to offer guidance and best
practices. Download your free copy.
Controlled Explosion: Keeping Big Data Contained with Security, Governance and Information Lifecycle Management
by James Kobielus
Can you control the big data explosion? James argues that you can, with careful attention to three vital types of controls.
Your Information is a Product
by Steven Adler
is the Chairman of IBM Data Governance Council. His view is that
companies who still treat their data as a raw resource rather than an
essential product that they produce, should prepare to be obsolete.
Steve has an ongoing blog related to the topic. You can find his blog
behind a registration page. Here is an entry to look for: The Data Governance Report Card.
Information Governance and Big Data
by Mike Martin
wrote this blog entry a year ago, but it is still as accurate today as
it was then: “I believe we should include Big Data in Information
Governance, after all one of the major issues we try to solve with
Information Governance is data volume and I consider Big Data solutions
to be yet another tool in the arsenal of Information Governance.”
Data Scientist: Exploration in the Age of the Unstructured
by James Kobielus
unstructured data (however defined) enters the enterprise big-data
picture, data management professionals begin to quake in their boots.
Governance of structured data is an established body of practices and
tools, focusing on enforcing controls on schemas and contents of data
deemed to constitute an official system of records in some subject area,
such as customers and finances.”
Infographic: Valentines: 4 database features we'd love to have in big data era.
Availability, Analytics, Governance, Security, Speed, and Compression. How do these values transfer to the Big Data era?
See also Database technology: Remember me? Big data will remind you!
summary of the conversations focused on how and why big data challenges
are posing a bit of a renaissance for databases. Some people see
databases as a commodity, something that gets taken for granted. But as
big data becomes a reality, databases are again taking center stage.
Follow us on twitter using the tag #bigdatamgmt and join in on the conversation.
Q5 Once you decide what data to keep, how do you make sure it goes to the right systems and people?
jeffreyfkelly From a developer perspective, Big Data app dev tools need to improve, make it easier to deliver insight to business users.
BTRG_MikeMartin You must increase control of wasteful data even with Big Data Management, archive/retire & dispose http://t.co/Ta361ASBkP http://t.co/904FjTm2yA
Again it's a matter of information liability and asset management.
Which is more critical to your organization? Cutting risk or mining
BTRG_MikeMartin Big Data doesn’t change retention. Keep the data you need, get rid of the rest . You can’t afford to keep it all. http://t.co/Ta361ASBkP
dvellante This is a metadata problem / opportunity
craigmullins Policies, procedures, automation and education are needed to ensure that Big Data makes its way to the right systems + people?
Big Data approach needs to include improved business outcomes which
requires people process & technology working in harmony.
BTRG_MikeMartin You need to instrument processes to not only govern but make the best use of valuable data.
Data is code in the new paradigm of new apps & services - lots of
issues so developer create & data can learn & be smart.
furrier The integration of data create new datasets - future is smart data and learning data - data is code.
jeffreyfkelly Exactly, and new data sets could be highly sensitive - need governance RT @furrier: the integration of data create new data sets
Betharonoff Data as a commodity already exists, so economy is only a few steps down the road.
furrier Meta data practices will be impacted in this data quality and data-as-code concept.
furrier One aspect of this chat is business competitiveness in integrating data as code into business lifecycle and processes.
BigDataAlex Metadata tags are aligned to role based systems - automated systems.
If you don’t improve processes with Big Data management and create
better business outcomes your Big Data initiative isn’t a success.
Kari_Agrawal When and how do we decide to discard the extremely old data? Or do we retain it as in Data Warehouse?
craigmullins You need policies and automated procedures based on retention requirements.
PPB13 How do practitioners overcome emerging skepticism in the marketplace? http://t.co/x0pHJCDcfN
@BTRG_MikeMartin: You need to instrument processes to not only govern but make the best use of valuable data
BigDataAlex Moving "beyond search"
TheSocialPitt ALWAYS start Big Data project by thinking+planning. More data does not fix bad process.
BTRG_MikeMartin What processes to improve: ediscovery, ECM, Data Governance, Data Security, data retention, and data quality.
IBMbigdata Who decides "best"? RT @BTRG_MikeMartin: You need to instrument processes to not only govern but make best use of valuable data
joycetompsett Data quality has to take on the idea it will be moving around different sys/APIs Big Data management > critical for security #RSAC
BTRG_MikeMartin - Without the right tools data retention with Big Data could be a nightmare
skenniston RT @furrier: We all want data retention but who owns it after it's retained..will a data marketplace economy develop?
BTRG_MikeMartin That's where determining business value, legal and regulations come in typically only 30% of data.
Kari_Agrawal How exactly do we begin to classify data in case of Big Data?
craigmullins MT @PPB13: How do practitioners overcome skepticism... <-- by continuing to do work that adds value to your company
dvellante @BigDataAlex yes re: search - it's sometimes used as a 'blunt instrument'
Aarti_Borkar Deciding what data to retain needs to start with business policies defined upfront - its not an "on the fly" decision.
praxsozi RT @jeffreyfkelly: Q5 Big Data requires rethink of business processes - this is NOT a trivial exercise
Natasha_D_G Culture also plays role RT @jeffreyfkelly: Q5 Big Data requires rethink of business processes - this is NOT a trivial exercise
TheSocialPitt Data antique dealers RT @furrier: We all want data retention but who owns it after it's retained.will data marketplace develop?
tomjkunkel @BTRG_MikeMartin Integrated effort with Legal, Finance, Sales, Marketing with IT serving through best architecture.
BTRG_TomNestor The process must lead to better data which should drive better business opportunities.
BTRG_MikeMartin Big Data is not immune to the laws of information economics: http://t.co/Ta361ASBkP #CGOC
Q6 How does Big Data affect data lifecycle management? Does big data introduce new stages to the info lifecycle?
Summary of top answers:
BigDataAlex Yes, new stages - stages we haven't even imagined yet. Data needs to update itself into authoritative sources.
craigmullins One issue that arises is "How can you create realistic test data for testing Big Data systems and applications?"
jeffreyfkelly Yes, but we are just starting to understand Big Data lifecycle mgt - need to build out best practices.
BTRG_MikeMartin Big Data might not create new stages in life cycle management, but certainly with new domains we have to extend the data lifecycle to new platforms.
I disagree - I think new stage of LCM includes emergence of new data
sets created from integration of other data sets and then yet new data
sets created from integrating new new data sets, and on and on and on.
Aarti_Borkar Big Data makes handing the lifecycle of data a far more complex problem than before.
Natasha_D_G Can u say more? RT @Aarti_Borkar: Big Data makes handing the lifecycle of data a far more complex problem than before
Aarti_Borkar Big Data does not create new stages - just new ways to apply the existing stages to different use cases.
Dmattcarter What are some of those new use cases?
Aarti_Borkar Test Data and Privacy for Big Data is critical - as we bring in more data potentially creating a bigger security threat.
BigDataAlex Is there a new data management paradigm emerging?
craigmullins A new paradigm may indeed be emerging.
BTRG_MikeMartin RT perhaps a refined one
TheSocialPitt One new stage = 'ephemeral'.
craigmullins Let's not burden Big Data with things little data has not yet mastered.
craigmullins Sometimes we forget that - in practice - many orgs do not follow a lifecycle, practice data governance, ensure quality, etc.
craigmullins So yes, Big Data should do these things, but it is not failing if it does not.
Big Data Management requires identification & deletion of
ROT-redundant, obsolete & trivial data, which reduces storage &
StevenDickens3 What role does the community see for the original Big Data system of record the mainframe?
BTRG_MikeMartin Consider impacts of eDiscovery, governance, security and #ILM on Big Data stores how do we move traditional methods to Big Data management.
BigDataAlex Many organizations can only afford to store 20 copies of the same data - they are looking for authoritative against process.
jeffreyfkelly definitely, data sprawl becomes an issue
Kari_Agrawal How do we deal with redundancy in case of Big Data?
StevenDickens3 What is the collective view of centralised data vs multiple federated copies ?
Could some Big Data mgmt stages be the elimination of stages? Using
data/ data analysis without constraint and eliminating steps.
Aarti_Borkar Masking test data is essential to Big Data development: what the enterprise considers private needs to always be privatized.
craigmullins My next two Tweets mentioned some of them. Not saying Big Data shouldn't just that our stds should not be too high
craigmullins @BigDataAlex Yes #littledata concepts apply to Big Data... but many orgs still struggle with managing little data
tomjkunkel @Kari_Agrawal Destroy it! I can provide insight on best practices
Dmattcarter Pretty intense data quality and Big Data conversation going on around Big Datamgmt chat!
is already a data problem with #smalldata carrying it over to Big
Datamgmt. Too costly to delete all data that has no value.
BTRG_MikeMartin You must increase control of wasteful data even w Big Datamgmt, archive/retire & dispose: http://t.co/Ta361ASBkP http://t.co/904FjTm2yA
Q7 Are new tools and platforms required to manage Big Data and the new dimensions of the data lifecycle?
Summary of top answers.
craigmullins Tools for performing advanced analytics on Big Data – though not new to the industry – will be new to many organizations.
BigDataAlex Yes. We need new tools, platforms, and systems....it is happening. Calling for massive innovation - love #DataAsCode.
BTRG_MikeMartin @craigmullins It's called defensible disposal http://t.co/Ta361ASBkP
craigmullins Hadoop-based products will need to be augmented with mission-critical DBMS capabilities to become de rigueur.
craigmullins But I think DB2 (and other RDBMS products) could be extended with Big Data capabilities before that happens.
BTRG_MikeMartin Flexibility & scalability of Big Data platforms will themselves assist in helping Big Datamgmt security & controls...
BigDataAlex We need DigitalDNA - anticipating the Internet of Things - World Wired Web.
Aarti_Borkar It’s a mix of new tools and enhancing existing tools. The core solution does not change it morphs
StevenDickens3 All depends where data resides today and whether the current platform/tools are fit for purpose, if yes why move or retool?
tomjkunkel Legacy storage assets can't handle the high availability,low latency applications and need to be displaced.
jeffreyfkelly Yes, a major topic at #strataconf is making Big Data enterprise ready -need better mgt, data gov, DQ capabilities.
Aarti_Borkar Key innovation is required to ensure that both traditional and#big data are uniformly governed.
BTRG_MikeMartin Infosphere Optim helps you get control of structured data to feed only the good into Big Datamgmt: http://t.co/Y5Jniunn6N
jeffreyfkelly And don't forget security - #RSAC - must keep Big Data secure
craigmullins Which brings up regulatory compliance... another big issue
Aarti_Borkar Big Data gov starts with a uniform set of data classification and policies that cover ALL data. Metadata is the magic here.
craigmullins If the Big Data contains PII then all the regulations that apply to PII still apply - doesn't matter how big the data set is.
BigDataAlex Does this spill over into machine learning? Can we reduce dimensionality of data through associative memory?
BTRG_MikeMartin Yes innovation & big ideas as well as change our paradigms.
Betharonoff Interesting query RT @BigDataAlex A7: Can we reduce dimensionality of data through associative memory?
Q8 How does Big Data impact data stewardship? Who “owns” particular data in a big data environment?
BigDataAlex Great question - ownership is beginning to blur - standard licensing models for data are being challenged.
craigmullins All data is owned by the company, whether it is Big Data or not…
jeffreyfkelly Ah, but is it? social data, market data etc.
Internal ownership of Big Data while beyond traditional areas should
still be based on business value, compliance or legal hold.
craigmullins Of course proper data governance policies need to be enacted by the corp to confer #datastewardship and ensure proper treatment
BTRG_MikeMartin Without good data stewardship & Big Datamgmt it will difficult to unlock the value of big data: http://t.co/hGJ3QkTiJf
Aarti_Borkar Ownership of replicated data is the original biz owner- governance of that data is still their problem.
jeffreyfkelly This is a really hard one, again new biz processes informed by Big Data will impact who owns the data.
Aarti_Borkar Stewardship does not change just because a new copy of the data was created.
craigmullins True, but some Big Data is all new.
craigmullins The word "own" is always so troublesome, isn't it?
BTRG_MikeMartin Yes it needs to be well defined .
Aarti_Borkar @craigmullins - Oh so right! .. think "Responsible for".. is better than "own"...
BigDataAlex If DataAsCode, then if DataAsCode is viral, can it be controlled? Do we want it to be controlled? What does ownership mean?
BigDataAlex How does OpenSource apply to our Data?
BTRG_MikeMartin For more Big Datamgmt resources Data Privacy and Security: http://t.co/UL0VNCiivP
is a lot of information! I hope you can follow the discussions. I
tried to clean up a little bit and hope that I didn’t change any content
from the participants.
Today I spent an hour taking part in the TweetChat at Big Datamgmt focused on governance to avoid a data landfill: http://t.co/j2wojSb9Hf: "Getting Control of Data in Big Data Era"
it went too fast for me to actually be a contributor, so I was
participating as a reader / listener. This kept me busy enough since by
the end we had generated a fair about of Big Data ourselves: 647 tweets, 180 users with reach of 136,229 & 1,506,585 impressions.
Who were the experts?
and facilitators / moderators:
There were 8 questions posed over the hour, but I'm only posting the first 4 here.
Q1 In this Big Data era, do traditional concepts data quality, data governance & data stewardship even apply?
A summary of the answers:
Big Data refers to datasets whose size, type and speed of creation make
it impractical to process and analyze with traditional tools. That Big
Data definition comes from wikibon; see http://t.co/awsPyuqXjZ. So given that, definitionally then, traditional concepts are at the very least “impractical”… no?
dvellante My belief is that ingest process & analysis of data changes with big data.
BigDataAlex Yes, I think they apply. Our clients are very concerned about these issues and it does apply.
jeffreyfkelly Absolutely, but vastly more complex.
Natasha_D_G Traditional concepts are even more critical in Big Data era especially in data governance.
craigmullins But, of course data quality, data governance and data stewardship SHOULD apply in the age of Big Data Management.
You still need clean and common policies for data taxonomies; but the
unstructured and semi-structured data texture requires some new thinking
and technology. Specifically ideas around function shipping, name value
pairs, Hadoop, etc - applying traditional concepts to new model.
Dmattcarter In order for Big Data to be enterprise-ready, it needs to include those traditional concepts.
jeffreyfkelly The challenge is applying DQ and governance to high velocity data - hard enough with "traditional" data, ie CRM, ERP.
craigmullins Failing to apply these concepts will result in poor data quality. Analytics performed on bad quality data produces bad results.
BigDataAlex I think transparency is important too in this era of Big Data and how we govern. I would suggest Big Data Ethics manager.
BTRG_MikeMartin IG concepts apply to Big Data even more so as the issues solved by Information governance are only exaggerated.
furrier Data quality has to take on the idea that it will be moving around different systems/APIs.
Yet there are issues and adaptations that will be required as we apply
data quality, data governance and data stewardship to Big Data
BigDataAlex Love the challenge on high velocity data....algorithms in streams.
jeffreyfkelly Big Data is experimenting with data sets, while governance is applying policies that sometimes restrict experimentation.
BTRG_MikeMartin You can’t make good business decisions on bad data. http://t.co/8J1pQPy6eW
Natasha_D_G Data quality is an issue as "94% biz believe some of their customer/prospect info is inaccurate".
Data governance is critical in the Big Data management era as it makes
small problems bigger. You need data quality to enable Biginsights http://t.co/yVTA9NpXIB
furrier Data as a resource for applications; ownership of data is important to individual and/or company.
BigDataAlex In health care sector, orgs are combining medical ethics with their CIOs.
Aarti_Borkar Governance is even more important with Big Data as the security and trust is a bigger business issue now.
dvellante In part this is a discussion around the balance between data being an asset an a liability - good DQ is important for both.
searchCIO Metadata practices are gaining momentum as companies tackle Big Data. http://t.co/DSkdH4Yk6S
Q2 With data at unprecedented speed/volume, how can data quality measures be applied in time for analysis?
A summary of the answers:
With data quality, cleansing can occur as humans eyeball the data -
most raw Big Data is not eyeballed. In some cases (e.g. medical
devices, automated metering, etc.) only rudimentary cleansing (if any)
may be needed. At least as long as the meters are calibrated and
BigDataAlex Real-time analytics is critical. We love Streams. The right algorithm at the right time.
Natasha_D_G Trust = Word we try to avoid. @Aarti_Borkar: Governance is even more important with Big Data as security & trust bigger biz issue.
To deal with Big Data, speed, and volume: be proactive by starting
Big Data Management across the enterprise now & maintain http://t.co/hGJ3QkTiJf
Aarti_Borkar Data Quality for Big Data can be handled right upfront before starting Big Data analysis
BigDataAlex A next-generation of KPIs for quality vs. quantity are being implemented to separate quality from quantity in real-time.
furrier Data quality is about the context of the application & what users experience for each use case is not always the same.
jeffreyfkelly Machine learning is required to improve data quality for Big Data - velocity too high for human methods IMHO
nenshad Variety of algorithms include semantics
zacharyjeans Ask your Big Data well crafted questions. Sloppy questions lead to sloppy answers.
craigmullins Speed + volume make data quality challenging…
searchCIO Data Quality is essential to master Big Data Management http://t.co/pxZ49Xgimm
BTRG_MikeMartin Start now on data quality because if you don’t have it in now Big Data only magnifies data issues http://t.co/hGJ3QkTiJf
Natasha_D_G Excellent question especially given social media data and its 18 minute life span
jeffreyfkelly Also with Big Data, volume of data can sometimes smooth over anomalies in data quality.
Aarti_Borkar Data quality should also be handled as the results of the analysis are merged back into the reporting marts.
BigDataAlex The right analytics at the right time against the systems of systems integration.
dvellante Perspectives from a former CIO on the importance of data quality http://t.co/mYPfqNCCjm
nenshad It’s all about the data first
dvellante In my view you can't deal with Big Data quality unless you can automate the classification of data at the point of creation.
Kari_Agrawal How exactly do we clean the data when it has no structure?
BTRG_MikeMartin You can’t make good decisions and enable business biginsights without high data quality.
furrier Dirty data equals poor user experience. I wrote about it in 2009 re: twitter facebook & social data http://t.co/vpkfB0xS3h
Aarti_Borkar Data quality should be handed as part of data integration as the Information Server customers do - its the same with Big Data.
Q3 How do data governance policies apply when the point of Big Data is to explore novel use cases?
A summary of the answers:
craigmullins Finding novel uses of data does not diminish the need for data governance policies.
Natasha_D_G True, but still need boundaries.
BTRG_MikeMartin Exploring Big Data still requires trusted data so you must secure and govern even more so. http://t.co/UL0VNCiivP
craigmullins The novel uses need to be documented as part of the data governance policies.
BigDataAlex The right policy at right time. I think you can agility with accountability.
craigmullins Keeping in mind that even under ideal circumstances data governance policies can be difficult to enact.
Big data isn't just for novel new business cases - it can also vastly
improve value in existing ones - i.e. R&D, cust service.
craigmullins Consider non-intrusive data governance; see this article by my friend Bob Seiner http://t.co/GogojXCcoV
Seiner states: data governance refers to the administering
(formalizing) of discipline (behavior) around the management of data.
craigmullins And data governance is an on-going process; it should formalize what already exists + address opportunities to improve.
jeffreyfkelly There is a need to set up boundaries but give analysts freedom to explore Big Data.
furrier Innovation will not come from regulations but creative developers to play with data -#slipperyslope
Q4 How does Big Data change data retention policies, ie, deciding what data to keep vs dispose?
A summary of the answers:
tomjkunkel Formal Data Destruction processes minimize the growing data landfill and need to be incorporated into Data Lifecycle Mgmt.
dvellante: Still must be able to defensibly delete data. you may not want WIP data hanging around - too much of a risk.
BTRG_MikeMartin Big Data is not immune to the laws of information economics: http://t.co/Ta361ASBkP
BigDataAlex Focus on workflow, business process, optimization. There is no set answer. Filtration - distillation
BTRG_MikeMartin Velocity of Big Data means current best data is changing rapidly, you want decisions on the best info.
BTRG_MikeMartin: It is important to have Big Data Management framework for good business outcomes inc. policy, security, ILM & quality.
Data is retained for internal + external reasons... Internal because
the org needs it for business – external because the law demands it.
tomjkunkel Isn't there also a need for Data Entrepreneurs (A business perspective with a knack for data)?
You may choose to retain more data for Big Data Management analytics
but be careful because data once retained is discoverable during court
furrier Big data complicates data retention policies - we have shadow IT and now "shadow data" or what I call "dark data".
Natasha_D_G Big Data can extend data retention esp in R&D. Pharmas can leverage old research to accelerate new research.
jeffreyfkelly This is a major issue: with hadoop you can now store all data inexpensively - not possible before and new challenge.
BTRG_MikeMartin NO still too costly.
Kari_Agrawal If we see the huge amount of IP packets flying around, can we process those packets to get something meaningful?
craigmullins There are over 150 different regulations (at the local, state, national, and international levels) that impact data retention.
Aarti_Borkar Retention is about storing what the business needs later vs everything - that core concept does not change with Big Data.
BigDataAlex Do we need to store everything? Can we, should we?
craigmullins No, no, and no to that last series of questions!
Natasha_D_G Data hoards say keep all! Fear of losing critical info.
jeffreyfkelly Nothing worse than looking for data you know you had only to remember you threw it away!
Aarti_Borkar Defensible disposal of data becomes harder if multiple copies are made as part of Big Data analytics.
craigmullins MT @Aarti_Borkar: Defensible disposal of data becomes harder if... hence the need for #datagovernance policies!
furrier We all want data retention but who owns it after it's retained..will a data marketplace economy develop?
TheSocialPitt Storage is a huge challenge, especially in cases with many streaming video feeds, e.g. defense.
Keep in mind regulations haven't caught up w the technology - industry
needs to be proactive on this issue or the government will.
Aarti_Borkar Big Data allows for pattern searches and trends in retained data that was not easy to do earlier.
is a lot of information! I hope you can follow the discussions. I
tried to clean up a little bit and hope that I didn’t change any content
from the participants.
To find out more about managing big data, join IBM for a free event: http://ibm.co/BigDataEvent
tweetchat happens when a group of people all tweet about the same topic
using a specific hashtag that allows it to be followed on Twitter.
We have one planned on Wednesday, February 27 at noon ET, so you can give it a try! Follow #bigdatamgmt to
follow a discussion with panel of experts discussing how data
governance can handle big data. Our panel includes experts from across
the spectrum, including:
and perhaps a few other people as well!
Follow along with the #bigdatamgmt to see the conversation and share your thoughts.
If you’re not yet on twitter, you can still listen to the conversation. Simply use Google Search on the term #bigdatamgmt and you’ll see what has been posted using this term.