Another great tweetchat on the hashtag #bigdatamgmt! The topic for this one was In-Memory for Big Data Management
again, we had an active group of experts leading the discussion and
were very pleased at the responses from other attendees. Thanks to all
who contributed their time and expertise! Here is a summary of the
Question 1: What is in-memory tech? How does it enable real-time speed-of-thought analytics?
BigDataAlex A1: In-Memory Computing (IMC) utilizes RAM-DRAM for extremely fast I/O, moving us away from slow, underutilized spinning disk
Natasha_D_G A1:In-memory tech enables businesses to utilize data stored in main memory vs fragmented/siloed trad databases
BTRG_MikeMartin To create business value, we must find new insights using data. In memory is “velocity w/ a vengeance” http://t.co/5KjRlOntQL
jeffreyfkelly A1 in-memory refers to storing data in main memory (DRAM) rather than spinning disk
Natasha_D_G A1: In simplest form: open book exams vs memorizing answers. Time it takes to search for answers test is over!
BigDataAlex A1: IMC reduces power and storage costs, revolutionizing access.
jameskobielus A1: In-memory puts data into RAM to enable interactive visualization exploration of patterns & real-time transactions
jeffreyfkelly A1 much faster to pull data from memory than disk - response time much quicker than spinning rusty metal allows
Mgmt has always been about "managing the bottlenecks". A major one has
always been the database itself. In-Memory helps a lot.
BTRG_MikeMartin In-memory enables you to increases the exploration aspect of Big Data
Natasha_D_G Without adding time to equation RT In-memory enables you to increases the exploration aspect of Big Data
cristianmolaro A1 memory access is way faster than disk I/O... even against SSD
cristianmolaro A1 faster data access enables real-time massive data processing: real-time #bigdata
jameskobielus A1: Speed of thought is any tech that doesn't have any architectural bottlenecks that arbitrarily slow people's explorations
CuneytG A1 In-memory means fast access to data
Natasha_D_G A1: Memory makes diff! Ability to deliver accurate answer w/o pregnant pauses impacts biz agility
cristianmolaro Combine massive parallel computing with on-memory processing and you will get a super super fast bigdata machine
BTRG_MikeMartin Solid redbook with a deeper dive on Big Data Tech from IBM: http://t.co/LMXvmFlTlK
Question 2: What are the killer apps of in-memory tech? Share examples for good reference models
BigDataAlex A2: Working with streaming data to analyze audio – processing in real-time 32 petabytes a day burn rate.
jeffreyfkelly A2 anything requiring speed-of-thought response time - allows for exploration of large data sets in near real-time
BTRG_MikeMartin How SAP HANA leverages in-memory: http://t.co/dBuy8BAnwB
zacharyjeans A2: Logistics. SAP HANA reduced a chinese bottled water company's calculation time from 24 hours to under a minute.
katsnelson Q2 Call Detail Records processing in memory. 9 billion CDRs per day. Can't think of a better case for memory
Natasha_D_G A2: In-memory tech = gold in #CX tactics and can drive proactive #custserv: up-sell, cross sell
cristianmolaro A2 I cannot think about any application that would not take advantage of faster processing...
BTRG_MikeMartin In-memory technology ensures cost savings, enhanced efficiency, and greater immediate visibility. http://t.co/R1ACWyCoAy
BigDataAlex A2: In 2000, Larry Page said he wanted to store the Internet in memory. Good idea. See: http://t.co/v7oiBQkxPb
InfoMgmtExec A2 - Apps such as High Frequency Trading and Real-time Risk/Fraud Analysis come to mind as strong users In-Memory. Many more.
BTRG_MikeMartin Benefits of In-Memory computing for insurance. http://t.co/5WZd4lHEOq
cristianmolaro A2 When you remove the I/O constraints by going on-memory you will hit the next performance wall: CPU
jameskobielus A2 Killer apps of in-mem $ visual exploration, modeling & scenario exploration. Data science, "spreadsheet on stereo
jeffreyfkelly A2 smart meter analytics
BTRG_MikeMartin How in-memory computing is helping Unilever to maximize profits http://t.co/bzaapUvG1I
katsnelson A2 many apps where data is not valuable enough to even store on disk. In Streams we process stuff in memory and discard
dfloyer The importance of Data in Memory (DRAM or FLASH) is increasing the number of DB calls and increasing the value to the App
CuneytG A2 fraud detection and investigation is a good candidate
Natasha_D_G Good for #finserv & #insurance RT @CuneytG: A2 fraud detection and investigation is a good candidate
TerraEchos Definitely has great security applications! RT @CuneytG: A2 fraud detection and investigation is a good candidate
zacharyjeans A2 - Apps such as High Frequency Trading and Real-time Risk/Fraud Analysis come to mind
BTRG_MikeMartin National security is another area in-memory is key, however we can't say more about it than that.
jeffreyfkelly A2 investigating network traffic issues, finding bottlenecks
cristianmolaro A2 on-memory allows applications to fully exploit today's more and more powerful CPUs... good news for bigdata!
cristianmolaro then scale with more CPU in parallel! :-) Then the next performance wall will be inter-CPU communication...
IBMbigdata Always a wall RT @cristianmolaro: Then the next performance wall will be inter-CPU communication...
jeffreyfkelly A2 analyzing high-velocity financial data in trading scenarios - no time to lose in this use case!
jameskobielus A2 Killer app of in-mem is ability to rapidly evaluate, iterate, & refine statistical models
dfloyer Memory in just DRAM limits the scope of the problem that can be tackled - large memory can impact L1/L2 caching performance
tomjkunkel And who are the 3 leading providers?
jeffreyfkelly A2 iterate, iterate, iterate
troycoleman Do you see any in-memory databases running on z/OS?
jameskobielus A2: Yes. Any transactional app that demands split-second response benefits from in-memory.
Question 3: What are in-memory's applications in transactional computing?
jeffreyfkelly A3 ad tech - analyzing user data, real-time bidding, delivering personalized content - in milliseconds
Orgs want entire Customer Base, Product Sku's & Pricing in Memory
for rapid transaction processing. Customers will not wait!
BTRG_MikeMartin In-Memory Computing Provides Real-Time Data Platform for Transactions. http://t.co/7xR4FB1lcj
katsnelson A3 many Streams apps are transactional and Streams is always in memory.
CuneytG A3 all oltp apps need to be fast. İn memory is fast too. So any oltp app is in the scope of inmemory
IBMbigdata Impatient souls RT @InfoMgmtExec: A3 Orgs want entire Customer Base, Product Skus & Pricing in Memory
BigDataAlex A3:Connecting the Internet of Things - IP addressable sensors to real-time calibrate our models for better predictive analytics
jameskobielus A3: faster transactions that result from caching more frequently used data in RAM at the server and/or client
BTRG_MikeMartin We have a customer who used in-memory to transform their OLTP systems in order to get paid faster http://t.co/1vr7SiC8iw
Natasha_D_G Needed in our "instant" mrkt RT @InfoMgmtExec: A3 Orgs want entire Customer Base, Product Skus & Pricing in Memory
zacharyjeans Is @Spotify an in memory application?
Ercan__Yilmaz @Spotify uses in-memory caching, so do millions of others
zacharyjeans I suspected as much. I can't imagine that @Spotify could deliver seamless streaming w/out in-memory caching.
jameskobielus A3: next best action, bridging analytics & transactions, could benefit from in-mem 4 low-latency data & execution
jeffreyfkelly A3 any transaction workload that requires real-time response in order to win/save/upsell the customer is in-memory candidate
johncrupi We have to treat in-memory as the new architectural tier for real-time analytic apps
jameskobielus A3: transactions are C (create), U (update), & delete (D) intensive...all can go faster if in-mem & no disk access
troycoleman Do customers tend to replicate data to the target platform that is running the in-memory database?
BigDataAlex A3:Working in the oil and gas industry-energy exploration requires millions of transactions a day for discovery of new resource
jasebell Think Point of Sale predicted coupons via receipts, customers don't want to wait. In memory wins.
rkeshavmurthy To increase analytics speed and reduce manual tuning. e.g custom coupon generation
furrier where are all the big data apps? they are already here. Analytics & in memory make them better
furrier A3: memory is making up for disk speed & is now becoming more important in software models-big oppty
johncrupi analytics is the killer use case for in-memory, IMO.
last night I spoke with SAP execs - apps are all the ones we know about
& they are all getting bigdata upgrades with new tech
rkeshavmurthy customers are using in-memory approach for simply accelerating traditional BI, recently with analysis of sensor data
In-Memory db will allow Predictive Models to be deployed into
Transactional Work Flows for real-time scoring & predictionzacharyjeans I imagine In-Memory computation could benefit @NASA's efforts to identify asteroids & their object oriented trajectories.