Recap of Tweetchat: "In-Memory for Big Data Management " - Part 3
svisser1 2700018UK9 Visits (3695)
This is part 3 of the In-Memory for Big Data Management Tweetchat Recap. See also: Recap of Tweetchat: "In-Memory for Big Data Management " - Part 1 and Part 2.
Question 7: Can in-memory techniques be applied to non-relational databases and/or Hadoop?
jeffreyfkelly A7 yes - see @aerospike, NoSQL, flash-optimized in-memory DB
dvellante A7. yes and @jeffreyfkelly - interesting Aerospike - that's an extension of memory using flash
InfoMgmtExec A7 -Time Series db's(Informix) will benefit substantially from In Memory. Critical to Smart Metering and Smart Grid strategies.
dfloyer A7 Of course! Databases such as Couchbase (Memcache) & Aerospike (Flash) use KV pairs in memory extensively for transactions
katsnelson Hadoop is about data on disk. Streams does the opposite i.e processes in-memory. IBM bundles Hadoop and Streams
BigDataAlex A7: InfoSphere #Streams brings "database" functions into IMC in real time for continuous query and calculations
jeffreyfkelly A7 like the DW question, in-memory DB can supplement Hadoop batch analytics w/ real-time analytic queries
kdnuggets Is there an upper limit to in-memory DB? Are there risks of instability/errors when memory is too large?
BTRG_MikeMartin RT of course @terraechos: an opportunity cost? @BTRG_MikeMartin: Like EVERYTHING it is about finding the cost/benefit.
Question 8: What is main role for in-memory in bigdata infrastructures? Where does flash memory fit?
dvellante: A8. The best IO is no IO
dvellante A8. But no IO is expensive so in-memory in #bigdata has to be used judiciously
cristianmolaro A8 I like the concept of multi-temperature storage: the hottest data stored on the faster (and more expensive) storage device
cristianmolaro A8 main role should be to accelerate access in relevant chunks... #bigdata is too big to be contained in memory...
BTRG_MikeMartin RT Favorite comment so far @dvellante: A8. The best IO is no IO
jeffreyfkelly A8 in-memory should be used strategically in #BigData infrastructure where speed, performance gains outweigh costs
dfloyer The L1/L2 cache hit rate become unpredictable as DRAM increases - adding more DRAM can decrease performance
dfloyer Yes, especially for DRAM - L1/L2 cache stability, RTO if backup data is on disk. Limits scope of problem addressed
jeffreyfkelly A7 I believe there are in-memory instances of #Cassandra - anybody have info?
dvellante RT @cristianmolaro: A8 I like concept of multi-temperature storage>> time to freshen up those page/swap algorithms in the OS?
Our next tweetchat is scheduled for Wednesday April 17 at noon: Organizations solving everyday problems with Analytics.
Watch for another blog entry that summarizes the reading recommendations found in this recap.