MapReduce and Columnar DB's

Transcript

1. MapReduce and Columnar DB’s Amant Stéphane @stephamant
2. Summary • MapReduce • Columnar DB’s • Practical Use Case
3. MapReduce
4. MapReduce - Definition • One of Google’s greatest contributions to computer science • MapReduce is an algorithmic framework for executing jobs in parallel over several nodes
5. MapReduce
6. MapReduce
7. MapReduce - Major Implementation • Almost always based on Hadoop - a Framework for the storage and processing of large scaled and distributed data supported by Apache • Itself inspired by Google BigTable Project
8. Columnar DB’s
9. Columnar DB’s - Definition Columnar databases are so named because the important aspect of their design is that data from a given column is stored together. (By contrast, a row-oriented database keeps information about a row together.) In column-oriented databases, adding columns is quite inexpensive.
10. Columnar DB’s - Definition
11. Columnar DB’s - Definition
12. Columnar DB’s - Queries get ‘t1′, ‘r1′, {COLUMN => ‘c1′} get ‘t1′, ‘r1′, {COLUMN => ['c1', 'c2', 'c3']} get ‘t1′, ‘r1′, {COLUMN => ‘c1′, TIMESTAMP => ts1} get ‘t1′, ‘r1′, {COLUMN => ‘c1′, TIMERANGE => [ts1, ts2], VERSIONS => 4} get ‘t1′, ‘r1′, {COLUMN => ‘c1′, TIMESTAMP => ts1, VERSIONS => 4}
13. Columnar DB’s - Major Implementation • Cassandra • Hypertable • HBase
14. Columnar DBs - Supporting Companies • Facebook • Yahoo • Ebay • Twitter • Amazon • Google • ...
15. Columnar DB’s - Pro’s • Horizontal scalability (replication and partitioning) • Versioning is trivial • No real storage cost for null values • Used mainly for Big Data / data mining / Business Intelligence analysis
16. Columnar DB’s - Con’s • Complexity (Installation, infrastructure and usage) • Design your schema based on how you plan to query the data • Some operations are really time expensive
17. Practical Use Case
18. Facebook Messaging Index Table Keyword #1 Keyword #2 Keyword #3 Keyword #... User ID #1 User ID #2 User ID #... Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id Timestamp Message_id
19. References Seven Databases in Seven Weeks: A Guide to Modern Databases and the NoSQL Movement by Eric Redmond and Jim R. Wilson
20. Thank you