Hybrid Datastore

NoSQL Matters 2012 : Building Hybrid Applications with MongoDB, RDBMS & Hadoop

1. Chris Harris Email : charris@10gen.com Twitter : cj_harris5 Hybrid Datastore Thursday, 31 May 12
2. Traditional Architecture Thursday, 31 May 12
3. Traditional Architecture Controllers Services Application Server Database Web Server HTML SQL Thursday, 31 May 12
4. Challenge #1 - HTML 5 Thursday, 31 May 12
5. Multiple Client Types Controllers Services Application Server Web Server HTML None HTML? Database SQL Thursday, 31 May 12
6. HTML 5 Services Application Server Web Server JSON JSON JSON Controllers Database SQL Thursday, 31 May 12
7. The controllers have moved to the client Expose small JSON Services The Move to Services + Death to the monolithic deployment Thursday, 31 May 12
8. Service Architecture Service #2 Application Server Web Server Service #1 Application Server ... SQL SQL JSON JSON JSON Thursday, 31 May 12
9. JSON Service becomes a API to client and other applications. The clients are not bound to the underlaying data store The Service becomes the API - Complexity due to mismatch between the JSON and SQL Thursday, 31 May 12
10. Migrating to MongoDB Service #2 Application Server Web Server Service #1 Application Server ... SQL JSON JSON JSON JSON Thursday, 31 May 12
11. Here is a “simple” SQL Model mysql> select * from book; +----+----------------------------------------------------------+ | id | title | +----+----------------------------------------------------------+ | 1 | The Demon-Haunted World: Science as a Candle in the Dark | | 2 | Cosmos | | 3 | Programming in Scala | +----+----------------------------------------------------------+ 3 rows in set (0.00 sec) mysql> select * from bookauthor; +---------+-----------+ | book_id | author_id | +---------+-----------+ | 1 | 1 | | 2 | 1 | | 3 | 2 | | 3 | 3 | | 3 | 4 | +---------+-----------+ 5 rows in set (0.00 sec) mysql> select * from author; +----+-----------+------------+-------------+-------------+---------------+ | id | last_name | first_name | middle_name | nationality | year_of_birth | +----+-----------+------------+-------------+-------------+---------------+ | 1 | Sagan | Carl | Edward | NULL | 1934 | | 2 | Odersky | Martin | NULL | DE | 1958 | | 3 | Spoon | Lex | NULL | NULL | NULL | | 4 | Venners | Bill | NULL | NULL | NULL | +----+-----------+------------+-------------+-------------+---------------+ 4 rows in set (0.00 sec) Thursday, 31 May 12
12. The Same Data in MongoDB { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" } ] } Thursday, 31 May 12
13. Demo #1 - JSON Service Thursday, 31 May 12
14. Challenge #2 - Write Volumes Thursday, 31 May 12
15. Users don’t just want to read content! They want to share and contribute to the content! Increase in Write Ratio Volume Writes! Thursday, 31 May 12
16. Need to Scale Datasource Web Server Service #1 Application Server SQL JSON JSON JSON Bottleneck! Thursday, 31 May 12
17. Need to Scale Datasource Web Server Service #1 Application Server SQL JSON JSON JSON Bottleneck! Thursday, 31 May 12
18. Application Cache? Web Server Service #1 Application Server SQL JSON JSON JSON App Cache Thursday, 31 May 12
19. Issues + Read Only data comes from a Cache - Writes slow down as need to update the Cache and the Database - Need to keep cache data in sync between Application Servers Thursday, 31 May 12
20. http://community.qlikview.com/cfs-filesystemfile.ashx/__key/CommunityServer.Blogs.Components.WeblogFiles/ theqlikviewblog/Cutting-Grass-with-Scissors-_2D00_-2.jpg Thursday, 31 May 12
21. http://www.bitquill.net/blog/wp-content/uploads/2008/07/pack_of_harvesters.jpg Thursday, 31 May 12
22. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Thursday, 31 May 12
23. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Thursday, 31 May 12
24. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Thursday, 31 May 12
25. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes • Each data node contains many chunks Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Thursday, 31 May 12
26. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes • Each data node contains many chunks • If a chunk gets too large or a node overloaded, data can be rebalanced Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Thursday, 31 May 12
27. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z Thursday, 31 May 12
28. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z MongoDB Sharding ( as well as HDFS ) breaks data into chunks (~64 mb) Thursday, 31 May 12
29. Large Dataset Primary Key as “username” Scaling Data Node 1 25% of chunks Data Node 2 25% of chunks Data Node 3 25% of chunks Data Node 4 25% of chunks a b c d e f g h s t u v w x y z Representing data as chunks allows many levels of scale across n data nodes Thursday, 31 May 12
30. Large Dataset Primary Key as “username” Scaling Data Node 1 25% of chunks Data Node 2 25% of chunks Data Node 3 25% of chunks Data Node 4 25% of chunks a b c d e f g h s t u v w x y z Representing data as chunks allows many levels of scale across n data nodes Thursday, 31 May 12
31. Scaling Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The set of chunks can be evenly distributed across n data nodes Thursday, 31 May 12
32. Scaling Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The set of chunks can be evenly distributed across n data nodes Thursday, 31 May 12
33. Add Nodes: Chunk Rebalancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The goal is equilibrium - an equal distribution. As nodes are added (or even removed) chunks can be redistributed for balance. Thursday, 31 May 12
34. Add Nodes: Chunk Rebalancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The goal is equilibrium - an equal distribution. As nodes are added (or even removed) chunks can be redistributed for balance. Thursday, 31 May 12
35. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Thursday, 31 May 12
36. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Thursday, 31 May 12
37. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z Writes are efficiently routed to the appropriate node & chunk Thursday, 31 May 12
38. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z Writes are efficiently routed to the appropriate node & chunk Thursday, 31 May 12
39. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Thursday, 31 May 12
40. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Thursday, 31 May 12
41. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Thursday, 31 May 12
42. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks z1 Thursday, 31 May 12
43. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks z1 Thursday, 31 May 12
44. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 z1 Each new part of the Z chunk (left & right) now contains half of the keys Thursday, 31 May 12
45. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 z1 As chunks continue to grow and split, they can be rebalanced to keep an equal share of data on each server. Thursday, 31 May 12
46. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z1 Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z2 Thursday, 31 May 12
47. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z1 z2 Thursday, 31 May 12
48. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z1 z2 Thursday, 31 May 12
49. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Keys “T”->”X” Reading multiple values by Primary Key Reads routed efficiently to specific chunks in range t u v w x z1 z2 Thursday, 31 May 12
50. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Keys “T”->”X” Reading multiple values by Primary Key Reads routed efficiently to specific chunks in range t u v w x z1 z2 Thursday, 31 May 12
51. Architecture Thursday, 31 May 12
52. Adding MongoS Web Server Service #2 Application Server JSON JSON JSON JSON MongoS Service #1 Application Server SQL Thursday, 31 May 12
53. Challenge #3 -Offline Processing Thursday, 31 May 12
54. Online / Offline Web Server Application Server JSON JSON JSON JSON Online Offline Service #2 MongoS Thursday, 31 May 12
55. MongoDB and Hadoop Thursday, 31 May 12
56. MongoDB Hadoop Adapter public void map(LongWritable key, Text value, Context context) throws ..{ String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } public void map(Object key, BSONObject value, Context context ) throws ....{ StringTokenizer itr = new StringTokenizer( value.get( "line" ).toString() ); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } Classic Hadoop Word Count - Map Thursday, 31 May 12
57. MongoDB Hadoop Adapter is the same code! public void reduce( Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException{ int sum = 0; for ( final IntWritable val : values ){ sum += val.get(); } context.write( key, new IntWritable(sum)); } Classic Hadoop Word Count - Reduce Thursday, 31 May 12
58. Demo #2 - MongoDB and Hadoop Thursday, 31 May 12
59. New Hybrid World Thursday, 31 May 12
60. New Hybrid World Application Server Web Server Service #1 Application Server ... SQL JSON JSON JSON JSON Online Offline Service #2 MongoS Thursday, 31 May 12
61. conferences, appearances http://www.10gen.com/events download at mongodb.org We’re Hiring ! Chris Harris Email : charris@10gen.com Twitter : cj_harris5 Thursday, 31 May 12