NDC : Hybrid Datastore

NDC Oslo 2012 : Hybrid Datastore

Transcript

1. Chris Harris Email : charris@10gen.com Twitter : cj_harris5 Hybrid Datastore Sunday, 10 June 12
2. Traditional Architecture Sunday, 10 June 12
3. Traditional Architecture Controllers Services Application Server Database Web Server HTML SQL Sunday, 10 June 12
4. Challenge #1 - HTML 5 Sunday, 10 June 12
5. Multiple Client Types Controllers Services Application Server Web Server HTML None HTML? Database SQL Sunday, 10 June 12
6. HTML 5 Services Application Server Web Server JSON JSON JSON Controllers Database SQL Sunday, 10 June 12
7. The controllers have moved to the client Expose small JSON Services The Move to Services + Death to the monolithic deployment Sunday, 10 June 12
8. Service Architecture Service #2 Application Server Web Server Service #1 Application Server ... SQL SQL JSON JSON JSON Sunday, 10 June 12
9. JSON Service becomes a API to client and other applications. The clients are not bound to the underlaying data store The Service becomes the API - Complexity due to mismatch between the JSON and SQL Sunday, 10 June 12
10. Migrating to MongoDB Service #2 Application Server Web Server Service #1 Application Server ... SQL JSON JSON JSON JSON Sunday, 10 June 12
11. Here is a “simple” SQL Model mysql> select * from book; +----+----------------------------------------------------------+ | id | title | +----+----------------------------------------------------------+ | 1 | The Demon-Haunted World: Science as a Candle in the Dark | | 2 | Cosmos | | 3 | Programming in Scala | +----+----------------------------------------------------------+ 3 rows in set (0.00 sec) mysql> select * from bookauthor; +---------+-----------+ | book_id | author_id | +---------+-----------+ | 1 | 1 | | 2 | 1 | | 3 | 2 | | 3 | 3 | | 3 | 4 | +---------+-----------+ 5 rows in set (0.00 sec) mysql> select * from author; +----+-----------+------------+-------------+-------------+---------------+ | id | last_name | first_name | middle_name | nationality | year_of_birth | +----+-----------+------------+-------------+-------------+---------------+ | 1 | Sagan | Carl | Edward | NULL | 1934 | | 2 | Odersky | Martin | NULL | DE | 1958 | | 3 | Spoon | Lex | NULL | NULL | NULL | | 4 | Venners | Bill | NULL | NULL | NULL | +----+-----------+------------+-------------+-------------+---------------+ 4 rows in set (0.00 sec) Sunday, 10 June 12
12. The Same Data in MongoDB { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" } ] } Example Query : db.books.find({title : "Programming in Scala" }); Sunday, 10 June 12
13. Demo #1 - Twitter Aggrgiation Sunday, 10 June 12
14. Migrating to MongoDB Service #2 Application Server Web Server Service #1 Application Server ... SQL JSON JSON JSON JSON Sunday, 10 June 12
15. Demo #2- JSON Service Sunday, 10 June 12
16. Challenge #2 - Write Volumes Sunday, 10 June 12
17. Users don’t just want to read content! They want to share and contribute to the content! Increase in Write Ratio Volume Writes! Sunday, 10 June 12
18. Need to Scale Datasource Web Server Service #1 Application Server SQL JSON JSON JSON Bottleneck! Sunday, 10 June 12
19. Need to Scale Datasource Web Server Service #1 Application Server SQL JSON JSON JSON Bottleneck! Sunday, 10 June 12
20. Application Cache? Web Server Service #1 Application Server SQL JSON JSON JSON App Cache Sunday, 10 June 12
21. Issues + Read Only data comes from a Cache - Writes slow down as need to update the Cache and the Database - Need to keep cache data in sync between Application Servers Sunday, 10 June 12
22. http://community.qlikview.com/cfs-filesystemfile.ashx/__key/CommunityServer.Blogs.Components.WeblogFiles/ theqlikviewblog/Cutting-Grass-with-Scissors-_2D00_-2.jpg Sunday, 10 June 12
23. http://www.bitquill.net/blog/wp-content/uploads/2008/07/pack_of_harvesters.jpg Sunday, 10 June 12
24. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Sunday, 10 June 12
25. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Sunday, 10 June 12
26. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Sunday, 10 June 12
27. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes • Each data node contains many chunks Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Sunday, 10 June 12
28. Big Data at a Glance • Systems like Google File System (which inspired Hadoop’s HDFS) and MongoDB’s Sharding handle the scale problem by chunking • Break up pieces of data into smaller chunks, spread across many data nodes • Each data node contains many chunks • If a chunk gets too large or a node overloaded, data can be rebalanced Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z ... Sunday, 10 June 12
29. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z Sunday, 10 June 12
30. Big Data at a Glance Large Dataset Primary Key as “username” a b c d e f g h s t u v w x y z MongoDB Sharding ( as well as HDFS ) breaks data into chunks (~64 mb) Sunday, 10 June 12
31. Large Dataset Primary Key as “username” Scaling Data Node 1 25% of chunks Data Node 2 25% of chunks Data Node 3 25% of chunks Data Node 4 25% of chunks a b c d e f g h s t u v w x y z Representing data as chunks allows many levels of scale across n data nodes Sunday, 10 June 12
32. Large Dataset Primary Key as “username” Scaling Data Node 1 25% of chunks Data Node 2 25% of chunks Data Node 3 25% of chunks Data Node 4 25% of chunks a b c d e f g h s t u v w x y z Representing data as chunks allows many levels of scale across n data nodes Sunday, 10 June 12
33. Scaling Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The set of chunks can be evenly distributed across n data nodes Sunday, 10 June 12
34. Scaling Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The set of chunks can be evenly distributed across n data nodes Sunday, 10 June 12
35. Add Nodes: Chunk Rebalancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The goal is equilibrium - an equal distribution. As nodes are added (or even removed) chunks can be redistributed for balance. Sunday, 10 June 12
36. Add Nodes: Chunk Rebalancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z The goal is equilibrium - an equal distribution. As nodes are added (or even removed) chunks can be redistributed for balance. Sunday, 10 June 12
37. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Sunday, 10 June 12
38. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Sunday, 10 June 12
39. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z Writes are efficiently routed to the appropriate node & chunk Sunday, 10 June 12
40. Writes Routed to Appropriate Chunk Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z Writes are efficiently routed to the appropriate node & chunk Sunday, 10 June 12
41. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z Write to key“ziggy” z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Sunday, 10 June 12
42. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Sunday, 10 June 12
43. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks Sunday, 10 June 12
44. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks z1 Sunday, 10 June 12
45. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 If a chunk gets too large (default in MongoDB - 64mb per chunk), It is split into two new chunks z1 Sunday, 10 June 12
46. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 z1 Each new part of the Z chunk (left & right) now contains half of the keys Sunday, 10 June 12
47. Chunk Splitting & Balancing Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z z2 z1 As chunks continue to grow and split, they can be rebalanced to keep an equal share of data on each server. Sunday, 10 June 12
48. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y z1 Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z2 Sunday, 10 June 12
49. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z1 z2 Sunday, 10 June 12
50. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Key “xavier” Reading a single value by Primary Key Read routed efficiently to specific chunk containing key z1 z2 Sunday, 10 June 12
51. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Keys “T”->”X” Reading multiple values by Primary Key Reads routed efficiently to specific chunks in range t u v w x z1 z2 Sunday, 10 June 12
52. Reads with Key Routed Efficiently Data Node 1 Data Node 2 Data Node 3 Data Node 4 Data Node 5 a b c d e f g h s t u v w x y Read Keys “T”->”X” Reading multiple values by Primary Key Reads routed efficiently to specific chunks in range t u v w x z1 z2 Sunday, 10 June 12
53. Architecture Sunday, 10 June 12
54. Adding MongoS Web Server Service #2 Application Server JSON JSON JSON JSON MongoS Service #1 Application Server SQL Sunday, 10 June 12
55. Challenge #3 -Offline Processing Sunday, 10 June 12
56. Online / Offline Web Server Application Server JSON JSON JSON JSON Online Offline Service #2 MongoS Sunday, 10 June 12
57. MongoDB and Hadoop Sunday, 10 June 12
58. MongoDB Hadoop Adapter public void map(LongWritable key, Text value, Context context) throws ..{ String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } public void map(Object key, BSONObject value, Context context ) throws ....{ StringTokenizer itr = new StringTokenizer( value.get( "line" ).toString() ); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } Classic Hadoop Word Count - Map Sunday, 10 June 12
59. MongoDB Hadoop Adapter is the same code! public void reduce( Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException{ int sum = 0; for ( final IntWritable val : values ){ sum += val.get(); } context.write( key, new IntWritable(sum)); } Classic Hadoop Word Count - Reduce Sunday, 10 June 12
60. Demo #3 - MongoDB and Hadoop Sunday, 10 June 12
61. New Hybrid World Sunday, 10 June 12
62. New Hybrid World Application Server Web Server Service #1 Application Server ... SQL JSON JSON JSON JSON Online Offline Service #2 MongoS Sunday, 10 June 12
63. conferences, appearances http://www.10gen.com/events download at mongodb.org We’re Hiring ! Chris Harris Email : charris@10gen.com Twitter : cj_harris5 Sunday, 10 June 12