Toronto MongoDB User Group

Presetation given on Toronto MongoDB User group

Transcript

1. MongoDB Toronto User group 1 Saturday, 23 February, 13
2. Hi. 2 Saturday, 23 February, 13
3. Hi. @arthurnn 2 Saturday, 23 February, 13
4. Hi. @arthurnn 2 Saturday, 23 February, 13
5. Hi. @arthurnn 2 Saturday, 23 February, 13
6. 3 Saturday, 23 February, 13
7. Tech 4 Saturday, 23 February, 13
8. Tech 4 Saturday, 23 February, 13
9. Tech 4 Saturday, 23 February, 13
10. Tech 4 Saturday, 23 February, 13
11. Tech 4 Saturday, 23 February, 13
12. RPC Tech 4 Saturday, 23 February, 13
13. Databases 5 Saturday, 23 February, 13
14. Databases 5 Saturday, 23 February, 13
15. Databases 5 Saturday, 23 February, 13
16. Databases Memcached 5 Saturday, 23 February, 13
17. Databases Memcached Sphinx -> ElasticSearch 5 Saturday, 23 February, 13
18. + 6 Saturday, 23 February, 13
19. + Activity Feed 6 Saturday, 23 February, 13
20. + Activity Feed User Settings 6 Saturday, 23 February, 13
21. + Activity Feed User Settings Push Notification 6 Saturday, 23 February, 13
22. + Activity Feed User Settings Push Notification Device track 6 Saturday, 23 February, 13
23. + Activity Feed User Settings Push Notification Device track Stats 6 Saturday, 23 February, 13
24. Servers 1.0 7 Saturday, 23 February, 13
25. Servers 1.0 mongod primary 7 Saturday, 23 February, 13
26. Servers 1.0 mongod primary mongod secondary 7 Saturday, 23 February, 13
27. Servers 1.0 mongod primary mongod secondary mongod secondary 7 Saturday, 23 February, 13
28. Servers 1.0 mongod primary mongod secondary mongod secondary mongod hidden 7 Saturday, 23 February, 13
29. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod mongoc mongod mongoc mongod mongoc Main 8 Saturday, 23 February, 13
30. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod mongoc mongod mongoc mongod mongoc mongod Main 8 Saturday, 23 February, 13
31. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod mongoc mongod mongoc mongod mongoc mongod mongod Main 8 Saturday, 23 February, 13
32. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13
33. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13
34. User Settings { "_id" : 123, "stats_page_empty" : true, "statistics_page_order" : [ 1, 4, 2, 3 ] } 9 Saturday, 23 February, 13
35. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications 10 Saturday, 23 February, 13
36. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"user_id": 1} 10 Saturday, 23 February, 13
37. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13
38. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13
39. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1} 10 Saturday, 23 February, 13
40. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id" : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1, "updated_at": -1} index: {"user_id": 1} 10 Saturday, 23 February, 13
41. Activity Feed - Flow 200M+ { "_id" : ObjectId(""), "aid" : 1127689, # Actor id "otype" : 9, # Object type for cast "v" : 7, # Verb of the action "oid" : 25000475, # Object id "uid" : 1127689 # User id } 11 Saturday, 23 February, 13
42. db.ensureIndex ??? { aid: 1 } { aid: 1, _id: 1 } { _id: 1, aid: 1 } { aid: 1, v: 1, _id: 1 } 12 Saturday, 23 February, 13
43. Data access 13 Saturday, 23 February, 13
44. Data access db.activities.find( aid: 22, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 13 Saturday, 23 February, 13
45. { aid: 1 } 14 Saturday, 23 February, 13
46. { aid: 1, v:1 } { aid: 1, v:1, _id: -1 } 15 Saturday, 23 February, 13
47. { aid: 1, _id: -1 } { aid: 1, _id: -1, v: 1 } 16 Saturday, 23 February, 13
48. 17 Saturday, 23 February, 13
49. db.activities.find( aid: {'$in': [1,10]}, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 17 Saturday, 23 February, 13
50. Shard key ??? 18 Saturday, 23 February, 13
51. Shard key ??? { aid: 1, v: 1, _id: 1 } 18 Saturday, 23 February, 13
52. Mistakes on shard 19 Saturday, 23 February, 13
53. Mistakes on shard Automatic shard 19 Saturday, 23 February, 13
54. Mistakes on shard Automatic shard SSDs on Primaries 19 Saturday, 23 February, 13
55. After sharding? 20 Saturday, 23 February, 13
56. After sharding? Good performance on writes 20 Saturday, 23 February, 13
57. After sharding? Good performance on User Flow reads Good performance on writes 20 Saturday, 23 February, 13
58. After sharding? Good performance on User Flow reads Bad performance on Friends Flow reads Good performance on writes 20 Saturday, 23 February, 13
59. Solutions: 21 Saturday, 23 February, 13
60. Solutions: #1: denormalize 21 Saturday, 23 February, 13
61. { "_id" : ObjectId(""), "user" : 123, # viewer "page" : 1, "feed" : [{activity}, {activity}] } Flow Feed by user 22 Saturday, 23 February, 13
62. { "_id" : ObjectId(""), "user" : 123, # viewer "page" : 1, "feed" : [{activity}, {activity}] } Flow Feed by user Fan out on write 22 Saturday, 23 February, 13
63. Solutions: #1: denormalize 23 Saturday, 23 February, 13
64. Solutions: #1: denormalize #2: cache 23 Saturday, 23 February, 13
65. key: user_id 24 Saturday, 23 February, 13
66. key: user_id prepend activity 24 Saturday, 23 February, 13
67. key: user_id prepend activity activity append 24 Saturday, 23 February, 13
68. key: user_id prepend activity activity append activity append 24 Saturday, 23 February, 13
69. Future Denormaziled solution. 25 Saturday, 23 February, 13
70. Thank you 26 Saturday, 23 February, 13