Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Toronto MongoDB User Group

Arthur Nogueira Neves
February 20, 2013
Technology
1
76

Toronto MongoDB User Group

Presetation given on Toronto MongoDB User group

Arthur Nogueira Neves

February 20, 2013
Tweet

More Decks by Arthur Nogueira Neves

See All by Arthur Nogueira Neves
Using multiple connections in ActiveRecord
arthurnn
3
910
Rails talk - RubyLightningTalksTO
arthurnn
0
90
Abril Pro Ruby
arthurnn
0
290

Other Decks in Technology

See All in Technology
How to Lead? Testimonial of a Lead Android Engineer
oleur
1
110
いいたいことちゃんという
tkengo
0
230
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
4
37k
推しは推せるときに推せ！ プロダクトにフィードバックしていこう
nakasho
0
460
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
3
620
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions （PSS）
pacificspatialsolutions
0
330
リテール金融（キャッシュレス・ネット銀行・ネット証券）の競争環境と経済圏
8maki
0
1.5k
コードや知識を組み込む / Incorporate Code and knowledge
ks91
PRO
0
140
Gradle Build Scanを使ってビルドのことを知ろう potatotips #87
tomorrowkey
2
150
Babylon.js JAPAN活動紹介 (2024/4)
limes2018
1
110
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
7
1.3k
GrafanaMeetup_AmazonManagedGrafanaのアクセス制御機能とマルチテナント環境下でのアクセス制御について
daitak
0
390

Featured

See All Featured
For a Future-Friendly Web
brad_frost
172
9k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
21
1.9k
Code Review Best Practice
trishagee
56
15k
YesSQL, Process and Tooling at Scale
rocio
165
13k
Fontdeck: Realign not Redesign
paulrobertlloyd
76
4.9k
Art, The Web, and Tiny UX
lynnandtonic
290
19k
It's Worth the Effort
3n
180
27k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
Producing Creativity
orderedlist
PRO
338
39k
The Invisible Side of Design
smashingmag
294
49k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
6
3.4k

Transcript

  1. MongoDB Toronto User group 1 Saturday, 23 February, 13

  2. Hi. 2 Saturday, 23 February, 13

  3. Hi. @arthurnn 2 Saturday, 23 February, 13

  4. Hi. @arthurnn 2 Saturday, 23 February, 13

  5. Hi. @arthurnn 2 Saturday, 23 February, 13

  6. 3 Saturday, 23 February, 13

  7. Tech 4 Saturday, 23 February, 13

  8. Tech 4 Saturday, 23 February, 13

  9. Tech 4 Saturday, 23 February, 13

  10. Tech 4 Saturday, 23 February, 13

  11. Tech 4 Saturday, 23 February, 13

  12. RPC Tech 4 Saturday, 23 February, 13

  13. Databases 5 Saturday, 23 February, 13

  14. Databases 5 Saturday, 23 February, 13

  15. Databases 5 Saturday, 23 February, 13

  16. Databases Memcached 5 Saturday, 23 February, 13

  17. Databases Memcached Sphinx -> ElasticSearch 5 Saturday, 23 February, 13

  18. + 6 Saturday, 23 February, 13

  19. + Activity Feed 6 Saturday, 23 February, 13

  20. + Activity Feed User Settings 6 Saturday, 23 February, 13

  21. + Activity Feed User Settings Push Notification 6 Saturday, 23

    February, 13

  22. + Activity Feed User Settings Push Notification Device track 6

    Saturday, 23 February, 13

  23. + Activity Feed User Settings Push Notification Device track Stats

    6 Saturday, 23 February, 13

  24. Servers 1.0 7 Saturday, 23 February, 13

  25. Servers 1.0 mongod primary 7 Saturday, 23 February, 13

  26. Servers 1.0 mongod primary mongod secondary 7 Saturday, 23 February,

    13

  27. Servers 1.0 mongod primary mongod secondary mongod secondary 7 Saturday,

    23 February, 13

  28. Servers 1.0 mongod primary mongod secondary mongod secondary mongod hidden

    7 Saturday, 23 February, 13

  29. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc Main 8 Saturday, 23 February, 13

  30. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod Main 8 Saturday, 23 February, 13

  31. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod Main 8 Saturday, 23 February, 13

  32. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  33. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  34. User Settings { "_id" : 123, "stats_page_empty" : true, "statistics_page_order"

    : [ 1, 4, 2, 3 ] } 9 Saturday, 23 February, 13

  35. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications 10 Saturday, 23 February, 13

  36. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"user_id": 1} 10 Saturday, 23 February, 13

  37. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  38. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  39. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  40. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1, "updated_at": -1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  41. Activity Feed - Flow 200M+ { "_id" : ObjectId(""), "aid"

    : 1127689, # Actor id "otype" : 9, # Object type for cast "v" : 7, # Verb of the action "oid" : 25000475, # Object id "uid" : 1127689 # User id } 11 Saturday, 23 February, 13

  42. db.ensureIndex ??? { aid: 1 } { aid: 1, _id:

    1 } { _id: 1, aid: 1 } { aid: 1, v: 1, _id: 1 } 12 Saturday, 23 February, 13

  43. Data access 13 Saturday, 23 February, 13

  44. Data access db.activities.find( aid: 22, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10)

    13 Saturday, 23 February, 13

  45. { aid: 1 } 14 Saturday, 23 February, 13

  46. { aid: 1, v:1 } { aid: 1, v:1, _id:

    -1 } 15 Saturday, 23 February, 13

  47. { aid: 1, _id: -1 } { aid: 1, _id:

    -1, v: 1 } 16 Saturday, 23 February, 13

  48. 17 Saturday, 23 February, 13

  49. db.activities.find( aid: {'$in': [1,10]}, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 17

    Saturday, 23 February, 13

  50. Shard key ??? 18 Saturday, 23 February, 13

  51. Shard key ??? { aid: 1, v: 1, _id: 1

    } 18 Saturday, 23 February, 13

  52. Mistakes on shard 19 Saturday, 23 February, 13

  53. Mistakes on shard Automatic shard 19 Saturday, 23 February, 13

  54. Mistakes on shard Automatic shard SSDs on Primaries 19 Saturday,

    23 February, 13

  55. After sharding? 20 Saturday, 23 February, 13

  56. After sharding? Good performance on writes 20 Saturday, 23 February,

    13

  57. After sharding? Good performance on User Flow reads Good performance

    on writes 20 Saturday, 23 February, 13

  58. After sharding? Good performance on User Flow reads Bad performance

    on Friends Flow reads Good performance on writes 20 Saturday, 23 February, 13

  59. Solutions: 21 Saturday, 23 February, 13

  60. Solutions: #1: denormalize 21 Saturday, 23 February, 13

  61. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user 22 Saturday, 23 February, 13

  62. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user Fan out on write 22 Saturday, 23 February, 13

  63. Solutions: #1: denormalize 23 Saturday, 23 February, 13

  64. Solutions: #1: denormalize #2: cache 23 Saturday, 23 February, 13

  65. key: user_id 24 Saturday, 23 February, 13

  66. key: user_id prepend activity 24 Saturday, 23 February, 13

  67. key: user_id prepend activity activity append 24 Saturday, 23 February,

    13

  68. key: user_id prepend activity activity append activity append 24 Saturday,

    23 February, 13

  69. Future Denormaziled solution. 25 Saturday, 23 February, 13

  70. Thank you 26 Saturday, 23 February, 13