Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Toronto MongoDB User Group

Avatar for Arthur Nogueira Neves Arthur Nogueira Neves
February 20, 2013
Technology
1
86

Toronto MongoDB User Group

Presetation given on Toronto MongoDB User group

Avatar for Arthur Nogueira Neves

Arthur Nogueira Neves

February 20, 2013
Tweet

More Decks by Arthur Nogueira Neves

See All by Arthur Nogueira Neves
Using multiple connections in ActiveRecord
Avatar for Arthur Nogueira Neves arthurnn
3
970
Rails talk - RubyLightningTalksTO
Avatar for Arthur Nogueira Neves arthurnn
0
95
Abril Pro Ruby
Avatar for Arthur Nogueira Neves arthurnn
0
350

Other Decks in Technology

See All in Technology
小規模チームによる衛星管制システムの開発とスケーラビリティの実現
Avatar for Takahiro Miyoshi sankichi92
0
170
AI駆動開発によるDDDの実践
Avatar for ディップ株式会社 dip_tech
PRO
0
160
AI/MLのマルチテナント基盤を支えるコンテナ技術
Avatar for Preferred Networks pfn
PRO
4
500
一億総業務改善を支える社内AIエージェント基盤の要諦
Avatar for Yuku Kotani yukukotani
7
2.3k
事業部のプロジェクト進行と開発チームの改善の “時間軸" のすり合わせ
Avatar for konifar konifar
9
2k
GitHub を組織的に使いこなすために ソニーが実践した全社展開のプラクティス
Avatar for ソニー株式会社 sony
20
11k
Contract One Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
9.8k
AI時代のインシデント対応 〜時代を切り抜ける、組織アーキテクチャ〜
Avatar for Kazuto Kusama jacopen
4
180
"なるべくスケジューリングしない" を実現する "PreferNoSchedule" taint
Avatar for Kazuki Suda superbrothers
0
130
私のRails開発環境
Avatar for Yasuo Honda yahonda
0
170
useEffectってなんで非推奨みたいなこと言われてるの？
Avatar for マグロ隊長kinTV maguroalternative
9
5.9k
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
Avatar for Sansan, Inc. sansan33
PRO
0
2.9k

Featured

See All Featured
Side Projects
Avatar for Sacha Greif sachag
455
43k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.9k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
1
67
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
34
2.5k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
210
24k
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
52
5.7k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
73
11k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.1k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Scaling GitHub
Avatar for Zach Holman holman
464
140k

Transcript

  1. MongoDB Toronto User group 1 Saturday, 23 February, 13

  2. Hi. 2 Saturday, 23 February, 13

  3. Hi. @arthurnn 2 Saturday, 23 February, 13

  4. Hi. @arthurnn 2 Saturday, 23 February, 13

  5. Hi. @arthurnn 2 Saturday, 23 February, 13

  6. 3 Saturday, 23 February, 13

  7. Tech 4 Saturday, 23 February, 13

  8. Tech 4 Saturday, 23 February, 13

  9. Tech 4 Saturday, 23 February, 13

  10. Tech 4 Saturday, 23 February, 13

  11. Tech 4 Saturday, 23 February, 13

  12. RPC Tech 4 Saturday, 23 February, 13

  13. Databases 5 Saturday, 23 February, 13

  14. Databases 5 Saturday, 23 February, 13

  15. Databases 5 Saturday, 23 February, 13

  16. Databases Memcached 5 Saturday, 23 February, 13

  17. Databases Memcached Sphinx -> ElasticSearch 5 Saturday, 23 February, 13

  18. + 6 Saturday, 23 February, 13

  19. + Activity Feed 6 Saturday, 23 February, 13

  20. + Activity Feed User Settings 6 Saturday, 23 February, 13

  21. + Activity Feed User Settings Push Notification 6 Saturday, 23

    February, 13

  22. + Activity Feed User Settings Push Notification Device track 6

    Saturday, 23 February, 13

  23. + Activity Feed User Settings Push Notification Device track Stats

    6 Saturday, 23 February, 13

  24. Servers 1.0 7 Saturday, 23 February, 13

  25. Servers 1.0 mongod primary 7 Saturday, 23 February, 13

  26. Servers 1.0 mongod primary mongod secondary 7 Saturday, 23 February,

    13

  27. Servers 1.0 mongod primary mongod secondary mongod secondary 7 Saturday,

    23 February, 13

  28. Servers 1.0 mongod primary mongod secondary mongod secondary mongod hidden

    7 Saturday, 23 February, 13

  29. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc Main 8 Saturday, 23 February, 13

  30. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod Main 8 Saturday, 23 February, 13

  31. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod Main 8 Saturday, 23 February, 13

  32. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  33. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  34. User Settings { "_id" : 123, "stats_page_empty" : true, "statistics_page_order"

    : [ 1, 4, 2, 3 ] } 9 Saturday, 23 February, 13

  35. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications 10 Saturday, 23 February, 13

  36. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"user_id": 1} 10 Saturday, 23 February, 13

  37. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  38. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  39. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  40. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1, "updated_at": -1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  41. Activity Feed - Flow 200M+ { "_id" : ObjectId(""), "aid"

    : 1127689, # Actor id "otype" : 9, # Object type for cast "v" : 7, # Verb of the action "oid" : 25000475, # Object id "uid" : 1127689 # User id } 11 Saturday, 23 February, 13

  42. db.ensureIndex ??? { aid: 1 } { aid: 1, _id:

    1 } { _id: 1, aid: 1 } { aid: 1, v: 1, _id: 1 } 12 Saturday, 23 February, 13

  43. Data access 13 Saturday, 23 February, 13

  44. Data access db.activities.find( aid: 22, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10)

    13 Saturday, 23 February, 13

  45. { aid: 1 } 14 Saturday, 23 February, 13

  46. { aid: 1, v:1 } { aid: 1, v:1, _id:

    -1 } 15 Saturday, 23 February, 13

  47. { aid: 1, _id: -1 } { aid: 1, _id:

    -1, v: 1 } 16 Saturday, 23 February, 13

  48. 17 Saturday, 23 February, 13

  49. db.activities.find( aid: {'$in': [1,10]}, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 17

    Saturday, 23 February, 13

  50. Shard key ??? 18 Saturday, 23 February, 13

  51. Shard key ??? { aid: 1, v: 1, _id: 1

    } 18 Saturday, 23 February, 13

  52. Mistakes on shard 19 Saturday, 23 February, 13

  53. Mistakes on shard Automatic shard 19 Saturday, 23 February, 13

  54. Mistakes on shard Automatic shard SSDs on Primaries 19 Saturday,

    23 February, 13

  55. After sharding? 20 Saturday, 23 February, 13

  56. After sharding? Good performance on writes 20 Saturday, 23 February,

    13

  57. After sharding? Good performance on User Flow reads Good performance

    on writes 20 Saturday, 23 February, 13

  58. After sharding? Good performance on User Flow reads Bad performance

    on Friends Flow reads Good performance on writes 20 Saturday, 23 February, 13

  59. Solutions: 21 Saturday, 23 February, 13

  60. Solutions: #1: denormalize 21 Saturday, 23 February, 13

  61. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user 22 Saturday, 23 February, 13

  62. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user Fan out on write 22 Saturday, 23 February, 13

  63. Solutions: #1: denormalize 23 Saturday, 23 February, 13

  64. Solutions: #1: denormalize #2: cache 23 Saturday, 23 February, 13

  65. key: user_id 24 Saturday, 23 February, 13

  66. key: user_id prepend activity 24 Saturday, 23 February, 13

  67. key: user_id prepend activity activity append 24 Saturday, 23 February,

    13

  68. key: user_id prepend activity activity append activity append 24 Saturday,

    23 February, 13

  69. Future Denormaziled solution. 25 Saturday, 23 February, 13

  70. Thank you 26 Saturday, 23 February, 13