Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Toronto MongoDB User Group

Avatar for Arthur Nogueira Neves Arthur Nogueira Neves
February 20, 2013
Technology
1
85

Toronto MongoDB User Group

Presetation given on Toronto MongoDB User group

Avatar for Arthur Nogueira Neves

Arthur Nogueira Neves

February 20, 2013
Tweet

More Decks by Arthur Nogueira Neves

See All by Arthur Nogueira Neves
Using multiple connections in ActiveRecord
Avatar for Arthur Nogueira Neves arthurnn
3
970
Rails talk - RubyLightningTalksTO
Avatar for Arthur Nogueira Neves arthurnn
0
94
Abril Pro Ruby
Avatar for Arthur Nogueira Neves arthurnn
0
340

Other Decks in Technology

See All in Technology
社内報はAIにやらせよう / Let AI handle the company newsletter
Avatar for Jumpei Sakatsu saka2jp
8
1.2k
AIAgentの限界を超え、 現場を動かすWorkflowAgentの設計と実践
Avatar for Koji Miyata miyatakoji
1
160
許しとアジャイル
Avatar for Jun Nakajima jnuank
1
140
神回のメカニズムと再現方法/Mechanisms and Playbook for Kamikai scrumat2025
Avatar for moriyuya moriyuya
4
680
セキュアな認可付きリモートMCPサーバーをAWSマネージドサービスでつくろう！ / Let's build an OAuth protected remote MCP server based on AWS managed services
Avatar for 株式会社カミナシ kaminashi
3
220
Function calling機能をPLaMo2に実装するには / PFN LLMセミナー
Avatar for Preferred Networks pfn
PRO
0
990
M5製品で作るポン置きセルラー対応カメラ
Avatar for Hisaya OKADA sayacom
0
170
VCC 2025 Write-up
Avatar for bata_24 bata_24
0
190
カンファレンスに託児サポートがあるということ / Having Childcare Support at Conferences
Avatar for nobu09 nobu09
1
430
やる気のない自分との向き合い方/How to Deal with Your Unmotivated Self
Avatar for Genki Sano sanogemaru
0
430
ACA でMAGI システムを社内で展開しようとした話
Avatar for Yuji Masaoka | まっぴぃ mappie_kochi
1
300
そのWAFのブロック、どう活かす？ サービスを守るための実践的多層防御と思考法 / WAF blocks defense decision
Avatar for 株式会社カミナシ kaminashi
0
110

Featured

See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
33
2.5k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
189
55k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
84
9.2k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
Docker and Python
Avatar for Tania Allard trallard
46
3.6k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
73
11k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
2.7k
Code Review Best Practice
Avatar for Trisha Gee trishagee
72
19k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
139
7.1k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
127
17k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
140
34k

Transcript

  1. MongoDB Toronto User group 1 Saturday, 23 February, 13

  2. Hi. 2 Saturday, 23 February, 13

  3. Hi. @arthurnn 2 Saturday, 23 February, 13

  4. Hi. @arthurnn 2 Saturday, 23 February, 13

  5. Hi. @arthurnn 2 Saturday, 23 February, 13

  6. 3 Saturday, 23 February, 13

  7. Tech 4 Saturday, 23 February, 13

  8. Tech 4 Saturday, 23 February, 13

  9. Tech 4 Saturday, 23 February, 13

  10. Tech 4 Saturday, 23 February, 13

  11. Tech 4 Saturday, 23 February, 13

  12. RPC Tech 4 Saturday, 23 February, 13

  13. Databases 5 Saturday, 23 February, 13

  14. Databases 5 Saturday, 23 February, 13

  15. Databases 5 Saturday, 23 February, 13

  16. Databases Memcached 5 Saturday, 23 February, 13

  17. Databases Memcached Sphinx -> ElasticSearch 5 Saturday, 23 February, 13

  18. + 6 Saturday, 23 February, 13

  19. + Activity Feed 6 Saturday, 23 February, 13

  20. + Activity Feed User Settings 6 Saturday, 23 February, 13

  21. + Activity Feed User Settings Push Notification 6 Saturday, 23

    February, 13

  22. + Activity Feed User Settings Push Notification Device track 6

    Saturday, 23 February, 13

  23. + Activity Feed User Settings Push Notification Device track Stats

    6 Saturday, 23 February, 13

  24. Servers 1.0 7 Saturday, 23 February, 13

  25. Servers 1.0 mongod primary 7 Saturday, 23 February, 13

  26. Servers 1.0 mongod primary mongod secondary 7 Saturday, 23 February,

    13

  27. Servers 1.0 mongod primary mongod secondary mongod secondary 7 Saturday,

    23 February, 13

  28. Servers 1.0 mongod primary mongod secondary mongod secondary mongod hidden

    7 Saturday, 23 February, 13

  29. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc Main 8 Saturday, 23 February, 13

  30. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod Main 8 Saturday, 23 February, 13

  31. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod Main 8 Saturday, 23 February, 13

  32. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  33. Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod

    mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13

  34. User Settings { "_id" : 123, "stats_page_empty" : true, "statistics_page_order"

    : [ 1, 4, 2, 3 ] } 9 Saturday, 23 February, 13

  35. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications 10 Saturday, 23 February, 13

  36. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"user_id": 1} 10 Saturday, 23 February, 13

  37. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  38. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13

  39. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  40. Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"

    : 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1, "updated_at": -1} index: {"user_id": 1} 10 Saturday, 23 February, 13

  41. Activity Feed - Flow 200M+ { "_id" : ObjectId(""), "aid"

    : 1127689, # Actor id "otype" : 9, # Object type for cast "v" : 7, # Verb of the action "oid" : 25000475, # Object id "uid" : 1127689 # User id } 11 Saturday, 23 February, 13

  42. db.ensureIndex ??? { aid: 1 } { aid: 1, _id:

    1 } { _id: 1, aid: 1 } { aid: 1, v: 1, _id: 1 } 12 Saturday, 23 February, 13

  43. Data access 13 Saturday, 23 February, 13

  44. Data access db.activities.find( aid: 22, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10)

    13 Saturday, 23 February, 13

  45. { aid: 1 } 14 Saturday, 23 February, 13

  46. { aid: 1, v:1 } { aid: 1, v:1, _id:

    -1 } 15 Saturday, 23 February, 13

  47. { aid: 1, _id: -1 } { aid: 1, _id:

    -1, v: 1 } 16 Saturday, 23 February, 13

  48. 17 Saturday, 23 February, 13

  49. db.activities.find( aid: {'$in': [1,10]}, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 17

    Saturday, 23 February, 13

  50. Shard key ??? 18 Saturday, 23 February, 13

  51. Shard key ??? { aid: 1, v: 1, _id: 1

    } 18 Saturday, 23 February, 13

  52. Mistakes on shard 19 Saturday, 23 February, 13

  53. Mistakes on shard Automatic shard 19 Saturday, 23 February, 13

  54. Mistakes on shard Automatic shard SSDs on Primaries 19 Saturday,

    23 February, 13

  55. After sharding? 20 Saturday, 23 February, 13

  56. After sharding? Good performance on writes 20 Saturday, 23 February,

    13

  57. After sharding? Good performance on User Flow reads Good performance

    on writes 20 Saturday, 23 February, 13

  58. After sharding? Good performance on User Flow reads Bad performance

    on Friends Flow reads Good performance on writes 20 Saturday, 23 February, 13

  59. Solutions: 21 Saturday, 23 February, 13

  60. Solutions: #1: denormalize 21 Saturday, 23 February, 13

  61. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user 22 Saturday, 23 February, 13

  62. { "_id" : ObjectId(""), "user" : 123, # viewer "page"

    : 1, "feed" : [{activity}, {activity}] } Flow Feed by user Fan out on write 22 Saturday, 23 February, 13

  63. Solutions: #1: denormalize 23 Saturday, 23 February, 13

  64. Solutions: #1: denormalize #2: cache 23 Saturday, 23 February, 13

  65. key: user_id 24 Saturday, 23 February, 13

  66. key: user_id prepend activity 24 Saturday, 23 February, 13

  67. key: user_id prepend activity activity append 24 Saturday, 23 February,

    13

  68. key: user_id prepend activity activity append activity append 24 Saturday,

    23 February, 13

  69. Future Denormaziled solution. 25 Saturday, 23 February, 13

  70. Thank you 26 Saturday, 23 February, 13