Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Toronto MongoDB User Group
Search
Arthur Nogueira Neves
February 20, 2013
Technology
1
85
Toronto MongoDB User Group
Presetation given on Toronto MongoDB User group
Arthur Nogueira Neves
February 20, 2013
Tweet
Share
More Decks by Arthur Nogueira Neves
See All by Arthur Nogueira Neves
Using multiple connections in ActiveRecord
arthurnn
3
960
Rails talk - RubyLightningTalksTO
arthurnn
0
94
Abril Pro Ruby
arthurnn
0
330
Other Decks in Technology
See All in Technology
Introduction to Bill One Development Engineer
sansan33
PRO
0
260
microCMSではじめるAIライティング
himaratsu
0
150
AI時代にも変わらぬ価値を発揮したい: インフラ・クラウドを切り口にユーザー価値と非機能要件に向き合ってエンジニアとしての地力を培う
netmarkjp
0
130
今だから言えるセキュリティLT_Wordpress5.7.2未満を一斉アップデートせよ
cuebic9bic
2
170
Bill One 開発エンジニア 紹介資料
sansan33
PRO
4
13k
CDKコード品質UP!ナイスな自作コンストラクタを作るための便利インターフェース
harukasakihara
2
230
アクセスピークを制するオートスケール再設計: 障害を乗り越えKEDAで実現したリソース管理の最適化
myamashii
1
660
Contract One Engineering Unit 紹介資料
sansan33
PRO
0
6.9k
Data Engineering Study#30 LT資料
tetsuroito
1
180
データ戦略部門 紹介資料
sansan33
PRO
1
3.3k
ソフトウェアテストのAI活用_ver1.25
fumisuke
1
610
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
520
KATA
mclloyd
30
14k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
990
Building Applications with DynamoDB
mza
95
6.5k
A better future with KSS
kneath
238
17k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
Side Projects
sachag
455
42k
Raft: Consensus for Rubyists
vanstee
140
7k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Transcript
MongoDB Toronto User group 1 Saturday, 23 February, 13
Hi. 2 Saturday, 23 February, 13
Hi. @arthurnn 2 Saturday, 23 February, 13
Hi. @arthurnn 2 Saturday, 23 February, 13
Hi. @arthurnn 2 Saturday, 23 February, 13
3 Saturday, 23 February, 13
Tech 4 Saturday, 23 February, 13
Tech 4 Saturday, 23 February, 13
Tech 4 Saturday, 23 February, 13
Tech 4 Saturday, 23 February, 13
Tech 4 Saturday, 23 February, 13
RPC Tech 4 Saturday, 23 February, 13
Databases 5 Saturday, 23 February, 13
Databases 5 Saturday, 23 February, 13
Databases 5 Saturday, 23 February, 13
Databases Memcached 5 Saturday, 23 February, 13
Databases Memcached Sphinx -> ElasticSearch 5 Saturday, 23 February, 13
+ 6 Saturday, 23 February, 13
+ Activity Feed 6 Saturday, 23 February, 13
+ Activity Feed User Settings 6 Saturday, 23 February, 13
+ Activity Feed User Settings Push Notification 6 Saturday, 23
February, 13
+ Activity Feed User Settings Push Notification Device track 6
Saturday, 23 February, 13
+ Activity Feed User Settings Push Notification Device track Stats
6 Saturday, 23 February, 13
Servers 1.0 7 Saturday, 23 February, 13
Servers 1.0 mongod primary 7 Saturday, 23 February, 13
Servers 1.0 mongod primary mongod secondary 7 Saturday, 23 February,
13
Servers 1.0 mongod primary mongod secondary mongod secondary 7 Saturday,
23 February, 13
Servers 1.0 mongod primary mongod secondary mongod secondary mongod hidden
7 Saturday, 23 February, 13
Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod
mongoc mongod mongoc mongod mongoc Main 8 Saturday, 23 February, 13
Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod
mongoc mongod mongoc mongod mongoc mongod Main 8 Saturday, 23 February, 13
Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod
mongoc mongod mongoc mongod mongoc mongod mongod Main 8 Saturday, 23 February, 13
Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod
mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13
Servers 2.0 mongod mongod mongod mongod mongod mongod Primary mongod
mongoc mongod mongoc mongod mongoc mongod mongod mongod Main 8 Saturday, 23 February, 13
User Settings { "_id" : 123, "stats_page_empty" : true, "statistics_page_order"
: [ 1, 4, 2, 3 ] } 9 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications 10 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"user_id": 1} 10 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} 10 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1} 10 Saturday, 23 February, 13
Push notifications { "_id" : ObjectId("4fe60f72a3a6a5654b000001"), "status" : 24, "user_id"
: 1, "photo_id" : 8026403, "actors" : [ 71 ], "action_id" : 0, "updated_at" : ISODate("2012-06-23T18:48:18Z"), "created_at" : ISODate("2012-06-23T18:48:18Z") } { "_id" : ObjectId("4ffc9eb55328f538ad00002d"), "permissions" : [ 0, 1, 5, 10 ], "type" : 1, "user_id" : 357534, "token" : "xxxx xxxx xxxx" } db.mobile_devices db.mobile_notifications index: {"token": 1}, {sparse: true} index: {"registration_id": 1}, {sparse: true} index: {"user_id": 1} index: {"user_id": 1, "updated_at": -1} index: {"user_id": 1} 10 Saturday, 23 February, 13
Activity Feed - Flow 200M+ { "_id" : ObjectId(""), "aid"
: 1127689, # Actor id "otype" : 9, # Object type for cast "v" : 7, # Verb of the action "oid" : 25000475, # Object id "uid" : 1127689 # User id } 11 Saturday, 23 February, 13
db.ensureIndex ??? { aid: 1 } { aid: 1, _id:
1 } { _id: 1, aid: 1 } { aid: 1, v: 1, _id: 1 } 12 Saturday, 23 February, 13
Data access 13 Saturday, 23 February, 13
Data access db.activities.find( aid: 22, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10)
13 Saturday, 23 February, 13
{ aid: 1 } 14 Saturday, 23 February, 13
{ aid: 1, v:1 } { aid: 1, v:1, _id:
-1 } 15 Saturday, 23 February, 13
{ aid: 1, _id: -1 } { aid: 1, _id:
-1, v: 1 } 16 Saturday, 23 February, 13
17 Saturday, 23 February, 13
db.activities.find( aid: {'$in': [1,10]}, v: {'$in': [2,3,4]}) .sort(_id: -1).limit(10) 17
Saturday, 23 February, 13
Shard key ??? 18 Saturday, 23 February, 13
Shard key ??? { aid: 1, v: 1, _id: 1
} 18 Saturday, 23 February, 13
Mistakes on shard 19 Saturday, 23 February, 13
Mistakes on shard Automatic shard 19 Saturday, 23 February, 13
Mistakes on shard Automatic shard SSDs on Primaries 19 Saturday,
23 February, 13
After sharding? 20 Saturday, 23 February, 13
After sharding? Good performance on writes 20 Saturday, 23 February,
13
After sharding? Good performance on User Flow reads Good performance
on writes 20 Saturday, 23 February, 13
After sharding? Good performance on User Flow reads Bad performance
on Friends Flow reads Good performance on writes 20 Saturday, 23 February, 13
Solutions: 21 Saturday, 23 February, 13
Solutions: #1: denormalize 21 Saturday, 23 February, 13
{ "_id" : ObjectId(""), "user" : 123, # viewer "page"
: 1, "feed" : [{activity}, {activity}] } Flow Feed by user 22 Saturday, 23 February, 13
{ "_id" : ObjectId(""), "user" : 123, # viewer "page"
: 1, "feed" : [{activity}, {activity}] } Flow Feed by user Fan out on write 22 Saturday, 23 February, 13
Solutions: #1: denormalize 23 Saturday, 23 February, 13
Solutions: #1: denormalize #2: cache 23 Saturday, 23 February, 13
key: user_id 24 Saturday, 23 February, 13
key: user_id prepend activity 24 Saturday, 23 February, 13
key: user_id prepend activity activity append 24 Saturday, 23 February,
13
key: user_id prepend activity activity append activity append 24 Saturday,
23 February, 13
Future Denormaziled solution. 25 Saturday, 23 February, 13
Thank you 26 Saturday, 23 February, 13