Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
月間10億PVから学んだMongoDBアンチパターン
Search
yujiosaka
November 11, 2014
Technology
19
5.5k
月間10億PVから学んだMongoDBアンチパターン
ZenClerkが月間10億PVを支えるまでの 過程で経験したアンチパターンを紹介
yujiosaka
November 11, 2014
Tweet
Share
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
260
Machine Learning with JavaScript
yujiosaka
0
180
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
450
ヘッドレスChromeでクローラを作った後の話
yujiosaka
3
680
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.3k
『XXX』のための管理画面
yujiosaka
1
1.3k
Enjoy Deep Learning by JavaScript
yujiosaka
1
350
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
570
Other Decks in Technology
See All in Technology
風が吹けばWHOISが使えなくなる~なぜWHOIS・RDAPはサーバー証明書のメール認証に使えなくなったのか~
orangemorishita
15
5.8k
ZOZOTOWNの大規模マーケティングメール配信を支えるアーキテクチャ
zozotech
PRO
0
330
Segment Anything Modelの最新動向:SAM2とその発展系
tenten0727
0
770
Strands Agents & Bedrock AgentCoreを1分でおさらい
minorun365
PRO
8
330
Oracle Exadata Database Service on Cloud@Customer X11M (ExaDB-C@C) サービス概要
oracle4engineer
PRO
2
6.3k
【新卒研修資料】数理最適化 / Mathematical Optimization
brainpadpr
27
13k
Serverless Meetup #21
yoshidashingo
1
120
生成AIによるデータサイエンスの変革
taka_aki
0
3k
Amazon Q と『音楽』-ゲーム音楽もAmazonQで作成してみた感想-
senseofunity129
0
140
AIエージェントを現場で使う / 2025.08.07 著者陣に聞く!現場で活用するためのAIエージェント実践入門(Findyランチセッション)
smiyawaki0820
6
1.1k
Google Agentspaceを実際に導入した効果と今後の展望
mixi_engineers
PRO
3
700
全員が手を動かす組織へ - 生成AIが変えるTVerの開発現場 / everyone-codes-genai-transforms-tver-development
tohae
0
180
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Designing Experiences People Love
moore
142
24k
How STYLIGHT went responsive
nonsquared
100
5.7k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
RailsConf 2023
tenderlove
30
1.2k
Scaling GitHub
holman
461
140k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Done Done
chrislema
185
16k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
760
Transcript
݄ؒ10ԯPV͔ΒֶΜͩ MongoDBΞϯνύλʔϯ db tech showcase ౦ژ 2014 speaker: “Yuji Isobe”
history: [ “violinist” “engineer” “security specialist” “socket.io contributor” “startup member
of ZenClerk ” ] name: “Yuji Isobe”
questions: -> MongoDBΛ༻ͨ͜͠ͱ͕͋Γ·͔͢ʁ -> ຊ൪ڥͰMongoDBΛ༻͍ͯ͠·͔͢ʁ
goal: -> ZenClerk͕݄ؒ10ԯPVΛࢧ͑Δ·Ͱͷ աఔͰܦݧͨ͠ΞϯνύλʔϯΛհ -> ओʹύϑΥʔϚϯενϡʔχϯάͷ ->ʮ݄ؒ10ԯPVΛࢧ͑ΔMongoDBʯͷͦͷઌ IUUQXXXTMJEFTIBSFOFUZVKJPTBLBQWNPOHPEC
None
ϦΞϧλΠϜ
ϦΞϧλΠϜੳ Ϗοάσʔλ
db.ZenClerk .stats() -> ݄ؒ10ԯPV -> ඵؒ400PV -> ಉ࣌ଓ5ສ -> ݄ؒ10TBอଘ
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
i2.2xlarge m1.large m3.large db.ZenClerk .aggregate() r3.large
ʙ1000ສPV mogngod
1000ສʙ1ԯPV delayed replica replica set
1ԯPVʙ10ԯPV replica set shard mongos replica set shard replica set
shard delayed replica delayed replica delayed replica mongoc
͜͜·ͰΞϓϦέʔγϣϯͷมߋ ΄ͱΜͲͳ͠ʢҰ෦ΫΤϦΛ࠷దԽʣ
we.use(“MongoDB”).explain() -> εΩʔϚϨε -> τϥϯβΫγϣϯෆཁ ʢ࠷ݶͷΞτϛοΫૢ࡞ʣ -> ॊೈͳΫΤϦ -> ༰қͳεέʔϧΞτ
-> Node.jsͱͷ૬ੑͷྑ͞
we.use(“MongoDB”).explain(true) -> ಋೖͷ༰қ͞ʢϗεςΟϯάαʔϏε༗ແʣ -> ֶशͷ༰қ͞ʢΤϯδχΞҎ֎ʹͱͬͯʣ
ࠓ͢ͷɾɾɾ ϕετϓϥΫςΟε Ξϯνύλʔϯ
ͪ͜Βʂ ϕετϓϥΫςΟε Ξϯνύλʔϯ
Ͱͦͷલʹɾɾɾ
ϕετϓϥΫςΟε
εΩʔϚσβΠϯ -> TwitterͷλΠϜϥΠϯΛอଘ͢Δ -> RDBMSͳΒਖ਼نԽ -> MongoDBͷϕετϓϥΫςΟεʁ
-> ಡΈࠐΈॏࢹ -> 1ΫΤϦ1ώοτ -> ίϝϯτʹԠͯ͡ ॻ͖ࠐΈ͕ॏ͘ͳΔ υΩϡϝϯτΛຒΊࠐΉ session: _id:
“session_id” device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”} ]
-> ॻ͖ࠐΈॏࢹ -> 2ΫΤϦN+1ώοτ -> ॻ͖ࠐΈεέʔϧ͢Δ υΩϡϝϯτΛຒΊࠐ·ͳ͍ session: _id: “session_id”
device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” ! tweet: _id: 1234 user_id: “yujiosaka” text: “MongoDB Rocks!”
timeline: _id: “yujiosaka-20141111” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”}
] -> όϥϯεܕ -> 1ΫΤϦώοτ -> ॻ͖ࠐΈ1·Ͱ ͔͠ॏ͘ͳΒͳ͍ ϋΠϒϦου
εΩʔϚϨε ≠ εΩʔϚఆ͕ٛෆཁ RDBMSΑΓબࢶ͕ଟ͍͘͠ tips: IUUQXXXNPOHPECDPNQSFTFOUBUJPOTTDIFNBEFTJHOTDBMF
Ξϯνύλʔϯ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶃΠϯσοΫεͷషΓա͗
MongoDBϝϞϦͷ͍ํ͕ ͋·Γݡ͋͘Γ·ͤΜ
before: user = new mongoose.Schema first: {type: String, required: true,
index: true} last: {type: String, required: true, index: true} city: {type: String, required: true, index: true} ... created_at: {type: Date, default: Date.now, index: true} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
after: user = new mongoose.Schema first: {type: String, required: true}
last: {type: String, required: true} city: {type: String, required: true} ... created_at: {type: Date, default: Date.now} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
ಡΈࠐΈɺΠϯσοΫε͕ ϝϞϦʹΒͳ͚Ε͘ͳΔ
tips: όʔδϣϯ2.8ʹPluggable Storage Engine͕༧ఆ͞Ε͍ͯΔͷͰɺ ݡ͍Τϯδϯ͕બΔΑ͏ʹͳΔ https://blog.compose.io/the-coming-of-the-mongodb-storage-engines/
ᶄจࣈྻͷΠϯσοΫε
ΠϯσοΫεͷߏ -> RDBMSͱ΄ͱΜͲಉ͡ -> ΠϯσοΫεʹB-TreeΛ࠾༻ -> B-TreeৗʹฏߧΛอ͍ͬͯΔ
จࣈྻͷΠϯσοΫε -> ϗοτσʔλ͕ࢄ -> ϝϞϦޮ͕ѱ͍
࣌ؒɾObjectIdͷΠϯσοΫε -> ϗοτσʔλ͕ूத -> ϝϞϦޮ͕ྑ͍
tips: Ͱ͖Δ͚ͩObjectIdΛ͍·͠ΐ͏ ޙ͔Βͷஔ͖͑େมͰ͢
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ
Ϣʔβʔใͷ͏ͪ ࢢใ͚͕ͩཉ͍͠ user: {_id: “yujiosaka”, first: “Yuji”, last: “Isobe”, city:
“Tokyo”} index: {_id:1, city: 1} // compound index
ϑΟʔϧυࢦఆͳ͠ -> SQLͰͷʮselect *ʯ -> ෆཁͳωοτϫʔΫίετ -> ෆཁͳσΟεΫΞΫηε var user
= db.users.findOne({_id: “yujiosaka”});
ϑΟʔϧυࢦఆ͋Γ -> SQLͰͷʮselect cityʯ -> ωοτϫʔΫίετͷઅ -> covered index ͕׆༻Ͱ͖Δ
var user = db.users.findOne({_id: “yujiosaka”}, {city: 1});
tips: covered indexΛ༗ޮ׆༻͢Εɺ σΟεΫͷΞΫηεҰͳ͠ʹ ϝϞϦ͚͔ͩΒใΛಡΈࠐΊΔ
ᶆίʔϧυσʔλͷΫΤϦ
MongoDBͷಛ -> ϗοτσʔλͷΞΫηεߴ -> ίʔϧυσʔλͷΞΫηε͍ -> શσʔλͷΞΫηεۤख
ZenClerkʹ͓͚Δूܭ -> 5͝ͱʹσʔλΛཁ -> 1࣌ؒ͝ͱʹ౷ܭσʔλΛूܭ -> 1͝ͱʹ౷ܭσʔλΛूܭ
tips: ৗʹϗοτσʔλΛҙࣝͨ͠ ͍ํΛ͠·͠ΐ͏
ᶇϓϥΠϚϦʹภͬͨΫΤϦ
ϨϓϦΧηοτ -> ΫΤϦΛࢄͤ͞Δ͜ͱ͕Ͱ͖Δ -> ηΧϯμϦʹΫΤϦΛ͛Δ͔ ϓϩάϥϚ͕ܾΊΒΕΔ -> σϑΥϧτͰϓϥΠϚϦʹ͛ΒΕΔ
σϑΥϧτ -> ϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλ͕อো͞ΕΔ var user = db.users.find({city: “Tokyo”});
ηΧϯμϦΛࢦఆ -> ηΧϯμϦ͕͋ΕηΧϯμϦ͔Βɺ ͳ͚ΕϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλอো͞Εͳ͍ var user = db.users.find({city:
“Tokyo”}) .readPref("secondaryPreferred")
tips: σʔλ͕࠷৽Ͱͳͯ͘Α͍߹ ಡΈࠐΈΛηΧϯμϦʹ͚Δ͜ͱͰ ΫΤϦ͕ࢄ͢Δ IUUQTTQFBLFSEFDLDPNNPOHPECQSBDUJDBM TDBMJOHBOETIBSEJOHFMJPUIPSPXJU[HFO
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶈඇޮͳΞοϓσʔτ
Α͘ݟ͔͚Δίʔυᶃ -> ඞͣΫΤϦ͕࣮ߦ͞ΕΔ -> ແବͳωοτϫʔΫίετ -> ඇΞτϛοΫͳॲཧ var user =
db.users.findOne({_id: “yujiosaka”}); user.count += 1; user.save();
Α͘ݟ͔͚Δίʔυᶃ -> updateͱ$incΛͬͯ·ͱΊΒΕΔ -> 1ճͷΞοϓσʔτͰࡁΉ -> ΞτϛοΫͳॲཧ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}});
Α͘ݟ͔͚Δίʔυᶄ -> σʔλ͕ݟ͔ͭͬͨΒߋ৽͠ɺݟ͔ͭΒͳ͔ͬͨΒ࡞͢Δ -> Ұݟෳࡶͳίʔυ var user = db.users.findOne({_id: “yujiosaka”});
if (user) { user.count += 1; user.save(); } else { db.users.insert({_id: “yujiosaka”, count: 1}); }
-> {upsert: true} Λ͏ͱ·ͱΊΒΕΔ -> ΑΓγϯϓϧͳίʔυ Α͘ݟ͔͚Δίʔυᶄ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}}, {upsert: true});
ᶉංେԽ͢ΔυΩϡϝϯτ
Ϛεͷ࠲ඪσʔλΛ อଘ͢Δ behavior: points: [{x: 151, y: 100} {x: 151,
y: 179} {x: 151, y: 266} … {x: 151, y: 340}]
MongoDBͷۤख -> ϑΟʔϧυ͕ංେԽ͢ΔΞοϓσʔτ͕͍ -> σʔλ͕ҰఆͷαΠζʹऩ·Βͳ͘ͳΔͱ σΟεΫͰσʔλͷҠಈ͕ى͖Δ -> RedisͰMongoDBͷۤखΛิ͏
Redis -> ΦϯϝϞϦܕͷKVS -> ಡΈࠐΈɾॻ͖ࠐΈ͕ૣ͍ -> جຊతʹσʔλͷӬଓԽ͠ͳ͍ -> σʔλܕ͕ඇৗʹ๛ ʢϦετɾηοτɾϋογϡʣ
RedisΛΘͳ͍߹ db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 100}}); db.col.update({_id:
1234}, {$push: {points: {x: 151, y: 179}}); db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 266}}); … db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 340}}); -> ࠲ඪ͕Ҡಈ͢ΔͨͼʹMongoDBʹॻ͖ࠐ·ΕΔ -> αΠζ͕ऩ·ΓΒͳ͍ͱɺσʔλͷҠಈ͕ى͖Δ
RedisΛͬͨ߹ db.col.insert({points: [ {x: 151, y: 100}, {x: 151, y:
179}, {x: 151, y: 266}, … {x: 151, y: 340} ]}); -> Ұ͔͠MongoDBʹॻ͖ࠐ·ͳ͍ -> υΩϡϝϯυͷαΠζ͜ΕҎ্େ͖͘ͳΒͳ͍
tips: ͯ͢ΛMongoDB͚ͩͰΖ͏ͱͤͣɺ ۤखͳ෦ଞͷٕज़Ͱิ͏͜ͱ͕େ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶊ٧ΊࠐΈա͗DB
σʔλΛҰͭͷDBʹ٧ΊࠐΜͰ ྑ͍͜ͱʢࠓͷͱ͜Ζʣ͋·Γͳ͍
ͳͥDBΛ͚Δͷ͔ʁ -> Reader-WriterϩοΫ͕ΘΕ͍ͯΔ -> 2.2.0Ҏ߱DB୯ҐͰϩοΫ͢Δ -> ϚγϯΛ͚ΕϝϞϦ͕૿͑Δ
ZenClerk ʹ͓͚ΔDBͷ༻్ -> ϝΠϯσʔλʢHotʣ -> ϝΠϯσʔλʢColdʣ -> ੜσʔλ -> ܭࢉ݁Ռσʔλ
-> ౷ܭσʔλ -> αΠτଐੑσʔλ
tips: όʔδϣϯ2.8ʹdocument-level locking 3.0Ҏ߱ʹpartitioned joins͕༧ఆ͞Ε͍ͯΔ ʢޙऀ͋·Γ͋ͯʹ͠ͳ͍ํ͕ແͦ͏ʣ http://www.slideshare.net/tetsutarowatanabe/mongodb-world-2014-37081258
ᶃΠϯσοΫεͷషΓա͗ ᶄจࣈྻͷΠϯσοΫε ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ ᶆίʔϧυσʔλͷΫΤϦ ᶇϓϥΠϚϦʹภͬͨΫΤϦ ᶈඇޮͳΞοϓσʔτ ᶉංେԽ͢ΔυΩϡϝϯτ ᶊ٧ΊࠐΈա͗DB Ξϯνύλʔϯ͓͞Β͍
Ξϯνύλʔϯʹ ؕΒͳ͍ͨΊʹ
̏ͭͷࢹ -> ϦΞϧλΠϜࢹ -> ݟฦͨ͢Ίͷࢹ -> ͍ΫΤϦͷࢹ
ϦΞϧλΠϜࢹ
-> ComposeʢMongoHQʣμογϡϘʔυ -> ݱࡏͷ݈߁νΣοΫ༻ -> ࣾʹઐ༻σΟεϓϨΠΛઃஔ -> ҟৗ͕͋Εʮ୭͔Mongoࡴͨ͠ʁʯ ϦΞϧλΠϜࢹ
ݟฦͨ͢Ίͷࢹ
-> MMSμογϡϘʔυ -> ఆظతͳ݈߁அ -> োൃੜ࣌ʹݟฦͯ͠ ɹʮͳΜͰ͜ΜͳϩοΫߴ͘ͳͬͯΜͷʁʯ ݟฦͨ͢Ίͷࢹ
͍ΫΤϦͷࢹ
-> explain(true) -> system.profile.find() -> Dex -> Compose (MongoHQ) Slow
Queries ͍ΫΤϦͷࢹ
tips: ObjectIdΛͬͨΫΤϦʹ 100msҎ্͔͔ͬͨΒةݥ৴߸
my.presentation.aggregate() -> MongoDBબࢶ͕ଟ͍ -> Ξϯνύλʔϯ͍͔ͭ͘ଘࡏ͢Δ -> ΞϯνύλʔϯʹؕΒͳ͍ͨΊʹࢹ͢Δ -> ࠷৽ใͷΩϟονΞοϓॏཁ
͍͞͝ʹ
we.are “hiring! :)” https://www.zenclerk.com/recruit.html
any.questions?