Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
月間10億PVから学んだMongoDBアンチパターン
Search
yujiosaka
November 11, 2014
Technology
19
5.4k
月間10億PVから学んだMongoDBアンチパターン
ZenClerkが月間10億PVを支えるまでの 過程で経験したアンチパターンを紹介
yujiosaka
November 11, 2014
Tweet
Share
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
220
Machine Learning with JavaScript
yujiosaka
0
150
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
420
ヘッドレスChromeでクローラを作った後の話
yujiosaka
3
610
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.2k
『XXX』のための管理画面
yujiosaka
1
1.2k
Enjoy Deep Learning by JavaScript
yujiosaka
1
310
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
530
Other Decks in Technology
See All in Technology
宇宙ベンチャーにおける最近の情シス取り組みについて
axelmizu
0
110
社内イベント管理システムを1週間でAKSからACAに移行した話し
shingo_kawahara
0
180
PHP ユーザのための OpenTelemetry 入門 / phpcon2024-opentelemetry
shin1x1
1
200
ブラックフライデーで購入したPixel9で、Gemini Nanoを動かしてみた
marchin1989
1
530
20241214_WACATE2024冬_テスト設計技法をチョット俯瞰してみよう
kzsuzuki
3
450
日本版とグローバル版のモバイルアプリ統合の開発の裏側と今後の展望
miichan
1
130
OpenAIの蒸留機能(Model Distillation)を使用して運用中のLLMのコストを削減する取り組み
pharma_x_tech
4
560
Snowflake女子会#3 Snowpipeの良さを5分で語るよ
lana2548
0
230
re:Invent 2024 Innovation Talks(NET201)で語られた大切なこと
shotashiratori
0
310
新機能VPCリソースエンドポイント機能検証から得られた考察
duelist2020jp
0
220
Jetpack Composeで始めるServer Cache State
ogaclejapan
2
170
LINEヤフーのフロントエンド組織・体制の紹介【24年12月】
lycorp_recruit_jp
0
530
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.9k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
111
49k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
GraphQLとの向き合い方2022年版
quramy
44
13k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Visualization
eitanlees
146
15k
Designing for Performance
lara
604
68k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
Fireside Chat
paigeccino
34
3.1k
Transcript
݄ؒ10ԯPV͔ΒֶΜͩ MongoDBΞϯνύλʔϯ db tech showcase ౦ژ 2014 speaker: “Yuji Isobe”
history: [ “violinist” “engineer” “security specialist” “socket.io contributor” “startup member
of ZenClerk ” ] name: “Yuji Isobe”
questions: -> MongoDBΛ༻ͨ͜͠ͱ͕͋Γ·͔͢ʁ -> ຊ൪ڥͰMongoDBΛ༻͍ͯ͠·͔͢ʁ
goal: -> ZenClerk͕݄ؒ10ԯPVΛࢧ͑Δ·Ͱͷ աఔͰܦݧͨ͠ΞϯνύλʔϯΛհ -> ओʹύϑΥʔϚϯενϡʔχϯάͷ ->ʮ݄ؒ10ԯPVΛࢧ͑ΔMongoDBʯͷͦͷઌ IUUQXXXTMJEFTIBSFOFUZVKJPTBLBQWNPOHPEC
None
ϦΞϧλΠϜ
ϦΞϧλΠϜੳ Ϗοάσʔλ
db.ZenClerk .stats() -> ݄ؒ10ԯPV -> ඵؒ400PV -> ಉ࣌ଓ5ສ -> ݄ؒ10TBอଘ
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
i2.2xlarge m1.large m3.large db.ZenClerk .aggregate() r3.large
ʙ1000ສPV mogngod
1000ສʙ1ԯPV delayed replica replica set
1ԯPVʙ10ԯPV replica set shard mongos replica set shard replica set
shard delayed replica delayed replica delayed replica mongoc
͜͜·ͰΞϓϦέʔγϣϯͷมߋ ΄ͱΜͲͳ͠ʢҰ෦ΫΤϦΛ࠷దԽʣ
we.use(“MongoDB”).explain() -> εΩʔϚϨε -> τϥϯβΫγϣϯෆཁ ʢ࠷ݶͷΞτϛοΫૢ࡞ʣ -> ॊೈͳΫΤϦ -> ༰қͳεέʔϧΞτ
-> Node.jsͱͷ૬ੑͷྑ͞
we.use(“MongoDB”).explain(true) -> ಋೖͷ༰қ͞ʢϗεςΟϯάαʔϏε༗ແʣ -> ֶशͷ༰қ͞ʢΤϯδχΞҎ֎ʹͱͬͯʣ
ࠓ͢ͷɾɾɾ ϕετϓϥΫςΟε Ξϯνύλʔϯ
ͪ͜Βʂ ϕετϓϥΫςΟε Ξϯνύλʔϯ
Ͱͦͷલʹɾɾɾ
ϕετϓϥΫςΟε
εΩʔϚσβΠϯ -> TwitterͷλΠϜϥΠϯΛอଘ͢Δ -> RDBMSͳΒਖ਼نԽ -> MongoDBͷϕετϓϥΫςΟεʁ
-> ಡΈࠐΈॏࢹ -> 1ΫΤϦ1ώοτ -> ίϝϯτʹԠͯ͡ ॻ͖ࠐΈ͕ॏ͘ͳΔ υΩϡϝϯτΛຒΊࠐΉ session: _id:
“session_id” device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”} ]
-> ॻ͖ࠐΈॏࢹ -> 2ΫΤϦN+1ώοτ -> ॻ͖ࠐΈεέʔϧ͢Δ υΩϡϝϯτΛຒΊࠐ·ͳ͍ session: _id: “session_id”
device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” ! tweet: _id: 1234 user_id: “yujiosaka” text: “MongoDB Rocks!”
timeline: _id: “yujiosaka-20141111” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”}
] -> όϥϯεܕ -> 1ΫΤϦώοτ -> ॻ͖ࠐΈ1·Ͱ ͔͠ॏ͘ͳΒͳ͍ ϋΠϒϦου
εΩʔϚϨε ≠ εΩʔϚఆ͕ٛෆཁ RDBMSΑΓબࢶ͕ଟ͍͘͠ tips: IUUQXXXNPOHPECDPNQSFTFOUBUJPOTTDIFNBEFTJHOTDBMF
Ξϯνύλʔϯ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶃΠϯσοΫεͷషΓա͗
MongoDBϝϞϦͷ͍ํ͕ ͋·Γݡ͋͘Γ·ͤΜ
before: user = new mongoose.Schema first: {type: String, required: true,
index: true} last: {type: String, required: true, index: true} city: {type: String, required: true, index: true} ... created_at: {type: Date, default: Date.now, index: true} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
after: user = new mongoose.Schema first: {type: String, required: true}
last: {type: String, required: true} city: {type: String, required: true} ... created_at: {type: Date, default: Date.now} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
ಡΈࠐΈɺΠϯσοΫε͕ ϝϞϦʹΒͳ͚Ε͘ͳΔ
tips: όʔδϣϯ2.8ʹPluggable Storage Engine͕༧ఆ͞Ε͍ͯΔͷͰɺ ݡ͍Τϯδϯ͕બΔΑ͏ʹͳΔ https://blog.compose.io/the-coming-of-the-mongodb-storage-engines/
ᶄจࣈྻͷΠϯσοΫε
ΠϯσοΫεͷߏ -> RDBMSͱ΄ͱΜͲಉ͡ -> ΠϯσοΫεʹB-TreeΛ࠾༻ -> B-TreeৗʹฏߧΛอ͍ͬͯΔ
จࣈྻͷΠϯσοΫε -> ϗοτσʔλ͕ࢄ -> ϝϞϦޮ͕ѱ͍
࣌ؒɾObjectIdͷΠϯσοΫε -> ϗοτσʔλ͕ूத -> ϝϞϦޮ͕ྑ͍
tips: Ͱ͖Δ͚ͩObjectIdΛ͍·͠ΐ͏ ޙ͔Βͷஔ͖͑େมͰ͢
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ
Ϣʔβʔใͷ͏ͪ ࢢใ͚͕ͩཉ͍͠ user: {_id: “yujiosaka”, first: “Yuji”, last: “Isobe”, city:
“Tokyo”} index: {_id:1, city: 1} // compound index
ϑΟʔϧυࢦఆͳ͠ -> SQLͰͷʮselect *ʯ -> ෆཁͳωοτϫʔΫίετ -> ෆཁͳσΟεΫΞΫηε var user
= db.users.findOne({_id: “yujiosaka”});
ϑΟʔϧυࢦఆ͋Γ -> SQLͰͷʮselect cityʯ -> ωοτϫʔΫίετͷઅ -> covered index ͕׆༻Ͱ͖Δ
var user = db.users.findOne({_id: “yujiosaka”}, {city: 1});
tips: covered indexΛ༗ޮ׆༻͢Εɺ σΟεΫͷΞΫηεҰͳ͠ʹ ϝϞϦ͚͔ͩΒใΛಡΈࠐΊΔ
ᶆίʔϧυσʔλͷΫΤϦ
MongoDBͷಛ -> ϗοτσʔλͷΞΫηεߴ -> ίʔϧυσʔλͷΞΫηε͍ -> શσʔλͷΞΫηεۤख
ZenClerkʹ͓͚Δूܭ -> 5͝ͱʹσʔλΛཁ -> 1࣌ؒ͝ͱʹ౷ܭσʔλΛूܭ -> 1͝ͱʹ౷ܭσʔλΛूܭ
tips: ৗʹϗοτσʔλΛҙࣝͨ͠ ͍ํΛ͠·͠ΐ͏
ᶇϓϥΠϚϦʹภͬͨΫΤϦ
ϨϓϦΧηοτ -> ΫΤϦΛࢄͤ͞Δ͜ͱ͕Ͱ͖Δ -> ηΧϯμϦʹΫΤϦΛ͛Δ͔ ϓϩάϥϚ͕ܾΊΒΕΔ -> σϑΥϧτͰϓϥΠϚϦʹ͛ΒΕΔ
σϑΥϧτ -> ϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλ͕อো͞ΕΔ var user = db.users.find({city: “Tokyo”});
ηΧϯμϦΛࢦఆ -> ηΧϯμϦ͕͋ΕηΧϯμϦ͔Βɺ ͳ͚ΕϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλอো͞Εͳ͍ var user = db.users.find({city:
“Tokyo”}) .readPref("secondaryPreferred")
tips: σʔλ͕࠷৽Ͱͳͯ͘Α͍߹ ಡΈࠐΈΛηΧϯμϦʹ͚Δ͜ͱͰ ΫΤϦ͕ࢄ͢Δ IUUQTTQFBLFSEFDLDPNNPOHPECQSBDUJDBM TDBMJOHBOETIBSEJOHFMJPUIPSPXJU[HFO
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶈඇޮͳΞοϓσʔτ
Α͘ݟ͔͚Δίʔυᶃ -> ඞͣΫΤϦ͕࣮ߦ͞ΕΔ -> ແବͳωοτϫʔΫίετ -> ඇΞτϛοΫͳॲཧ var user =
db.users.findOne({_id: “yujiosaka”}); user.count += 1; user.save();
Α͘ݟ͔͚Δίʔυᶃ -> updateͱ$incΛͬͯ·ͱΊΒΕΔ -> 1ճͷΞοϓσʔτͰࡁΉ -> ΞτϛοΫͳॲཧ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}});
Α͘ݟ͔͚Δίʔυᶄ -> σʔλ͕ݟ͔ͭͬͨΒߋ৽͠ɺݟ͔ͭΒͳ͔ͬͨΒ࡞͢Δ -> Ұݟෳࡶͳίʔυ var user = db.users.findOne({_id: “yujiosaka”});
if (user) { user.count += 1; user.save(); } else { db.users.insert({_id: “yujiosaka”, count: 1}); }
-> {upsert: true} Λ͏ͱ·ͱΊΒΕΔ -> ΑΓγϯϓϧͳίʔυ Α͘ݟ͔͚Δίʔυᶄ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}}, {upsert: true});
ᶉංେԽ͢ΔυΩϡϝϯτ
Ϛεͷ࠲ඪσʔλΛ อଘ͢Δ behavior: points: [{x: 151, y: 100} {x: 151,
y: 179} {x: 151, y: 266} … {x: 151, y: 340}]
MongoDBͷۤख -> ϑΟʔϧυ͕ංେԽ͢ΔΞοϓσʔτ͕͍ -> σʔλ͕ҰఆͷαΠζʹऩ·Βͳ͘ͳΔͱ σΟεΫͰσʔλͷҠಈ͕ى͖Δ -> RedisͰMongoDBͷۤखΛิ͏
Redis -> ΦϯϝϞϦܕͷKVS -> ಡΈࠐΈɾॻ͖ࠐΈ͕ૣ͍ -> جຊతʹσʔλͷӬଓԽ͠ͳ͍ -> σʔλܕ͕ඇৗʹ๛ ʢϦετɾηοτɾϋογϡʣ
RedisΛΘͳ͍߹ db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 100}}); db.col.update({_id:
1234}, {$push: {points: {x: 151, y: 179}}); db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 266}}); … db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 340}}); -> ࠲ඪ͕Ҡಈ͢ΔͨͼʹMongoDBʹॻ͖ࠐ·ΕΔ -> αΠζ͕ऩ·ΓΒͳ͍ͱɺσʔλͷҠಈ͕ى͖Δ
RedisΛͬͨ߹ db.col.insert({points: [ {x: 151, y: 100}, {x: 151, y:
179}, {x: 151, y: 266}, … {x: 151, y: 340} ]}); -> Ұ͔͠MongoDBʹॻ͖ࠐ·ͳ͍ -> υΩϡϝϯυͷαΠζ͜ΕҎ্େ͖͘ͳΒͳ͍
tips: ͯ͢ΛMongoDB͚ͩͰΖ͏ͱͤͣɺ ۤखͳ෦ଞͷٕज़Ͱิ͏͜ͱ͕େ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶊ٧ΊࠐΈա͗DB
σʔλΛҰͭͷDBʹ٧ΊࠐΜͰ ྑ͍͜ͱʢࠓͷͱ͜Ζʣ͋·Γͳ͍
ͳͥDBΛ͚Δͷ͔ʁ -> Reader-WriterϩοΫ͕ΘΕ͍ͯΔ -> 2.2.0Ҏ߱DB୯ҐͰϩοΫ͢Δ -> ϚγϯΛ͚ΕϝϞϦ͕૿͑Δ
ZenClerk ʹ͓͚ΔDBͷ༻్ -> ϝΠϯσʔλʢHotʣ -> ϝΠϯσʔλʢColdʣ -> ੜσʔλ -> ܭࢉ݁Ռσʔλ
-> ౷ܭσʔλ -> αΠτଐੑσʔλ
tips: όʔδϣϯ2.8ʹdocument-level locking 3.0Ҏ߱ʹpartitioned joins͕༧ఆ͞Ε͍ͯΔ ʢޙऀ͋·Γ͋ͯʹ͠ͳ͍ํ͕ແͦ͏ʣ http://www.slideshare.net/tetsutarowatanabe/mongodb-world-2014-37081258
ᶃΠϯσοΫεͷషΓա͗ ᶄจࣈྻͷΠϯσοΫε ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ ᶆίʔϧυσʔλͷΫΤϦ ᶇϓϥΠϚϦʹภͬͨΫΤϦ ᶈඇޮͳΞοϓσʔτ ᶉංେԽ͢ΔυΩϡϝϯτ ᶊ٧ΊࠐΈա͗DB Ξϯνύλʔϯ͓͞Β͍
Ξϯνύλʔϯʹ ؕΒͳ͍ͨΊʹ
̏ͭͷࢹ -> ϦΞϧλΠϜࢹ -> ݟฦͨ͢Ίͷࢹ -> ͍ΫΤϦͷࢹ
ϦΞϧλΠϜࢹ
-> ComposeʢMongoHQʣμογϡϘʔυ -> ݱࡏͷ݈߁νΣοΫ༻ -> ࣾʹઐ༻σΟεϓϨΠΛઃஔ -> ҟৗ͕͋Εʮ୭͔Mongoࡴͨ͠ʁʯ ϦΞϧλΠϜࢹ
ݟฦͨ͢Ίͷࢹ
-> MMSμογϡϘʔυ -> ఆظతͳ݈߁அ -> োൃੜ࣌ʹݟฦͯ͠ ɹʮͳΜͰ͜ΜͳϩοΫߴ͘ͳͬͯΜͷʁʯ ݟฦͨ͢Ίͷࢹ
͍ΫΤϦͷࢹ
-> explain(true) -> system.profile.find() -> Dex -> Compose (MongoHQ) Slow
Queries ͍ΫΤϦͷࢹ
tips: ObjectIdΛͬͨΫΤϦʹ 100msҎ্͔͔ͬͨΒةݥ৴߸
my.presentation.aggregate() -> MongoDBબࢶ͕ଟ͍ -> Ξϯνύλʔϯ͍͔ͭ͘ଘࡏ͢Δ -> ΞϯνύλʔϯʹؕΒͳ͍ͨΊʹࢹ͢Δ -> ࠷৽ใͷΩϟονΞοϓॏཁ
͍͞͝ʹ
we.are “hiring! :)” https://www.zenclerk.com/recruit.html
any.questions?