Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
月間10億PVから学んだMongoDBアンチパターン
Search
yujiosaka
November 11, 2014
Technology
19
5.2k
月間10億PVから学んだMongoDBアンチパターン
ZenClerkが月間10億PVを支えるまでの 過程で経験したアンチパターンを紹介
yujiosaka
November 11, 2014
Tweet
Share
More Decks by yujiosaka
See All by yujiosaka
Machine Learning with JavaScript
yujiosaka
0
130
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
390
ヘッドレスChromeでクローラを作った後の話
yujiosaka
3
540
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1k
『XXX』のための管理画面
yujiosaka
1
1.2k
Enjoy Deep Learning by JavaScript
yujiosaka
1
280
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
520
第二回もんご祭 パネルディスカッション
yujiosaka
0
810
Other Decks in Technology
See All in Technology
Tebiki株式会社 エンジニア採用資料
tebiki
0
4.1k
SPI原点回帰論:事業課題とFour Keysの結節点を見出す実践的ソフトウェアプロセス改善 / DevOpsDays Tokyo 2024
visional_engineering_and_design
4
1.5k
巨大なテーブルのテーブル定義を無停止で安全に誰でも変更できるようにする / Table-definitions-for-huge-tables-can-be-modified-by-anyone-safely-and-non-disruptively
freee
1
740
Postman v10リリース後を振り返る
nagix
0
130
WebアプリケーションにおけるPDOの使い方入門 / phpcon odawara 2024
meihei3
2
430
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
14
35k
なぜ NOT A HOTEL が Web3 に取り組むのか - NOT A HOTEL TECH TALK
ynunokawa
0
160
0→1開発における技術選定において一番大切なこと
bicstone
1
320
検証を通して見えてきたTiDBの性能特性
lycorptech_jp
PRO
6
3.3k
2024/4/26 コンピュータ歴史博物館解説告知
toshi_atsumi
0
200
反実仮想機械学習とは何か
usaito
PRO
6
1.9k
The CloudCompare project by Dr. Daniel Girardeau-Montaut
kentaitakura
0
500
Featured
See All Featured
jQuery: Nuts, Bolts and Bling
dougneiner
59
7.1k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
240
1.2M
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
321
20k
Faster Mobile Websites
deanohume
297
30k
Designing for Performance
lara
602
67k
What's new in Ruby 2.0
geeforr
337
31k
Typedesign – Prime Four
hannesfritz
36
2.1k
Building Adaptive Systems
keathley
30
1.8k
The Cult of Friendly URLs
andyhume
74
5.7k
Writing Fast Ruby
sferik
620
60k
The Pragmatic Product Professional
lauravandoore
24
5.8k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
18
1.7k
Transcript
݄ؒ10ԯPV͔ΒֶΜͩ MongoDBΞϯνύλʔϯ db tech showcase ౦ژ 2014 speaker: “Yuji Isobe”
history: [ “violinist” “engineer” “security specialist” “socket.io contributor” “startup member
of ZenClerk ” ] name: “Yuji Isobe”
questions: -> MongoDBΛ༻ͨ͜͠ͱ͕͋Γ·͔͢ʁ -> ຊ൪ڥͰMongoDBΛ༻͍ͯ͠·͔͢ʁ
goal: -> ZenClerk͕݄ؒ10ԯPVΛࢧ͑Δ·Ͱͷ աఔͰܦݧͨ͠ΞϯνύλʔϯΛհ -> ओʹύϑΥʔϚϯενϡʔχϯάͷ ->ʮ݄ؒ10ԯPVΛࢧ͑ΔMongoDBʯͷͦͷઌ IUUQXXXTMJEFTIBSFOFUZVKJPTBLBQWNPOHPEC
None
ϦΞϧλΠϜ
ϦΞϧλΠϜੳ Ϗοάσʔλ
db.ZenClerk .stats() -> ݄ؒ10ԯPV -> ඵؒ400PV -> ಉ࣌ଓ5ສ -> ݄ؒ10TBอଘ
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
i2.2xlarge m1.large m3.large db.ZenClerk .aggregate() r3.large
ʙ1000ສPV mogngod
1000ສʙ1ԯPV delayed replica replica set
1ԯPVʙ10ԯPV replica set shard mongos replica set shard replica set
shard delayed replica delayed replica delayed replica mongoc
͜͜·ͰΞϓϦέʔγϣϯͷมߋ ΄ͱΜͲͳ͠ʢҰ෦ΫΤϦΛ࠷దԽʣ
we.use(“MongoDB”).explain() -> εΩʔϚϨε -> τϥϯβΫγϣϯෆཁ ʢ࠷ݶͷΞτϛοΫૢ࡞ʣ -> ॊೈͳΫΤϦ -> ༰қͳεέʔϧΞτ
-> Node.jsͱͷ૬ੑͷྑ͞
we.use(“MongoDB”).explain(true) -> ಋೖͷ༰қ͞ʢϗεςΟϯάαʔϏε༗ແʣ -> ֶशͷ༰қ͞ʢΤϯδχΞҎ֎ʹͱͬͯʣ
ࠓ͢ͷɾɾɾ ϕετϓϥΫςΟε Ξϯνύλʔϯ
ͪ͜Βʂ ϕετϓϥΫςΟε Ξϯνύλʔϯ
Ͱͦͷલʹɾɾɾ
ϕετϓϥΫςΟε
εΩʔϚσβΠϯ -> TwitterͷλΠϜϥΠϯΛอଘ͢Δ -> RDBMSͳΒਖ਼نԽ -> MongoDBͷϕετϓϥΫςΟεʁ
-> ಡΈࠐΈॏࢹ -> 1ΫΤϦ1ώοτ -> ίϝϯτʹԠͯ͡ ॻ͖ࠐΈ͕ॏ͘ͳΔ υΩϡϝϯτΛຒΊࠐΉ session: _id:
“session_id” device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”} ]
-> ॻ͖ࠐΈॏࢹ -> 2ΫΤϦN+1ώοτ -> ॻ͖ࠐΈεέʔϧ͢Δ υΩϡϝϯτΛຒΊࠐ·ͳ͍ session: _id: “session_id”
device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” ! tweet: _id: 1234 user_id: “yujiosaka” text: “MongoDB Rocks!”
timeline: _id: “yujiosaka-20141111” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”}
] -> όϥϯεܕ -> 1ΫΤϦώοτ -> ॻ͖ࠐΈ1·Ͱ ͔͠ॏ͘ͳΒͳ͍ ϋΠϒϦου
εΩʔϚϨε ≠ εΩʔϚఆ͕ٛෆཁ RDBMSΑΓબࢶ͕ଟ͍͘͠ tips: IUUQXXXNPOHPECDPNQSFTFOUBUJPOTTDIFNBEFTJHOTDBMF
Ξϯνύλʔϯ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶃΠϯσοΫεͷషΓա͗
MongoDBϝϞϦͷ͍ํ͕ ͋·Γݡ͋͘Γ·ͤΜ
before: user = new mongoose.Schema first: {type: String, required: true,
index: true} last: {type: String, required: true, index: true} city: {type: String, required: true, index: true} ... created_at: {type: Date, default: Date.now, index: true} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
after: user = new mongoose.Schema first: {type: String, required: true}
last: {type: String, required: true} city: {type: String, required: true} ... created_at: {type: Date, default: Date.now} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
ಡΈࠐΈɺΠϯσοΫε͕ ϝϞϦʹΒͳ͚Ε͘ͳΔ
tips: όʔδϣϯ2.8ʹPluggable Storage Engine͕༧ఆ͞Ε͍ͯΔͷͰɺ ݡ͍Τϯδϯ͕બΔΑ͏ʹͳΔ https://blog.compose.io/the-coming-of-the-mongodb-storage-engines/
ᶄจࣈྻͷΠϯσοΫε
ΠϯσοΫεͷߏ -> RDBMSͱ΄ͱΜͲಉ͡ -> ΠϯσοΫεʹB-TreeΛ࠾༻ -> B-TreeৗʹฏߧΛอ͍ͬͯΔ
จࣈྻͷΠϯσοΫε -> ϗοτσʔλ͕ࢄ -> ϝϞϦޮ͕ѱ͍
࣌ؒɾObjectIdͷΠϯσοΫε -> ϗοτσʔλ͕ूத -> ϝϞϦޮ͕ྑ͍
tips: Ͱ͖Δ͚ͩObjectIdΛ͍·͠ΐ͏ ޙ͔Βͷஔ͖͑େมͰ͢
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ
Ϣʔβʔใͷ͏ͪ ࢢใ͚͕ͩཉ͍͠ user: {_id: “yujiosaka”, first: “Yuji”, last: “Isobe”, city:
“Tokyo”} index: {_id:1, city: 1} // compound index
ϑΟʔϧυࢦఆͳ͠ -> SQLͰͷʮselect *ʯ -> ෆཁͳωοτϫʔΫίετ -> ෆཁͳσΟεΫΞΫηε var user
= db.users.findOne({_id: “yujiosaka”});
ϑΟʔϧυࢦఆ͋Γ -> SQLͰͷʮselect cityʯ -> ωοτϫʔΫίετͷઅ -> covered index ͕׆༻Ͱ͖Δ
var user = db.users.findOne({_id: “yujiosaka”}, {city: 1});
tips: covered indexΛ༗ޮ׆༻͢Εɺ σΟεΫͷΞΫηεҰͳ͠ʹ ϝϞϦ͚͔ͩΒใΛಡΈࠐΊΔ
ᶆίʔϧυσʔλͷΫΤϦ
MongoDBͷಛ -> ϗοτσʔλͷΞΫηεߴ -> ίʔϧυσʔλͷΞΫηε͍ -> શσʔλͷΞΫηεۤख
ZenClerkʹ͓͚Δूܭ -> 5͝ͱʹσʔλΛཁ -> 1࣌ؒ͝ͱʹ౷ܭσʔλΛूܭ -> 1͝ͱʹ౷ܭσʔλΛूܭ
tips: ৗʹϗοτσʔλΛҙࣝͨ͠ ͍ํΛ͠·͠ΐ͏
ᶇϓϥΠϚϦʹภͬͨΫΤϦ
ϨϓϦΧηοτ -> ΫΤϦΛࢄͤ͞Δ͜ͱ͕Ͱ͖Δ -> ηΧϯμϦʹΫΤϦΛ͛Δ͔ ϓϩάϥϚ͕ܾΊΒΕΔ -> σϑΥϧτͰϓϥΠϚϦʹ͛ΒΕΔ
σϑΥϧτ -> ϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλ͕อো͞ΕΔ var user = db.users.find({city: “Tokyo”});
ηΧϯμϦΛࢦఆ -> ηΧϯμϦ͕͋ΕηΧϯμϦ͔Βɺ ͳ͚ΕϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλอো͞Εͳ͍ var user = db.users.find({city:
“Tokyo”}) .readPref("secondaryPreferred")
tips: σʔλ͕࠷৽Ͱͳͯ͘Α͍߹ ಡΈࠐΈΛηΧϯμϦʹ͚Δ͜ͱͰ ΫΤϦ͕ࢄ͢Δ IUUQTTQFBLFSEFDLDPNNPOHPECQSBDUJDBM TDBMJOHBOETIBSEJOHFMJPUIPSPXJU[HFO
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶈඇޮͳΞοϓσʔτ
Α͘ݟ͔͚Δίʔυᶃ -> ඞͣΫΤϦ͕࣮ߦ͞ΕΔ -> ແବͳωοτϫʔΫίετ -> ඇΞτϛοΫͳॲཧ var user =
db.users.findOne({_id: “yujiosaka”}); user.count += 1; user.save();
Α͘ݟ͔͚Δίʔυᶃ -> updateͱ$incΛͬͯ·ͱΊΒΕΔ -> 1ճͷΞοϓσʔτͰࡁΉ -> ΞτϛοΫͳॲཧ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}});
Α͘ݟ͔͚Δίʔυᶄ -> σʔλ͕ݟ͔ͭͬͨΒߋ৽͠ɺݟ͔ͭΒͳ͔ͬͨΒ࡞͢Δ -> Ұݟෳࡶͳίʔυ var user = db.users.findOne({_id: “yujiosaka”});
if (user) { user.count += 1; user.save(); } else { db.users.insert({_id: “yujiosaka”, count: 1}); }
-> {upsert: true} Λ͏ͱ·ͱΊΒΕΔ -> ΑΓγϯϓϧͳίʔυ Α͘ݟ͔͚Δίʔυᶄ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}}, {upsert: true});
ᶉංେԽ͢ΔυΩϡϝϯτ
Ϛεͷ࠲ඪσʔλΛ อଘ͢Δ behavior: points: [{x: 151, y: 100} {x: 151,
y: 179} {x: 151, y: 266} … {x: 151, y: 340}]
MongoDBͷۤख -> ϑΟʔϧυ͕ංେԽ͢ΔΞοϓσʔτ͕͍ -> σʔλ͕ҰఆͷαΠζʹऩ·Βͳ͘ͳΔͱ σΟεΫͰσʔλͷҠಈ͕ى͖Δ -> RedisͰMongoDBͷۤखΛิ͏
Redis -> ΦϯϝϞϦܕͷKVS -> ಡΈࠐΈɾॻ͖ࠐΈ͕ૣ͍ -> جຊతʹσʔλͷӬଓԽ͠ͳ͍ -> σʔλܕ͕ඇৗʹ๛ ʢϦετɾηοτɾϋογϡʣ
RedisΛΘͳ͍߹ db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 100}}); db.col.update({_id:
1234}, {$push: {points: {x: 151, y: 179}}); db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 266}}); … db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 340}}); -> ࠲ඪ͕Ҡಈ͢ΔͨͼʹMongoDBʹॻ͖ࠐ·ΕΔ -> αΠζ͕ऩ·ΓΒͳ͍ͱɺσʔλͷҠಈ͕ى͖Δ
RedisΛͬͨ߹ db.col.insert({points: [ {x: 151, y: 100}, {x: 151, y:
179}, {x: 151, y: 266}, … {x: 151, y: 340} ]}); -> Ұ͔͠MongoDBʹॻ͖ࠐ·ͳ͍ -> υΩϡϝϯυͷαΠζ͜ΕҎ্େ͖͘ͳΒͳ͍
tips: ͯ͢ΛMongoDB͚ͩͰΖ͏ͱͤͣɺ ۤखͳ෦ଞͷٕज़Ͱิ͏͜ͱ͕େ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶊ٧ΊࠐΈա͗DB
σʔλΛҰͭͷDBʹ٧ΊࠐΜͰ ྑ͍͜ͱʢࠓͷͱ͜Ζʣ͋·Γͳ͍
ͳͥDBΛ͚Δͷ͔ʁ -> Reader-WriterϩοΫ͕ΘΕ͍ͯΔ -> 2.2.0Ҏ߱DB୯ҐͰϩοΫ͢Δ -> ϚγϯΛ͚ΕϝϞϦ͕૿͑Δ
ZenClerk ʹ͓͚ΔDBͷ༻్ -> ϝΠϯσʔλʢHotʣ -> ϝΠϯσʔλʢColdʣ -> ੜσʔλ -> ܭࢉ݁Ռσʔλ
-> ౷ܭσʔλ -> αΠτଐੑσʔλ
tips: όʔδϣϯ2.8ʹdocument-level locking 3.0Ҏ߱ʹpartitioned joins͕༧ఆ͞Ε͍ͯΔ ʢޙऀ͋·Γ͋ͯʹ͠ͳ͍ํ͕ແͦ͏ʣ http://www.slideshare.net/tetsutarowatanabe/mongodb-world-2014-37081258
ᶃΠϯσοΫεͷషΓա͗ ᶄจࣈྻͷΠϯσοΫε ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ ᶆίʔϧυσʔλͷΫΤϦ ᶇϓϥΠϚϦʹภͬͨΫΤϦ ᶈඇޮͳΞοϓσʔτ ᶉංେԽ͢ΔυΩϡϝϯτ ᶊ٧ΊࠐΈա͗DB Ξϯνύλʔϯ͓͞Β͍
Ξϯνύλʔϯʹ ؕΒͳ͍ͨΊʹ
̏ͭͷࢹ -> ϦΞϧλΠϜࢹ -> ݟฦͨ͢Ίͷࢹ -> ͍ΫΤϦͷࢹ
ϦΞϧλΠϜࢹ
-> ComposeʢMongoHQʣμογϡϘʔυ -> ݱࡏͷ݈߁νΣοΫ༻ -> ࣾʹઐ༻σΟεϓϨΠΛઃஔ -> ҟৗ͕͋Εʮ୭͔Mongoࡴͨ͠ʁʯ ϦΞϧλΠϜࢹ
ݟฦͨ͢Ίͷࢹ
-> MMSμογϡϘʔυ -> ఆظతͳ݈߁அ -> োൃੜ࣌ʹݟฦͯ͠ ɹʮͳΜͰ͜ΜͳϩοΫߴ͘ͳͬͯΜͷʁʯ ݟฦͨ͢Ίͷࢹ
͍ΫΤϦͷࢹ
-> explain(true) -> system.profile.find() -> Dex -> Compose (MongoHQ) Slow
Queries ͍ΫΤϦͷࢹ
tips: ObjectIdΛͬͨΫΤϦʹ 100msҎ্͔͔ͬͨΒةݥ৴߸
my.presentation.aggregate() -> MongoDBબࢶ͕ଟ͍ -> Ξϯνύλʔϯ͍͔ͭ͘ଘࡏ͢Δ -> ΞϯνύλʔϯʹؕΒͳ͍ͨΊʹࢹ͢Δ -> ࠷৽ใͷΩϟονΞοϓॏཁ
͍͞͝ʹ
we.are “hiring! :)” https://www.zenclerk.com/recruit.html
any.questions?