Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Google File System
Search
hama_du
October 19, 2018
Technology
0
75
Google File System
Google File Systemの中身を紹介
hama_du
October 19, 2018
Tweet
Share
More Decks by hama_du
See All by hama_du
Distinct Query using HyperLogLog
hamadu
2
76
木の上を歩こう
hamadu
1
840
linear-algebra-in-n-minutes
hamadu
0
240
Other Decks in Technology
See All in Technology
AOAI Dev Day - Opening Session
yoshidashingo
2
440
簡単に始めるSnowflakeの機械学習
nayuts
1
190
楽しくGoを学び合う、LayerXの勉強会文化 / LayerX's study culture of having fun and learning Go together
ar_tama
2
350
AI研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
130
What is DRE? - Road to SRE NEXT@広島
chanyou0311
3
630
Matterport を使ってクラスメソッド各拠点のバーチャルオフィスツアーを作成してみた
wakatsuki
0
160
可視化プラットフォームGrafanaの基本と活用方法の全て
hamadakoji
0
230
20240717_イケコパ代表Copilot_in_Teams会社でこう使ってます
ponponmikankan
2
430
初中級者用如何使用backlog -VALE TUDOEDITION-
in0u
0
140
コンテナ・K8s研修 - 後半 Kubernetes 基礎&ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
1
120
セキュリティ研修 Day1【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
How to Think Like a Performance Engineer
csswizardry
4
590
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
291
20k
Fashionably flexible responsive web design (full day workshop)
malarkey
399
65k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
149
45k
Statistics for Hackers
jakevdp
792
220k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
360
22k
Automating Front-end Workflow
addyosmani
1362
200k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
24
1.8k
Designing with Data
zakiwarfel
96
5k
Building a Modern Day E-commerce SEO Strategy
aleyda
25
6.7k
Music & Morning Musume
bryan
43
5.9k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
34
1.9k
The Invisible Side of Design
smashingmag
294
50k
Transcript
The Google File System SDDษڧձ@r-n-i 2018/10/19
֓ཁ • Google ͷϓϩμΫτ෦ͰΘΕ͍ͯͨ େنσʔλ͚ͷϑΝΠϧγεςϜ • ΦʔϓϯιʔεͰͳ͍ • HadoopͷࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠ •
ઍ ͷαʔό͔ͨͪΒΔ
※2003ͷจͰ͢
࣌ͷঢ়گ(จ͔Βൈਮ) • Dual 1.4GHz Pentium III Processor • 2GB Memory
• 80GB 5400 rpm disk ×2 • 100Mbps full-duplex Ethernet
ϋʔυΣΞͷલఏ • γεςϜΛߏ͢Δύʔπ յΕΔ • ࣓ؾσΟεΫ • ωοτϫʔΫέʔϒϧ • ͰϥοΫյ
• ҆ՁͳͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
Ϣʔεέʔεͷલఏ • Read ΫΤϦ >> WriteΫΤϦ • ॻ͖͑ຆͲͳ͍ɺAppend(ه) ͷΈ
ΫϥΠΞϯτ͔Βݟͨ Google File System
ϑΝΠϧγεςϜʁ • ԾతͳπϦʔσΟϨΫτϦΛ࣋ͭ • ΫϥΠΞϯτ͔Βී௨ͷϑΝΠϧʹݟ͍͑ͯΔ • ϑΝΠϧͷ࣮ମ֤αʔόʹࢄஔ • POSIXޓͰͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ •
GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱ ԾϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
ߏཁૉͷհ
ΞʔΩςΫνϟ
νϟϯΫ / Chunk • 1ԾϑΝΠϧ has many νϟϯΫ • 1νϟϯΫ
= 64MB • ෳͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ • 1ͭͷϓϥΠϚϦʔ • NݸͷηΧϯμϦʔ
νϟϯΫαʔό / Chunk server • νϟϯΫͨͪΛ࣮ࡍʹཧతʹ࣋ͭ • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ ͰΞΫηε͢Δ •
ଳҬͷઅͷͨΊ
Ϛελʔαʔό / Master server • ԾϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔ • νϟϯΫαʔόͷࢮ׆ࢹ
• ૢ࡞ϩάΛه • Ϋϥογϡͯ͠෮ݩͰ͖ΔΑ͏ʹ
Read Operation
None
खॱ 1. (ԾϑΝΠϧ໊,νϟϯΫ൪߸) Λ ϚελʔʹϦΫΤετ 2. ֘νϟϯΫ͕͋ΔαʔόͷॴΛڭ͑ͯΒ͏ 3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ,
νϟϯΫͰ ཉ͍͠byteͷൣғ)ΛϦΫΤετ 4. νϟϯΫαʔό͔Βσʔλ͕Β͑Δ
Write Operation
None
खॱ 1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ 2. ֘νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ ηΧϯμϦʔϨϓϦΧͷॴΛڭ͑ͯΒ͏ 3. σʔλΛҙͷϨϓϦΧʹૹΔ σʔλΛड͚औͬͨϨϓϦΧ࣍ͷϨϓϦΧʹྲྀ͢
4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
खॱ 5. ϓϥΠϚϦʔWriteϦΫΤετΛडɺ ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ) ·ͣϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ 6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ ֤ϨϓϦΧϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ 7. ηΧϯμϦʔૢ࡞͕ऴΘͬͨΒͦͷࢫΛ
ϓϥΠϚϦʔʹ͑Δ 8. ϓϥΠϚϦʔ͕ૢ࡞ྃΛΫϥΠΞϯτʹ͑Δ
Write Operationͷࣦഊ • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖ ΫϥΠΞϯτ σʔλૹ৴ૢ࡞͔ΒΓ͢ • ֤ͦͦϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ อূ͍ͯ͠ͳ͍ •
Appendͷͱ͖֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
Snapshot Operation
Snapshot Operation • ԾϑΝΠϧσΟϨΫτϦͷίϐʔΛ࡞Δ • Copy-on-Writeઓུ • ରνϟϯΫʹ Write Operation
͕དྷͨͱ͖ ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷॴʹ࡞ΒΕΔ • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ ͍Ͱ͍Δ
ࢀߟจݙ • The Google File System