Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Google File System
Search
hama_du
October 19, 2018
Technology
0
72
Google File System
Google File Systemの中身を紹介
hama_du
October 19, 2018
Tweet
Share
More Decks by hama_du
See All by hama_du
Distinct Query using HyperLogLog
hamadu
2
71
木の上を歩こう
hamadu
1
780
linear-algebra-in-n-minutes
hamadu
0
240
Other Decks in Technology
See All in Technology
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
360
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
350
Reducing Cross-Zone Egress at Spotify with Custom gRPC Load Balancing Recap
koh_naga
0
210
Azure犬駆動開発の記録/GlobalAzureFukuoka2024_20240420
nina01
1
220
プラットフォームってつくることより計測することが重要なんじゃないかという話 / Platform Engineering Meetup #8
taishin
1
370
KubeCon EU 2024 Recap “Kubernetes Policy Time Machine: Where to Next?”
ryysud
0
220
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
2.1k
プロトタイピングによる不確実性の低減 / Reducing Uncertainty through Prototyping
ohbarye
5
390
どうするコスト最適化のトレードオフ
tetsuyaooooo
1
530
JSON攻略法.pdf
miyakemito
8
5.1k
AWSに詳しくない人でも始められるコスト最適化ガイド
yuhta28
1
250
Featured
See All Featured
Optimizing for Happiness
mojombo
370
69k
Building Effective Engineering Teams - LeadDev
addyosmani
28
1.8k
How to train your dragon (web standard)
notwaldorf
73
5.2k
Clear Off the Table
cherdarchuk
84
310k
Building a Scalable Design System with Sketch
lauravandoore
456
32k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
241
1.2M
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
17
1.4k
Producing Creativity
orderedlist
PRO
337
39k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
A designer walks into a library…
pauljervisheath
200
23k
Learning to Love Humans: Emotional Interface Design
aarron
267
39k
Designing the Hi-DPI Web
ddemaree
276
33k
Transcript
The Google File System SDDษڧձ@r-n-i 2018/10/19
֓ཁ • Google ͷϓϩμΫτ෦ͰΘΕ͍ͯͨ େنσʔλ͚ͷϑΝΠϧγεςϜ • ΦʔϓϯιʔεͰͳ͍ • HadoopͷࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠ •
ઍ ͷαʔό͔ͨͪΒΔ
※2003ͷจͰ͢
࣌ͷঢ়گ(จ͔Βൈਮ) • Dual 1.4GHz Pentium III Processor • 2GB Memory
• 80GB 5400 rpm disk ×2 • 100Mbps full-duplex Ethernet
ϋʔυΣΞͷલఏ • γεςϜΛߏ͢Δύʔπ յΕΔ • ࣓ؾσΟεΫ • ωοτϫʔΫέʔϒϧ • ͰϥοΫյ
• ҆ՁͳͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
Ϣʔεέʔεͷલఏ • Read ΫΤϦ >> WriteΫΤϦ • ॻ͖͑ຆͲͳ͍ɺAppend(ه) ͷΈ
ΫϥΠΞϯτ͔Βݟͨ Google File System
ϑΝΠϧγεςϜʁ • ԾతͳπϦʔσΟϨΫτϦΛ࣋ͭ • ΫϥΠΞϯτ͔Βී௨ͷϑΝΠϧʹݟ͍͑ͯΔ • ϑΝΠϧͷ࣮ମ֤αʔόʹࢄஔ • POSIXޓͰͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ •
GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱ ԾϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
ߏཁૉͷհ
ΞʔΩςΫνϟ
νϟϯΫ / Chunk • 1ԾϑΝΠϧ has many νϟϯΫ • 1νϟϯΫ
= 64MB • ෳͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ • 1ͭͷϓϥΠϚϦʔ • NݸͷηΧϯμϦʔ
νϟϯΫαʔό / Chunk server • νϟϯΫͨͪΛ࣮ࡍʹཧతʹ࣋ͭ • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ ͰΞΫηε͢Δ •
ଳҬͷઅͷͨΊ
Ϛελʔαʔό / Master server • ԾϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔ • νϟϯΫαʔόͷࢮ׆ࢹ
• ૢ࡞ϩάΛه • Ϋϥογϡͯ͠෮ݩͰ͖ΔΑ͏ʹ
Read Operation
None
खॱ 1. (ԾϑΝΠϧ໊,νϟϯΫ൪߸) Λ ϚελʔʹϦΫΤετ 2. ֘νϟϯΫ͕͋ΔαʔόͷॴΛڭ͑ͯΒ͏ 3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ,
νϟϯΫͰ ཉ͍͠byteͷൣғ)ΛϦΫΤετ 4. νϟϯΫαʔό͔Βσʔλ͕Β͑Δ
Write Operation
None
खॱ 1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ 2. ֘νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ ηΧϯμϦʔϨϓϦΧͷॴΛڭ͑ͯΒ͏ 3. σʔλΛҙͷϨϓϦΧʹૹΔ σʔλΛड͚औͬͨϨϓϦΧ࣍ͷϨϓϦΧʹྲྀ͢
4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
खॱ 5. ϓϥΠϚϦʔWriteϦΫΤετΛडɺ ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ) ·ͣϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ 6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ ֤ϨϓϦΧϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ 7. ηΧϯμϦʔૢ࡞͕ऴΘͬͨΒͦͷࢫΛ
ϓϥΠϚϦʔʹ͑Δ 8. ϓϥΠϚϦʔ͕ૢ࡞ྃΛΫϥΠΞϯτʹ͑Δ
Write Operationͷࣦഊ • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖ ΫϥΠΞϯτ σʔλૹ৴ૢ࡞͔ΒΓ͢ • ֤ͦͦϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ อূ͍ͯ͠ͳ͍ •
Appendͷͱ͖֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
Snapshot Operation
Snapshot Operation • ԾϑΝΠϧσΟϨΫτϦͷίϐʔΛ࡞Δ • Copy-on-Writeઓུ • ରνϟϯΫʹ Write Operation
͕དྷͨͱ͖ ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷॴʹ࡞ΒΕΔ • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ ͍Ͱ͍Δ
ࢀߟจݙ • The Google File System