Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Google File System
Search
hama_du
October 19, 2018
Technology
0
80
Google File System
Google File Systemの中身を紹介
hama_du
October 19, 2018
Tweet
Share
More Decks by hama_du
See All by hama_du
Distinct Query using HyperLogLog
hamadu
2
81
木の上を歩こう
hamadu
1
980
linear-algebra-in-n-minutes
hamadu
0
240
Other Decks in Technology
See All in Technology
話題の MCP と巡る OCI RAG ソリューションの旅 - Select AI with RAG と Generative AI Agents ディープダイブ
oracle4engineer
PRO
5
110
SREのためのeBPF活用ステップアップガイド
egmc
2
900
[ JAWS-UG千葉支部 x 彩の国埼玉支部 ]ムダ遣い卒業!FinOpsで始めるAWSコスト最適化の第一歩
sh_fk2
2
150
ロールが細分化された組織でSREは何をするか?
tgidgd
1
200
United Airlines Customer Service– Call 1-833-341-3142 Now!
airhelp
0
180
20250707-AI活用の個人差を埋めるチームづくり
shnjtk
6
4.1k
いつの間にか入れ替わってる!?新しいAWS Security Hubとは?
cmusudakeisuke
0
160
推し書籍📚 / Books and a QA Engineer
ak1210
0
120
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
55
22k
アクセスピークを制するオートスケール再設計: 障害を乗り越えKEDAで実現したリソース管理の最適化
myamashii
1
330
60以上のプロダクトを持つ組織における開発者体験向上への取り組み - チームAPIとBackstageで構築する組織の可視化基盤 - / sre next 2025 Efforts to Improve Developer Experience in an Organization with Over 60 Products
vtryo
3
980
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
1
390
Featured
See All Featured
Visualization
eitanlees
146
16k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
How to train your dragon (web standard)
notwaldorf
96
6.1k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
A designer walks into a library…
pauljervisheath
207
24k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
7
330
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
How STYLIGHT went responsive
nonsquared
100
5.6k
A Tale of Four Properties
chriscoyier
160
23k
Transcript
The Google File System SDDษڧձ@r-n-i 2018/10/19
֓ཁ • Google ͷϓϩμΫτ෦ͰΘΕ͍ͯͨ େنσʔλ͚ͷϑΝΠϧγεςϜ • ΦʔϓϯιʔεͰͳ͍ • HadoopͷࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠ •
ઍ ͷαʔό͔ͨͪΒΔ
※2003ͷจͰ͢
࣌ͷঢ়گ(จ͔Βൈਮ) • Dual 1.4GHz Pentium III Processor • 2GB Memory
• 80GB 5400 rpm disk ×2 • 100Mbps full-duplex Ethernet
ϋʔυΣΞͷલఏ • γεςϜΛߏ͢Δύʔπ յΕΔ • ࣓ؾσΟεΫ • ωοτϫʔΫέʔϒϧ • ͰϥοΫյ
• ҆ՁͳͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
Ϣʔεέʔεͷલఏ • Read ΫΤϦ >> WriteΫΤϦ • ॻ͖͑ຆͲͳ͍ɺAppend(ه) ͷΈ
ΫϥΠΞϯτ͔Βݟͨ Google File System
ϑΝΠϧγεςϜʁ • ԾతͳπϦʔσΟϨΫτϦΛ࣋ͭ • ΫϥΠΞϯτ͔Βී௨ͷϑΝΠϧʹݟ͍͑ͯΔ • ϑΝΠϧͷ࣮ମ֤αʔόʹࢄஔ • POSIXޓͰͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ •
GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱ ԾϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
ߏཁૉͷհ
ΞʔΩςΫνϟ
νϟϯΫ / Chunk • 1ԾϑΝΠϧ has many νϟϯΫ • 1νϟϯΫ
= 64MB • ෳͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ • 1ͭͷϓϥΠϚϦʔ • NݸͷηΧϯμϦʔ
νϟϯΫαʔό / Chunk server • νϟϯΫͨͪΛ࣮ࡍʹཧతʹ࣋ͭ • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ ͰΞΫηε͢Δ •
ଳҬͷઅͷͨΊ
Ϛελʔαʔό / Master server • ԾϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔ • νϟϯΫαʔόͷࢮ׆ࢹ
• ૢ࡞ϩάΛه • Ϋϥογϡͯ͠෮ݩͰ͖ΔΑ͏ʹ
Read Operation
None
खॱ 1. (ԾϑΝΠϧ໊,νϟϯΫ൪߸) Λ ϚελʔʹϦΫΤετ 2. ֘νϟϯΫ͕͋ΔαʔόͷॴΛڭ͑ͯΒ͏ 3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ,
νϟϯΫͰ ཉ͍͠byteͷൣғ)ΛϦΫΤετ 4. νϟϯΫαʔό͔Βσʔλ͕Β͑Δ
Write Operation
None
खॱ 1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ 2. ֘νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ ηΧϯμϦʔϨϓϦΧͷॴΛڭ͑ͯΒ͏ 3. σʔλΛҙͷϨϓϦΧʹૹΔ σʔλΛड͚औͬͨϨϓϦΧ࣍ͷϨϓϦΧʹྲྀ͢
4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
खॱ 5. ϓϥΠϚϦʔWriteϦΫΤετΛडɺ ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ) ·ͣϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ 6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ ֤ϨϓϦΧϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ 7. ηΧϯμϦʔૢ࡞͕ऴΘͬͨΒͦͷࢫΛ
ϓϥΠϚϦʔʹ͑Δ 8. ϓϥΠϚϦʔ͕ૢ࡞ྃΛΫϥΠΞϯτʹ͑Δ
Write Operationͷࣦഊ • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖ ΫϥΠΞϯτ σʔλૹ৴ૢ࡞͔ΒΓ͢ • ֤ͦͦϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ อূ͍ͯ͠ͳ͍ •
Appendͷͱ͖֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
Snapshot Operation
Snapshot Operation • ԾϑΝΠϧσΟϨΫτϦͷίϐʔΛ࡞Δ • Copy-on-Writeઓུ • ରνϟϯΫʹ Write Operation
͕དྷͨͱ͖ ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷॴʹ࡞ΒΕΔ • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ ͍Ͱ͍Δ
ࢀߟจݙ • The Google File System