Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Google File System
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
hama_du
October 19, 2018
Technology
100
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Google File System
Google File Systemの中身を紹介
hama_du
October 19, 2018
Transcript
24 slides
The Google File System SDDษڧձ@r-n-i 2018/10/19
֓ཁ • Google ͷϓϩμΫτ෦ͰΘΕ͍ͯͨ େنσʔλ͚ͷϑΝΠϧγεςϜ • ΦʔϓϯιʔεͰͳ͍ • HadoopͷࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠ •
ઍ ͷαʔό͔ͨͪΒΔ
※2003ͷจͰ͢
࣌ͷঢ়گ(จ͔Βൈਮ) • Dual 1.4GHz Pentium III Processor • 2GB Memory
• 80GB 5400 rpm disk ×2 • 100Mbps full-duplex Ethernet
ϋʔυΣΞͷલఏ • γεςϜΛߏ͢Δύʔπ յΕΔ • ࣓ؾσΟεΫ • ωοτϫʔΫέʔϒϧ • ͰϥοΫյ
• ҆ՁͳͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
Ϣʔεέʔεͷલఏ • Read ΫΤϦ >> WriteΫΤϦ • ॻ͖͑ຆͲͳ͍ɺAppend(ه) ͷΈ
ΫϥΠΞϯτ͔Βݟͨ Google File System
ϑΝΠϧγεςϜʁ • ԾతͳπϦʔσΟϨΫτϦΛ࣋ͭ • ΫϥΠΞϯτ͔Βී௨ͷϑΝΠϧʹݟ͍͑ͯΔ • ϑΝΠϧͷ࣮ମ֤αʔόʹࢄஔ • POSIXޓͰͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ •
GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱ ԾϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
ߏཁૉͷհ
ΞʔΩςΫνϟ
νϟϯΫ / Chunk • 1ԾϑΝΠϧ has many νϟϯΫ • 1νϟϯΫ
= 64MB • ෳͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ • 1ͭͷϓϥΠϚϦʔ • NݸͷηΧϯμϦʔ
νϟϯΫαʔό / Chunk server • νϟϯΫͨͪΛ࣮ࡍʹཧతʹ࣋ͭ • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ ͰΞΫηε͢Δ •
ଳҬͷઅͷͨΊ
Ϛελʔαʔό / Master server • ԾϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔ • νϟϯΫαʔόͷࢮ׆ࢹ
• ૢ࡞ϩάΛه • Ϋϥογϡͯ͠෮ݩͰ͖ΔΑ͏ʹ
Read Operation
None
खॱ 1. (ԾϑΝΠϧ໊,νϟϯΫ൪߸) Λ ϚελʔʹϦΫΤετ 2. ֘νϟϯΫ͕͋ΔαʔόͷॴΛڭ͑ͯΒ͏ 3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ,
νϟϯΫͰ ཉ͍͠byteͷൣғ)ΛϦΫΤετ 4. νϟϯΫαʔό͔Βσʔλ͕Β͑Δ
Write Operation
None
खॱ 1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ 2. ֘νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ ηΧϯμϦʔϨϓϦΧͷॴΛڭ͑ͯΒ͏ 3. σʔλΛҙͷϨϓϦΧʹૹΔ σʔλΛड͚औͬͨϨϓϦΧ࣍ͷϨϓϦΧʹྲྀ͢
4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
खॱ 5. ϓϥΠϚϦʔWriteϦΫΤετΛडɺ ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ) ·ͣϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ 6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ ֤ϨϓϦΧϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ 7. ηΧϯμϦʔૢ࡞͕ऴΘͬͨΒͦͷࢫΛ
ϓϥΠϚϦʔʹ͑Δ 8. ϓϥΠϚϦʔ͕ૢ࡞ྃΛΫϥΠΞϯτʹ͑Δ
Write Operationͷࣦഊ • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖ ΫϥΠΞϯτ σʔλૹ৴ૢ࡞͔ΒΓ͢ • ֤ͦͦϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ อূ͍ͯ͠ͳ͍ •
Appendͷͱ͖֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
Snapshot Operation
Snapshot Operation • ԾϑΝΠϧσΟϨΫτϦͷίϐʔΛ࡞Δ • Copy-on-Writeઓུ • ରνϟϯΫʹ Write Operation
͕དྷͨͱ͖ ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷॴʹ࡞ΒΕΔ • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ ͍Ͱ͍Δ
ࢀߟจݙ • The Google File System
More Decks by hama_du
See All by hama_du
Distinct Query using HyperLogLog
hamadu
2
97
木の上を歩こう
hamadu
1
1.1k
linear-algebra-in-n-minutes
hamadu
0
260
Other Decks in Technology
See All in Technology
SORACOM × AWS IoT Core でSREする
butadora
0
110
エンタープライズ事業未経験のデータ屋、 AI時代にN回撃沈する.pdf
ryoskdara_
0
150
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
10
5.6k
閉域網でのAPI公開の選択肢全部比較!
takano0131
4
530
常時稼働エージェントの作り方(Claude公式ブログを資料化)
minorun365
PRO
16
27k
落とし穴にハマってわかった 開発生産性を測るときに気をつけたいこと
munakata
3
400
サーバーラックは作る時代! 3Dプリンターで作る10インチラックのススメ
jacopen
2
450
gradle-best-practices 使ってみた
mikanichinose
0
120
IP重複は"ランディングVPC"で受け止める ― PrivateLink と Transit Gatewayによるマルチテナント接続基盤の設計
hatahata021
3
250
Shared Signals Framework 仕様解説
oidfj
0
480
LLM 基盤と kagent v0 を全部 k8s に入れてみた / Deploy both LLM and kagent v0 in k8s
kota2and3kan
0
240
AI時代における医療情報システムのセキュリティ / Security of Healthcare IT Systems in the AI Era
henryofficial
0
190
Featured
See All Featured
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
akademiya2063
PRO
1
460
A better future with KSS
kneath
240
18k
Intergalactic Javascript Robots from Outer Space
tanoku
273
28k
Optimising Largest Contentful Paint
csswizardry
37
4k
The Pragmatic Product Professional
lauravandoore
37
7.5k
Claude Code のすすめ
schroneko
67
230k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
techseoconnect
PRO
0
750
Bash Introduction
62gerente
615
220k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
56
3.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
128
56k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
17k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
141
35k