Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google File System

Eaa974d2082a6bad3672eabfd6445d02?s=47 hama_du
October 19, 2018

Google File System

Google File Systemの中身を紹介

Eaa974d2082a6bad3672eabfd6445d02?s=128

hama_du

October 19, 2018
Tweet

Transcript

  1. The Google File System SDDษڧձ@r-n-i 2018/10/19

  2. ֓ཁ • Google ͷϓϩμΫτ಺෦Ͱ࢖ΘΕ͍ͯͨ
 େن໛σʔλ޲͚ͷϑΝΠϧγεςϜ • ΦʔϓϯιʔεͰ͸ͳ͍ • Hadoopͷ෼ࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠ •

    ਺ઍ୆ ͷαʔό͔ͨͪΒ੒Δ
  3. ※2003೥ͷ࿦จͰ͢

  4. ౰࣌ͷঢ়گ(࿦จ͔Βൈਮ) • Dual 1.4GHz Pentium III Processor • 2GB Memory

    • 80GB 5400 rpm disk ×2 • 100Mbps full-duplex Ethernet
  5. ϋʔυ΢ΣΞͷલఏ • γεςϜΛߏ੒͢Δύʔπ͸ յΕΔ • ࣓ؾσΟεΫ • ωοτϫʔΫέʔϒϧ • ஍਒ͰϥοΫ౗յ

    • ҆Ձͳ΋ͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
  6. Ϣʔεέʔεͷલఏ • Read ΫΤϦ਺ >> WriteΫΤϦ਺ • ॻ͖׵͑͸ຆͲͳ͍ɺAppend(௥ه) ͷΈ

  7. ΫϥΠΞϯτ͔Βݟͨ Google File System

  8. ϑΝΠϧγεςϜʁ • Ծ૝తͳπϦʔσΟϨΫτϦΛ࣋ͭ • ΫϥΠΞϯτ͔Β͸ී௨ͷϑΝΠϧʹݟ͍͑ͯΔ • ϑΝΠϧͷ࣮ମ͸֤αʔόʹ෼ࢄ഑ஔ • POSIXޓ׵Ͱ͸ͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ •

    GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱
 Ծ૝ϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
  9. ߏ੒ཁૉͷ঺հ

  10. ΞʔΩςΫνϟ

  11. νϟϯΫ / Chunk • 1Ծ૝ϑΝΠϧ has many νϟϯΫ • 1νϟϯΫ

    = 64MB • ෳ਺ͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ • 1ͭͷϓϥΠϚϦʔ • NݸͷηΧϯμϦʔ
  12. νϟϯΫαʔό / Chunk server • νϟϯΫͨͪΛ࣮ࡍʹ෺ཧతʹ࣋ͭ • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ
 ௚ͰΞΫηε͢Δ •

    ଳҬͷઅ໿ͷͨΊ
  13. Ϛελʔαʔό / Master server • Ծ૝ϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔ • νϟϯΫαʔόͷࢮ׆؂ࢹ

    • ૢ࡞ϩάΛه࿥ • Ϋϥογϡͯ͠΋෮ݩͰ͖ΔΑ͏ʹ
  14. Read Operation

  15. None
  16. खॱ 1. (Ծ૝ϑΝΠϧ໊,νϟϯΫ൪߸) Λ
 ϚελʔʹϦΫΤετ 2. ֘౰νϟϯΫ͕͋Δαʔόͷ৔ॴΛڭ͑ͯ΋Β͏ 3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ,

    νϟϯΫ಺Ͱ ཉ͍͠byteͷൣғ)ΛϦΫΤετ 4. νϟϯΫαʔό͔Βσʔλ͕΋Β͑Δ
  17. Write Operation

  18. None
  19. खॱ 1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ 2. ֘౰νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ
 ηΧϯμϦʔϨϓϦΧͷ৔ॴΛڭ͑ͯ΋Β͏ 3. σʔλΛ೚ҙͷϨϓϦΧʹૹΔ
 σʔλΛड͚औͬͨϨϓϦΧ͸࣍ͷϨϓϦΧʹྲྀ͢

    4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ
 WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
  20. खॱ 5. ϓϥΠϚϦʔ͸WriteϦΫΤετΛड୚ɺ
 ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ)
 ·ͣ͸ϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ 6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ
 ֤ϨϓϦΧ͸ϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ 7. ηΧϯμϦʔ͸ૢ࡞͕ऴΘͬͨΒͦͷࢫΛ


    ϓϥΠϚϦʔʹ఻͑Δ 8. ϓϥΠϚϦʔ͕ૢ࡞׬ྃΛΫϥΠΞϯτʹ఻͑Δ
  21. Write Operationͷࣦഊ • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖
 ΫϥΠΞϯτ͸ σʔλૹ৴ૢ࡞͔Β΍Γ௚͢ • ͦ΋ͦ΋֤ϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ
 อূ͍ͯ͠ͳ͍ •

    Appendͷͱ͖͸֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
  22. Snapshot Operation

  23. Snapshot Operation • Ծ૝ϑΝΠϧ΍σΟϨΫτϦͷίϐʔΛ࡞Δ • Copy-on-Writeઓུ • ର৅νϟϯΫʹ Write Operation

    ͕དྷͨͱ͖
 ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷ৔ॴʹ࡞ΒΕΔ • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ ๷͍Ͱ͍Δ
  24. ࢀߟจݙ • The Google File System