Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google File System

hama_du
October 19, 2018

Google File System

Google File Systemの中身を紹介

hama_du

October 19, 2018
Tweet

More Decks by hama_du

Other Decks in Technology

Transcript

  1. The Google File System
    SDDษڧձ@r-n-i
    2018/10/19

    View Slide

  2. ֓ཁ
    • Google ͷϓϩμΫτ಺෦Ͱ࢖ΘΕ͍ͯͨ

    େن໛σʔλ޲͚ͷϑΝΠϧγεςϜ
    • ΦʔϓϯιʔεͰ͸ͳ͍
    • Hadoopͷ෼ࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠
    • ਺ઍ୆ ͷαʔό͔ͨͪΒ੒Δ

    View Slide

  3. ※2003೥ͷ࿦จͰ͢

    View Slide

  4. ౰࣌ͷঢ়گ(࿦จ͔Βൈਮ)
    • Dual 1.4GHz Pentium III Processor
    • 2GB Memory
    • 80GB 5400 rpm disk ×2
    • 100Mbps full-duplex Ethernet

    View Slide

  5. ϋʔυ΢ΣΞͷલఏ
    • γεςϜΛߏ੒͢Δύʔπ͸ յΕΔ
    • ࣓ؾσΟεΫ
    • ωοτϫʔΫέʔϒϧ
    • ஍਒ͰϥοΫ౗յ
    • ҆Ձͳ΋ͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠

    View Slide

  6. Ϣʔεέʔεͷલఏ
    • Read ΫΤϦ਺ >> WriteΫΤϦ਺
    • ॻ͖׵͑͸ຆͲͳ͍ɺAppend(௥ه) ͷΈ

    View Slide

  7. ΫϥΠΞϯτ͔Βݟͨ
    Google File System

    View Slide

  8. ϑΝΠϧγεςϜʁ
    • Ծ૝తͳπϦʔσΟϨΫτϦΛ࣋ͭ
    • ΫϥΠΞϯτ͔Β͸ී௨ͷϑΝΠϧʹݟ͍͑ͯΔ
    • ϑΝΠϧͷ࣮ମ͸֤αʔόʹ෼ࢄ഑ஔ
    • POSIXޓ׵Ͱ͸ͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ
    • GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱

    Ծ૝ϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢

    View Slide

  9. ߏ੒ཁૉͷ঺հ

    View Slide

  10. ΞʔΩςΫνϟ

    View Slide

  11. νϟϯΫ / Chunk
    • 1Ծ૝ϑΝΠϧ has many νϟϯΫ
    • 1νϟϯΫ = 64MB
    • ෳ਺ͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ
    • 1ͭͷϓϥΠϚϦʔ
    • NݸͷηΧϯμϦʔ

    View Slide

  12. νϟϯΫαʔό / Chunk server
    • νϟϯΫͨͪΛ࣮ࡍʹ෺ཧతʹ࣋ͭ
    • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ

    ௚ͰΞΫηε͢Δ
    • ଳҬͷઅ໿ͷͨΊ

    View Slide

  13. Ϛελʔαʔό / Master server
    • Ծ૝ϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ
    • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔
    • νϟϯΫαʔόͷࢮ׆؂ࢹ
    • ૢ࡞ϩάΛه࿥
    • Ϋϥογϡͯ͠΋෮ݩͰ͖ΔΑ͏ʹ

    View Slide

  14. Read Operation

    View Slide

  15. View Slide

  16. खॱ
    1. (Ծ૝ϑΝΠϧ໊,νϟϯΫ൪߸) Λ

    ϚελʔʹϦΫΤετ
    2. ֘౰νϟϯΫ͕͋Δαʔόͷ৔ॴΛڭ͑ͯ΋Β͏
    3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ, νϟϯΫ಺Ͱ
    ཉ͍͠byteͷൣғ)ΛϦΫΤετ
    4. νϟϯΫαʔό͔Βσʔλ͕΋Β͑Δ

    View Slide

  17. Write Operation

    View Slide

  18. View Slide

  19. खॱ
    1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ
    2. ֘౰νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ

    ηΧϯμϦʔϨϓϦΧͷ৔ॴΛڭ͑ͯ΋Β͏
    3. σʔλΛ೚ҙͷϨϓϦΧʹૹΔ

    σʔλΛड͚औͬͨϨϓϦΧ͸࣍ͷϨϓϦΧʹྲྀ͢
    4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ

    WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ

    View Slide

  20. खॱ
    5. ϓϥΠϚϦʔ͸WriteϦΫΤετΛड୚ɺ

    ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ)

    ·ͣ͸ϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ
    6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ

    ֤ϨϓϦΧ͸ϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ
    7. ηΧϯμϦʔ͸ૢ࡞͕ऴΘͬͨΒͦͷࢫΛ

    ϓϥΠϚϦʔʹ఻͑Δ
    8. ϓϥΠϚϦʔ͕ૢ࡞׬ྃΛΫϥΠΞϯτʹ఻͑Δ

    View Slide

  21. Write Operationͷࣦഊ
    • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖

    ΫϥΠΞϯτ͸ σʔλૹ৴ૢ࡞͔Β΍Γ௚͢
    • ͦ΋ͦ΋֤ϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ

    อূ͍ͯ͠ͳ͍
    • Appendͷͱ͖͸֤ϨϓϦΧͷ࠷େoffsetΛΈΔ

    View Slide

  22. Snapshot Operation

    View Slide

  23. Snapshot Operation
    • Ծ૝ϑΝΠϧ΍σΟϨΫτϦͷίϐʔΛ࡞Δ
    • Copy-on-Writeઓུ
    • ର৅νϟϯΫʹ Write Operation ͕དྷͨͱ͖

    ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷ৔ॴʹ࡞ΒΕΔ
    • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ
    ๷͍Ͱ͍Δ

    View Slide

  24. ࢀߟจݙ
    • The Google File System

    View Slide