Google File System

Google File Systemの中身を紹介

Transcript

1. The Google File System
   SDDษڧձ@r-n-i
   2018/10/19
   

   View Slide

2. ֓ཁ
   • Google ͷϓϩμΫτ಺෦Ͱ࢖ΘΕ͍ͯͨ

   େن໛σʔλ޲͚ͷϑΝΠϧγεςϜ
   • ΦʔϓϯιʔεͰ͸ͳ͍
   • Hadoopͷ෼ࢄϑΝΠϧεςϜ͕ࢀߟʹͨ͠
   • ਺ઍ୆ ͷαʔό͔ͨͪΒ੒Δ
   

   View Slide

3. ※2003೥ͷ࿦จͰ͢
   

   View Slide

4. ౰࣌ͷঢ়گ(࿦จ͔Βൈਮ)
   • Dual 1.4GHz Pentium III Processor
   • 2GB Memory
   • 80GB 5400 rpm disk ×2
   • 100Mbps full-duplex Ethernet
   

   View Slide

5. ϋʔυ΢ΣΞͷલఏ
   • γεςϜΛߏ੒͢Δύʔπ͸ յΕΔ
   • ࣓ؾσΟεΫ
   • ωοτϫʔΫέʔϒϧ
   • ஍਒ͰϥοΫ౗յ
   • ҆Ձͳ΋ͷΛར༻ͯ͠ͳΜͱ͔͍ͨ͠
   

   View Slide

6. Ϣʔεέʔεͷલఏ
   • Read ΫΤϦ਺ >> WriteΫΤϦ਺
   • ॻ͖׵͑͸ຆͲͳ͍ɺAppend(௥ه) ͷΈ
   

   View Slide

7. ΫϥΠΞϯτ͔Βݟͨ
   Google File System
   

   View Slide

8. ϑΝΠϧγεςϜʁ
   • Ծ૝తͳπϦʔσΟϨΫτϦΛ࣋ͭ
   • ΫϥΠΞϯτ͔Β͸ී௨ͷϑΝΠϧʹݟ͍͑ͯΔ
   • ϑΝΠϧͷ࣮ମ͸֤αʔόʹ෼ࢄ഑ஔ
   • POSIXޓ׵Ͱ͸ͳ͍͕ͦΕͬΆ͍APIΛ࣋ͭ
   • GFS্ͷ(ΫϥΠΞϯτ͔Βݟͨ)ϑΝΠϧͷ͜ͱΛҎ߱

   Ծ૝ϑΝΠϧ ͱݺͿ͜ͱʹ͠·͢
   

   View Slide

9. ߏ੒ཁૉͷ঺հ
   

   View Slide

10. ΞʔΩςΫνϟ
    

    View Slide

11. νϟϯΫ / Chunk
    • 1Ծ૝ϑΝΠϧ has many νϟϯΫ
    • 1νϟϯΫ = 64MB
    • ෳ਺ͷϨϓϦΧ͕νϟϯΫαʔόʹଘࡏ
    • 1ͭͷϓϥΠϚϦʔ
    • NݸͷηΧϯμϦʔ
    

    View Slide

12. νϟϯΫαʔό / Chunk server
    • νϟϯΫͨͪΛ࣮ࡍʹ෺ཧతʹ࣋ͭ
    • ΫϥΠΞϯτ͕σʔλऔಘ࣌ʹ

    ௚ͰΞΫηε͢Δ
    • ଳҬͷઅ໿ͷͨΊ
    

    View Slide

13. Ϛελʔαʔό / Master server
    • Ծ૝ϑΝΠϧ(ͱͦͷΦϑηοτ)ͱνϟϯΫͷϚοϓ
    • ֤νϟϯΫ͕ͲͷνϟϯΫαʔόʹ͋Δ͔
    • νϟϯΫαʔόͷࢮ׆؂ࢹ
    • ૢ࡞ϩάΛه࿥
    • Ϋϥογϡͯ͠΋෮ݩͰ͖ΔΑ͏ʹ
    

    View Slide

14. Read Operation
    

    View Slide

15. View Slide

16. खॱ
    1. (Ծ૝ϑΝΠϧ໊,νϟϯΫ൪߸) Λ

    ϚελʔʹϦΫΤετ
    2. ֘౰νϟϯΫ͕͋Δαʔόͷ৔ॴΛڭ͑ͯ΋Β͏
    3. νϟϯΫαʔόʹ (νϟϯΫϋϯυϧ, νϟϯΫ಺Ͱ
    ཉ͍͠byteͷൣғ)ΛϦΫΤετ
    4. νϟϯΫαʔό͔Βσʔλ͕΋Β͑Δ
    

    View Slide

17. Write Operation
    

    View Slide

18. View Slide

19. खॱ
    1. νϟϯΫϋϯυϧ ΛϚελʔʹϦΫΤετ
    2. ֘౰νϟϯΫͷϓϥΠϚϦʔϨϓϦΧͱ

    ηΧϯμϦʔϨϓϦΧͷ৔ॴΛڭ͑ͯ΋Β͏
    3. σʔλΛ೚ҙͷϨϓϦΧʹૹΔ

    σʔλΛड͚औͬͨϨϓϦΧ͸࣍ͷϨϓϦΧʹྲྀ͢
    4. શϨϓϦΧ͕σʔλΛೝࣝͨ͠Βɺ

    WriteϦΫΤετΛϓϥΠϚϦʔʹૹΔ
    

    View Slide

20. खॱ
    5. ϓϥΠϚϦʔ͸WriteϦΫΤετΛड୚ɺ

    ϦΫΤετʹγϦΞϧ൪߸Λ͚ͭΔ(ૢ࡞ͷॱংΛܾఆ)

    ·ͣ͸ϓϥΠϚϦʔͰૢ࡞Λ࣮ߦ
    6. ϓϥΠϚϦʔ͕WriteϦΫΤετΛଞͷϨϓϦΧʹૹΔɻ

    ֤ϨϓϦΧ͸ϦΫΤετΛγϦΞϧ൪߸ॱʹॲཧ
    7. ηΧϯμϦʔ͸ૢ࡞͕ऴΘͬͨΒͦͷࢫΛ

    ϓϥΠϚϦʔʹ఻͑Δ
    8. ϓϥΠϚϦʔ͕ૢ࡞׬ྃΛΫϥΠΞϯτʹ఻͑Δ
    

    View Slide

21. Write Operationͷࣦഊ
    • ॻ͖ࠐΈʹࣦഊͨ͠ϨϓϦΧ͕͋Δͱ͖

    ΫϥΠΞϯτ͸ σʔλૹ৴ૢ࡞͔Β΍Γ௚͢
    • ͦ΋ͦ΋֤ϨϓϦΧͷbyteྻ͕ಉҰͰ͋Δ͜ͱΛ

    อূ͍ͯ͠ͳ͍
    • Appendͷͱ͖͸֤ϨϓϦΧͷ࠷େoffsetΛΈΔ
    

    View Slide

22. Snapshot Operation
    

    View Slide

23. Snapshot Operation
    • Ծ૝ϑΝΠϧ΍σΟϨΫτϦͷίϐʔΛ࡞Δ
    • Copy-on-Writeઓུ
    • ର৅νϟϯΫʹ Write Operation ͕དྷͨͱ͖

    ৽νϟϯΫ͕ݩνϟϯΫͱಉҰͷ৔ॴʹ࡞ΒΕΔ
    • νϟϯΫσʔλ͕ωοτϫʔΫΛඈͼަ͏͜ͱΛ
    ๷͍Ͱ͍Δ
    

    View Slide

24. ࢀߟจݙ
    • The Google File System
    

    View Slide