分散リソースマネージメントミドルウェアの設計と実装

分散リソースマネージメントミドルウェアの設計と実装

公立はこだて未来大学 松原研究室 訪問

2019/04/24
さくらインターネット株式会社
さくらインターネット研究所
上級研究員
松本亮介 / まつもとりー / @matsumotory

Transcript

1. ͘͞ΒΠϯλʔωοτגࣜձࣾ
   (C) Copyright 1996-2019 SAKURA Internet Inc
   ͘͞ΒΠϯλʔωοτݚڀॴ
   ෼ࢄϦιʔεϚωʔδϝϯτϛυϧ΢ΣΞ
   ͷઃܭͱ࣮૷
   2019/04/23 ্ڃݚڀһ দຊ ྄հ
   ެཱ͸ͩͯ͜ະདྷେֶ দݪݚڀࣨ ๚໰
   

   View Slide

2. 2
   ɾ͘͞ΒΠϯλʔωοτݚڀॴ ্ڃݚڀһ
   ɾגࣜձࣾGrooves Forkewll ٕज़ސ໰
   ɾϖύϘݚڀॴ ٬һݚڀһ ݚڀސ໰
   ɾηΩϡϦςΟɾΩϟϯϓߨࢣ
   ɾ৘ใॲཧֶձ Πϯλʔωοτͱӡ༻ٕज़ݚڀձ ֤छҕһ
   ɾژ౎େֶത࢜ʢ৘ใֶʣ
   দຊ྄հ / ·ͭ΋ͱΓʔ / @matsumotory
   

   View Slide

3. 3
   1. ͸͡Ίʹ
   2. ϦιʔεϚωʔδϝϯτϛυϧ΢ΣΞ
   3. ϦιʔεϚωʔδϝϯτγεςϜͷ෼ࢄԽͷઃܭͱ࣮૷
   4. ·ͱΊ
   ໨࣍
   

   View Slide

4. 4
   ௒ݸମతσʔληϯλʔʹ޲͚ͨऔΓ૊Έ
   • Ϋϥ΢υωΠςΟϒɾϚϧνΫϥ΢υɾಛఆOSSͷґଘͷഉআ
   • ίϯςφ୯ҐͰͷαʔϏε΍ػೳɺ؅ཧख๏ͷ෼ࢄԽ
   • ϚΠΫϩαʔϏεͷΑ͏ͳSOAࢤ޲ͷઃܭख๏΁
   • ૊৫΍νʔϜͱͯ͠ͷࣗ཯෼ࢄԽ΋ؚΉ
   • ͘͞ΒΠϯλʔωοτݚڀॴʹΑΔ௒ݸମతσʔληϯλʔͷऔΓ૊Έ
   • ϨΠςϯγͷ௿ݮͱࣾձʹ༹͚ࠐΉίϯϐϡʔςΟϯά
   • ਓͷۙ͘ʹͲΜͲΜͱΫϥ΢υͷϚγϯύϫʔΛد͍ͤͯ͘
   

   View Slide

5. 5
   ӡ༻ٕज़ͷ෼ࢄԽ
   • ϚΠΫϩαʔϏε୯ҐͰνʔϜͱͯ͠ӡ༻͢Δํ޲ੑ͕ͻͱͭ
   • ࣗ཯෼ࢄ͍ͯ͘͠αʔϏε΍ίϯςφΛӡ༻͢Δٕज़ͷਐԽ
   • k8sΛ࢝Ίͱ͢ΔΦʔέετϨʔγϣϯ૚ʹΑΔ؅ཧ
   • k8sʹΑͬͯΫϥ΢υ΍OSΛந৅Խͯ͠Ϋϥ΢υOSͱͳΔੈք
   • k8sʹґଘ͠ͳ͍෼ࢄͨ͠ίϯςφ΍VM؀ڥͷϦιʔεϚωʔδϝϯτ
   • k8sΛΑΓ௿͍૚͔Βࢧ͑Δج൫ٕज़։ൃͱ΋ݴ͑Δ?
   • ΦʔέετϨʔγϣϯ૚ʹ࣮૷͢ΔͨΊͷӡ༻ٕज़ͷ෼ࢄԽ
   

   View Slide

6. 1.
   ϦιʔεϚωʔδϝϯτϛυϧ΢ΣΞ
   

   View Slide

7. 7
   k2i
   • ίϯςφΛऔΓר͘෼ࢄγεςϜʹ͓͚Δϓϩηε৘ใΛಁաతʹऔಘ
   • ίϯςφͷऩ༰ϗετͷϓϩηε৘ใΛϦϞʔτ͔ΒऔಘɾௐࠪՄೳ
   • ֤ऩ༰ϗετʹىಈ͓ͤͯ͘͜͞ͱͰಁաతʹϓϩηε৘ใΛऔಘՄೳ
   • ࣮૷͸RustͷHTTPαʔό+libprocpsͷRust bindingʹΑ࣮ͬͯ૷
   • Ͱ͖Δ͚ͩߴ଎ʹಈ࡞͢ΔΑ͏ʹ
   • https://github.com/matsumotory/k2i
   • https://github.com/matsumotory/procps-sys
   

   View Slide

8. 8
   drcond
   • ϦϞʔτ͔Βऩ༰ϗετͷ֤छϓϩηεͷ֤छϦιʔε࢖༻ྔΛ੍ޚ
   • localݶఆͷcliπʔϧͱͯ͠͸rconͱ͍͏πʔϧΛҎલ࡞ͬͨ
   • ෳ਺ͷऩ༰ϗετʹରͯ͠drcond͕ಈ͍͍ͯΕ͹ϦϞʔτ͔Β੍ޚՄೳ
   • ֤छϛυϧ΢ΣΞ͔ΒҰ࣌తʹϦιʔεΛ੍ޚ͍ͨ͠৔߹΋ར༻Մೳ
   • ྫ: HTTPϦΫΤετ୯ҐͰಛఆͷϦΫΤετॲཧ͚ͩΛ੍ޚ͢Δͱ͔※1
   • drcond͕͋Ε͹SMTPηογϣϯ୯ҐͰ΋൚༻తʹ੍ޚՄೳʹͳΔ
   ※1 দຊ྄հ, ܀ྛ݈ଠ࿠, Ԭ෦णஉ, ϦΫΤετ୯ҐͰԾ૝తʹϋʔυ΢ΣΞϦιʔεΛ෼཭͢ΔWebαʔόͷϦιʔ
   ε੍ޚΞʔΩςΫνϟ, ৘ใॲཧֶձ࿦จࢽ, Vol.59, No.3, pp.1016-1025, 2018೥3݄.
   

   View Slide

9. 9
   drcond
   • ϦΫΤετ୯Ґͱ͔ηογϣϯ୯ҐͰͷҰ࣌తͳ੍ݶͳͲͷέʔε
   • Ϧιʔε੍ޚॲཧࣗମ͕ΦʔόʔϔουʹͳΒͳ͍Α͏ͳੑೳ͕ඞཁ
   • Trusterdͱ͍͏ߴ଎ʹmrubyͰಈ࡞͢ΔHTTP/2αʔόΛར༻
   • https://github.com/matsumotory/trusterd
   • nginxΑΓ໿3ഒ଎͍ͱ͍͏ݕূ݁Ռ΋ग़ͨ https://hb.matsumoto-r.jp/
   entry/2015/12/16/000114
   • trusterdʹmruby-cgroupΛ૊ΈࠐΜͰRubyͰrcondΛ࣮૷͍ͯ͠Δ్த
   • https://github.com/matsumotory/mruby-cgroup
   

   View Slide

10. drcond is a distributed resource control
    middle-ware.
    

    View Slide

11. distributed????
    

    View Slide

12. 2.
    ϦιʔεϚωʔδϝϯτϛυϧ΢ΣΞͷ
    ෼ࢄԽ
    

    View Slide

13. 13
    ͜͜·Ͱ঺հͨ͠ϛυϧ΢ΣΞ
    • ֤ऩ༰αʔόʹىಈͤ͞Δϛυϧ΢ΣΞ
    • ෳ਺ʹىಈͤͨ͞ΓɺࣗಈͰσϓϩΠ͞ΕͨVM΍෺ཧαʔόͰىಈͨ͠৔߹
    ʹͦΕͧΕͷ৘ใΛूΊͨΓҾ਺ʹ౉࣮ͯ͠ߦ͢Δͷ͸෼ࢄతͰͳ͍
    • ෼ࢄγεςϜͷͨΊͷӡ༻ٕज़ͳͷͰ͋ΔͷͰɺͦ͜Ͱඞཁͳπʔϧ΍ϛυϧ
    ΢ΣΞ΋෼ࢄԽ͍ͤͨ͞
    • ίϯηϯαεΞϧΰϦζϜʹΑͬͯྑ͍ײ͡ʹ͢Δʂ
    

    View Slide

14. 14
    ίϯηϯαεΞϧΰϦζϜͰ΍Γ͍ͨ͜ͱ
    • ෼ࢄγεςϜͰ໋ྩʹର͢Δ߹ҙΛͱͬͯશମʹಉ໋͡ྩΛ࣮ߦ͍ͨ͠
    • ͦͷͨΊʹ΋֤αʔόΛϝϯόγοϓͱͯ͠Έͳͯ͠ϝϯόʔγοϓ؅ཧ
    • ϦʔμʔΛҰਓબఆͯ͠ΫϥΠΞϯτ͔Β͸Ϧʔμʔ͋Δ͍͸ϑΥϩϫʔʹ
    ͍ͬͯ΋ϦʔμʔϊʔυʹϦμΠϨΫτ͞ΕΔΑ͏ʹ͍ͨ͠
    

    View Slide

15. 15
    ίϯηϯαεΞϧΰϦζϜͰ΍Γ͍ͨ͜ͱ
    • ͦͷ্ͰϦʔμʔʹର໋ͯ͠ྩΛ౤͛ΔͱશϑΥϩϫʔͰ΋໋ྩΛ࣮ߦ
    • ෼ࢄτϥϯβΫγϣϯతͳཁૉ΍ϩάอଘ͸༏ઌ౓͸௿ΊͰݕ౼
    • ૝ఆ͍ͯ͠Δk2i΍drcondͰͷ໋ྩ͕લޙͱؔ܎ͷͳ͍୯Ұ໋ྩ͔ͭτϥ
    ϯβΫγϣϯͱͯ͠ѻΘͳͯ͘΋ྑ͍͔΋͠Εͳ͍͔Β
    • ͍΍Ͱ΋Ϧʔμʔબग़ʹ݁ہτϥϯβΫγϣϯͷϧʔϧ͕ೖΓͦ͏ʁ
    

    View Slide

16. 16
    k2i΍drcondͷ෼ࢄԽ
    leader
    k2i
    follower
    k2i
    follower
    k2i
    client
    

    View Slide

17. 17
    ίϯηϯαεΞϧΰϦζϜͱ࣮૷
    • Paxos, Raft, Zab, 2PC౳ɺ࣮૷͸etcd, consul, CockroachDB, TiDB౳
    • Paxos
    • ੲ͔Β͋ΔΞϧΰϦζϜͰ͋Δ͕ཧ࿦͕೉ͯ͘͠Paxos-baseΈ͍ͨͳ࣮૷
    ͕େྔʹੜ·ΕΔ
    • ͦͷͨΊ࣮ࡍʹҰ؏ੑΛอূͰ͖͍ͯΔͷ͔͕ෆ໌ͰࠔΔ͜ͱ͕ଟ਺
    • Raft
    • In search of an understandable consensus algorithm in USENIX 2014
    • PaxosΛ࣮༻తʹ͸໰୊ແ͍ϨϕϧͰੑೳͱཧղͷ͠΍͢͞Λվળ
    

    View Slide

18. 18
    Raftͷ໘ന͍ͱ͜Ζ
    • ෳࡶͳ෼ࢄ߹ҙܗ੒Λ࣮༻্໰୊ͳ͍ͱ͍͏લఏͰཧ࿦Λߏங
    • ΞϧΰϦζϜͰ͋ͬͯ΋࣮ફΛલఏʹ͍ͯ͠Δͱ͜Ζ͕͓͘͢͝΋͠Ζ͍
    • Ϧʔμʔ͕ແݶʹܾ·Βͳ͍໰୊΋λΠϜΞ΢τΛཚ਺ʹ͢Ε͹࣮༻্໰
    ୊ແ͍ϨϕϧͰϥΠϒϩοΫʹͳͬͯϦʔμʔΛܾΊΒΕΔͱ͔
    • ϩάͷ࠶ૹͱ্ॻ͖ͷޮ཰΋ࣦഊͨ͠Βલʹ໭ͬͯ΍Γ௚͠Λ܁Γฦ͢
    ͷͰҰݟޮ཰͕ѱͦ͏͚ͩͲ࣮༻্͸ͦ͜·Ͱͷ໰୊͕ى͖ʹ͍͘ͱ͔
    • https://www.usenix.org/system/files/conference/atc14/atc14-paper-
    ongaro.pdf
    

    View Slide

19. 19
    ෼ࢄτϥϯβΫγϣϯͱ͔ϩάอଘ
    • ίϯηϯαεΞϧΰϦζϜͰॏཁͳཁૉ
    • ͨͩ๻ͷ໨తʹ͸ͦΜͳʹඞཁͳ͍ͷͰ͸ͳ͍͔
    • ໋ྩΛϩάʹͯ͠Ұ؏ੑΛอͭͱ͍͏ػೳ΋k2i΍drcondʹ͓͍ͯ͸໋ྩࣗମ
    ͕εςʔτϨε
    • ͱ͸͍͑ɺ࣮ߦઌͰͷ࣮ߦ଎౓ͳͲ͸ෛՙʹΑͬͯมΘΔͷͰߟྀ͸ඞཁ
    • KVSͷΑ͏ʹϩάΛͨΊͨΓίϯύΫγϣϯΛ͢Δͱ͍͏ͷ͸୹࣌ؒͰྑ͍
    • ετϨʔδ΁ͷอଘΛߟྀ͠ͳ͚Ε͹ߋʹraft͸ߴ͍ੑೳ͕ͰΔ ※1
    ※1 ෼ࢄ߹ҙΞϧΰϦζϜRaftͷௐࠪ http://db-event.jpn.org/deim2017/papers/386.pdf
    

    View Slide

20. 20
    Ԡ༻ྫ: Koordinator
    • A Service Approach for Replicating Docker Containers in Kubernetes
    • k8sͰѻ͏ίϯςφ͕εςʔτΛอͭ৔߹ʹreplicaͷҰ؏ੑΛอͭ
    • ϚϧνίϯςφͰwriteͷ௚ޙʹམͪΔͱଞͷίϯςφͱঢ়ଶ͕มΘΔ
    • proxyͱίϯςφؒʹkoordinator૚Λ࡞Δ
    • koordinator૚ͰίϯηϯαεΞϧΰϦζϜΛ࢖໋ͬͯྩΛorderԽͯ͠ෳ
    ਺ͷίϯςφʹ໋ྩॱʹಉ࣮࣌ߦ →Ұ؏ੑΛΑΓڧ͍΋ͷʹ͢Δ
    Hylson Vescovi Netto, Aldelir Fernando Luiz, Miguel Correia, Luciana de Oliveira Rech, Caio Pereira Oliveira,
    Koordinator: A Service Approach for Replicating Docker Containers in Kubernetes, 2018 IEEE Symposium on
    Computers and Communications (ISCC) Year: 2018, Volume: 1, Pages: 58-51
    

    View Slide

21. Ԡ༻ྫɿKoordinator
    21
    client
    FW
    proxy
    proxy
    proxy
    koordinator
    koordinator
    koordinator
    app
    app
    app
    client
    FW
    proxy
    proxy
    proxy
    koordinator
    koordinator
    koordinator
    app
    app
    app
    $POTFOTVT"MHPSJUIN
    0SEFSJOH
    &YFDVUJPO
    XSJUFSFRVFTU SFBESFRVFTU
    

    View Slide

22. 22
    k2i΍drcondͷ෼ࢄԽ: Multiraft
    leader
    follower
    follower
    client
    leader
    follower
    follower
    leader
    follower
    follower
    Multiraftͱ͔RaftΫϥελͷ෼ࢄΈ͍ͨͳ࿩୊΋͋Δ
    1. https://www.cockroachlabs.com/blog/scaling-raft/
    2. http://db-event.jpn.org/deim2017/papers/386.pdf
    3. http://sergeiturukin.com/2017/06/09/multiraft.html
    

    View Slide

23. 23
    ࣮૷༧ఆ
    • Raftͷ࣮૷͸͍ͬͺ͍͋Δ CͰ΋RustͰ΋
    • drcond͸mrubyͰ࡞͍ͬͯΔͷͰCͷϥΠϒϥϦΛ࢖͏ͭ΋Γ
    • k2i͸RustͰॻ͍͍ͯΔͷͰRustΛૉ௚ʹ࢖͏͔
    • ͱ͸͍͑ίϯηϯαεΞϧΰϦζϜͰKVS΍σʔλϕʔεΛ࡞Δͱ͍͏Θ͚Ͱ
    ͸ͳ͍ͷͰɺ֤छϥΠϒϥϦΛ࢖͍ͳ͕Β໨తʹ͋ͬͨՕॴͷ࣮૷Λ͍ͨ͠
    

    View Slide

24. 3.
    ·ͱΊ
    

    View Slide

25. 25
    ίϯςφͷ෼ࢄγεςϜΛࢧ͑Δӡ༻ٕज़
    • ෼ࢄγεςϜʹ߹Θͤͯӡ༻πʔϧ΋෼ࢄԽ
    • ෼ࢄϦιʔεϚωʔδϝϯτϛυϧ΢ΣΞͷઃܭͱ࣮૷ͷ঺հ
    • ίϯηϯαεΞϧΰϦζϜʹΑΔ֤छϛυϧ΢ΣΞͷ෼ࢄԽରԠ
    • k8s͕Ͳ͜·Ͱ΍ͬͯͲ͜·Ͱӡ༻πʔϧͰରԠ͢΂͖͔
    • k8sͱ෼ࢄӡ༻πʔϧͷཱͪҐஔ͸Ͳ͏͋Δ΂͖͔
    

    View Slide