Upgrade to Pro — share decks privately, control downloads, hide ads and more …

kuroko2の近況とクックパッドのバッチ周りの概況

 kuroko2の近況とクックパッドのバッチ周りの概況

Eisuke Oishi

June 07, 2017
Tweet

More Decks by Eisuke Oishi

Other Decks in Technology

Transcript

  1. LVSPLPͷۙگͱ
    ΫοΫύουͷόονपΓͷ
    ֓گ

    $PPLQBE5FDI,JUDIFO

    View Slide

  2. ࣗݾ঺հ
    • େੴӳհ
    • ٕज़෦ ։ൃج൫άϧʔϓ
    • ڞ௨ܾࡁج൫΍kuroko2ͷӡ༻
    ͳͲ
    • ෳ਺αʔϏεؒͷ੔߹ੑͷऔ
    Γ૊Έʹ͍ͭͯ

    View Slide

  3. "HFOEB
    • Kuroko2 ͷۙگ
    • ΫοΫύουͷόονपΓͷ֓گ
    • ߏ੒ɺӡ༻ํ๏ɺσϓϩΠํ๏ͳͲʹ͍ͭͯ

    View Slide

  4. LVSPLPͷۙگ

    View Slide

  5. LVSPLPͱ͸
    • web-based job scheduler / workflow engine.
    • ࢀߟࢿྉ
    • repository: https://github.com/cookpad/kuroko2
    • ΫοΫύουͷδϣϒ؅ཧγεςϜ kuroko2 ͷ঺հ
    • The Design Philosophy of Kuroko2
    • kuroko2 documentation

    View Slide

  6. ΞʔΩςΫνϟ
    • Overview https://github.com/cookpad/kuroko2/blob/master/docs/index.md

    View Slide

  7. LVSPLPͷԊֵ
    • 2015/04 ౰ٕ࣌ज़෦௕ͩͬͨ Naoto Takai (ݱΤϯδχΞ౷ׅϚωʔδϟʔ )ͷεΧϯΫ
    ϫʔΫʹΑͬͯ։ൃ͞Εɺࣾ಺ʹಋೖ͞ΕΔ
    • 2015/05 େੴ͕ kuroko2 ͷϝϯςΛҾ͖ܧ͙
    • 2015/12 ΫοΫύου։ൃऀϒϩάʮΫοΫύουͷδϣϒ؅ཧγεςϜ kuroko2 ͷ঺
    հʯ
    • OSSʹ͸͠ͳ͍ͭ΋Γͩͬͨ
    • 2016/10 ಥવͷOSSԽ
    • ॾൠͷࣄ৘ʹΑΓαΠϨϯτͳϦϦʔε
    • ʮOSS ʹͳͬͨ Kuroko2 ΛͲ͜ΑΓ΋ૣ͘ಋೖͨ͠ͷͰ঺հ͍ͨ͠ϒϩάʯͰ
    OSSԽ͕ੈؒʹ஌Ε౉Δ
    • 2017/01 Naoto Takai ͷࣾ಺޲͚LTʮThe Design Philosophy of Kuroko2ʯެ։

    View Slide

  8. LVSPLPͷํ਑
    • ݱ࣮తͳӡ༻Λݟਾ͑ͨ҆ఆͨ͠ઃܭΛอͭ (The Design
    Philosophy of Kuroko2 p.20)
    • ൥ࡶʹͳΓ͕ͪͳόονδϣϒ؅ཧͷ໰୊ΛUI/UXͳͲͰղ
    ܾ͢Δ
    • γϯϓϦγςΟ
    • εέδϡʔϥʔͱ൚༻ੑͷߴ͍ϫʔΫϑϩʔ؅ཧʹప͢Δ
    • ͏·͘֎ͷػߏͱ࿈ܞ͢Δ (Bricolage, Hako)
    • ඞཁͳ෦෼͚ͩʹ։͔Ε֦ͨுੑ

    View Slide

  9. ֦ுੑ
    • ͨ͘͞Μ঺հ͍ͨ͠ػೳ͸͋Γ·͕͢ࠓճ͸࣮ࡍʹ
    ݱ৔ʹ kuroko2 Λ౤ೖ͢Δࡍʹ໾ཱͪͦ͏ͳɺ
    kuroko2 Λ֦ு͢Δํ๏Λ঺հ͠·͢

    View Slide

  10. ΧελϜλεΫ
    1. λεΫͷ࡞੒
    $ cd your_kurko2_rails_apps/
    $ mkdir -p lib/kuroko2/workflow/task/
    $ cat > my_project_runner.rb
    module Kuroko2
    module Workflow
    module Task
    class MyProjectRunner < Execute
    def chdir
    '/home/alice/my_project'
    end
    def shell
    "bundle exec ./bin/rails runner -e
    production #{Shellwords.escape(option)}"
    end
    end
    end
    end
    end
    2. kuroko2.yml ΁એݴ
    custom_tasks:
    my_project_runner: MyProjectRunner
    3. kuroko2 script
    env: VAL1=A
    env: VAL2=B
    my_project_runner: MyProject::Batch1.run

    View Slide

  11. ΧελϜλεΫͷ࢖͍Ͳ͜Ζ
    • γϣʔτΧοτͷ࡞੒
    • kuroko2ຊମʹखΛ͍Εͳͯ͘΋ӡ༻͍ͯ͠Δυϝ
    Πϯʹ͋Θͤͨઐ༻ͷλεΫ͕࡞ΕΔ
    • ࣮ݧతλεΫΛࢼ͢

    View Slide

  12. ,VSPLP"QQMJDBUJPO$POUSPMMFS֦ு
    1. extentionͷ࡞੒
    $ cd your_kurko2_rails_apps/
    $ cat > lib/controller_extention.rb
    module ControllerExtention
    extend ActiveSupport::Concern
    included do
    before_action :additional_before_action
    end
    private
    def additional_before_action
    if signed_in?
    # do something
    end
    end
    end
    2. kuroko2.yml ΁એݴ
    extentions:
    controller:
    - ControllerExtention

    View Slide

  13. ,VSPLP"QQMJDBUJPO$POUSPMMFS֦ுͷ࢖͍ॴ
    • ΫοΫύουͰ͸ Raven::Transports::Fluentd ΁ͷί
    ϯςΩετ௥ՃͳͲͰར༻ (Τϥʔ௨஌)
    • ϩάͳͲͷτϨʔγϯά
    • ϢʔβʔຖͷΞΫηε੍ݶ
    • Controller͚͍ͩ·ͷͱ͜ΖରԠ͍ͯ͠Δ͕ɺ΋͠ඞ
    ཁέʔε͕͋Ε͹֦ுϙΠϯτΛ૿΍͢͜ͱ͸ݕ౼͠
    ͍ͨ

    View Slide

  14. ͜Ε͔Β
    • ·ͩ·ͩະ੔උ
    • υΩϡϝϯτ
    • ΋ͬͱރΕ͍ͤͨ͞
    • ϫʔΫϑϩʔͷػೳڧԽ
    • Hako (Docker) ͱͷޮ཰తͳ࿈ܞ

    View Slide

  15. View Slide

  16. ΫοΫύουͷόονपΓͷ֓گ

    View Slide

  17. Α͋͘Δ࣭໰
    • OSSͷ kuroko2 ͸ΫοΫύουࣾ಺Ͱ࢖͍ͬͯΔͷʁ

    View Slide

  18. Α͋͘Δ࣭໰
    • OSSͷ kuroko2 ͸ΫοΫύουࣾ಺Ͱ࢖͍ͬͯΔͷʁ
    • YES
    • ࣾ಺όʔδϣϯ͸ଘࡏ͍ͯ͠ͳ͍
    • masterϒϥϯνͷϦϦʔε௚લͷ΋ͷΛৗʹ࢖͏Α͏
    ʹͯ͠ɺࣾ಺Ͱ໰୊ͳ͍͔֬ೝͯ͠ϦϦʔε͍ͯ͠Δ

    View Slide

  19. ن໛
    • ໿700ݸͷόονδϣϒ (2017/06/01࣌఺)
    • Ұ೔͋ͨΓͷ࣮ߦճ਺͸ ໿6000
    • kuroko2 ͷ্Ͱ͢΂ͯ؅ཧ
    • αʔϏεຖʹ෼͚ͣʹίϯιʔϧ͸͚̍ͭͩ
    • ͦ͜·Ͱେن໛Ͱ΋ͳ͍ (ͱࢥ͍ͬͯΔ)

    View Slide

  20. ߏ੒
    • worker (commnad-executor) ͷ਺ ໿230
    • 34ݸͷworker༻Πϯελϯε
    • ൚༻ worker༻Πϯελϯε x 10
    • CPUΛ࢖͏όον༻Πϯελϯε x 2
    • Hako (docker) ΞϓϦέʔγϣϯ༻ x 3
    • େྔϝʔϧ഑৴༻ x 8
    • ϓογϡ௨஌༻ x 4
    • DWH༻ x 1
    • ͦͷଞ x 7

    View Slide

  21. ؂ࢹ
    • CPU, Disk, Network, ࢮ׆؂ࢹ ͳͲͷఆܗͷ΋ͷʹՃ͑ͯɺδϣϒͷঢ়گΛ؂ࢹ
    • OSS kuroko2 ʹ΋͋ΔҎԼͷAPIΛఆظతʹୟ͍͍ͯΔ
    • /v1/stats/waiting_execution
    • /v1/stats/instance

    View Slide

  22. LVSPLPͷσϓϩΠ
    • kuroko2ຊମ͸ capistrano ͰσϓϩΠ
    • console, scheduler, processor, command-
    executer ͦΕͧΕʹroleΛ෼͚ͯσϓϩΠͯ͠
    worker Λ restart
    • HUPγάφϧʹΑΔ graceful shutdown

    View Slide

  23. όονδϣϒͷσϓϩΠ
    • όονͷσϓϩΠ͸ͦΕͧΕͷΞϓϦέʔγϣϯຖʹ
    ߦ͍ͬͯΔ
    • ࠷ۙͷ΋ͷ͸ hako oneshot Ͱ ECS λεΫΛ࣮ߦ͠
    ͍ͯΔͷͰ worker ʹରͯ͠഑෍͸͍Βͳ͍
    • ݹ͍ΞϓϦέʔγϣϯ͸ worker಺΁഑෍͍ͯ͠Δ

    View Slide

  24. όονδϣϒͷ؅ཧ
    • όονδϣϒ͸֤։ൃऀɺ෦ॺ͕੹೚Λ΋ͭ
    • δϣϒ͕ࣦഊͨ͠৔߹͸ɺ୲౰ऀʹϝʔϧ΍SlackͰ௨஌ɺ֤୲౰෦ॺͰϦ
    τϥΠͳͲͷରԠ͢Δ
    • δϣϒͷ description ʹো֐࣌ͷӨڹɺରԠํ๏ɺϦτϥΠͷඞཁੑͳͲΛ
    ॻ͍͓ͯ͘͜ͱͰେ͖ͳো֐࣌ʹଞͷ෦ॺ(SRE ͳͲ)Ͱ΋ରԠͰ͖ΔΑ͏ʹ
    ͍ͯ͠Δ
    • ϩά͸ kuroko2 ͔Β AWS CloudWatch Logs ʹอଘͭͭ͠ɺΞϓϦέʔγϣ
    ϯ͕ͦΕͧΕ؅ཧ͍ͯ͠Δϩά΋ଘࡏ͍ͯ͠Δ
    • ෦ॺ΍δϣϒʹ·͕ͨͬͨδϣϒͷ࿈ܞʹ͸ wait λεΫ͕Α͘࢖ΘΕ͍ͯΔ

    View Slide

  25. ·ͱΊ
    • OSS ͷ kuroko2 Λ࢖͍ͬͯ·͢
    • αʔϏεຖʹ෼͚ΔͷͰ͸ͳͯ͘ɺ΄΅͢΂ͯͷδϣϒΛ1ͭͷkuroko2
    ΞϓϦͰ؅ཧ
    • தن໛͘Β͍?
    • 700ݸͷδϣϒ
    • 6000࣮ߦ/೔
    • 230 worker
    • kuroko2 ͷػೳΛར༻ͨ͠όονઐ༻ͷ؂ࢹ߲໨͕͋Δ
    • kuroko2 ͷ job definition ͷςϯϓϨ௨Γʹӡ༻

    View Slide