Slide 1

Slide 1 text

LVSPLPͷۙگͱ ΫοΫύουͷόονपΓͷ ֓گ  $PPLQBE5FDI,JUDIFO

Slide 2

Slide 2 text

ࣗݾ঺հ • େੴӳհ • ٕज़෦ ։ൃج൫άϧʔϓ • ڞ௨ܾࡁج൫΍kuroko2ͷӡ༻ ͳͲ • ෳ਺αʔϏεؒͷ੔߹ੑͷऔ Γ૊Έʹ͍ͭͯ

Slide 3

Slide 3 text

"HFOEB • Kuroko2 ͷۙگ • ΫοΫύουͷόονपΓͷ֓گ • ߏ੒ɺӡ༻ํ๏ɺσϓϩΠํ๏ͳͲʹ͍ͭͯ

Slide 4

Slide 4 text

LVSPLPͷۙگ

Slide 5

Slide 5 text

LVSPLPͱ͸ • web-based job scheduler / workflow engine. • ࢀߟࢿྉ • repository: https://github.com/cookpad/kuroko2 • ΫοΫύουͷδϣϒ؅ཧγεςϜ kuroko2 ͷ঺հ • The Design Philosophy of Kuroko2 • kuroko2 documentation

Slide 6

Slide 6 text

ΞʔΩςΫνϟ • Overview https://github.com/cookpad/kuroko2/blob/master/docs/index.md

Slide 7

Slide 7 text

LVSPLPͷԊֵ • 2015/04 ౰ٕ࣌ज़෦௕ͩͬͨ Naoto Takai (ݱΤϯδχΞ౷ׅϚωʔδϟʔ )ͷεΧϯΫ ϫʔΫʹΑͬͯ։ൃ͞Εɺࣾ಺ʹಋೖ͞ΕΔ • 2015/05 େੴ͕ kuroko2 ͷϝϯςΛҾ͖ܧ͙ • 2015/12 ΫοΫύου։ൃऀϒϩάʮΫοΫύουͷδϣϒ؅ཧγεςϜ kuroko2 ͷ঺ հʯ • OSSʹ͸͠ͳ͍ͭ΋Γͩͬͨ • 2016/10 ಥવͷOSSԽ • ॾൠͷࣄ৘ʹΑΓαΠϨϯτͳϦϦʔε • ʮOSS ʹͳͬͨ Kuroko2 ΛͲ͜ΑΓ΋ૣ͘ಋೖͨ͠ͷͰ঺հ͍ͨ͠ϒϩάʯͰ OSSԽ͕ੈؒʹ஌Ε౉Δ • 2017/01 Naoto Takai ͷࣾ಺޲͚LTʮThe Design Philosophy of Kuroko2ʯެ։

Slide 8

Slide 8 text

LVSPLPͷํ਑ • ݱ࣮తͳӡ༻Λݟਾ͑ͨ҆ఆͨ͠ઃܭΛอͭ (The Design Philosophy of Kuroko2 p.20) • ൥ࡶʹͳΓ͕ͪͳόονδϣϒ؅ཧͷ໰୊ΛUI/UXͳͲͰղ ܾ͢Δ • γϯϓϦγςΟ • εέδϡʔϥʔͱ൚༻ੑͷߴ͍ϫʔΫϑϩʔ؅ཧʹప͢Δ • ͏·͘֎ͷػߏͱ࿈ܞ͢Δ (Bricolage, Hako) • ඞཁͳ෦෼͚ͩʹ։͔Ε֦ͨுੑ

Slide 9

Slide 9 text

֦ுੑ • ͨ͘͞Μ঺հ͍ͨ͠ػೳ͸͋Γ·͕͢ࠓճ͸࣮ࡍʹ ݱ৔ʹ kuroko2 Λ౤ೖ͢Δࡍʹ໾ཱͪͦ͏ͳɺ kuroko2 Λ֦ு͢Δํ๏Λ঺հ͠·͢

Slide 10

Slide 10 text

ΧελϜλεΫ 1. λεΫͷ࡞੒ $ cd your_kurko2_rails_apps/ $ mkdir -p lib/kuroko2/workflow/task/ $ cat > my_project_runner.rb module Kuroko2 module Workflow module Task class MyProjectRunner < Execute def chdir '/home/alice/my_project' end def shell "bundle exec ./bin/rails runner -e production #{Shellwords.escape(option)}" end end end end end 2. kuroko2.yml ΁એݴ custom_tasks: my_project_runner: MyProjectRunner 3. kuroko2 script env: VAL1=A env: VAL2=B my_project_runner: MyProject::Batch1.run

Slide 11

Slide 11 text

ΧελϜλεΫͷ࢖͍Ͳ͜Ζ • γϣʔτΧοτͷ࡞੒ • kuroko2ຊମʹखΛ͍Εͳͯ͘΋ӡ༻͍ͯ͠Δυϝ Πϯʹ͋Θͤͨઐ༻ͷλεΫ͕࡞ΕΔ • ࣮ݧతλεΫΛࢼ͢

Slide 12

Slide 12 text

,VSPLP"QQMJDBUJPO$POUSPMMFS֦ு 1. extentionͷ࡞੒ $ cd your_kurko2_rails_apps/ $ cat > lib/controller_extention.rb module ControllerExtention extend ActiveSupport::Concern included do before_action :additional_before_action end private def additional_before_action if signed_in? # do something end end end 2. kuroko2.yml ΁એݴ extentions: controller: - ControllerExtention

Slide 13

Slide 13 text

,VSPLP"QQMJDBUJPO$POUSPMMFS֦ுͷ࢖͍ॴ • ΫοΫύουͰ͸ Raven::Transports::Fluentd ΁ͷί ϯςΩετ௥ՃͳͲͰར༻ (Τϥʔ௨஌) • ϩάͳͲͷτϨʔγϯά • ϢʔβʔຖͷΞΫηε੍ݶ • Controller͚͍ͩ·ͷͱ͜ΖରԠ͍ͯ͠Δ͕ɺ΋͠ඞ ཁέʔε͕͋Ε͹֦ுϙΠϯτΛ૿΍͢͜ͱ͸ݕ౼͠ ͍ͨ

Slide 14

Slide 14 text

͜Ε͔Β • ·ͩ·ͩະ੔උ • υΩϡϝϯτ • ΋ͬͱރΕ͍ͤͨ͞ • ϫʔΫϑϩʔͷػೳڧԽ • Hako (Docker) ͱͷޮ཰తͳ࿈ܞ

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

ΫοΫύουͷόονपΓͷ֓گ

Slide 17

Slide 17 text

Α͋͘Δ࣭໰ • OSSͷ kuroko2 ͸ΫοΫύουࣾ಺Ͱ࢖͍ͬͯΔͷʁ

Slide 18

Slide 18 text

Α͋͘Δ࣭໰ • OSSͷ kuroko2 ͸ΫοΫύουࣾ಺Ͱ࢖͍ͬͯΔͷʁ • YES • ࣾ಺όʔδϣϯ͸ଘࡏ͍ͯ͠ͳ͍ • masterϒϥϯνͷϦϦʔε௚લͷ΋ͷΛৗʹ࢖͏Α͏ ʹͯ͠ɺࣾ಺Ͱ໰୊ͳ͍͔֬ೝͯ͠ϦϦʔε͍ͯ͠Δ

Slide 19

Slide 19 text

ن໛ • ໿700ݸͷόονδϣϒ (2017/06/01࣌఺) • Ұ೔͋ͨΓͷ࣮ߦճ਺͸ ໿6000 • kuroko2 ͷ্Ͱ͢΂ͯ؅ཧ • αʔϏεຖʹ෼͚ͣʹίϯιʔϧ͸͚̍ͭͩ • ͦ͜·Ͱେن໛Ͱ΋ͳ͍ (ͱࢥ͍ͬͯΔ)

Slide 20

Slide 20 text

ߏ੒ • worker (commnad-executor) ͷ਺ ໿230 • 34ݸͷworker༻Πϯελϯε • ൚༻ worker༻Πϯελϯε x 10 • CPUΛ࢖͏όον༻Πϯελϯε x 2 • Hako (docker) ΞϓϦέʔγϣϯ༻ x 3 • େྔϝʔϧ഑৴༻ x 8 • ϓογϡ௨஌༻ x 4 • DWH༻ x 1 • ͦͷଞ x 7

Slide 21

Slide 21 text

؂ࢹ • CPU, Disk, Network, ࢮ׆؂ࢹ ͳͲͷఆܗͷ΋ͷʹՃ͑ͯɺδϣϒͷঢ়گΛ؂ࢹ • OSS kuroko2 ʹ΋͋ΔҎԼͷAPIΛఆظతʹୟ͍͍ͯΔ • /v1/stats/waiting_execution • /v1/stats/instance

Slide 22

Slide 22 text

LVSPLPͷσϓϩΠ • kuroko2ຊମ͸ capistrano ͰσϓϩΠ • console, scheduler, processor, command- executer ͦΕͧΕʹroleΛ෼͚ͯσϓϩΠͯ͠ worker Λ restart • HUPγάφϧʹΑΔ graceful shutdown

Slide 23

Slide 23 text

όονδϣϒͷσϓϩΠ • όονͷσϓϩΠ͸ͦΕͧΕͷΞϓϦέʔγϣϯຖʹ ߦ͍ͬͯΔ • ࠷ۙͷ΋ͷ͸ hako oneshot Ͱ ECS λεΫΛ࣮ߦ͠ ͍ͯΔͷͰ worker ʹରͯ͠഑෍͸͍Βͳ͍ • ݹ͍ΞϓϦέʔγϣϯ͸ worker಺΁഑෍͍ͯ͠Δ

Slide 24

Slide 24 text

όονδϣϒͷ؅ཧ • όονδϣϒ͸֤։ൃऀɺ෦ॺ͕੹೚Λ΋ͭ • δϣϒ͕ࣦഊͨ͠৔߹͸ɺ୲౰ऀʹϝʔϧ΍SlackͰ௨஌ɺ֤୲౰෦ॺͰϦ τϥΠͳͲͷରԠ͢Δ • δϣϒͷ description ʹো֐࣌ͷӨڹɺରԠํ๏ɺϦτϥΠͷඞཁੑͳͲΛ ॻ͍͓ͯ͘͜ͱͰେ͖ͳো֐࣌ʹଞͷ෦ॺ(SRE ͳͲ)Ͱ΋ରԠͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • ϩά͸ kuroko2 ͔Β AWS CloudWatch Logs ʹอଘͭͭ͠ɺΞϓϦέʔγϣ ϯ͕ͦΕͧΕ؅ཧ͍ͯ͠Δϩά΋ଘࡏ͍ͯ͠Δ • ෦ॺ΍δϣϒʹ·͕ͨͬͨδϣϒͷ࿈ܞʹ͸ wait λεΫ͕Α͘࢖ΘΕ͍ͯΔ

Slide 25

Slide 25 text

·ͱΊ • OSS ͷ kuroko2 Λ࢖͍ͬͯ·͢ • αʔϏεຖʹ෼͚ΔͷͰ͸ͳͯ͘ɺ΄΅͢΂ͯͷδϣϒΛ1ͭͷkuroko2 ΞϓϦͰ؅ཧ • தن໛͘Β͍? • 700ݸͷδϣϒ • 6000࣮ߦ/೔ • 230 worker • kuroko2 ͷػೳΛར༻ͨ͠όονઐ༻ͷ؂ࢹ߲໨͕͋Δ • kuroko2 ͷ job definition ͷςϯϓϨ௨Γʹӡ༻