AirflowからKuroko2に乗り換えたわけ

Tokyo Airflow Meetup #1

Transcript

1. Airflow͔ΒKuroko2
   ʹ৐Γ׵͑ͨΘ͚
   by hatappi @Tokyo Airflow Meetup #1
   1
   

   View Slide

2. ໨࣍
   • ࣗݾ঺հ
   • AirflowͷಋೖܦҢ
   • AirflowΛӡ༻ͯ͠Έͯ
   • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
   • ·ͱΊ
   2
   

   View Slide

3. ໨࣍
   • ࣗݾ঺հ
   • AirflowͷಋೖܦҢ
   • AirflowΛӡ༻ͯ͠Έͯ
   • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
   • ·ͱΊ
   3
   

   View Slide

4. ࣗݾ঺հ
   • ໊લ: ാத༔࡞
   • ID: hatappi
   • ձࣾ: גࣜձࣾSpeee
   • ΍͍ͬͯΔ͜ͱ
   • UZOU
   • Angular + RailsͰ࡞੒͢Δސ٬؅ཧγεςϜ
   4
   

   View Slide

5. ໨࣍
   • ࣗݾ঺հ
   • AirflowͷಋೖܦҢ
   • AirflowΛӡ༻ͯ͠Έͯ
   • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
   • ·ͱΊ
   5
   

   View Slide

6. AirflowΛ͍ΕΔલ
   6
   ఆظతʹόονΛ࣮ߦ͍ͨ͠
   cronΛ࢖͑͹Ͱ͖ΔͶ
   

   View Slide

7. cronӡ༻Ͱى͖ͨ͜ͱ
   • ͳ͔ͥࢦఆͨ࣌ؒ͠ʹىಈ͠ͳ͍
   • ࣮ߦݖݶʁ࣌ؒʁcrondىಈͯ͠Δ?
   • ࣮ߦϑΝΠϧͰϩάग़ͯ͠ͳ͍͔ΒԿ͕ى͖͔ͨݪ
   Ҽ͕෼͔Βͳ͍
   7
   

   View Slide

8. ΍Γ͍ͨ͜ͱΛ੔ཧ
   8
   ఆظ࣮ߦ λεΫͷؔ࿈ੑ
   ࣮ߦঢ়ଶͷ֬ೝ EngҎ֎Ͱ΋ϦτϥΠ
   

   View Slide

9. 9
   Airflowͷ࠾༻
   

   View Slide

10. Airflowͱ͸?
    • ϫʔΫϑϩʔϚωδϝϯτγεςϜ
    • OpenAir 2015ͰAirbnb͕ࣾࣗࣾͰ։ൃ͍ͯͨ͠΋
    ͷΛΦʔϓϯιʔεԽͱൃද
    • ݱࡏ͸Apache IncubatorͷΦʔϓϯιʔεϓϩ
    δΣΫτ
    10
    

    View Slide

11. 11
    DAG
    TASK
    

    View Slide

12. Airflowͷ࠾༻
    • ৄ͘͠͸ϒϩά΁
    • http://tech.speee.jp/entry/2016/07/07/050000
    12
    

    View Slide

13. 13
    ॻ͍ͯͳ͍
    

    View Slide

14. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    14
    

    View Slide

15. ྑ͔ͬͨ͜ͱ
    • λεΫͷґଘؔ܎͕ՄࢹԽ͞Εɺ࣮ߦ͞Ε͍ͯΔ࣌
    ͸Ͳ͜·ͰऴΘ͍ͬͯΔͷ͔෼͔Δ
    15
    

    View Slide

16. ྑ͔ͬͨ͜ͱ
    • ϩά͸ඪ४ग़ྗ, ඪ४Τϥʔग़ྗΛग़ྗͯ͘͠ΕΔ
    ͷͰԿ͔ஔ͖ͨͱ͖ʹσόοά͠΍͘͢ͳͬͨ
    16
    

    View Slide

17. ྑ͔ͬͨ͜ͱ
    • ϓϩάϥϜΛमਖ਼Λඞཁͱ͠ͳ͍ݶΓ࠶࣮ߦ͸Eng
    Ҏ֎Ͱ΋ग़དྷΔΑ͏ʹ
    17
    

    View Slide

18. ͓͖ͨ໰୊
    • ϩάΠϯػೳ
    • UTC
    • DAGͷฤू
    • DAGϑΝΠϧͷංେԽ
    18
    

    View Slide

19. ϩάΠϯػೳ
    • σϑΥϧτͰ͸FalseͳͷͰઃఆ͕ඞཁ
    • http://qiita.com/hatappi/items/
    1b8a3a2d1f4b01c844ef
    19
    

    View Slide

20. UTC
    • ͜Ε͸͍࣮ͭߦ͞ΕΔͰ͠ΐ͏͔
    20
    

    View Slide

21. 21
    Airflow͕ಈ͍͍ͯΔϚγ
    ϯͷλΠϜκʔϯʹΑΔ
    

    View Slide

22. Ϛγϯ͕JSTͷ࣌
    • JSTͰ10:00ʹ࣮ߦ͞ΕΔͷͰը໘্ͷUTCදهͰ
    ͸1:00ʹ࣮ߦ͞ΕΔ
    • ը໘ͰݟΔ࣌͸ຖճ+9࣌ؒ͠ͳ͍ͱ͍͚ͳ͍͆
    22
    

    View Slide

23. ໰౴ແ༻ͷUTC
    • UTCʹͳΔΑ͏ʹϋʔυίʔσΟϯά͞Ε͍ͯΔ
    • ৄ͘͠͸
    • http://qiita.com/hatappi/items/2a4d41ca14746a3c98a6
    23
    

    View Slide

24. DAGͷฤू
    • ৽ن௥Ճɺฤू͸ίϚϯυϥΠϯ্͔Βߦ͏
    • ੲ͸೔ຊޠίϝϯτ͕ग़དྷͳ͔ͬͨΑ͏͕ͩv1.8.0
    Ͱ֬ೝͨ͠ͱ͜Ζग़དྷͨ
    24
    

    View Slide

25. DAGϑΝΠϧͷංେԽ
    • PythonͰఆٛ͢ΔͷͰAPI௨৴Ͱ΋ԿͰ΋ग़དྷΔ
    • ϩδοΫ͸λεΫ΍εΫϦϓτʹఆٛͯ͠DAGʹ
    ͸λεΫͷґଘؔ܎΍εέδϡʔϧʹઐ೦ͤ͞Δ΂
    ͖ͩͬͨ
    25
    

    View Slide

26. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    26
    

    View Slide

27. Kuroko2ͱ͸
    • cookpad੡ͷδϣϒ؅ཧπʔϧ
    • RailsʹͷͤΔܗͰ࢖༻͢Δ
    • ϩάΠϯ͸GoogleOAuthೝূ͕͍͍ͭͯΔ
    • ࣌ࠁ͸Railsͷઃఆ͕൓ө͞ΕΔ
    • ϒϥ΢β্͔Β৽ن௥Ճɺฤू͕ग़དྷΔ
    27
    

    View Slide

28. ॻ͖׳ΕͨݴޠͰ͋Δ͜ͱ
    • ࣾ಺ͷϝΠϯݴޠ͕RubyͳͷͰ৽͍͠ํ͕ೖͬͯ
    ΋ཧղ͠΍͍͢
    • ίʔυ͸RubyͰ͔͔Ε͍ͯΔͷͰίʔυ΋ಡΊΔ
    ͠ͳ͓ͤΔ
    • ΄͍͠ػೳ͕͋Ε͹ࣗ෼Ͱ։ൃ΋ग़དྷΔ
    • https://github.com/hatappi/kuroko2_read_only
    28
    

    View Slide

29. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    29
    

    View Slide

30. ·ͱΊ
    • AirflowͰ΋Kuroko2Ͱ΋ࣗ෼ͷ໨తʹ͋ͬͨ΋ͷ
    Λ࢖༻͢Δͱྑͦ͞͏Ͱ͢
    • OSSͳͷͰԿ͔͋Ε͹commit͘Β͍ͷؾ࣋ͪ
    • DAG͸ઈରʹංେԽͤ͞ͳ͍ʂ
    30
    

    View Slide

31. 31
    ͓ΘΓ
    

    View Slide