$30 off During Our Annual Pro Sale. View Details »

AirflowからKuroko2に乗り換えたわけ

 AirflowからKuroko2に乗り換えたわけ

Tokyo Airflow Meetup #1

hatappi

May 16, 2017
Tweet

More Decks by hatappi

Other Decks in Technology

Transcript

  1. Airflow͔ΒKuroko2
    ʹ৐Γ׵͑ͨΘ͚
    by hatappi @Tokyo Airflow Meetup #1
    1

    View Slide

  2. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    2

    View Slide

  3. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    3

    View Slide

  4. ࣗݾ঺հ
    • ໊લ: ാத༔࡞
    • ID: hatappi
    • ձࣾ: גࣜձࣾSpeee
    • ΍͍ͬͯΔ͜ͱ
    • UZOU
    • Angular + RailsͰ࡞੒͢Δސ٬؅ཧγεςϜ
    4

    View Slide

  5. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    5

    View Slide

  6. AirflowΛ͍ΕΔલ
    6
    ఆظతʹόονΛ࣮ߦ͍ͨ͠
    cronΛ࢖͑͹Ͱ͖ΔͶ

    View Slide

  7. cronӡ༻Ͱى͖ͨ͜ͱ
    • ͳ͔ͥࢦఆͨ࣌ؒ͠ʹىಈ͠ͳ͍
    • ࣮ߦݖݶʁ࣌ؒʁcrondىಈͯ͠Δ?
    • ࣮ߦϑΝΠϧͰϩάग़ͯ͠ͳ͍͔ΒԿ͕ى͖͔ͨݪ
    Ҽ͕෼͔Βͳ͍
    7

    View Slide

  8. ΍Γ͍ͨ͜ͱΛ੔ཧ
    8
    ఆظ࣮ߦ λεΫͷؔ࿈ੑ
    ࣮ߦঢ়ଶͷ֬ೝ EngҎ֎Ͱ΋ϦτϥΠ

    View Slide

  9. 9
    Airflowͷ࠾༻

    View Slide

  10. Airflowͱ͸?
    • ϫʔΫϑϩʔϚωδϝϯτγεςϜ
    • OpenAir 2015ͰAirbnb͕ࣾࣗࣾͰ։ൃ͍ͯͨ͠΋
    ͷΛΦʔϓϯιʔεԽͱൃද
    • ݱࡏ͸Apache IncubatorͷΦʔϓϯιʔεϓϩ
    δΣΫτ
    10

    View Slide

  11. 11
    DAG
    TASK

    View Slide

  12. Airflowͷ࠾༻
    • ৄ͘͠͸ϒϩά΁
    • http://tech.speee.jp/entry/2016/07/07/050000
    12

    View Slide

  13. 13
    ॻ͍ͯͳ͍

    View Slide

  14. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    14

    View Slide

  15. ྑ͔ͬͨ͜ͱ
    • λεΫͷґଘؔ܎͕ՄࢹԽ͞Εɺ࣮ߦ͞Ε͍ͯΔ࣌
    ͸Ͳ͜·ͰऴΘ͍ͬͯΔͷ͔෼͔Δ
    15

    View Slide

  16. ྑ͔ͬͨ͜ͱ
    • ϩά͸ඪ४ग़ྗ, ඪ४Τϥʔग़ྗΛग़ྗͯ͘͠ΕΔ
    ͷͰԿ͔ஔ͖ͨͱ͖ʹσόοά͠΍͘͢ͳͬͨ
    16

    View Slide

  17. ྑ͔ͬͨ͜ͱ
    • ϓϩάϥϜΛमਖ਼Λඞཁͱ͠ͳ͍ݶΓ࠶࣮ߦ͸Eng
    Ҏ֎Ͱ΋ग़དྷΔΑ͏ʹ
    17

    View Slide

  18. ͓͖ͨ໰୊
    • ϩάΠϯػೳ
    • UTC
    • DAGͷฤू
    • DAGϑΝΠϧͷංେԽ
    18

    View Slide

  19. ϩάΠϯػೳ
    • σϑΥϧτͰ͸FalseͳͷͰઃఆ͕ඞཁ
    • http://qiita.com/hatappi/items/
    1b8a3a2d1f4b01c844ef
    19

    View Slide

  20. UTC
    • ͜Ε͸͍࣮ͭߦ͞ΕΔͰ͠ΐ͏͔
    20

    View Slide

  21. 21
    Airflow͕ಈ͍͍ͯΔϚγ
    ϯͷλΠϜκʔϯʹΑΔ

    View Slide

  22. Ϛγϯ͕JSTͷ࣌
    • JSTͰ10:00ʹ࣮ߦ͞ΕΔͷͰը໘্ͷUTCදهͰ
    ͸1:00ʹ࣮ߦ͞ΕΔ
    • ը໘ͰݟΔ࣌͸ຖճ+9࣌ؒ͠ͳ͍ͱ͍͚ͳ͍͆
    22

    View Slide

  23. ໰౴ແ༻ͷUTC
    • UTCʹͳΔΑ͏ʹϋʔυίʔσΟϯά͞Ε͍ͯΔ
    • ৄ͘͠͸
    • http://qiita.com/hatappi/items/2a4d41ca14746a3c98a6
    23

    View Slide

  24. DAGͷฤू
    • ৽ن௥Ճɺฤू͸ίϚϯυϥΠϯ্͔Βߦ͏
    • ੲ͸೔ຊޠίϝϯτ͕ग़དྷͳ͔ͬͨΑ͏͕ͩv1.8.0
    Ͱ֬ೝͨ͠ͱ͜Ζग़དྷͨ
    24

    View Slide

  25. DAGϑΝΠϧͷංେԽ
    • PythonͰఆٛ͢ΔͷͰAPI௨৴Ͱ΋ԿͰ΋ग़དྷΔ
    • ϩδοΫ͸λεΫ΍εΫϦϓτʹఆٛͯ͠DAGʹ
    ͸λεΫͷґଘؔ܎΍εέδϡʔϧʹઐ೦ͤ͞Δ΂
    ͖ͩͬͨ
    25

    View Slide

  26. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    26

    View Slide

  27. Kuroko2ͱ͸
    • cookpad੡ͷδϣϒ؅ཧπʔϧ
    • RailsʹͷͤΔܗͰ࢖༻͢Δ
    • ϩάΠϯ͸GoogleOAuthೝূ͕͍͍ͭͯΔ
    • ࣌ࠁ͸Railsͷઃఆ͕൓ө͞ΕΔ
    • ϒϥ΢β্͔Β৽ن௥Ճɺฤू͕ग़དྷΔ
    27

    View Slide

  28. ॻ͖׳ΕͨݴޠͰ͋Δ͜ͱ
    • ࣾ಺ͷϝΠϯݴޠ͕RubyͳͷͰ৽͍͠ํ͕ೖͬͯ
    ΋ཧղ͠΍͍͢
    • ίʔυ͸RubyͰ͔͔Ε͍ͯΔͷͰίʔυ΋ಡΊΔ
    ͠ͳ͓ͤΔ
    • ΄͍͠ػೳ͕͋Ε͹ࣗ෼Ͱ։ൃ΋ग़དྷΔ
    • https://github.com/hatappi/kuroko2_read_only
    28

    View Slide

  29. ໨࣍
    • ࣗݾ঺հ
    • AirflowͷಋೖܦҢ
    • AirflowΛӡ༻ͯ͠Έͯ
    • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔
    • ·ͱΊ
    29

    View Slide

  30. ·ͱΊ
    • AirflowͰ΋Kuroko2Ͱ΋ࣗ෼ͷ໨తʹ͋ͬͨ΋ͷ
    Λ࢖༻͢Δͱྑͦ͞͏Ͱ͢
    • OSSͳͷͰԿ͔͋Ε͹commit͘Β͍ͷؾ࣋ͪ
    • DAG͸ઈରʹංେԽͤ͞ͳ͍ʂ
    30

    View Slide

  31. 31
    ͓ΘΓ

    View Slide