Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AirflowからKuroko2に乗り換えたわけ

C582b722e015633f7900083f8ea75732?s=47 hatappi
May 16, 2017
Technology
0
560

 AirflowからKuroko2に乗り換えたわけ

Tokyo Airflow Meetup #1

C582b722e015633f7900083f8ea75732?s=128

hatappi

May 16, 2017
Tweet

More Decks by hatappi

See All by hatappi
RubyではじめるGraphQL
C582b722e015633f7900083f8ea75732?s=48 hatappi
0
460
RubyでChainerつくってます!!
C582b722e015633f7900083f8ea75732?s=48 hatappi
2
1.2k
TDDな個人開発
C582b722e015633f7900083f8ea75732?s=48 hatappi
0
140
できるだけ楽して楽しくRails開発しよう
C582b722e015633f7900083f8ea75732?s=48 hatappi
2
250
EKSにRailsをのせた
C582b722e015633f7900083f8ea75732?s=48 hatappi
1
780
RubyとApache Arrow
C582b722e015633f7900083f8ea75732?s=48 hatappi
0
1.2k
Red Chainerを
なぜ作って今後どうするのか
C582b722e015633f7900083f8ea75732?s=48 hatappi
2
1.7k
Fargateで夢は見られるのか
C582b722e015633f7900083f8ea75732?s=48 hatappi
1
1.5k
個人サービスをFargateに移行したよ
C582b722e015633f7900083f8ea75732?s=48 hatappi
1
440

Other Decks in Technology

See All in Technology
Learning to Solve Hard Minimal Problems
2c88806cbaab6024dc95b9a654c99d86?s=48 takmin
1
340
私のAWS愛を聞け！ ~ここが好きだよStep Functions~ #devio2022
808ce3d41280c085f8bdc27dce7ea8fb?s=48 kongmingstrap
0
290
Istioを活用したセキュアなマイクロサービスの実現/Secure Microservices with Istio
00851927294532e0742c2c174730dff0?s=48 ido_kara_deru
3
420
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
3115a782126be714b5f94d24073c957d?s=48 oracle4engineer
PRO
10
19k
ログ集約基盤をCloudWatchからOpenSearchに変えてみた
098ad475722e3697ec2fba28c8654f9f?s=48 yuhta28
0
130
大声で伝えたい！定時に帰る方法
61c616e83bef28a1cd2666439ebf334b?s=48 sbtechnight
0
230
Settlement simulation testing to ensure correct settlement processing
23e9a5be8e46efd1cf6a07694a047237?s=48 applepine1125
2
1.2k
VS Code Meetup #21 - もう一度知りたい基礎編 - ファイル操作、コーディングの基本編
Fc815be82d09a2e922d4000b65b9f83b?s=48 74th
0
190
MySQL v5.7 勉強会/study-mysql-ver-5-7
8847086af047cbf895ab3277b59529fe?s=48 andpad
0
2k
塩漬けにしているMySQL 8.0.xxをバージョンアップしたくなる、ここ数年でのMySQL 8.0の改善点 / MySQL Update 202208
7fb060398b26ed622d34921bd64e4f5d?s=48 yoshiakiyamasaki
1
680
DMMプラットフォーム ゼロから始めるKubernetes運用 課題と改善
F91225895fc7411d415604147af75cab?s=48 pospome
0
400
ここが好きだよAWS管理ポリシー_devio2022/i_am_iam_lover
325ce6fcd0a74ff78990b8632817da55?s=48 yukihirochiba
0
3.2k

Featured

See All Featured
Robots, Beer and Maslow
9375a9529679f1b42b567a640d775e7d?s=48 schacon
152
7.1k
Unsuck your backbone
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
659
55k
Docker and Python
Ecdea9b9714877b86cee08458f085481?s=48 trallard
27
1.6k
Happy Clients
C51b39fa3c79ccb99dcb8a076bc98287?s=48 brianwarren
89
5.6k
GitHub's CSS Performance
64827b31aef81498662e8af4bd6d5157?s=48 jonrohan
1020
420k
Facilitating Awesome Meetings
245cee81a9c424266e5e401d844ea881?s=48 lara
29
4.1k
Product Roadmaps are Hard
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
35
6.8k
Debugging Ruby Performance
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
65
10k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
95d9f37115fbb0d13135d0f77132f856?s=48 morganepeng
107
16k
GraphQLとの向き合い方2022年版
893f54413c2bd9ba41d11d753aacaf2c?s=48 quramy
16
8.4k
Raft: Consensus for Rubyists
B6a8f005f39d23ffc930508ac9da68b9?s=48 vanstee
127
5.5k
A Tale of Four Properties
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
149
21k

Transcript

  1. Airflow͔ΒKuroko2 ʹ৐Γ׵͑ͨΘ͚ by hatappi @Tokyo Airflow Meetup #1 1

  2. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 2

  3. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 3

  4. ࣗݾ঺հ • ໊લ: ാத༔࡞ • ID: hatappi • ձࣾ: גࣜձࣾSpeee

    • ΍͍ͬͯΔ͜ͱ • UZOU • Angular + RailsͰ࡞੒͢Δސ٬؅ཧγεςϜ 4

  5. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 5

  6. AirflowΛ͍ΕΔલ 6 ఆظతʹόονΛ࣮ߦ͍ͨ͠ cronΛ࢖͑͹Ͱ͖ΔͶ

  7. cronӡ༻Ͱى͖ͨ͜ͱ • ͳ͔ͥࢦఆͨ࣌ؒ͠ʹىಈ͠ͳ͍ • ࣮ߦݖݶʁ࣌ؒʁcrondىಈͯ͠Δ? • ࣮ߦϑΝΠϧͰϩάग़ͯ͠ͳ͍͔ΒԿ͕ى͖͔ͨݪ Ҽ͕෼͔Βͳ͍ 7

  8. ΍Γ͍ͨ͜ͱΛ੔ཧ 8 ఆظ࣮ߦ λεΫͷؔ࿈ੑ ࣮ߦঢ়ଶͷ֬ೝ EngҎ֎Ͱ΋ϦτϥΠ

  9. 9 Airflowͷ࠾༻

  10. Airflowͱ͸? • ϫʔΫϑϩʔϚωδϝϯτγεςϜ • OpenAir 2015ͰAirbnb͕ࣾࣗࣾͰ։ൃ͍ͯͨ͠΋ ͷΛΦʔϓϯιʔεԽͱൃද • ݱࡏ͸Apache IncubatorͷΦʔϓϯιʔεϓϩ

    δΣΫτ 10

  11. 11 DAG TASK

  12. Airflowͷ࠾༻ • ৄ͘͠͸ϒϩά΁ • http://tech.speee.jp/entry/2016/07/07/050000 12

  13. 13 ॻ͍ͯͳ͍

  14. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 14

  15. ྑ͔ͬͨ͜ͱ • λεΫͷґଘؔ܎͕ՄࢹԽ͞Εɺ࣮ߦ͞Ε͍ͯΔ࣌ ͸Ͳ͜·ͰऴΘ͍ͬͯΔͷ͔෼͔Δ 15

  16. ྑ͔ͬͨ͜ͱ • ϩά͸ඪ४ग़ྗ, ඪ४Τϥʔग़ྗΛग़ྗͯ͘͠ΕΔ ͷͰԿ͔ஔ͖ͨͱ͖ʹσόοά͠΍͘͢ͳͬͨ 16

  17. ྑ͔ͬͨ͜ͱ • ϓϩάϥϜΛमਖ਼Λඞཁͱ͠ͳ͍ݶΓ࠶࣮ߦ͸Eng Ҏ֎Ͱ΋ग़དྷΔΑ͏ʹ 17

  18. ͓͖ͨ໰୊ • ϩάΠϯػೳ • UTC • DAGͷฤू • DAGϑΝΠϧͷංେԽ 18

  19. ϩάΠϯػೳ • σϑΥϧτͰ͸FalseͳͷͰઃఆ͕ඞཁ • http://qiita.com/hatappi/items/ 1b8a3a2d1f4b01c844ef 19

  20. UTC • ͜Ε͸͍࣮ͭߦ͞ΕΔͰ͠ΐ͏͔ 20

  21. 21 Airflow͕ಈ͍͍ͯΔϚγ ϯͷλΠϜκʔϯʹΑΔ

  22. Ϛγϯ͕JSTͷ࣌ • JSTͰ10:00ʹ࣮ߦ͞ΕΔͷͰը໘্ͷUTCදهͰ ͸1:00ʹ࣮ߦ͞ΕΔ • ը໘ͰݟΔ࣌͸ຖճ+9࣌ؒ͠ͳ͍ͱ͍͚ͳ͍͆ 22

  23. ໰౴ແ༻ͷUTC • UTCʹͳΔΑ͏ʹϋʔυίʔσΟϯά͞Ε͍ͯΔ • ৄ͘͠͸ • http://qiita.com/hatappi/items/2a4d41ca14746a3c98a6 23

  24. DAGͷฤू • ৽ن௥Ճɺฤू͸ίϚϯυϥΠϯ্͔Βߦ͏ • ੲ͸೔ຊޠίϝϯτ͕ग़དྷͳ͔ͬͨΑ͏͕ͩv1.8.0 Ͱ֬ೝͨ͠ͱ͜Ζग़དྷͨ 24

  25. DAGϑΝΠϧͷංେԽ • PythonͰఆٛ͢ΔͷͰAPI௨৴Ͱ΋ԿͰ΋ग़དྷΔ • ϩδοΫ͸λεΫ΍εΫϦϓτʹఆٛͯ͠DAGʹ ͸λεΫͷґଘؔ܎΍εέδϡʔϧʹઐ೦ͤ͞Δ΂ ͖ͩͬͨ 25

  26. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 26

  27. Kuroko2ͱ͸ • cookpad੡ͷδϣϒ؅ཧπʔϧ • RailsʹͷͤΔܗͰ࢖༻͢Δ • ϩάΠϯ͸GoogleOAuthೝূ͕͍͍ͭͯΔ • ࣌ࠁ͸Railsͷઃఆ͕൓ө͞ΕΔ •

    ϒϥ΢β্͔Β৽ن௥Ճɺฤू͕ग़དྷΔ 27

  28. ॻ͖׳ΕͨݴޠͰ͋Δ͜ͱ • ࣾ಺ͷϝΠϯݴޠ͕RubyͳͷͰ৽͍͠ํ͕ೖͬͯ ΋ཧղ͠΍͍͢ • ίʔυ͸RubyͰ͔͔Ε͍ͯΔͷͰίʔυ΋ಡΊΔ ͠ͳ͓ͤΔ • ΄͍͠ػೳ͕͋Ε͹ࣗ෼Ͱ։ൃ΋ग़དྷΔ •

    https://github.com/hatappi/kuroko2_read_only 28

  29. ໨࣍ • ࣗݾ঺հ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳͥ࢖͍͸͡Ίͨͷ͔ •

    ·ͱΊ 29

  30. ·ͱΊ • AirflowͰ΋Kuroko2Ͱ΋ࣗ෼ͷ໨తʹ͋ͬͨ΋ͷ Λ࢖༻͢Δͱྑͦ͞͏Ͱ͢ • OSSͳͷͰԿ͔͋Ε͹commit͘Β͍ͷؾ࣋ͪ • DAG͸ઈରʹංେԽͤ͞ͳ͍ʂ 30

  31. 31 ͓ΘΓ