Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AirflowからKuroko2に乗り換えたわけ
Search
hatappi
May 16, 2017
Technology
0
690
AirflowからKuroko2に乗り換えたわけ
Tokyo Airflow Meetup #1
hatappi
May 16, 2017
Tweet
Share
More Decks by hatappi
See All by hatappi
RubyではじめるGraphQL
hatappi
0
690
RubyでChainerつくってます!!
hatappi
2
1.3k
TDDな個人開発
hatappi
0
210
できるだけ楽して楽しくRails開発しよう
hatappi
2
280
EKSにRailsをのせた
hatappi
1
970
RubyとApache Arrow
hatappi
0
1.9k
Red Chainerを なぜ作って今後どうするのか
hatappi
2
2k
Fargateで夢は見られるのか
hatappi
1
1.9k
個人サービスをFargateに移行したよ
hatappi
1
560
Other Decks in Technology
See All in Technology
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
150
DevOpsDays History and my DevOps story
kawaguti
PRO
9
2.3k
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
240
Google Cloud の AI を支える裏側のインフラを垣間見る!
maroon1st
0
320
20240416_devopsdaystokyo
kzkmaeda
1
200
[PlatformCon 24] Platform Orchestrators: The Missing Middle of Internal Developer Platforms?
danielbryantuk
1
800
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
2
110
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
890
SPI原点回帰論:事業課題とFour Keysの結節点を見出す実践的ソフトウェアプロセス改善 / DevOpsDays Tokyo 2024
visional_engineering_and_design
4
1.8k
アクセス制御にまつわる改善 / Improving access control
itkq
0
470
日本におけるデータエンジニアリングのこれまでとこれから
foursue
16
4k
継続的な改善 x ⾮連続的な進化
sansantech
PRO
3
120
Featured
See All Featured
Dealing with People You Can't Stand - Big Design 2015
cassininazir
356
22k
Agile that works and the tools we love
rasmusluckow
324
20k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Art, The Web, and Tiny UX
lynnandtonic
288
19k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
Fireside Chat
paigeccino
20
2.6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
30
6k
Writing Fast Ruby
sferik
620
60k
Designing on Purpose - Digital PM Summit 2013
jponch
110
6.4k
Building Your Own Lightsaber
phodgson
98
5.7k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
658
120k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
Transcript
Airflow͔ΒKuroko2 ʹΓ͑ͨΘ͚ by hatappi @Tokyo Airflow Meetup #1 1
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 2
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 3
ࣗݾհ • ໊લ: ാத༔࡞ • ID: hatappi • ձࣾ: גࣜձࣾSpeee
• ͍ͬͯΔ͜ͱ • UZOU • Angular + RailsͰ࡞͢Δސ٬ཧγεςϜ 4
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 5
AirflowΛ͍ΕΔલ 6 ఆظతʹόονΛ࣮ߦ͍ͨ͠ cronΛ͑Ͱ͖ΔͶ
cronӡ༻Ͱى͖ͨ͜ͱ • ͳ͔ͥࢦఆͨ࣌ؒ͠ʹىಈ͠ͳ͍ • ࣮ߦݖݶʁ࣌ؒʁcrondىಈͯ͠Δ? • ࣮ߦϑΝΠϧͰϩάग़ͯ͠ͳ͍͔ΒԿ͕ى͖͔ͨݪ Ҽ͕͔Βͳ͍ 7
Γ͍ͨ͜ͱΛཧ 8 ఆظ࣮ߦ λεΫͷؔ࿈ੑ ࣮ߦঢ়ଶͷ֬ೝ EngҎ֎ͰϦτϥΠ
9 Airflowͷ࠾༻
Airflowͱ? • ϫʔΫϑϩʔϚωδϝϯτγεςϜ • OpenAir 2015ͰAirbnb͕ࣾࣗࣾͰ։ൃ͍ͯͨ͠ ͷΛΦʔϓϯιʔεԽͱൃද • ݱࡏApache IncubatorͷΦʔϓϯιʔεϓϩ
δΣΫτ 10
11 DAG TASK
Airflowͷ࠾༻ • ৄ͘͠ϒϩά • http://tech.speee.jp/entry/2016/07/07/050000 12
13 ॻ͍ͯͳ͍
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 14
ྑ͔ͬͨ͜ͱ • λεΫͷґଘ͕ؔՄࢹԽ͞Εɺ࣮ߦ͞Ε͍ͯΔ࣌ Ͳ͜·ͰऴΘ͍ͬͯΔͷ͔͔Δ 15
ྑ͔ͬͨ͜ͱ • ϩάඪ४ग़ྗ, ඪ४Τϥʔग़ྗΛग़ྗͯ͘͠ΕΔ ͷͰԿ͔ஔ͖ͨͱ͖ʹσόοά͘͢͠ͳͬͨ 16
ྑ͔ͬͨ͜ͱ • ϓϩάϥϜΛमਖ਼Λඞཁͱ͠ͳ͍ݶΓ࠶࣮ߦEng Ҏ֎Ͱग़དྷΔΑ͏ʹ 17
͓͖ͨ • ϩάΠϯػೳ • UTC • DAGͷฤू • DAGϑΝΠϧͷංେԽ 18
ϩάΠϯػೳ • σϑΥϧτͰFalseͳͷͰઃఆ͕ඞཁ • http://qiita.com/hatappi/items/ 1b8a3a2d1f4b01c844ef 19
UTC • ͜Ε͍࣮ͭߦ͞ΕΔͰ͠ΐ͏͔ 20
21 Airflow͕ಈ͍͍ͯΔϚγ ϯͷλΠϜκʔϯʹΑΔ
Ϛγϯ͕JSTͷ࣌ • JSTͰ10:00ʹ࣮ߦ͞ΕΔͷͰը໘্ͷUTCදهͰ 1:00ʹ࣮ߦ͞ΕΔ • ը໘ͰݟΔ࣌ຖճ+9࣌ؒ͠ͳ͍ͱ͍͚ͳ͍͆ 22
ແ༻ͷUTC • UTCʹͳΔΑ͏ʹϋʔυίʔσΟϯά͞Ε͍ͯΔ • ৄ͘͠ • http://qiita.com/hatappi/items/2a4d41ca14746a3c98a6 23
DAGͷฤू • ৽نՃɺฤूίϚϯυϥΠϯ্͔Βߦ͏ • ੲຊޠίϝϯτ͕ग़དྷͳ͔ͬͨΑ͏͕ͩv1.8.0 Ͱ֬ೝͨ͠ͱ͜Ζग़དྷͨ 24
DAGϑΝΠϧͷංେԽ • PythonͰఆٛ͢ΔͷͰAPI௨৴ͰԿͰग़དྷΔ • ϩδοΫλεΫεΫϦϓτʹఆٛͯ͠DAGʹ λεΫͷґଘؔεέδϡʔϧʹઐ೦ͤ͞Δ ͖ͩͬͨ 25
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 26
Kuroko2ͱ • cookpadͷδϣϒཧπʔϧ • RailsʹͷͤΔܗͰ༻͢Δ • ϩάΠϯGoogleOAuthೝূ͕͍͍ͭͯΔ • ࣌ࠁRailsͷઃఆ͕ө͞ΕΔ •
ϒϥβ্͔Β৽نՃɺฤू͕ग़དྷΔ 27
ॻ͖׳ΕͨݴޠͰ͋Δ͜ͱ • ࣾͷϝΠϯݴޠ͕RubyͳͷͰ৽͍͠ํ͕ೖͬͯ ཧղ͍͢͠ • ίʔυRubyͰ͔͔Ε͍ͯΔͷͰίʔυಡΊΔ ͠ͳ͓ͤΔ • ΄͍͠ػೳ͕͋ΕࣗͰ։ൃग़དྷΔ •
https://github.com/hatappi/kuroko2_read_only 28
࣍ • ࣗݾհ • AirflowͷಋೖܦҢ • AirflowΛӡ༻ͯ͠Έͯ • Kuroko2Λͳ͍ͥ͡Ίͨͷ͔ •
·ͱΊ 29
·ͱΊ • AirflowͰKuroko2Ͱࣗͷతʹ͋ͬͨͷ Λ༻͢Δͱྑͦ͞͏Ͱ͢ • OSSͳͷͰԿ͔͋Εcommit͘Β͍ͷؾ࣋ͪ • DAGઈରʹංେԽͤ͞ͳ͍ʂ 30
31 ͓ΘΓ