Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Digdagを仕事で使ってみて良かったこと、ハマったこと / Using Digdag in production environment
Takehiro Shiozaki
June 07, 2017
Technology
1
1.2k
Digdagを仕事で使ってみて良かったこと、ハマったこと / Using Digdag in production environment
Takehiro Shiozaki
June 07, 2017
Tweet
Share
More Decks by Takehiro Shiozaki
See All by Takehiro Shiozaki
shiozaki
5
4.1k
shiozaki
14
8.8k
shiozaki
0
320
shiozaki
3
990
shiozaki
1
480
shiozaki
5
21k
shiozaki
0
760
shiozaki
0
4.8k
shiozaki
15
23k
Other Decks in Technology
See All in Technology
humank
0
220
kraj
0
4k
torisoup
0
300
yukitodate
2
330
shotakashihara
1
1.3k
nkjzm
1
850
iqbocchi
0
540
y0hgi
1
390
kyonmm
1
2.2k
unifa_dev
0
390
hirosys
0
140
muras
0
100
Featured
See All Featured
keathley
18
640
zakiwarfel
88
3.3k
schacon
145
6.6k
wjessup
338
16k
notwaldorf
13
1.5k
dougneiner
55
5.4k
thoeni
4
550
jrom
114
7.1k
qrush
285
18k
geeforr
332
29k
malarkey
119
16k
rmw
11
740
Transcript
%JHEBHΛࣄͰͬͯΈͯ ྑ͔ͬͨ͜ͱɺϋϚͬͨ͜ͱ 8PSLqPX&OHJOFT/JHIU VASILY,Inc. Ԙ㟒݈߂
Ԙ㟒݈߂ w ʹ7"4*-:ʹ৽ଔೖࣾ w ීஈͷ͓ࣄ3BJMTͰXFC"1*࡞ͬͨΓɺ4PMSͷϝϯςͨ͠Γ 5BCMFBV#JH2VFSZͰμογϡϘʔυ࡞ͬͨΓ w %JHEBHྺ·ͩिؒ͘Β͍ 7"4*-: *OD
TPGUXBSFFOHJOFFS
*20/ Ҏ্ͷϑΝογϣϯ&$αΠτ͔Β ྦྷܭ ສΛ͑ΔΛܝࡌ ݄ؒສਓҎ্͕ར༻͢Δຊ࠷େڃͷϑΝογϣϯαΠτ
w ͱ͋ΔҊ݅ͰΫϩʔϥʔΛ࡞Γ·ͨ͠ w αΠτͷใΛΫϩʔϧ͠ɺ ͦΕΒͷใΛϚʔδͨ͠ޙʹ4ʹอଘ ͲΜͳͷΛ࡞ͬͨͷ͔
%"(Ͱॻ͘ͱ͜Μͳײ͡ $SBXM $SBXM $SBXM/ .FSHF 6QMPBE ɾɾɾ
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload ͠ɺҎʹऴΘΒͳ͔ͬͨΒʁ
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload ͜ͷॲཧ͕ࣦഊͨ͠Βɺ ͜ΕҎ߱ͷॲཧετοϓ͢Δ͖ʁ
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload ॲཧ͕ࣦഊͨ࣌͠ʹϦτϥΠ͍ͨ͠
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload ্ྲྀͷॲཧ͕ϦτϥΠͨ࣌͠ʹɺ ॲཧͷ։࢝࣌ࠁΛΒ͍ͤͨ
DSPOͰཧ͍ͯͨ͠ࠒ 0 0 * * * crawl-1 0 0 *
* * crawl-2 0 0 * * * crawl-N # crawl処理は30分以内に終わるはず 0 30 * * * merge # merge処理は10分以内に終わるはず 0 40 * * * upload Τϥʔ͕ى͖ͨΒ௨͕ཉ͍͠
ͭΒ͍
w લड़ͨ͠ʮͭΒ͞ʯΛղফ͢ΔͨΊʹಋೖ w %JHEBHಋೖͷཧ༝ w ৴པͱ࣮ͷ5SFBTVSF%BUB044 w "JSqPX-VJHJͱൺΔͱઃఆ͕γϯϓϧ w Ϟάϥ͕νϟʔϛϯά
%JHEBHಋೖ
͔͜͜Βຊ %JHEBHΛಋೖͯ͠Α͔ͬͨ͜ͱ ˍ %JHEBHͷಋೖͰϋϚͬͨ͜ͱ
w "JSqPX-VJHJͱൺΔͱࣗ༝͕͍ ˢͦͦߴ͍ࣗ༝ෆཁ Α͔ͬͨ͜ͱઃఆϑΝΠϧ͕γϯϓϧ timezone: UTC +setup: echo>: start ${session_time}
+disp_current_date: echo>: ${moment(session_time).utc().format('YYYY-MM-DD HH:mm:ss Z')} +repeat: for_each>: order: [first, second, third] animal: [dog, cat] _do: echo>: ${order} ${animal} _parallel: true +teardown: echo>: finish ${session_time}
w %JHEBHαʔόʔʹ%PDLFS͚ͩΛΠϯετʔϧ w ΞϓϦέʔγϣϯͷ࣮ߦڥίϯςφʹด͡ࠐΊ w %JHEBHαʔόʔͷߏ͕γϯϓϧʹͳΔ Α͔ͬͨ͜ͱ%PDLFSαϙʔτ
w /tmp/digdag-tempdir* ͕λεΫຖʹੜ͞ΕΔ w ଞͷλεΫ͔ΒͷӨڹΛ࠷খԽͰ͖Δ w ˞%JHEBHTFSWFSݶఆ Α͔ͬͨ͜ͱҰ࣌σΟϨΫτϦͰλεΫΛ࣮ߦ
w QZΦϖϨʔλʔQZUIPOίϚϯυΛݺͼग़͢ w 6CVOUVͰQZUIPOίϚϯυͰ1ZUIPO͕ɺ QZUIPOίϚϯυͰ1ZUIPO͕࣮ߦ͞ΕΔ w ͍͔ͨͬͨͷ1ZUIPOͷํ w ղܾࡦ w
1ZUIPOͷ%PDLFSΠϝʔδΛ༻ ϋϚͬͨ͜ͱ1ZUIPOͷόʔδϣϯ
w ղܾࡦ w 4ʹҰ࣌ϑΝΠϧΛஔ͍ͯड͚͠Λ͢Δ ϋϚͬͨ͜ͱҰ࣌ϑΝΠϧͷड͚͠ _export: docker: image: python:3.6.1 +crawl:
_parallel: true +crawl1: sh>: crawl.sh > temp_result_1.json +crawl2: sh>: crawl.sh > temp_result_2.json +merge: sh>: merge.sh temp_result1.json temp_result2.json &3303 ϑΝΠϧ͕ݟ͔ͭΒͳ͍
w υΩϡϝϯτʹॻ͔Ε͍ͯͳ͍ػೳ͕͋ͬͨΓ w TFSWFSϞʔυͱTDIFEVMFSϞʔυͷ͍͚ͷج४ w ϓϥάΠϯϚωʔδϟ embulk gemతͳ ͜Ε͔Βʹظ
w %JHEBHศར w DSPOӡ༻ͷʮͭΒ͞ʯΛղফͯ͘͠ΕΔϞάϥ w ·ͩࢼͨ͜͠ͱ͕ͳ͍ਓɺҎԼͷίϚϯυΛ࣮ߦͯ͠Πϯετʔϧ • $ curl -o
~/bin/digdag --create-dirs \ -L "https://dl.digdag.io/digdag-latest" $ chmod +x ~/bin/digdag $ echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc ·ͱΊ