Digdagへ日次バッチを移行して幸せになるお話

DMM.comラボではビッグデータ基盤の日次バッチ（ETL処理）のワークフローエンジンとしてDigdagを採用しました。
ワークフローの運用は、障害時のリカバリ、進捗状況に応じた他システム連携、JOB実行時間の局所化など様々な課題があります。
Digdag選定理由や設計の勘所（サーバ構成・ワークフローの設計方針）、Digdag Plugin実装、他チーム連携のためのClient実装など、課題解決のために取り組んだ内容について紹介させていただきました！

Transcript

1. %..DPN
   -BCP
   Ϗοάσʔλ෦
   %3&νʔϜ
   ླ໦
   ᠳଠ 
   !
   8PSLqPX
   &OHJOFT
   /JHIU
   
   
   
   8PSLqPX&OHJOFT/JHIU %JHEBH΁೔࣍όονΛҠߦͯ͠
   ޾ͤʹͳΔ͓࿩
2. 2 Copyright © since 1998 DMM All Rights Reserved. ࣗݾ঺հ ೥ʹ
   %..DPN
   -BCP
   ʹ৽ଔೖࣾ
   /PEFKT
   Ͱ಺੡#*πʔϧ։ൃ
   1SFTUPಋೖ
   )BEPPQΫϥελӡ༻
   ೔࣍όονҠߦ
   
   ΠϚίί ླ໦
   ᠳଠ
   !J@T[ZO
   %..DPN
   -BCP
   γεςϜຊ෦
   Ϗοάσʔλ෦
   %3&νʔϜ
3. 3 Copyright © since 1998 DMM All Rights Reserved. %..DPNʹ͍ͭͯ %..DPNͱάϧʔϓʹ͍ͭͯ
4. 4 Copyright © since 1998 DMM All Rights Reserved. αʔϏεʹ͍ͭͯ %..DPN͸೥͔Βଓ͘࿝ฮ αʔϏεαΠτͰ͢ɻ࣌୅ͷχʔζ ʹ߹Θͤͨଟ࠼ͳίϯςϯπΛɺ ೥ؒഓͬͨಠࣗϓϥοτϑΥʔϜͰ ҆ఆతʹఏڙ͍ͯ͠·͢ɻ
   "CPVU
   %..DPN Ҏ্ͷ෯޿͍αʔϏεΛల։
5. 5 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ʹ͍ͭͯ ϛογϣϯɾ͓࢓ࣄɾ͜Ε·Ͱͷ׆ಈ
6. 6 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ͷϛογϣϯ ϏοάσʔλΛ༻͍ͯࣄۀͷ,1*ʢॏཁܦӦࢦඪʣ޲্΁ߩݙ͢Δ͜ͱ σʔλυϦϒϯϚʔέςΟϯά ϚʔέςΟϯάΦʔτϝʔγϣϯ ࢪࡦͷޮՌͳͲࠓޙͷܭըΛ
   ࿅ΔͨΊͷ৘ใΛఏڙ Ϩίϝϯυ΍ݕࡧΤϯδϯͳͲΛ௨ͯ͠
   Ϣʔβʔʹద੾ͳ৘ใΛఏڙ
7. 7 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ͷ͓࢓ࣄ
   )BEPPQ
   ج൫ӡ༻
   
   ࢲͷ୲౰
   
   Ϩίϝϯυ
   
   ݕࡧ
   
   σʔλ෼ੳ
   
   ϝϧϚΨ
   
   ΞϑΟϦΤΠτ
8. 8 Copyright © since 1998 DMM All Rights Reserved. ΞδΣϯμ ͳͥ%JHEBHͳͷ͔ʁ
   γεςϜߏ੒
   8PSLqPXઃܭ
   ϓϥάΠϯͷ࿩
   νʔϜؒ࿈ܞ
   ϋϚΓͲ͜Ζ
   ·ͱΊ
9. 9 Copyright © since 1998 DMM All Rights Reserved. ͳͥ%JHEBHͳͷ͔ʁ ೔࣍όονͷ՝୊ͱ%JHEBHͷબఆʹࢸΔ·Ͱ
10. 10 Copyright © since 1998 DMM All Rights Reserved. ଞࣄۀ෦ͷσʔλऔΓࠐΈॲཧ
    
    σʔλՃ޻ॲཧ
    
    #*Ϩϙʔτ޲͚σʔλूܭॲཧ
    
    ूܭσʔλͷసૹॲཧ νʔϜ಺ʹ͓͚Δ೔࣍όον
11. 11 Copyright © since 1998 DMM All Rights Reserved. ଞࣄۀ෦ͷσʔλऔΓࠐΈॲཧ
    &YUSBDU
    σʔλՃ޻ॲཧ
    5SBOTGPSN
    #*Ϩϙʔτ޲͚σʔλूܭॲཧ
    5SBOTGPSN
    ूܭσʔλͷసૹॲཧ
    -PBE νʔϜ಺ʹ͓͚Δ೔࣍όον
12. 12 Copyright © since 1998 DMM All Rights Reserved. ͍ΘΏΔ&5- νʔϜ಺ʹ͓͚Δ೔࣍όον σʔλऩूπʔϧͰ࣮ߦ
    
    1ZUIPO੡ Ճ޻ͨ͠σʔλΛ
    )JWF
    
    .BSJB%#ʹసૹ )JWFΫΤϦͰ
    σʔλՃ޻ &YUSBDU σʔλऔಘॲཧ
    ˞
    TRPPQ
    
    NZTRM 5SBOTGPSN औಘσʔλՃ޻ॲཧ -PBE %8)
    
    %BUB
    .BSU΁
    σʔλసૹ
13. 13 Copyright © since 1998 DMM All Rights Reserved. ͜Ε·Ͱͷόονॲཧ͸ʜ
    #VJME
    'MPX
    1MVHJO
    
    1BSBNFUFSJ[FE
    5SJHHFS
    1MVHJO
    Λۦ࢖͠ɺ
    ґଘؔ܎΍ॲཧΛఆٛ νʔϜ಺ʹ͓͚Δ೔࣍όον ˞
    Wܥ
14. 14 Copyright © since 1998 DMM All Rights Reserved.
15. 15 Copyright © since 1998 DMM All Rights Reserved. ˣ೔෼ͷδϣϒϑϩʔ
16. 16 Copyright © since 1998 DMM All Rights Reserved. +FOLJOTґଘ
    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍
    όονࣦഊ࣌ͷϦτϥΠ͕େม
    ॲཧ͕࣌ؒґଘ
    ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠ ೔࣍όονʹ͓͚Δ՝୊
17. 17 Copyright © since 1998 DMM All Rights Reserved. ͜ΕΒͷ՝୊Λղܾ͢ΔͨΊɺ+FOLJOT͔Βͷ୤٫Λܾҙ
    ໨త͸ʜ
    ೔࣍όονॲཧΛ৽ϫʔΫϑϩʔΤϯδϯ΁ϦϓϨΠε ೔࣍όονʹ͓͚Δ՝୊
18. 18 Copyright © since 1998 DMM All Rights Reserved. ͨ͘͞Μͷબ୒ࢶ͕͋Γ·ͨ͠
19. 19 Copyright © since 1998 DMM All Rights Reserved. ཁ݅͸ओʹͭ
    Մ༻ੑ
    εέʔϥϏϦςΟ
    ෼ࢄ࣮ߦ
    
    ฒྻॲཧ
    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖Δ
    ࣦഊͨ͠λεΫ͔ΒϦτϥΠՄೳ
    ਐḿ֬ೝ͕Ͱ͖ΔΠϯλʔϑΣʔε
    3&45"1* ϫʔΫϑϩʔΤϯδϯͷબఆϙΠϯτ
20. 20 Copyright © since 1998 DMM All Rights Reserved.
21. 21 Copyright © since 1998 DMM All Rights Reserved. Մ༻ੑ
    ‎
    "
    "
    "
    
    εέʔϥϏϦςΟ
    ෼ࢄ࣮ߦ
    
    ฒྻॲཧ
    ‎
    "
    "
    "
    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖Δ
    ‎
    "
    ࣦഊͨ͠λεΫ͔ΒϦτϥΠՄೳ
    ‎
    "
    ਐḿ֬ೝ͕Ͱ͖ΔΠϯλʔϑΣʔε
    3&45"1*
    ‎
    " ϫʔΫϑϩʔΤϯδϯͷબఆϙΠϯτ
22. 22 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒ %JHEBHͱͦΕΛऔΓר͘γεςϜ શͯΦϯϓϨͰ͢
23. 23 Copyright © since 1998 DMM All Rights Reserved. ࠾༻͍ͯ͠Δϛυϧ΢ΣΞ %JHEBH
    
    1PTUHSF42-
    
    1HQPPM**
    
    ˞
    
    ࣌఺
24. 24 Copyright © since 1998 DMM All Rights Reserved. Streaming Replication Watchdog Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync
25. 25 Copyright © since 1998 DMM All Rights Reserved. Streaming Replication Watchdog Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync
26. 26 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒
    
    %JHEBH αʔό݉ΫϥΠΞϯτͷ୆ߏ੒
    )"ͷͨΊ
    TVQFSWJTPS
    ͰσʔϞϯԽ
    λεΫ͸෼ࢄ࣮ͯ͠ߦͯ͘͠ΕΔ
    ࣮ߦϩά΋෼ࢄͯ͠͠·͏
    -TZODE
    
    STZOD
    ͰλεΫϩάΛ૒ํ޲ಉظ
27. 27 Copyright © since 1998 DMM All Rights Reserved. Watchdog Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync Streaming Replication
28. 28 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒
    
    1PTUHSF42- %JHEBHΛෳ਺୆ͷαʔόΫϥΠΞϯτߏ੒ʹ͢ΔͨΊ
    ϫʔΫϑϩʔͷ࣮ମ؅ཧ΍λεΫͷεςʔτ؅ཧ
    1SJNBSZ
    
    4UBOECZͰ୆ߏ੒
    4USFBNJOH
    3FQMJDBUJPO
29. 29 Copyright © since 1998 DMM All Rights Reserved. Streaming Replication Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync Watchdog
30. 30 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒
    
    1HQPPM** ໨త͸ෛՙ෼ࢄͰ͸ͳ͘1PTUHSF42-ͷ৑௕Խ
    1HQPPM**
    ΋৑௕Խ͢ΔͨΊ୆ߏ੒
    "DUJWF
    
    4UBOECZ
    ͨͩ͠1PTUHSF42-ͱಉډ
    8BUDIEPH
    ػೳͰ7JSUVBM
    *1Λ੍ޚ
31. 31 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ ద੾ͳཻ౓Ͱ֤ϑϩʔͷݟ௨͠Λվળ
32. 32 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ ࣄۀ୯ҐͰͭͷϫʔΫϑϩʔΛ࡞੒͢ΔΑ͏ʹ੔ཧ
    ྫ
    ΦϯϥΠϯήʔϜ
    
    ӳձ࿩
    ʜ
    FUD
    ֤ϫʔΫϑϩʔͰ&5-Λߦ͏Α͏ʹ
33. 33 Copyright © since 1998 DMM All Rights Reserved. &5-ॲཧͷΠϝʔδ &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE ήʔϜ
    ࣄۀ෦
    ϫʔΫϑϩʔ ˚˚
    ࣄۀ෦
    ϫʔΫϑϩʔ ˘˘
    ࣄۀ෦
    ϫʔΫϑϩʔ
34. 34 Copyright © since 1998 DMM All Rights Reserved. &5-ॲཧͷΠϝʔδ &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE ήʔϜ
    ࣄۀ෦
    ϫʔΫϑϩʔ ˚˚
    ࣄۀ෦
    ϫʔΫϑϩʔ ˘˘
    ࣄۀ෦
    ϫʔΫϑϩʔ
35. 35 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ ήʔϜࣄۀ෦ͷ৔߹ 4MBDL௨஌ςϯϓϨʔτ HBNFࣄۀ෦ϫʔΫϑϩʔ
    &5-ΛDBMM͢Δ ύϥϝʔλ HBNFࣄۀ෦ͷ࣮ॲཧ
    HBNFEJH͔ΒDBMM͞ΕΔ
36. 36 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ ήʔϜࣄۀ෦ͷ৔߹ ڞ௨ύϥϝʔλಡΈࠐΈ HBNFEJH Τϥʔ࣌ͷ௨஌ HBNFࣄۀ෦ͷॲཧΛDBMM
    ˞
    FYUSBDUΛݺΜͰ͍Δ εέδϡʔϧઃఆ 4-"࣌ͷ௨஌
37. 37 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ ࣄۀ୯ҐͰ੔ཧͨ͜͠ͱͰશମͷݟ௨͕͠վળ
    ϧʔϧΛܾΊͨ͜ͱͰ୭͕ݟͯ΋Θ͔ΔΑ͏ʹ
    :BNMܗࣜͳͷͰಡΈ΍͍͢
    Կ͔վम͕ೖΔ࣌ɺखೖΕ͠΍͍͢
38. 38 Copyright © since 1998 DMM All Rights Reserved. ݸʑͷλεΫʹԿ෼͔͔Δͷ͔ࢹ֮తʹ೺ѲՄೳʹ
39. 39 Copyright © since 1998 DMM All Rights Reserved. %JHEBHͰ͸ओʹ3VCZ
    
    4IFMM
    4DSJQUΛར༻
    Α͘࢖͏ΦϖϨʔλʔ
    
    SC
    3VCZͷλεΫΛ࣮ߦ
    TI
    4IFMM
    4DSJQUΛ࣮ߦ
    σʔλऩूπʔϧͷ࣮ߦ
    TMBDL
    
    4MBDL΁ͷ௨஌
    ˞
    ޙड़ 8PSLqPXઃܭ
40. 40 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯͷ࿩ %JHEBHͷϓϥάΠϯͷ࡞੒ࣄྫͷ঺հ
41. 41 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯΛࣗ࡞ͨ͠ࣄྫ 4MBDL௨஌Λ࣮ݱ͢Δ
    EJHEBHTMBDL
    Λ࡞ͬͨ
    TMBDL
    ΦϖϨʔλʔ͕࢖͑ΔΑ͏ʹ
    
    :BNMܗࣜͷςϯϓϨʔτ͔Β4MBDL௨஌
    ม਺ͷຒΊࠐΈ΋Մೳ
42. 42 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌ͷྫ 
    ݺͼग़͢
    UFNQMBUFTTVDDFTTZNM
    Λ࡞੒ 
    TMBDLEJH
    Λ࡞੒
43. 43 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌ ϓϥάΠϯͷαϯϓϧ ௨஌͕དྷΔ
44. 44 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌ ϓϥάΠϯͷαϯϓϧ Τϥʔ࣌ 4-"ʢࢦఆܦա࣌ؒ಺ʹ׬ྃ͠ͳ͍࣌ʣ
45. 45 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯ ͳͥ࡞͔ͬͨʁ
    Ϧονͳ௨஌͕͔ͨͬͨ͠
    λεΫ੒ޭ࣌
    
    Τϥʔ࣌
    
    4-"ͷ௨஌ʹར༻
    ࢼ͠ʹ࡞ͬͯΈͨΒαΫοͱͰ͖ͨʢ+BWBʣ
    NZVJEJHEBHQMVHJOFYBNQMF
    Λࢀߟʹ࣮૷
    +JUQBDLʹެ։͢Ε͹ར༻΋ָʑ
46. 46 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯ (JUIVCʹͯެ։த
    (JUIVCɿ
    IUUQTHJUIVCDPNT[ZOEJHEBHTMBDL
    ϒϩάɿ
    IUUQJSPJMPHOFUEJHEBHTMBDLɹ
47. 47 Copyright © since 1998 DMM All Rights Reserved. ৄࡉ͸ϒϩάʹͯʜ
48. 48 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ (PݴޠͷΫϥΠΞϯτ
    NPH
    ͷࣄྫ঺հ
49. 49 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ &5-ॲཧ͕ऴΘΔ·Ͱ
    ଴ػ 4VDFTT ΋͠΋ɺ໨తͷλεΫ͕
    ੒ޭʹͳͬͨΒ ޙଓॲཧελʔτ 1PMMJOH 4UBSU ࣌ؒʹґଘ͠ͳ͍ͨΊͷϑϩʔΛߟ͑ͨ ̋̋ϓϩδΣΫτͷ˚˚ϫʔΫϑϩʔͷ˘˘ͱ͍͏
    λεΫ͕ऴΘ͍ͬͯͨΒɺδϣϒΛ࣮ߦ͍ͨ͠ʂ
50. 50 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ (PݴޠͰ$-*πʔϧ
    
    
    NPH
    
    Λ࡞੒͠ఏڙ
    NPH
    Ͱ͸ಛఆͷϓϩδΣΫτͷϫʔΫϑϩʔΛɺ
    λεΫ໊ϕʔεͰεςʔλε֬ೝ͕Մೳʹ
    ϫʔΫϑϩʔΛελʔτ
    
    ϦτϥΠ͢Δػೳ΋
51. 51 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ ࢖༻ྫ NPHͰ͸λεΫͷεςʔλεऔಘ
    ϫʔΫϑϩʔͷελʔτ
    ϦτϥΠػೳ
    Λఏڙ
52. 52 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ ࢖༻ྫ TBNQMF
    ϓϩδΣΫτ
    TMBDL
    ϫʔΫϑϩʔ
    TMBDL TUFQ
    λεΫ
    εςʔλεऔಘͷྫ
53. 53 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ ࢖༻ྫ λεΫͷελʔτ
    
    ϦτϥΠ΋Մೳ
    ҎԼ͸ϫʔΫϑϩʔͷελʔτྫ
54. 54 Copyright © since 1998 DMM All Rights Reserved. ϋϚΓͲ͜Ζ %JHEBHಋೖͰϋϚͬͨͱ͜Ζ
55. 55 Copyright © since 1998 DMM All Rights Reserved. %JHEBH
    4FSWFSͰࣗ࡞ϓϥάΠϯ͕࢖͑ͳ͍
    λεΫ͕Ұ޲ʹਐ·ͳ͍ɾҟৗʹ͕͔͔࣌ؒΔ ϋϚΓͲ͜Ζ
56. 56 Copyright © since 1998 DMM All Rights Reserved. %JHEBH
    4FSWFSͰࣗ࡞ϓϥάΠϯ͕࢖͑ͳ͍
    ‎ݱঢ়͸%JHEBH
    4FSWFSΛىಈͨ͠ΧϨϯτσΟϨΫτϦʹ
    ϓϥάΠϯΛμ΢ϯϩʔυ͢Δ࢓༷
    σΟϨΫτϦݖݶͷݟ௚͠ͰରԠ ϋϚΓͲ͜Ζ
57. 57 Copyright © since 1998 DMM All Rights Reserved. λεΫ͕Ұ޲ʹਐ·ͳ͍ɾҟৗʹ͕͔͔࣌ؒΔ
    ‎1HQPPM**Ͱ-PBE
    CBMBODF
    NPEFΛ༗ޮʹ͍ͯͨ͠
    ‎%JHEBHͷ%BUBCBTF઀ଓ͕4PDLFU
    5JNFPVU
    1HQPPM**ͷ
    MPBE@CBMBODF@NPEF
    Λ
    GBMTF
    ʹ
    ෳ਺୆ߏ੒Ͱ௕࣌ؒͷόονΛ࣮ߦ͢Δࡍʹ͸ɺ EBUBCBTFTPDLFU5JNFPVU
    EFGBVMU
    
    ʹ஫ҙ
    ϋϚΓͲ͜Ζ
58. 58 Copyright © since 1998 DMM All Rights Reserved. ·ͱΊ %JHEBHಋೖͰ޾ͤʹͳͬͨͱ͜Ζ
59. 59 Copyright © since 1998 DMM All Rights Reserved. ͜ͷΑ͏ͳ՝୊͕͋Γ·ͨ͠
    +FOLJOTґଘ
    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍
    ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠
    όονࣦഊ࣌ͷϦτϥΠ͕େม
    ॲཧ͕࣌ؒґଘ ·ͱΊ
60. 60 Copyright © since 1998 DMM All Rights Reserved. +FOLJOTґଘ
    +FOLJOT͔Βͷ୤٫ʹ੒ޭ
    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍
    શͯίʔυͰ؅ཧՄೳʹ
    ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠
    ίʔυͰґଘؔ܎΋දݱ
    όονࣦഊ࣌ͷϦτϥΠ͕େม
    όονࣦഊ࣌ͷϦτϥΠ΋ָͪΜʹ
    ॲཧ͕࣌ؒґଘ
    NPH
    $-*ͷఏڙʹΑΓ࣌ؒґଘͷॲཧΛഉআ ·ͱΊ
61. 61 Copyright © since 1998 DMM All Rights Reserved. ߋʹخ͔ͬͨ͠ͱ͜Ζʜ
    ฒྻॲཧͰόονॲཧ͕ߴ଎ʹ
    ݸʑͷλεΫʹԿ෼͔͔Δͷ͔ࢹ֮తʹ೺ѲՄೳʹ ·ͱΊ
62. 62 Copyright © since 1998 DMM All Rights Reserved. pOBMMZ
    UIBOLT