Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Digdagへ日次バッチを移行して幸せになるお話

 Digdagへ日次バッチを移行して幸せになるお話

DMM.comラボではビッグデータ基盤の日次バッチ(ETL処理)のワークフローエンジンとしてDigdagを採用しました。
ワークフローの運用は、障害時のリカバリ、進捗状況に応じた他システム連携、JOB実行時間の局所化など様々な課題があります。
Digdag選定理由や設計の勘所(サーバ構成・ワークフローの設計方針)、Digdag Plugin実装、他チーム連携のためのClient実装など、課題解決のために取り組んだ内容について紹介させていただきました!

B6d9eb9a3825f15c84b5e7588c56a405?s=128

DMM.com

June 08, 2017
Tweet

Transcript

  1. %..DPN-BCPϏοάσʔλ෦%3&νʔϜླ໦ᠳଠ !8PSLqPX&OHJOFT/JHIU 8PSLqPX&OHJOFT/JHIU %JHEBH΁೔࣍όονΛҠߦͯ͠ ޾ͤʹͳΔ͓࿩

  2. 2 Copyright © since 1998 DMM All Rights Reserved. ࣗݾ঺հ

    ೥ʹ%..DPN-BCPʹ৽ଔೖࣾ /PEFKTͰ಺੡#*πʔϧ։ൃ 1SFTUPಋೖ )BEPPQΫϥελӡ༻ ೔࣍όονҠߦΠϚίί ླ໦ᠳଠ !J@T[ZO  %..DPN-BCPγεςϜຊ෦Ϗοάσʔλ෦%3&νʔϜ
  3. 3 Copyright © since 1998 DMM All Rights Reserved. %..DPNʹ͍ͭͯ

    %..DPNͱάϧʔϓʹ͍ͭͯ
  4. 4 Copyright © since 1998 DMM All Rights Reserved. αʔϏεʹ͍ͭͯ

    %..DPN͸೥͔Βଓ͘࿝ฮ αʔϏεαΠτͰ͢ɻ࣌୅ͷχʔζ ʹ߹Θͤͨଟ࠼ͳίϯςϯπΛɺ ೥ؒഓͬͨಠࣗϓϥοτϑΥʔϜͰ ҆ఆతʹఏڙ͍ͯ͠·͢ɻ "CPVU%..DPN Ҏ্ͷ෯޿͍αʔϏεΛల։
  5. 5 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ʹ͍ͭͯ

    ϛογϣϯɾ͓࢓ࣄɾ͜Ε·Ͱͷ׆ಈ
  6. 6 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ͷϛογϣϯ

    ϏοάσʔλΛ༻͍ͯࣄۀͷ,1*ʢॏཁܦӦࢦඪʣ޲্΁ߩݙ͢Δ͜ͱ σʔλυϦϒϯϚʔέςΟϯά ϚʔέςΟϯάΦʔτϝʔγϣϯ ࢪࡦͷޮՌͳͲࠓޙͷܭըΛ ࿅ΔͨΊͷ৘ใΛఏڙ Ϩίϝϯυ΍ݕࡧΤϯδϯͳͲΛ௨ͯ͠ Ϣʔβʔʹద੾ͳ৘ใΛఏڙ
  7. 7 Copyright © since 1998 DMM All Rights Reserved. Ϗοάσʔλ෦ͷ͓࢓ࣄ

    )BEPPQج൫ӡ༻ࢲͷ୲౰ Ϩίϝϯυ ݕࡧ σʔλ෼ੳ ϝϧϚΨ ΞϑΟϦΤΠτ
  8. 8 Copyright © since 1998 DMM All Rights Reserved. ΞδΣϯμ

    ͳͥ%JHEBHͳͷ͔ʁ γεςϜߏ੒ 8PSLqPXઃܭ ϓϥάΠϯͷ࿩ νʔϜؒ࿈ܞ ϋϚΓͲ͜Ζ ·ͱΊ
  9. 9 Copyright © since 1998 DMM All Rights Reserved. ͳͥ%JHEBHͳͷ͔ʁ

    ೔࣍όονͷ՝୊ͱ%JHEBHͷબఆʹࢸΔ·Ͱ
  10. 10 Copyright © since 1998 DMM All Rights Reserved. ଞࣄۀ෦ͷσʔλऔΓࠐΈॲཧ

    σʔλՃ޻ॲཧ #*Ϩϙʔτ޲͚σʔλूܭॲཧ ूܭσʔλͷసૹॲཧ νʔϜ಺ʹ͓͚Δ೔࣍όον
  11. 11 Copyright © since 1998 DMM All Rights Reserved. ଞࣄۀ෦ͷσʔλऔΓࠐΈॲཧ

    &YUSBDU  σʔλՃ޻ॲཧ 5SBOTGPSN  #*Ϩϙʔτ޲͚σʔλूܭॲཧ 5SBOTGPSN  ूܭσʔλͷసૹॲཧ -PBE νʔϜ಺ʹ͓͚Δ೔࣍όον
  12. 12 Copyright © since 1998 DMM All Rights Reserved. ͍ΘΏΔ&5-

    νʔϜ಺ʹ͓͚Δ೔࣍όον σʔλऩूπʔϧͰ࣮ߦ  1ZUIPO੡ Ճ޻ͨ͠σʔλΛ )JWF.BSJB%#ʹసૹ )JWFΫΤϦͰ σʔλՃ޻ &YUSBDU σʔλऔಘॲཧ ˞TRPPQNZTRM 5SBOTGPSN औಘσʔλՃ޻ॲཧ -PBE %8)%BUB.BSU΁ σʔλసૹ
  13. 13 Copyright © since 1998 DMM All Rights Reserved. ͜Ε·Ͱͷόονॲཧ͸ʜ

    #VJME'MPX1MVHJO1BSBNFUFSJ[FE5SJHHFS1MVHJOΛۦ࢖͠ɺ ґଘؔ܎΍ॲཧΛఆٛ νʔϜ಺ʹ͓͚Δ೔࣍όον ˞Wܥ
  14. 14 Copyright © since 1998 DMM All Rights Reserved.

  15. 15 Copyright © since 1998 DMM All Rights Reserved. ˣ೔෼ͷδϣϒϑϩʔ

  16. 16 Copyright © since 1998 DMM All Rights Reserved. +FOLJOTґଘ

    ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍ όονࣦഊ࣌ͷϦτϥΠ͕େม ॲཧ͕࣌ؒґଘ ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠ ೔࣍όονʹ͓͚Δ՝୊
  17. 17 Copyright © since 1998 DMM All Rights Reserved. ͜ΕΒͷ՝୊Λղܾ͢ΔͨΊɺ+FOLJOT͔Βͷ୤٫Λܾҙ

    ໨త͸ʜ ೔࣍όονॲཧΛ৽ϫʔΫϑϩʔΤϯδϯ΁ϦϓϨΠε ೔࣍όονʹ͓͚Δ՝୊
  18. 18 Copyright © since 1998 DMM All Rights Reserved. ͨ͘͞Μͷબ୒ࢶ͕͋Γ·ͨ͠

  19. 19 Copyright © since 1998 DMM All Rights Reserved. ཁ݅͸ओʹͭ

    Մ༻ੑ εέʔϥϏϦςΟ ෼ࢄ࣮ߦฒྻॲཧ  ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖Δ ࣦഊͨ͠λεΫ͔ΒϦτϥΠՄೳ ਐḿ֬ೝ͕Ͱ͖ΔΠϯλʔϑΣʔε 3&45"1* ϫʔΫϑϩʔΤϯδϯͷબఆϙΠϯτ
  20. 20 Copyright © since 1998 DMM All Rights Reserved.

  21. 21 Copyright © since 1998 DMM All Rights Reserved. Մ༻ੑ‎"""

    εέʔϥϏϦςΟ ෼ࢄ࣮ߦฒྻॲཧ ‎""" ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖Δ‎" ࣦഊͨ͠λεΫ͔ΒϦτϥΠՄೳ‎" ਐḿ֬ೝ͕Ͱ͖ΔΠϯλʔϑΣʔε 3&45"1* ‎" ϫʔΫϑϩʔΤϯδϯͷબఆϙΠϯτ
  22. 22 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒

    %JHEBHͱͦΕΛऔΓר͘γεςϜ શͯΦϯϓϨͰ͢
  23. 23 Copyright © since 1998 DMM All Rights Reserved. ࠾༻͍ͯ͠Δϛυϧ΢ΣΞ

    %JHEBH   1PTUHSF42-   1HQPPM**   ˞࣌఺
  24. 24 Copyright © since 1998 DMM All Rights Reserved. Streaming

    Replication Watchdog Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync
  25. 25 Copyright © since 1998 DMM All Rights Reserved. Streaming

    Replication Watchdog Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync
  26. 26 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒%JHEBH

    αʔό݉ΫϥΠΞϯτͷ୆ߏ੒ )"ͷͨΊ  TVQFSWJTPSͰσʔϞϯԽ λεΫ͸෼ࢄ࣮ͯ͠ߦͯ͘͠ΕΔ ࣮ߦϩά΋෼ࢄͯ͠͠·͏ -TZODE STZODͰλεΫϩάΛ૒ํ޲ಉظ
  27. 27 Copyright © since 1998 DMM All Rights Reserved. Watchdog

    Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync Streaming Replication
  28. 28 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒1PTUHSF42-

    %JHEBHΛෳ਺୆ͷαʔόΫϥΠΞϯτߏ੒ʹ͢ΔͨΊ ϫʔΫϑϩʔͷ࣮ମ؅ཧ΍λεΫͷεςʔτ؅ཧ 1SJNBSZ4UBOECZͰ୆ߏ੒ 4USFBNJOH3FQMJDBUJPO
  29. 29 Copyright © since 1998 DMM All Rights Reserved. Streaming

    Replication Digdag Server (兼 Client) PostgreSQL (Primary) pgpool-II (Active) VIP PostgreSQL (Standby) pgpool-II (Standby) Digdag Server (兼 Client) Lsyncd & rync Watchdog
  30. 30 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒1HQPPM**

    ໨త͸ෛՙ෼ࢄͰ͸ͳ͘1PTUHSF42-ͷ৑௕Խ 1HQPPM**΋৑௕Խ͢ΔͨΊ୆ߏ੒ "DUJWF4UBOECZ  ͨͩ͠1PTUHSF42-ͱಉډ 8BUDIEPHػೳͰ7JSUVBM*1Λ੍ޚ
  31. 31 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ

    ద੾ͳཻ౓Ͱ֤ϑϩʔͷݟ௨͠Λվળ
  32. 32 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ

    ࣄۀ୯ҐͰͭͷϫʔΫϑϩʔΛ࡞੒͢ΔΑ͏ʹ੔ཧ ྫΦϯϥΠϯήʔϜӳձ࿩ʜFUD ֤ϫʔΫϑϩʔͰ&5-Λߦ͏Α͏ʹ
  33. 33 Copyright © since 1998 DMM All Rights Reserved. &5-ॲཧͷΠϝʔδ

    &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE ήʔϜࣄۀ෦ ϫʔΫϑϩʔ ˚˚ࣄۀ෦ ϫʔΫϑϩʔ ˘˘ࣄۀ෦ ϫʔΫϑϩʔ
  34. 34 Copyright © since 1998 DMM All Rights Reserved. &5-ॲཧͷΠϝʔδ

    &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE &YUSBDU 5SBOTGPSN -PBE ήʔϜࣄۀ෦ ϫʔΫϑϩʔ ˚˚ࣄۀ෦ ϫʔΫϑϩʔ ˘˘ࣄۀ෦ ϫʔΫϑϩʔ
  35. 35 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ

    ήʔϜࣄۀ෦ͷ৔߹ 4MBDL௨஌ςϯϓϨʔτ HBNFࣄۀ෦ϫʔΫϑϩʔ &5-ΛDBMM͢Δ ύϥϝʔλ HBNFࣄۀ෦ͷ࣮ॲཧ HBNFEJH͔ΒDBMM͞ΕΔ
  36. 36 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ

    ήʔϜࣄۀ෦ͷ৔߹ ڞ௨ύϥϝʔλಡΈࠐΈ HBNFEJH Τϥʔ࣌ͷ௨஌ HBNFࣄۀ෦ͷॲཧΛDBMM ˞FYUSBDUΛݺΜͰ͍Δ εέδϡʔϧઃఆ 4-"࣌ͷ௨஌
  37. 37 Copyright © since 1998 DMM All Rights Reserved. 8PSLqPXઃܭ

    ࣄۀ୯ҐͰ੔ཧͨ͜͠ͱͰશମͷݟ௨͕͠վળ ϧʔϧΛܾΊͨ͜ͱͰ୭͕ݟͯ΋Θ͔ΔΑ͏ʹ :BNMܗࣜͳͷͰಡΈ΍͍͢ Կ͔վम͕ೖΔ࣌ɺखೖΕ͠΍͍͢
  38. 38 Copyright © since 1998 DMM All Rights Reserved. ݸʑͷλεΫʹԿ෼͔͔Δͷ͔ࢹ֮తʹ೺ѲՄೳʹ

  39. 39 Copyright © since 1998 DMM All Rights Reserved. %JHEBHͰ͸ओʹ3VCZ4IFMM4DSJQUΛར༻

    Α͘࢖͏ΦϖϨʔλʔ SC3VCZͷλεΫΛ࣮ߦ TI4IFMM4DSJQUΛ࣮ߦ σʔλऩूπʔϧͷ࣮ߦ TMBDL4MBDL΁ͷ௨஌ ˞ޙड़ 8PSLqPXઃܭ
  40. 40 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯͷ࿩

    %JHEBHͷϓϥάΠϯͷ࡞੒ࣄྫͷ঺հ
  41. 41 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯΛࣗ࡞ͨ͠ࣄྫ

    4MBDL௨஌Λ࣮ݱ͢ΔEJHEBHTMBDLΛ࡞ͬͨ TMBDLΦϖϨʔλʔ͕࢖͑ΔΑ͏ʹ :BNMܗࣜͷςϯϓϨʔτ͔Β4MBDL௨஌ ม਺ͷຒΊࠐΈ΋Մೳ
  42. 42 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌ͷྫ

    ݺͼग़͢UFNQMBUFTTVDDFTTZNMΛ࡞੒ TMBDLEJHΛ࡞੒
  43. 43 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌

    ϓϥάΠϯͷαϯϓϧ ௨஌͕དྷΔ
  44. 44 Copyright © since 1998 DMM All Rights Reserved. 4MBDL΁ͷ௨஌

    ϓϥάΠϯͷαϯϓϧ Τϥʔ࣌ 4-"ʢࢦఆܦա࣌ؒ಺ʹ׬ྃ͠ͳ͍࣌ʣ
  45. 45 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯ

    ͳͥ࡞͔ͬͨʁ Ϧονͳ௨஌͕͔ͨͬͨ͠ λεΫ੒ޭ࣌Τϥʔ࣌4-"ͷ௨஌ʹར༻ ࢼ͠ʹ࡞ͬͯΈͨΒαΫοͱͰ͖ͨʢ+BWBʣ NZVJEJHEBHQMVHJOFYBNQMFΛࢀߟʹ࣮૷ +JUQBDLʹެ։͢Ε͹ར༻΋ָʑ
  46. 46 Copyright © since 1998 DMM All Rights Reserved. ϓϥάΠϯ

    (JUIVCʹͯެ։த (JUIVCɿIUUQTHJUIVCDPNT[ZOEJHEBHTMBDL ϒϩάɿIUUQJSPJMPHOFUEJHEBHTMBDLɹ
  47. 47 Copyright © since 1998 DMM All Rights Reserved. ৄࡉ͸ϒϩάʹͯʜ

  48. 48 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ

    (PݴޠͷΫϥΠΞϯτ NPH ͷࣄྫ঺հ
  49. 49 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ

    &5-ॲཧ͕ऴΘΔ·Ͱ ଴ػ 4VDFTT ΋͠΋ɺ໨తͷλεΫ͕ ੒ޭʹͳͬͨΒ ޙଓॲཧελʔτ 1PMMJOH 4UBSU ࣌ؒʹґଘ͠ͳ͍ͨΊͷϑϩʔΛߟ͑ͨ ̋̋ϓϩδΣΫτͷ˚˚ϫʔΫϑϩʔͷ˘˘ͱ͍͏ λεΫ͕ऴΘ͍ͬͯͨΒɺδϣϒΛ࣮ߦ͍ͨ͠ʂ
  50. 50 Copyright © since 1998 DMM All Rights Reserved. νʔϜؒ࿈ܞ

    (PݴޠͰ$-*πʔϧ NPH Λ࡞੒͠ఏڙ NPHͰ͸ಛఆͷϓϩδΣΫτͷϫʔΫϑϩʔΛɺ λεΫ໊ϕʔεͰεςʔλε֬ೝ͕Մೳʹ ϫʔΫϑϩʔΛελʔτϦτϥΠ͢Δػೳ΋
  51. 51 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ

    ࢖༻ྫ NPHͰ͸λεΫͷεςʔλεऔಘ ϫʔΫϑϩʔͷελʔτ ϦτϥΠػೳΛఏڙ
  52. 52 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ

    ࢖༻ྫ TBNQMFϓϩδΣΫτ TMBDLϫʔΫϑϩʔ TMBDL TUFQλεΫεςʔλεऔಘͷྫ
  53. 53 Copyright © since 1998 DMM All Rights Reserved. NPHͷ঺հ

    ࢖༻ྫ λεΫͷελʔτϦτϥΠ΋Մೳ ҎԼ͸ϫʔΫϑϩʔͷελʔτྫ
  54. 54 Copyright © since 1998 DMM All Rights Reserved. ϋϚΓͲ͜Ζ

    %JHEBHಋೖͰϋϚͬͨͱ͜Ζ
  55. 55 Copyright © since 1998 DMM All Rights Reserved. %JHEBH4FSWFSͰࣗ࡞ϓϥάΠϯ͕࢖͑ͳ͍

    λεΫ͕Ұ޲ʹਐ·ͳ͍ɾҟৗʹ͕͔͔࣌ؒΔ ϋϚΓͲ͜Ζ
  56. 56 Copyright © since 1998 DMM All Rights Reserved. %JHEBH4FSWFSͰࣗ࡞ϓϥάΠϯ͕࢖͑ͳ͍

    ‎ݱঢ়͸%JHEBH4FSWFSΛىಈͨ͠ΧϨϯτσΟϨΫτϦʹ ϓϥάΠϯΛμ΢ϯϩʔυ͢Δ࢓༷ σΟϨΫτϦݖݶͷݟ௚͠ͰରԠ ϋϚΓͲ͜Ζ
  57. 57 Copyright © since 1998 DMM All Rights Reserved. λεΫ͕Ұ޲ʹਐ·ͳ͍ɾҟৗʹ͕͔͔࣌ؒΔ

    ‎1HQPPM**Ͱ-PBECBMBODFNPEFΛ༗ޮʹ͍ͯͨ͠ ‎%JHEBHͷ%BUBCBTF઀ଓ͕4PDLFU5JNFPVU 1HQPPM**ͷMPBE@CBMBODF@NPEFΛGBMTFʹ ෳ਺୆ߏ੒Ͱ௕࣌ؒͷόονΛ࣮ߦ͢Δࡍʹ͸ɺ EBUBCBTFTPDLFU5JNFPVU EFGBVMU ʹ஫ҙ ϋϚΓͲ͜Ζ
  58. 58 Copyright © since 1998 DMM All Rights Reserved. ·ͱΊ

    %JHEBHಋೖͰ޾ͤʹͳͬͨͱ͜Ζ
  59. 59 Copyright © since 1998 DMM All Rights Reserved. ͜ͷΑ͏ͳ՝୊͕͋Γ·ͨ͠

    +FOLJOTґଘ ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍ ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠ όονࣦഊ࣌ͷϦτϥΠ͕େม ॲཧ͕࣌ؒґଘ ·ͱΊ
  60. 60 Copyright © since 1998 DMM All Rights Reserved. +FOLJOTґଘ

    +FOLJOT͔Βͷ୤٫ʹ੒ޭ ॲཧϑϩʔ͕ίʔυͰ؅ཧͰ͖͍ͯͳ͍ શͯίʔυͰ؅ཧՄೳʹ ֤σʔλιʔεͷґଘؔ܎ͷೝ͕ࣝ೉͍͠ ίʔυͰґଘؔ܎΋දݱ όονࣦഊ࣌ͷϦτϥΠ͕େม όονࣦഊ࣌ͷϦτϥΠ΋ָͪΜʹ ॲཧ͕࣌ؒґଘ NPH$-*ͷఏڙʹΑΓ࣌ؒґଘͷॲཧΛഉআ ·ͱΊ
  61. 61 Copyright © since 1998 DMM All Rights Reserved. ߋʹخ͔ͬͨ͠ͱ͜Ζʜ

    ฒྻॲཧͰόονॲཧ͕ߴ଎ʹ ݸʑͷλεΫʹԿ෼͔͔Δͷ͔ࢹ֮తʹ೺ѲՄೳʹ ·ͱΊ
  62. 62 Copyright © since 1998 DMM All Rights Reserved. pOBMMZUIBOLT