DigdagͰETLॲཧΛ͢Δ σʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #2 2019.07.19 த໺ᠳଠ(@tosametal)
 גࣜձࣾϚΠΫϩΞυ ΞϓϦέʔγϣϯΤϯδχΞ

ϚΠΫϩΞυʹ͓͚Δػցֶश ޿ࠂ഑৴γεςϜʹ͓͚ΔCTR༧ଌɺCVR༧ଌɺෆਖ਼ΫϦοΫͷݕग़ͳͲ

ϩάج൫ͷߏ੒ Imp
 Server Click
 Server RTB Server Kafka Hadoop (σʔλ΢ΣΞϋ΢ε) Digdag Hadoop (෼ੳج൫)

ϩάج൫ͷߏ੒ Imp
 Server Click
 Server RTB Server Kafka Hadoop (σʔλ΢ΣΞϋ΢ε) Digdag Hadoop (෼ੳج൫) at least once ϢχʔΫͳIDʹΑΔॏෳഉআ sessionͰ؅ཧ ႈ౳ͳॲཧ Kafka secondaryͰ kafkaΛࢦఆ jsonܗࣜͷ ߏ଄Խσʔλ

Digdagͱ͸ digϑΝΠϧʹએݴతʹϫʔΫϑϩʔΛهड़ Workflow as code εέδϡʔϧ࣮ߦɺϦΧόϦ UI͔Βਐḿͷ֬ೝ΍࠶࣮ߦ͕Մೳ ΦϖϨʔλΛࣗ࡞Մೳ

PostgreSQL ࣮ߦཤྺͳͲΛอଘ Task͝ͱʹhadoopΫϥΠΞϯτ ͱͳΔίϯςφΛ্ཱͪ͛Δ εέʔϧΞ΢τՄೳ όον࣮ߦج൫ߏ੒

ෳࡶͳґଘؔ܎Λ੍ޚͭͭ͠ ϫʔΫϑϩʔͷՄಡੑΛอͭ

ϓϩδΣΫτΛػೳ୯ҐͰ෼ׂ ϓϩδΣΫτͱ͸ In Digdag, workflows are packaged together with other files used in the workflows. The files can be anything such as SQL scripts, Python/Ruby/Shell scripts, configuration files, etc. This set of the workflow definitions is called project. ެࣜυΩϡϝϯτ(http://docs.digdag.io/)ΑΓҾ༻ ϚΠΫϩΞυͰ͸ݱࡏ໿60ݸͷϓϩδΣΫτ͕ಈ͍͍ͯΔ

ϓϩδΣΫτ಺ͷґଘؔ܎ schedule: daily>: 12:00:00 +task1: _parallel: true +subtask1: call>: subtask1.dig +subtask2: call>: subtask2.dig +task2: echo>: task finished successfully ●callΦϖϨʔλΛ࢖͏͜ͱͰdigϑΝΠϧ ͷ෼ׂΛߦ͏͜ͱ͕Մೳ ●requireΛ࢖͏ͱ΋͏গ͠ෳࡶͳDAGͷ දݱ΋Մೳ subtask1 subtask2 task2

ϓϩδΣΫτؒͷґଘؔ܎ ϓϩδΣΫτA ϓϩδΣΫτB ௚઀ଞͷϓϩδΣ Ϋτͷ݁ՌΛݟΔ ͜ͱ͸ग़དྷͳ͍

ϓϩδΣΫτؒͷґଘؔ܎ +touch_task: s3_touch>: bucket/flag/fileX +wait_task: s3_wait>: bucket/flag/fileX ϓϩδΣΫτB ϓϩδΣΫτA fileX ࣗ࡞ΦϖϨʔλ ࢀߟ:https://github.com/ tosametal/digdag-plugins

ͦͷଞ ϫʔΫϑϩʔશମΛႈ౳ʹ͢Δ ● hiveΫΤϦ͸insert overwrite ● distcp͸overwrite deleteΦϓγϣϯΛࢦఆ ϦτϥΠΛઃఆ͢Δ ● exponential interval

·ͱΊ ● ϓϩδΣΫτ͸ංେԽ͠ͳ͍Α͏ʹػೳͰ෼ׂ ● ϓϩδΣΫτؒͷґଘ͸s3_waitͰղܾ ● Α͘࢖͏ػೳ͸ϓϥάΠϯΛ࡞Ζ͏

No content