Digdagを本番導入したので社内勉強会で発表した資料です。
Introduction of Digdag.
View Slide
Who am I.• Twitter/GitHub Account• @smdmts• Main Fields• Scala & Java8 & React.js & Python• DDD CleanArchitecture @ akka-http• Workflow• Hive/Presto
Agenda• Digdag = Workflow automation system.
ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅• ఆظతͳλεΫͷ࣮ߦ• λεΫͷॱ࣮࣍ߦ• γεςϜؒͷσʔλࣗಈ࿈ܞ• όονʹΑΔσʔλूܭͷࣗಈԽ• όονδϣϒྃޙͷϝʔϧ/SlackͳͲ௨• ϦτϥΠ࣌ʹ͓͚Δႈੑ
Digdagͱ• DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑϩʔΤϯδϯ• YAMLͰDAGΛදݱ͢ΔͨΊఆٛମGitཧՄೳʢWorkflow as Codeʣ• LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ͰՔಇͤ͞Δ• Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ߦՄೳ
DigdagͱʢClient/ServerϞʔυʣ• PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ• αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳͰ࣮ߦڥ͕εέʔϧՄೳ• Workflowͷ࣮ମPostgreSQLʹӬଓԽ͞ΕΔ• Client͕ίϚϯυͰWorkflowΛpush͢Δ• Workflowੈཧ͞ΕΔ• ࠶ىಈෆཁͰδϣϒొʗ࠶࣮ߦՄೳ
DAG (Directed acyclic graph)ͱ• DAGʢ༗ඇ८ճάϥϑʣͱ ʢwikipedia)άϥϑཧʹ͓͚Δด࿏ͷͳ͍༗άϥϑͷࣄ༗άϥϑͱ༗ลʢํΛࣔ͢ҹ͖ͷลʣ͔ΒͳΓɺลಉ࢜Λͭͳ͙͕ɺ͋Δ v ͔Βग़ൃ͠ɺลΛͨͲΓɺ v ʹͬͯ͜ͳ͍ͷ͕༗ඇ८ճάϥϑͰ͋Δɻ
DAG (Directed acyclic graph)ͱ• DAGʢ༗ඇ८ճάϥϑʣͱ• తʹݴ͏ͱऴ͕ଘࡏ͠։࢝ʹͬͯ͜ͳ͍άϥϑ
DigdagͰͷදݱํ๏• YAMLͰΦϖϨʔλΛఆٛtimezone: UTC_export:mail:..... # Definition of mail+step1_input:py>: tasks.load_error:mail>: body.txtsubject: input error!to: [[email protected]]+step2_process:sh>: echo process.+step2_report:sh>: echo report.
δϣϒϑϩʔߏུ֓ਤ
δϣϒϑϩʔߏུ֓ਤʹ͓͚Δఆٛtimezone: UTC+prepare_load_aws_env:py>: tasks.load_aws_env+step1_produce_tasks:# Generate SQL Queries for Redshift.!include : 'child_tasks/produce_tasks/bootstrap.dig'+step2_create_redshift_buffer:# Internal S3 or TreasureData to Redshift temporary buffer.!include : 'child_tasks/create_redshift_buffer/bootstrap.dig'+step3_create_publisher_s3:# Create Redshift buffer to publisher s3 bucket.!include : 'child_tasks/create_publisher_s3/bootstrap.dig'
։ൃ/ӡ༻ͯ͠Έͨײ• Workflow͕ίʔυͰදݱ͞Ε σόοά༰қ ͳͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ• Πϯετʔϧͷ؆қੑɺ࠶ىಈෆཁͷδϣϒ࠶ొʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ• ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ͢ΔࣄΛܾఆ
։ൃ࣌ʹൃੜͨ͠/՝• py operatorར༻࣌ʹগ͠ϋϚͬͨ• ςετίʔυΛॻͨ͘Ίʹimport digdagͷϞοΫίʔυ͕ඞཁ• !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱPythonεΫϦϓτؒͷґଘղܾͷ࣮͕ඞཁ• λεΫؒͷม࿈ܞdigdag.env.storeͰɺશλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ΓʢಉҰ໊শͰ্ॻ͖͕ൃੜ͢Δ߹༗Γʣ
։ൃ࣌ʹൃੜͨ͠/՝ʢิʣ• rb operatorར༻ͯ͠·ͤΜ
ӡ༻࣌ʹൃੜͨ͠/՝• ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ• ֬ೝը໘ (ίϚϯυͰճආத)• ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ• ϩάͷS3ӬଓԽ (S3FSͰճආத)
·ͱΊ• ࣗಈԽਖ਼ٛʂ