digdag-Introduction - Speaker Deck

Slide 1

Slide 1 text

Introduction of Digdag.

Slide 2

Slide 2 text

Who am I. • Twitter/GitHub Account • @smdmts • Main Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto

Slide 3

Slide 3 text

Agenda • Digdag = Workflow automation system.

Slide 4

Slide 4 text

ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ • όονδϣϒ׬ྃޙͷϝʔϧ/SlackͳͲ΁௨஌ • ϦτϥΠ࣌ʹ͓͚Δႈ౳ੑ

Slide 5

Slide 5 text

Digdagͱ͸ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମ͸Git؅ཧՄೳ ʢWorkflow as Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ

Slide 6

Slide 6 text

Digdagͱ͸ʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ਺ ୆Ͱ࣮ߦ؀ڥ͕εέʔϧՄೳ • Workflowͷ࣮ମ͸PostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ • Workflow͸ੈ୅؅ཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొ࿥ʗ࠶࣮ߦՄೳ

Slide 7

Slide 7 text

DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ ʢwikipedia) άϥϑཧ࿦ʹ͓͚Δด࿏ͷͳ͍༗޲άϥϑͷࣄ ༗޲άϥϑ͸௖఺ͱ༗޲ลʢํ޲Λࣔ͢໼ҹ෇͖ ͷลʣ͔ΒͳΓɺล͸௖఺ಉ࢜Λͭͳ͙͕ɺ͋Δ ௖఺ v ͔Βग़ൃ͠ɺลΛͨͲΓɺ௖఺ v ʹ໭ͬͯ ͜ͳ͍ͷ͕༗޲ඇ८ճάϥϑͰ͋Δɻ

Slide 8

Slide 8 text

DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ • ୺తʹݴ͏ͱऴ఺͕ଘࡏ͠։࢝఺ʹ໭ͬͯ͜ͳ ͍άϥϑ

Slide 9

Slide 9 text

DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Deﬁnition of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [[email protected]] +step2_process: sh>: echo process. +step2_report: sh>: echo report.

Slide 10

Slide 10 text

δϣϒϑϩʔߏ੒ུ֓ਤ

Slide 11

Slide 11 text

δϣϒϑϩʔߏ੒ུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'

Slide 12

Slide 12 text

։ൃ/ӡ༻ͯ͠Έͨײ૝ • Workflow͕ίʔυͰදݱ͞Ε σόοά΋༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ • Πϯετʔϧͷ؆қੑ΍ɺ࠶ىಈෆཁͷδϣϒ࠶ ొ࿥ʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ৅ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ؅͢Δ ࣄΛܾఆ

Slide 13

Slide 13 text

։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹ͸import digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ PythonεΫϦϓτؒͷґଘղܾͷ࣮૷͕ඞཁ • λεΫؒͷม਺࿈ܞdigdag.env.storeͰ͸ɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ͸্ॻ͖͕ൃੜ͢Δ৔߹༗Γʣ

Slide 14

Slide 14 text

։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ʢิ଍ʣ • rb operator͸ར༻ͯ͠·ͤΜ

Slide 15

Slide 15 text

ӡ༻࣌ʹൃੜͨ͠໰୊/՝୊ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ (S3FSͰճආத)

Slide 16

Slide 16 text

·ͱΊ • ࣗಈԽ͸ਖ਼ٛʂ