digdag-Introduction

Transcript

1. Introduction of Digdag.

2. Who am I. • Twitter/GitHub Account • @smdmts • Main

   Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto

3. Agenda • Digdag = Workflow automation system.

4. ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ •

   όονδϣϒ׬ྃޙͷϝʔϧ/SlackͳͲ΁௨஌ • ϦτϥΠ࣌ʹ͓͚Δႈ౳ੑ

5. Digdagͱ͸ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମ͸Git؅ཧՄೳ ʢWorkflow as

   Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ

6. Digdagͱ͸ʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ਺ ୆Ͱ࣮ߦ؀ڥ͕εέʔϧՄೳ • Workflowͷ࣮ମ͸PostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ

   • Workflow͸ੈ୅؅ཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొ࿥ʗ࠶࣮ߦՄೳ

7. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ ʢwikipedia) άϥϑཧ࿦ʹ͓͚Δด࿏ͷͳ͍༗޲άϥϑͷࣄ ༗޲άϥϑ͸௖఺ͱ༗޲ลʢํ޲Λࣔ͢໼ҹ෇͖ ͷลʣ͔ΒͳΓɺล͸௖఺ಉ࢜Λͭͳ͙͕ɺ͋Δ

   ௖఺ v ͔Βग़ൃ͠ɺลΛͨͲΓɺ௖఺ v ʹ໭ͬͯ ͜ͳ͍ͷ͕༗޲ඇ८ճάϥϑͰ͋Δɻ

8. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ • ୺తʹݴ͏ͱऴ఺͕ଘࡏ͠։࢝఺ʹ໭ͬͯ͜ͳ ͍άϥϑ

9. DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition

   of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [me@example.com] +step2_process: sh>: echo process. +step2_report: sh>: echo report.

10. δϣϒϑϩʔߏ੒ུ֓ਤ

11. δϣϒϑϩʔߏ੒ུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL

    Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'

12. ։ൃ/ӡ༻ͯ͠Έͨײ૝ • Workflow͕ίʔυͰදݱ͞Ε σόοά΋༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •

    Πϯετʔϧͷ؆қੑ΍ɺ࠶ىಈෆཁͷδϣϒ࠶ ొ࿥ʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ৅ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ؅͢Δ ࣄΛܾఆ

13. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹ͸import digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ

    PythonεΫϦϓτؒͷґଘղܾͷ࣮૷͕ඞཁ • λεΫؒͷม਺࿈ܞdigdag.env.storeͰ͸ɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ͸্ॻ͖͕ൃੜ͢Δ৔߹༗Γʣ

14. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ʢิ଍ʣ • rb operator͸ར༻ͯ͠·ͤΜ

15. ӡ༻࣌ʹൃੜͨ͠໰୊/՝୊ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ

    (S3FSͰճආத)

16. ·ͱΊ • ࣗಈԽ͸ਖ਼ٛʂ