Upgrade to Pro — share decks privately, control downloads, hide ads and more …

digdag-Introduction

 digdag-Introduction

Digdagを本番導入したので社内勉強会で発表した資料です。

53c93eb80b5958590b18205dc5fd3634?s=128

Masatoshi Shimada

August 19, 2016
Tweet

Transcript

  1. Introduction of Digdag.

  2. Who am I. • Twitter/GitHub Account • @smdmts • Main

    Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
  3. Agenda • Digdag = Workflow automation system.

  4. ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ •

    όονδϣϒ׬ྃޙͷϝʔϧ/SlackͳͲ΁௨஌ • ϦτϥΠ࣌ʹ͓͚Δႈ౳ੑ
  5. Digdagͱ͸ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମ͸Git؅ཧՄೳ ʢWorkflow as

    Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
  6. Digdagͱ͸ʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ਺ ୆Ͱ࣮ߦ؀ڥ͕εέʔϧՄೳ • Workflowͷ࣮ମ͸PostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ

    • Workflow͸ੈ୅؅ཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొ࿥ʗ࠶࣮ߦՄೳ
  7. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ ʢwikipedia) άϥϑཧ࿦ʹ͓͚Δด࿏ͷͳ͍༗޲άϥϑͷࣄ ༗޲άϥϑ͸௖఺ͱ༗޲ลʢํ޲Λࣔ͢໼ҹ෇͖ ͷลʣ͔ΒͳΓɺล͸௖఺ಉ࢜Λͭͳ͙͕ɺ͋Δ

    ௖఺ v ͔Βग़ൃ͠ɺลΛͨͲΓɺ௖఺ v ʹ໭ͬͯ ͜ͳ͍ͷ͕༗޲ඇ८ճάϥϑͰ͋Δɻ
  8. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ • ୺తʹݴ͏ͱऴ఺͕ଘࡏ͠։࢝఺ʹ໭ͬͯ͜ͳ ͍άϥϑ

  9. DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition

    of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [me@example.com] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
  10. δϣϒϑϩʔߏ੒ུ֓ਤ

  11. δϣϒϑϩʔߏ੒ུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL

    Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
  12. ։ൃ/ӡ༻ͯ͠Έͨײ૝ • Workflow͕ίʔυͰදݱ͞Ε σόοά΋༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •

    Πϯετʔϧͷ؆қੑ΍ɺ࠶ىಈෆཁͷδϣϒ࠶ ొ࿥ʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ৅ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ؅͢Δ ࣄΛܾఆ
  13. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹ͸import digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ

    PythonεΫϦϓτؒͷґଘղܾͷ࣮૷͕ඞཁ • λεΫؒͷม਺࿈ܞdigdag.env.storeͰ͸ɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ͸্ॻ͖͕ൃੜ͢Δ৔߹༗Γʣ
  14. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ʢิ଍ʣ • rb operator͸ར༻ͯ͠·ͤΜ

  15. ӡ༻࣌ʹൃੜͨ͠໰୊/՝୊ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ

    (S3FSͰճආத)
  16. ·ͱΊ • ࣗಈԽ͸ਖ਼ٛʂ