Upgrade to Pro — share decks privately, control downloads, hide ads and more …

digdag-Introduction

 digdag-Introduction

Digdagを本番導入したので社内勉強会で発表した資料です。

Masatoshi Shimada

August 19, 2016
Tweet

More Decks by Masatoshi Shimada

Other Decks in Programming

Transcript

  1. Who am I. • Twitter/GitHub Account • @smdmts • Main

    Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
  2. Digdagͱ͸ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମ͸Git؅ཧՄೳ ʢWorkflow as

    Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
  3. DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition

    of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [me@example.com] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
  4. δϣϒϑϩʔߏ੒ུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL

    Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
  5. ։ൃ/ӡ༻ͯ͠Έͨײ૝ • Workflow͕ίʔυͰදݱ͞Ε σόοά΋༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •

    Πϯετʔϧͷ؆қੑ΍ɺ࠶ىಈෆཁͷδϣϒ࠶ ొ࿥ʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ৅ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ؅͢Δ ࣄΛܾఆ
  6. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹ͸import digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ

    PythonεΫϦϓτؒͷґଘղܾͷ࣮૷͕ඞཁ • λεΫؒͷม਺࿈ܞdigdag.env.storeͰ͸ɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ͸্ॻ͖͕ൃੜ͢Δ৔߹༗Γʣ