Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
digdag-Introduction
Search
Masatoshi Shimada
August 19, 2016
Programming
1
1.2k
digdag-Introduction
Digdagを本番導入したので社内勉強会で発表した資料です。
Masatoshi Shimada
August 19, 2016
Tweet
Share
More Decks by Masatoshi Shimada
See All by Masatoshi Shimada
Delta Lakeを用いた LLM処理基盤 / Delta Lake with LLM on Dataplatform
smdmts
3
5.3k
Lakehouseプラットフォームを 採用するまでの話/Lakehouse Platform Adoption
smdmts
1
760
Sparkから利用するAirframe/Spark-With-Airframe
smdmts
0
1.5k
Redashで何をみるのか/What Do You Wanna See Redash?
smdmts
1
1.5k
DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization
smdmts
0
1.5k
DatabricksとSparkではじめる [データ分析/機械学習] 実践入門 / Databrick and Spark with Data Analyze and ML for newbie.
smdmts
5
2.1k
作らない分析基板のススメ/DWH For Startup With YAGNI
smdmts
1
680
エンジニアのためのドメイン駆動設計実践入門 / DDD for Engineer newbie
smdmts
18
3.4k
How to growth the delish kitchen team to data-driven team
smdmts
1
1.1k
Other Decks in Programming
See All in Programming
Using Livebook to build and deploy internal tools @ ElixirConf 2024
hugobarauna
0
230
全力の跳躍を捉える計測アプリを作る
ogijun2018
1
1.2k
ECMAScript仕様を読むのに必要な知識 - ダイジェスト版
syumai
4
2.6k
Mastering AsyncSequence - 使う・作る・他のデザインパターン(クロージャ、Delegate など)から移行する
treastrain
4
1.5k
令和トラベルにおけるLLM活用事例:社内ツール開発から得た学びと実践
ippo012
0
120
Amazon Neptuneで始める初めてのグラフDB ー グラフDBを使う意味を考える ー
satoshi256kbyte
2
220
長期運用プロダクトの開発速度を維持し続けるためのリファクタリング実践例
wataruss
8
2.6k
Amebaチョイス立ち上げの裏側 ~依存システムとの闘い~
daichi_igarashi
0
220
現代のVueとTypeScript - 型安全の活用術
minako__ph
4
3.1k
React + TextAliveでカッコいいLyric Applicatioinを作ろう!!
tosuri13
0
370
Rechartsで楽にゴリゴリにカスタマイズする!
10tera
1
140
Scala におけるコンパイラエラーとの付き合い方
chencmd
2
350
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
354
29k
A designer walks into a library…
pauljervisheath
201
24k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
363
22k
StorybookのUI Testing Handbookを読んだ
zakiyama
25
5k
How To Stay Up To Date on Web Technology
chriscoyier
785
250k
The Invisible Side of Design
smashingmag
295
50k
The Art of Programming - Codeland 2020
erikaheidi
48
13k
YesSQL, Process and Tooling at Scale
rocio
167
14k
GraphQLの誤解/rethinking-graphql
sonatard
65
9.8k
BBQ
matthewcrist
83
9.1k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
278
13k
What's new in Ruby 2.0
geeforr
340
31k
Transcript
Introduction of Digdag.
Who am I. • Twitter/GitHub Account • @smdmts • Main
Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
Agenda • Digdag = Workflow automation system.
ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ •
όονδϣϒྃޙͷϝʔϧ/SlackͳͲ௨ • ϦτϥΠ࣌ʹ͓͚Δႈੑ
Digdagͱ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମGitཧՄೳ ʢWorkflow as
Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
DigdagͱʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ Ͱ࣮ߦڥ͕εέʔϧՄೳ • Workflowͷ࣮ମPostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ
• Workflowੈཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొʗ࠶࣮ߦՄೳ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ ʢwikipedia) άϥϑཧʹ͓͚Δด࿏ͷͳ͍༗άϥϑͷࣄ ༗άϥϑͱ༗ลʢํΛࣔ͢ҹ͖ ͷลʣ͔ΒͳΓɺลಉ࢜Λͭͳ͙͕ɺ͋Δ
v ͔Βग़ൃ͠ɺลΛͨͲΓɺ v ʹͬͯ ͜ͳ͍ͷ͕༗ඇ८ճάϥϑͰ͋Δɻ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ • తʹݴ͏ͱऴ͕ଘࡏ͠։࢝ʹͬͯ͜ͳ ͍άϥϑ
DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition
of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [
[email protected]
] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
δϣϒϑϩʔߏུ֓ਤ
δϣϒϑϩʔߏུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL
Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
։ൃ/ӡ༻ͯ͠Έͨײ • Workflow͕ίʔυͰදݱ͞Ε σόοά༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •
Πϯετʔϧͷ؆қੑɺ࠶ىಈෆཁͷδϣϒ࠶ ొʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ͢Δ ࣄΛܾఆ
։ൃ࣌ʹൃੜͨ͠/՝ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹimport digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ
PythonεΫϦϓτؒͷґଘղܾͷ࣮͕ඞཁ • λεΫؒͷม࿈ܞdigdag.env.storeͰɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ্ॻ͖͕ൃੜ͢Δ߹༗Γʣ
։ൃ࣌ʹൃੜͨ͠/՝ʢิʣ • rb operatorར༻ͯ͠·ͤΜ
ӡ༻࣌ʹൃੜͨ͠/՝ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ
(S3FSͰճආத)
·ͱΊ • ࣗಈԽਖ਼ٛʂ