Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
digdag-Introduction
Search
Masatoshi Shimada
August 19, 2016
Programming
1
1.3k
digdag-Introduction
Digdagを本番導入したので社内勉強会で発表した資料です。
Masatoshi Shimada
August 19, 2016
Tweet
Share
More Decks by Masatoshi Shimada
See All by Masatoshi Shimada
データプラットフォーム技術におけるメダリオンアーキテクチャという考え方/DataPlatformWithMedallionArchitecture
smdmts
6
730
Delta Lakeを用いた LLM処理基盤 / Delta Lake with LLM on Dataplatform
smdmts
3
7.5k
Lakehouseプラットフォームを 採用するまでの話/Lakehouse Platform Adoption
smdmts
1
980
Sparkから利用するAirframe/Spark-With-Airframe
smdmts
0
1.7k
Redashで何をみるのか/What Do You Wanna See Redash?
smdmts
1
1.6k
DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization
smdmts
0
1.7k
DatabricksとSparkではじめる [データ分析/機械学習] 実践入門 / Databrick and Spark with Data Analyze and ML for newbie.
smdmts
5
2.3k
作らない分析基板のススメ/DWH For Startup With YAGNI
smdmts
1
760
エンジニアのためのドメイン駆動設計実践入門 / DDD for Engineer newbie
smdmts
18
3.8k
Other Decks in Programming
See All in Programming
Goで作る、開発・CI環境
sin392
0
260
はじめてのWeb API体験 ー 飲食店検索アプリを作ろうー
akinko_0915
0
140
ソフトウェア品質を数字で捉える技術。事業成長を支えるシステム品質の マネジメント
takuya542
2
15k
Hack Claude Code with Claude Code
choplin
7
2.5k
RailsGirls IZUMO スポンサーLT
16bitidol
0
200
オンコール⼊⾨〜ページャーが鳴る前に、あなたが備えられること〜 / Before The Pager Rings
yktakaha4
2
980
Agentic Coding: The Future of Software Development with Agents
mitsuhiko
0
130
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
260
Model Pollution
hschwentner
1
160
Python型ヒント完全ガイド 初心者でも分かる、現代的で実践的な使い方
mickey_kubo
1
240
iOS 26にアップデートすると実機でのHot Reloadができない?
umigishiaoi
0
140
GPUを計算資源として使おう!
primenumber
1
230
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Gamification - CAS2011
davidbonilla
81
5.4k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
How GitHub (no longer) Works
holman
314
140k
Designing Experiences People Love
moore
142
24k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Transcript
Introduction of Digdag.
Who am I. • Twitter/GitHub Account • @smdmts • Main
Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
Agenda • Digdag = Workflow automation system.
ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ •
όονδϣϒྃޙͷϝʔϧ/SlackͳͲ௨ • ϦτϥΠ࣌ʹ͓͚Δႈੑ
Digdagͱ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମGitཧՄೳ ʢWorkflow as
Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
DigdagͱʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ Ͱ࣮ߦڥ͕εέʔϧՄೳ • Workflowͷ࣮ମPostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ
• Workflowੈཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొʗ࠶࣮ߦՄೳ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ ʢwikipedia) άϥϑཧʹ͓͚Δด࿏ͷͳ͍༗άϥϑͷࣄ ༗άϥϑͱ༗ลʢํΛࣔ͢ҹ͖ ͷลʣ͔ΒͳΓɺลಉ࢜Λͭͳ͙͕ɺ͋Δ
v ͔Βग़ൃ͠ɺลΛͨͲΓɺ v ʹͬͯ ͜ͳ͍ͷ͕༗ඇ८ճάϥϑͰ͋Δɻ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ • తʹݴ͏ͱऴ͕ଘࡏ͠։࢝ʹͬͯ͜ͳ ͍άϥϑ
DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition
of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [me@example.com] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
δϣϒϑϩʔߏུ֓ਤ
δϣϒϑϩʔߏུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL
Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
։ൃ/ӡ༻ͯ͠Έͨײ • Workflow͕ίʔυͰදݱ͞Ε σόοά༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •
Πϯετʔϧͷ؆қੑɺ࠶ىಈෆཁͷδϣϒ࠶ ొʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ͢Δ ࣄΛܾఆ
։ൃ࣌ʹൃੜͨ͠/՝ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹimport digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ
PythonεΫϦϓτؒͷґଘղܾͷ࣮͕ඞཁ • λεΫؒͷม࿈ܞdigdag.env.storeͰɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ্ॻ͖͕ൃੜ͢Δ߹༗Γʣ
։ൃ࣌ʹൃੜͨ͠/՝ʢิʣ • rb operatorར༻ͯ͠·ͤΜ
ӡ༻࣌ʹൃੜͨ͠/՝ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ
(S3FSͰճආத)
·ͱΊ • ࣗಈԽਖ਼ٛʂ