Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
digdag-Introduction
Search
Masatoshi Shimada
August 19, 2016
Programming
1
1.3k
digdag-Introduction
Digdagを本番導入したので社内勉強会で発表した資料です。
Masatoshi Shimada
August 19, 2016
Tweet
Share
More Decks by Masatoshi Shimada
See All by Masatoshi Shimada
データプラットフォーム技術におけるメダリオンアーキテクチャという考え方/DataPlatformWithMedallionArchitecture
smdmts
6
2.5k
Delta Lakeを用いた LLM処理基盤 / Delta Lake with LLM on Dataplatform
smdmts
3
8k
Lakehouseプラットフォームを 採用するまでの話/Lakehouse Platform Adoption
smdmts
1
1k
Sparkから利用するAirframe/Spark-With-Airframe
smdmts
0
1.7k
Redashで何をみるのか/What Do You Wanna See Redash?
smdmts
1
1.6k
DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization
smdmts
0
1.7k
DatabricksとSparkではじめる [データ分析/機械学習] 実践入門 / Databrick and Spark with Data Analyze and ML for newbie.
smdmts
5
2.3k
作らない分析基板のススメ/DWH For Startup With YAGNI
smdmts
1
770
エンジニアのためのドメイン駆動設計実践入門 / DDD for Engineer newbie
smdmts
18
3.8k
Other Decks in Programming
See All in Programming
Back to the Future: Let me tell you about the ACP protocol
terhechte
0
120
CSC509 Lecture 01
javiergs
PRO
1
430
大規模アプリにおけるXcode Previews実用化までの道のり
ikesyo
0
970
実践AIチャットボットUI実装入門
syumai
7
2.4k
Server Less Code More - コードを書かない時代に生きるサーバーレスデザイン / server-less-code-more
gawa
5
1.9k
メモリ不足との戦い〜大量データを扱うアプリでの実践例〜
kwzr
1
670
Reduxモダナイズ 〜コードのモダン化を通して、将来のライブラリ移行に備える〜
pvcresin
2
650
iOSDC.pdf
chronos2500
2
640
Serena MCPのすすめ
wadakatu
4
830
いま中途半端なSwift 6対応をするより、Default ActorやApproachable Concurrencyを有効にしてからでいいんじゃない?
yimajo
2
230
なぜGoのジェネリクスはこの形なのか? Featherweight Goが明かす設計の核心
ryotaros
7
990
Swift Concurrency - 状態監視の罠
objectiveaudio
2
320
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
4k
GraphQLとの向き合い方2022年版
quramy
49
14k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Embracing the Ebb and Flow
colly
88
4.8k
Speed Design
sergeychernyshev
32
1.1k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Building Better People: How to give real-time feedback that sticks.
wjessup
368
20k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Transcript
Introduction of Digdag.
Who am I. • Twitter/GitHub Account • @smdmts • Main
Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
Agenda • Digdag = Workflow automation system.
ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ •
όονδϣϒྃޙͷϝʔϧ/SlackͳͲ௨ • ϦτϥΠ࣌ʹ͓͚Δႈੑ
Digdagͱ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମGitཧՄೳ ʢWorkflow as
Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
DigdagͱʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ Ͱ࣮ߦڥ͕εέʔϧՄೳ • Workflowͷ࣮ମPostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ
• Workflowੈཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొʗ࠶࣮ߦՄೳ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ ʢwikipedia) άϥϑཧʹ͓͚Δด࿏ͷͳ͍༗άϥϑͷࣄ ༗άϥϑͱ༗ลʢํΛࣔ͢ҹ͖ ͷลʣ͔ΒͳΓɺลಉ࢜Λͭͳ͙͕ɺ͋Δ
v ͔Βग़ൃ͠ɺลΛͨͲΓɺ v ʹͬͯ ͜ͳ͍ͷ͕༗ඇ८ճάϥϑͰ͋Δɻ
DAG (Directed acyclic graph)ͱ • DAGʢ༗ඇ८ճάϥϑʣͱ • తʹݴ͏ͱऴ͕ଘࡏ͠։࢝ʹͬͯ͜ͳ ͍άϥϑ
DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition
of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [
[email protected]
] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
δϣϒϑϩʔߏུ֓ਤ
δϣϒϑϩʔߏུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL
Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
։ൃ/ӡ༻ͯ͠Έͨײ • Workflow͕ίʔυͰදݱ͞Ε σόοά༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ •
Πϯετʔϧͷ؆қੑɺ࠶ىಈෆཁͷδϣϒ࠶ ొʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ͢Δ ࣄΛܾఆ
։ൃ࣌ʹൃੜͨ͠/՝ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹimport digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ
PythonεΫϦϓτؒͷґଘղܾͷ࣮͕ඞཁ • λεΫؒͷม࿈ܞdigdag.env.storeͰɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ্ॻ͖͕ൃੜ͢Δ߹༗Γʣ
։ൃ࣌ʹൃੜͨ͠/՝ʢิʣ • rb operatorར༻ͯ͠·ͤΜ
ӡ༻࣌ʹൃੜͨ͠/՝ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ
(S3FSͰճආத)
·ͱΊ • ࣗಈԽਖ਼ٛʂ