Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
trinity で Cloud Composer に ワークフローを簡単デプロイ / Easy...
Search
Hiroka Zaitsu
October 25, 2019
Technology
0
880
trinity で Cloud Composer に ワークフローを簡単デプロイ / Easy workflow deployment to Cloud Composer with trinity
2019.10.25 Fukuoka.go#14+Umeda.go
https://fukuokago.connpass.com/event/146447/
Hiroka Zaitsu
October 25, 2019
Tweet
Share
More Decks by Hiroka Zaitsu
See All by Hiroka Zaitsu
ビジネス職が分析も担う事業部制組織でのデータ活用の仕組みづくり / Enabling Data Analytics in Business-Led Divisional Organizations
zaimy
1
440
Vertex AI Matching Engine と CLIP を使って EC サービスの類似画像検索機能を作る / Development of similar image search function for EC services using Vertex AI Matching Engine and CLIP
zaimy
0
740
BigQuery の日本語データを Dataflow と Vertex AI でトピックモデリング / Topic modeling of Japanese data in BigQuery with Dataflow and Vertex AI
zaimy
1
5.8k
データサイエンティストの仕事紹介 / Data Scientist Job Introduction
zaimy
1
600
GMOペパボのサービスと研究開発を支えるデータ基盤の裏側 / Inside Story of Data Infrastructure Supporting GMO Pepabo's Services and R&D
zaimy
1
1.8k
正則化とロジスティック回帰/machine-learning-lecture-regularization-and-logistic-regression
zaimy
0
8.7k
ECサイトにおける閲覧履歴を用いた購買に繋がる行動の変化検出 / Change Detection in Behavior Followed by Possible Purchase Using Electronic Commerce Site Browsing History
zaimy
1
930
ハンドメイド作品を対象としたECサイトにおける大量生産品の検出 / Detection of Mass-produced Goods at EC Site to Trade Handmade Goods
zaimy
3
4.8k
キャリアキーノート2018 / Career Keynote 2018
zaimy
1
2.2k
Other Decks in Technology
See All in Technology
Maintainer Meetupで「生の声」を聞く ~講演だけじゃないKubeCon
logica0419
0
140
室長の逆襲 :データ活用の陣地を増やすためのヒント
masatoshi0205
0
130
AI時代にも変わらぬ価値を発揮したい: インフラ・クラウドを切り口にユーザー価値と非機能要件に向き合ってエンジニアとしての地力を培う
netmarkjp
0
180
Shadow DOM & Security - Exploring the boundary between light and shadow
masatokinugawa
0
580
データ戦略部門 紹介資料
sansan33
PRO
1
3.3k
BEYOND THE RAG🚀 ~とりあえずRAG?を超えていけ! 本当に使えるAIエージェント&生成AIプロダクトを目指して~ / BEYOND-THE-RAG-Toward Practical-GenerativeAI-Products-AOAI-DevDay-2025
jnymyk
4
180
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
6
3.4k
「現場で活躍するAIエージェント」を実現するチームと開発プロセス
tkikuchi1002
5
830
P2P通信の標準化 WebRTCを知ろう
faithandbrave
6
1.8k
Digitization部 紹介資料
sansan33
PRO
1
4.6k
今だから言えるセキュリティLT_Wordpress5.7.2未満を一斉アップデートせよ
cuebic9bic
2
180
QAを早期に巻き込む”って どうやるの? モヤモヤから抜け出す実践知
moritamasami
2
140
Featured
See All Featured
YesSQL, Process and Tooling at Scale
rocio
173
14k
Making Projects Easy
brettharned
116
6.3k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Git: the NoSQL Database
bkeepers
PRO
431
65k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
We Have a Design System, Now What?
morganepeng
53
7.7k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Transcript
ࡒେՆ / Pepabo R&D Institute, GMO Pepabo, Inc. 2019.10.25 Fukuoka.go#14+Umeda.go
trinity Ͱ Cloud Composer ʹ ϫʔΫϑϩʔΛ؆୯σϓϩΠ
σʔλαΠΤϯςΟετ ࡒ େՆ / @zaimy 2 Hiroka Zaitsu ϖύϘݚڀॴ ݚڀһ
1. Cloud Composer ͱ 2. Cloud Composer ͷσϓϩΠ࣌ͷࠔΓ͝ͱ 3. trinity
ʹΑΔղܾͷࢼΈ 4. ࠓޙΔ͜ͱ 3 ࣍
1. Cloud Composer ͱ
• GCP ͷ "ϑϧϚωʔδυͷϫʔΫϑϩʔ ΦʔέετϨʔγϣϯ αʔϏε" • Apache Airflow Λ
GCP ্ʹߏங͢Δ • ϖύϘͷϩάج൫ʢDWHʣΛ Treasure Data ͔Β GCP Ҡߦத • ϫʔΫϑϩʔαʔϏε Treasure Workflow (Ϛωʔδυ Digdag) ͔Β Cloud Composer Ҡߦத 5 Cloud Composer ͷ֓ཁ
ϫʔΫϑϩʔͷίʔυϕʔε repository └ dags ɹ ├ workflowA ɹ │ ├
main.py ɹ │ └ hoge.sql ɹ └ workflowB ɹ ɹ ├ main.py ɹ ɹ └ piyo.sql 6 • dags σΟϨΫτϦԼʹϫʔΫϑϩʔ୯ҐͰ αϒσΟϨΫτϦΛΔ • ϫʔΫϑϩʔຊମʢDAGʣͷ python ίʔυ • ϫʔΫϑϩʔͰར༻͢ΔΫΤϦ • ઃఆϑΝΠϧɹͳͲ ※σΟϨΫτϦߏΛ Cloud Storage ͱ߹ΘͤΔ߹
ϫʔΫϑϩʔͷσϓϩΠʢՃͱߋ৽ʣ $ gcloud composer environments storage dags import \ --environment
ENVIRONMENT_NAME \ --location LOCATION \ --source LOCAL_FILE_TO_UPLOAD 7 ίʔυϕʔε $MPVE4UPSBHF "JSqPX HDMPVEDPNQPTFSJNQPSU
ϫʔΫϑϩʔͷআ ͦͷ1 - Cloud Storage ͔Βআ $ gcloud composer environments
storage dags delete \ --environment ENVIRONMENT_NAME \ --location LOCATION \ DAG_NAME.py 8 ίʔυϕʔε $MPVE4UPSBHF "JSqPX HDMPVEDPNQPTFSEFMFUF
ϫʔΫϑϩʔͷআ ͦͷ2 - Airflow ͔Βআ $ gcloud composer environments run
--location LOCATION \ ENVIRONMENT_NAME delete_dag -- DAG_NAME 9 ίʔυϕʔε $MPVE4UPSBHF "JSqPX HDMPVEDPNQPTFSEFMFUF@EBH
2. Cloud Composer ͷ σϓϩΠ࣌ͷࠔΓ͝ͱ
• ϫʔΫϑϩʔͷՃͱߋ৽ • import ϫʔΫϑϩʔ୯ҐͰͷ࣮ߦ • ࠩͷ͋ΔϫʔΫϑϩʔʹରͯ͠ݸผʹ࣮ߦ͢Δඞཁ͕͋Δ • import
Cloud Storage ͷϑΝΠϧΛ্ॻ͖͢Δ • ίʔυϕʔεͰআͨ͠ϑΝΠϧ ݸผʹআ͠ͳ͍ݶΓ Cloud Storage ʹΔ 11 gcloud ίϚϯυΛͦͷ··ӡ༻ʹ͏ͱେม
• ϫʔΫϑϩʔͷআ • delete ͱ Airflow ͷ dag_delete ͷ2ճίϚϯυΛ࣮ߦ͢Δඞཁ͕͋Δ •
delete ϑΝΠϧ୯Ґ, dag_delete ϫʔΫϑϩʔ୯ҐͰͷ࣮ߦ • ࠩͷ͋ΔϑΝΠϧ/ϫʔΫϑϩʔʹରͯ͠ݸผʹ࣮ߦ͢Δඞཁ͕͋Δ • ։ൃʹΑΓेݸͷϫʔΫϑϩʔʹʑ͕ࠩੜ·Ε͍ͯ͘ • ࠩΛػցతʹݕग़ͯ͠ Cloud Composer ʹಉظ͍ͨ͠ 12 gcloud ίϚϯυΛͦͷ··ӡ༻ʹ͏ͱେม
• όέοτ/σΟϨΫτϦؒͰϑΝΠϧΛಉظ͢Δ Cloud Storage ͷίϚϯυ • ϑΝΠϧͷߋ৽࣌ࠁʹࠩҟ͕͋Εಉظରͱఆ͞ΕΔ • ༰͕มߋ͞Ε͍ͯͳͯ͘ॲཧରʹͳͬͯ͠·͏ •
Cloud Storage ʹґଘ͢Δ • Airflow GCP Ҏ֎ͰߏஙͰ͖ΔͷͰଞͷετϨʔδʹରԠ͍ͨ͠ 13 gsutil rsync Ͳ͏͔ͳ
• ಛఆͷ git ϦϙδτϦͱಉظ͢Δ Airflow ͷػೳ • ୯ҰͷϒϥϯνͷΈࢦఆՄೳ • ຊ൪ڥʹ
master ͷίʔυΛಉظ͢Δʹྑͦ͞͏ • ςετڥ CI Ͱ feature branch ͷίʔυΛσϓϩΠ͍ͨ͠ 14 Airflow sync Ͳ͏͔ͳ
3. trinity ʹΑΔղܾͷࢼΈ
• ίʔυϕʔεͱ Cloud Storage ͱ Airflow ͷ3ͭΛಉظ͢Δ • ϫʔΫϑϩʔ୯ҐͰɺσΟϨΫτϦߏͱϑΝΠϧ༰͔ΒϋογϡΛܭࢉ •
͋Δ࣌ͷϫʔΫϑϩʔఆٛΛද͢ϋογϡ • ίʔυϕʔε͔Βܭࢉͨ͠ϋογϡͱ Cloud Storage ʹอଘ͞Ε͍ͯΔ ϋογϡ͕ҟͳΔϫʔΫϑϩʔΛಉظૢ࡞ͷରʹ͢Δ 16 trinity ͷํ
• https://github.com/zaimy/trinity • A tool to synchronize workflows between Codebase,
Cloud Storage and Airflow metadata. • ͳͥ Goʁ • ΫϩείϯύΠϧͰ Mac, Linux, Windows ʹରԠͰ͖Δ • ϫʔΫϑϩʔ୯ҐͰॲཧ͕ՄೳͳͷͰฒྻԽ͍ͨ͠ 17 trinity ͷ࣮ $ trinity --bucket=BUCKET_NAME \ --composer-env=COMPOSER_ENV_NAME
1. ίʔυϕʔεͰϋογϡΛܭࢉͯ͠ϫʔΫϑϩʔ͝ͱʹอଘ 2. ίʔυϕʔεͱ Cloud Storage ͷϫʔΫϑϩʔΛϦετͯ͠ൺֱ i. ίʔυϕʔεʹ͔͠ͳ͚Ε Cloud
Storage ʹΞοϓϩʔυʢՃʣ ii. Cloud Storage ʹ͔͠ͳ͚Ε Cloud Storage ͱ Airflow ͔Βআ iii. ྆ํʹ͋Είʔυϕʔεͱ Cloud Storage ͷϋογϡΛൺֱ a. ࠩҟ͕͋Ε Cloud Storage ͷϫʔΫϑϩʔΛஔʢߋ৽ʣ 18 ॲཧͷྲྀΕ
؆୯ʹಉظతͳσϓϩΠ͕ Ͱ͖ΔΑ͏ʹͳͬͨ !
• ςετՃͱϦϑΝΫλϦϯά • Go ͷ࡞๏ߟ͑ํʹԊ͍͖͍ͬͯͨ • ػೳՃ • Airflow ʹ
dags Ҏ֎ʹ plugins ͋ΔͷͰରԠ͢Δ • dry-run 20 ࠓޙΔ͜ͱ
None