Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Airflowのチュートリアルやってみた
Search
iwamot
PRO
June 30, 2023
Technology
0
390
Airflowのチュートリアルやってみた
2023-06-30
ENECHANGE Tech Talk(社内勉強会)
iwamot
PRO
June 30, 2023
Tweet
Share
More Decks by iwamot
See All by iwamot
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
220
2年でここまで成長!AWSで育てたAI Slack botの軌跡
iwamot
PRO
4
930
名単体テスト 禁断の傀儡(モック)
iwamot
PRO
1
470
クォータ監視、AWS Organizations環境でも楽勝です✌️
iwamot
PRO
2
470
Cline、めっちゃ便利、お金が飛ぶ💸
iwamot
PRO
22
21k
開発組織を進化させる!AWSで実践するチームトポロジー
iwamot
PRO
3
1.2k
始めないともったいない!SLO運用で得られる3つのメリット
iwamot
PRO
1
150
あなたの人生も変わるかも?AWS認定2つで始まったウソみたいな話
iwamot
PRO
3
8k
効率的な技術組織が作れる!書籍『チームトポロジー』要点まとめ
iwamot
PRO
2
370
Other Decks in Technology
See All in Technology
Getting to Know Your Legacy (System) with AI-Driven Software Archeology (WeAreDevelopers World Congress 2025)
feststelltaste
1
180
ソフトウェアテストのAI活用_ver1.25
fumisuke
1
560
Copilot coding agentにベットしたいCTOが開発組織で取り組んだこと / GitHub Copilot coding agent in Team
tnir
0
150
cdk initで生成されるあのファイル達は何なのか/cdk-init-generated-files
tomoki10
1
540
話題の MCP と巡る OCI RAG ソリューションの旅 - Select AI with RAG と Generative AI Agents ディープダイブ
oracle4engineer
PRO
5
110
事例で学ぶ!B2B SaaSにおけるSREの実践例/SRE for B2B SaaS: A Real-World Case Study
bitkey
1
340
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
340
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
280
SRE不在の開発チームが障害対応と 向き合った100日間 / 100 days dealing with issues without SREs
shin1988
2
1.5k
Delegating the chores of authenticating users to Keycloak
ahus1
0
180
AWS CDKの仕組み / how-aws-cdk-works
gotok365
10
890
「Chatwork」のEKS環境を支えるhelmfileを使用したマニフェスト管理術
hanayo04
1
240
Featured
See All Featured
RailsConf 2023
tenderlove
30
1.1k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Adopting Sorbet at Scale
ufuk
77
9.5k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Rails Girls Zürich Keynote
gr2m
95
14k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Designing Experiences People Love
moore
142
24k
Site-Speed That Sticks
csswizardry
10
700
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
BBQ
matthewcrist
89
9.7k
Transcript
Airflowのチュートリアルやってみた 2023-06-30 ENECHANGE Tech Talk(社内勉強会) CTO室 岩本隆史
Airflow案件に関わることに
よい機会なのでMWAAを試したい https://aws.amazon.com/jp/managed-workflows-for-apache-airflow/
チュートリアルをやってみよう https://docs.aws.amazon.com/mwaa/latest/userguide/quick-start.html
めっちゃ時間かかった… https://docs.aws.amazon.com/mwaa/latest/userguide/quick-start.html#quick-start- createstack
Dockerだと数分で構築完了 curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.6.2/docker-compose.yaml' mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env docker compose up airflow-init docker compose up https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/
サンプルDAGも豊富
tutorial DAGを実行
成功
3つのタスク
タスク1=日付の出力 t1 = BashOperator( task_id="print_date", bash_command="date", ) [2023-06-22, 06:52:22 UTC]
{subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'date'] [2023-06-22, 06:52:22 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:22 UTC] {subprocess.py:93} INFO - Thu Jun 22 06:52:22 UTC 2023 [2023-06-22, 06:52:22 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク2=スリープ t2 = BashOperator( task_id="sleep", depends_on_past=False, bash_command="sleep 5", retries=3, )
[2023-06-22, 06:52:25 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'sleep 5'] [2023-06-22, 06:52:25 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:30 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク3=テンプレートの利用 templated_command = dedent( """ {% for i in range(5)
%} echo "{{ ds }}" echo "{{ macros.ds_add(ds, 7)}}" {% endfor %} """ ) t3 = BashOperator( task_id="templated", depends_on_past=False, bash_command=templated_command, )
10個のechoにレンダリング echo "2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29" echo
"2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29"
10個の日付が出力 [2023-06-22, 06:52:25 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:25
UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク依存関係は演算子で指定 t1 >> [t2, t3]
別のチュートリアルも実行
Extract @task() def extract(): data_string = '{"1001": 301.27, "1002": 433.21,
"1003": 502.22}' order_data_dict = json.loads(data_string) return order_data_dict Key Value return_value {'1001': 301.27, '1002': 433.21, '1003': 502.22}
Transform @task(multiple_outputs=True) def transform(order_data_dict: dict): total_order_value = 0 for value
in order_data_dict.values(): total_order_value += value return {"total_order_value": total_order_value} Key Value total_order_value 1236.7 return_value {'total_order_value': 1236.7}
Load @task() def load(total_order_value: float): print(f"Total order value is: {total_order_value:.2f}")
[2023-06-22, 07:55:00 UTC] {logging_mixin.py:149} INFO - Total order value is: 1236.70
タスク依存関係は自動解決 order_data = extract() order_summary = transform(order_data) load(order_summary["total_order_value"])
実はAirflow 2.0の新機能 @task def hello_name(name: str): print(f'Hello {name}!') hello_name('Airflow users')
Dockerで気軽に試そう