Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Airflowのチュートリアルやってみた
Search
iwamot
June 30, 2023
Technology
0
150
Airflowのチュートリアルやってみた
2023-06-30
ENECHANGE Tech Talk(社内勉強会)
iwamot
June 30, 2023
Tweet
Share
More Decks by iwamot
See All by iwamot
ECS on FargateへのSeekable OCI導入レポート
iwamot
0
340
サービスクォータ、ちゃんと監視してますか?
iwamot
0
620
AWS Well-Architected Toolで信頼性をレビューした結果
iwamot
0
320
テックカンパニーとしてのENECHANGEの中期目標
iwamot
0
54
AIの活用状況と今後の展望
iwamot
0
48
ENECHANGE社のAWSセキュリティ改革ストーリー
iwamot
0
120
ALBのアクセスログをAthenaで分析してSLOをゆるく決める
iwamot
2
1k
ChatGPT in SlackでAI Slackbotを楽しく運用する
iwamot
1
340
Function callingによるSlackbotの機能拡張
iwamot
2
150
Other Decks in Technology
See All in Technology
EM完全に理解した と思ったけど、 やっぱり何も分からなかった話 / EM Night Fukuoka #1
hirutas
0
130
Android Target SDK 35 (Android 15) 対応の概要
akkie76
0
130
Kernel MemoryでAzure OpenAI Serviceとお手軽データソース連携
mitsuzono
1
270
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
Cracking the KubeCon CfP
inductor
2
260
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
320
一生覚えておきたい「システム開発=コミュニケーション」〜初めての実務案件振り返りLT〜
maimyyym
2
240
Grafana x PagerDuty Better Together
jacopen
1
200
EMとして2023年度に頑張ったこと / What we did well in FY2023 as a EM
pauli
1
190
Improve Your Development Workflow with Gemini Code Assist
meteatamel
0
120
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
6
1.3k
require(ESM)とECMAScript仕様
uhyo
4
910
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
It's Worth the Effort
3n
180
27k
In The Pink: A Labor of Love
frogandcode
138
21k
Done Done
chrislema
178
15k
Atom: Resistance is Futile
akmur
260
25k
Building Better People: How to give real-time feedback that sticks.
wjessup
356
18k
Happy Clients
brianwarren
92
6.4k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
How to name files
jennybc
65
93k
Teambox: Starting and Learning
jrom
128
8.4k
Building Your Own Lightsaber
phodgson
100
5.7k
A Philosophy of Restraint
colly
197
16k
Transcript
Airflowのチュートリアルやってみた 2023-06-30 ENECHANGE Tech Talk(社内勉強会) CTO室 岩本隆史
Airflow案件に関わることに
よい機会なのでMWAAを試したい https://aws.amazon.com/jp/managed-workflows-for-apache-airflow/
チュートリアルをやってみよう https://docs.aws.amazon.com/mwaa/latest/userguide/quick-start.html
めっちゃ時間かかった… https://docs.aws.amazon.com/mwaa/latest/userguide/quick-start.html#quick-start- createstack
Dockerだと数分で構築完了 curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.6.2/docker-compose.yaml' mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env docker compose up airflow-init docker compose up https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/
サンプルDAGも豊富
tutorial DAGを実行
成功
3つのタスク
タスク1=日付の出力 t1 = BashOperator( task_id="print_date", bash_command="date", ) [2023-06-22, 06:52:22 UTC]
{subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'date'] [2023-06-22, 06:52:22 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:22 UTC] {subprocess.py:93} INFO - Thu Jun 22 06:52:22 UTC 2023 [2023-06-22, 06:52:22 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク2=スリープ t2 = BashOperator( task_id="sleep", depends_on_past=False, bash_command="sleep 5", retries=3, )
[2023-06-22, 06:52:25 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'sleep 5'] [2023-06-22, 06:52:25 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:30 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク3=テンプレートの利用 templated_command = dedent( """ {% for i in range(5)
%} echo "{{ ds }}" echo "{{ macros.ds_add(ds, 7)}}" {% endfor %} """ ) t3 = BashOperator( task_id="templated", depends_on_past=False, bash_command=templated_command, )
10個のechoにレンダリング echo "2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29" echo
"2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29" echo "2023-06-22" echo "2023-06-29"
10個の日付が出力 [2023-06-22, 06:52:25 UTC] {subprocess.py:86} INFO - Output: [2023-06-22, 06:52:25
UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-22 [2023-06-22, 06:52:25 UTC] {subprocess.py:93} INFO - 2023-06-29 [2023-06-22, 06:52:25 UTC] {subprocess.py:97} INFO - Command exited with return code 0
タスク依存関係は演算子で指定 t1 >> [t2, t3]
別のチュートリアルも実行
Extract @task() def extract(): data_string = '{"1001": 301.27, "1002": 433.21,
"1003": 502.22}' order_data_dict = json.loads(data_string) return order_data_dict Key Value return_value {'1001': 301.27, '1002': 433.21, '1003': 502.22}
Transform @task(multiple_outputs=True) def transform(order_data_dict: dict): total_order_value = 0 for value
in order_data_dict.values(): total_order_value += value return {"total_order_value": total_order_value} Key Value total_order_value 1236.7 return_value {'total_order_value': 1236.7}
Load @task() def load(total_order_value: float): print(f"Total order value is: {total_order_value:.2f}")
[2023-06-22, 07:55:00 UTC] {logging_mixin.py:149} INFO - Total order value is: 1236.70
タスク依存関係は自動解決 order_data = extract() order_summary = transform(order_data) load(order_summary["total_order_value"])
実はAirflow 2.0の新機能 @task def hello_name(name: str): print(f'Hello {name}!') hello_name('Airflow users')
Dockerで気軽に試そう