Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cloud Composer 入門
Search
HIROKI OKUMURA
September 05, 2019
Technology
0
340
Cloud Composer 入門
2019/08/03(土)【埼玉】GCPUG Saitama 第二回【GCP】 GKE & Next Tokyo recap のLTの資料です。
HIROKI OKUMURA
September 05, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
Context Engineeringの取り組み
nutslove
0
410
[CV勉強会@関東 World Model 読み会] Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models (Mousakhan+, NeurIPS 2025)
abemii
0
160
Greatest Disaster Hits in Web Performance
guaca
0
310
プロダクト成長を支える開発基盤とスケールに伴う課題
yuu26
4
1.4k
Embedded SREの終わりを設計する 「なんとなく」から計画的な自立支援へ
sansantech
PRO
3
2.7k
Exadata Fleet Update
oracle4engineer
PRO
0
1.1k
10Xにおける品質保証活動の全体像と改善 #no_more_wait_for_test
nihonbuson
PRO
2
360
OpenShiftでllm-dを動かそう!
jpishikawa
0
150
pool.ntp.orgに ⾃宅サーバーで 参加してみたら...
tanyorg
0
1.8k
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
15
93k
Amazon Bedrock Knowledge Basesチャンキング解説!
aoinoguchi
0
170
GitHub Issue Templates + Coding Agentで簡単みんなでIaC/Easy IaC for Everyone with GitHub Issue Templates + Coding Agent
aeonpeople
1
290
Featured
See All Featured
svc-hook: hooking system calls on ARM64 by binary rewriting
retrage
1
110
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
37
6.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
46
2.7k
Bootstrapping a Software Product
garrettdimon
PRO
307
120k
Groundhog Day: Seeking Process in Gaming for Health
codingconduct
0
100
Raft: Consensus for Rubyists
vanstee
141
7.3k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
659
61k
jQuery: Nuts, Bolts and Bling
dougneiner
65
8.4k
Neural Spatial Audio Processing for Sound Field Analysis and Control
skoyamalab
0
180
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4.2k
Rebuilding a faster, lazier Slack
samanthasiow
85
9.4k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
130k
Transcript
Cloud Composer 入門 2019/08/03 GCPUG Saitama 第二回 @hirokiokumura
Agenda • What’s Cloud Composer ? • Operators ◦ airflow.operators
◦ airflow.contrib.operators • Demo
Cloud Composer Version List https://cloud.google.com/composer/docs/concepts/versioning/composer-versions から抜粋
Apache Airflow https://github.com/apache/airflow から抜粋
airflow.operators https://github.com/apache/airflow/tree/master/airflow/operators から抜粋
airflow.contrib.operators https://github.com/apache/airflow/tree/master/airflow/contrib/operators から抜粋
Cloud Composer
Kubernetes Engine
Private IP Cloud Composer : Disable Endpoint
Private IP Cloud Composer : Nat Gateway
Private IP Cloud Composer : Node pool
UI
Connect to External API Server On-Premises Cloud NAT Cloud Composer
Web Server Firewall
BashOperator
Graph View : BashOperator
Log
BashOperator & PythonOperator
Graph View : BashOperator & PythonOperator
Use container image Cloud Composer Cloud Build Container Registry Logging
KubernetesPodOperator
Graph View : KubernetesPodOperator
Stackdriver Logging
Data Mart Cloud Composer BigQuery Cloud Pub/Sub
Graph View : BigQueryOperator & PubSubPublishOperator
Data Pipeline Cloud Composer BigQuery Cloud Storage Cloud Dataflow Cloud
Functions Cloud Pub/Sub
Graph View : BashOperator & DataflowTemplateOperator
Demo
まとめ • プライベートなCloud Composer環境の構成で運用できる • GCPのマネージドサービスを連携するOperatorが用意されている ◦ GKEのマスターエンドポイントが有効であればkubectlコマンド 利用できる •
DWHを構築するなどワークフローが必要なケースでは便利