Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Composer 入門

Avatar for HIROKI OKUMURA HIROKI OKUMURA
September 05, 2019
Technology
0
340

Cloud Composer 入門

2019/08/03(土)【埼玉】GCPUG Saitama 第二回【GCP】 GKE & Next Tokyo recap のLTの資料です。

Avatar for HIROKI OKUMURA

HIROKI OKUMURA

September 05, 2019
Tweet

Other Decks in Technology

See All in Technology
Context Engineeringの取り組み
Avatar for nutslove nutslove
0
410
[CV勉強会@関東 World Model 読み会] Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models (Mousakhan+, NeurIPS 2025)
Avatar for abemii_ abemii
0
160
Greatest Disaster Hits in Web Performance
Avatar for Estela Franco guaca
0
310
プロダクト成長を支える開発基盤とスケールに伴う課題
Avatar for yuu26 yuu26
4
1.4k
Embedded SREの終わりを設計する　「なんとなく」から計画的な自立支援へ
Avatar for SansanTech sansantech
PRO
3
2.7k
Exadata Fleet Update
Avatar for oracle4engineer oracle4engineer
PRO
0
1.1k
10Xにおける品質保証活動の全体像と改善 #no_more_wait_for_test
Avatar for nihonbuson nihonbuson
PRO
2
360
OpenShiftでllm-dを動かそう！
Avatar for jpishikawa jpishikawa
0
150
pool.ntp.orgに ⾃宅サーバーで 参加してみたら...
Avatar for Fuminori -Tany- Tanizaki(谷崎文義) tanyorg
0
1.8k
Oracle Base Database Service 技術詳細
Avatar for oracle4engineer oracle4engineer
PRO
15
93k
Amazon Bedrock Knowledge Basesチャンキング解説！
Avatar for 野口碧生 aoinoguchi
0
170
GitHub Issue Templates + Coding Agentで簡単みんなでIaC/Easy IaC for Everyone with GitHub Issue Templates + Coding Agent
Avatar for AEON aeonpeople
1
290

Featured

See All Featured
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
1
110
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.3k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.7k
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
120k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
100
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
141
7.3k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
61k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
65
8.4k
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
180
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.2k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.4k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k

Transcript

  1. Cloud Composer 入門 2019/08/03 GCPUG Saitama 第二回 @hirokiokumura

  2. Agenda • What’s Cloud Composer ? • Operators ◦ airflow.operators

    ◦ airflow.contrib.operators • Demo

  3. Cloud Composer Version List https://cloud.google.com/composer/docs/concepts/versioning/composer-versions から抜粋

  4. Apache Airflow https://github.com/apache/airflow　から抜粋

  5. airflow.operators https://github.com/apache/airflow/tree/master/airflow/operators　から抜粋

  6. airflow.contrib.operators https://github.com/apache/airflow/tree/master/airflow/contrib/operators　から抜粋

  7. Cloud Composer

  8. Kubernetes Engine

  9. Private IP Cloud Composer : Disable Endpoint

  10. Private IP Cloud Composer : Nat Gateway

  11. Private IP Cloud Composer : Node pool

  12. UI

  13. Connect to External API Server On-Premises Cloud NAT Cloud Composer

    Web Server Firewall

  14. BashOperator

  15. Graph View : BashOperator

  16. Log

  17. BashOperator & PythonOperator

  18. Graph View : BashOperator & PythonOperator

  19. Use container image Cloud Composer Cloud Build Container Registry Logging

  20. KubernetesPodOperator

  21. Graph View : KubernetesPodOperator

  22. Stackdriver Logging

  23. Data Mart Cloud Composer BigQuery Cloud Pub/Sub

  24. Graph View : BigQueryOperator & PubSubPublishOperator

  25. Data Pipeline Cloud Composer BigQuery Cloud Storage Cloud Dataflow Cloud

    Functions Cloud Pub/Sub

  26. Graph View : BashOperator & DataflowTemplateOperator

  27. Demo

  28. まとめ • プライベートなCloud Composer環境の構成で運用できる • GCPのマネージドサービスを連携するOperatorが用意されている ◦ GKEのマスターエンドポイントが有効であればkubectlコマンド 利用できる •

    DWHを構築するなどワークフローが必要なケースでは便利