Cloud Composer 入門

2019/08/03(土)【埼玉】GCPUG Saitama 第二回【GCP】 GKE & Next Tokyo recap のLTの資料です。

Transcript

1. Cloud Composer 入門 2019/08/03 GCPUG Saitama 第二回 @hirokiokumura
2. Agenda • What’s Cloud Composer ? • Operators ◦ airflow.operators ◦ airflow.contrib.operators • Demo
3. Cloud Composer Version List https://cloud.google.com/composer/docs/concepts/versioning/composer-versions から抜粋
4. Apache Airflow https://github.com/apache/airflow　から抜粋
5. airflow.operators https://github.com/apache/airflow/tree/master/airflow/operators　から抜粋
6. airflow.contrib.operators https://github.com/apache/airflow/tree/master/airflow/contrib/operators　から抜粋
7. Cloud Composer
8. Kubernetes Engine
9. Private IP Cloud Composer : Disable Endpoint
10. Private IP Cloud Composer : Nat Gateway
11. Private IP Cloud Composer : Node pool
12. UI
13. Connect to External API Server On-Premises Cloud NAT Cloud Composer Web Server Firewall
14. BashOperator
15. Graph View : BashOperator
16. Log
17. BashOperator & PythonOperator
18. Graph View : BashOperator & PythonOperator
19. Use container image Cloud Composer Cloud Build Container Registry Logging
20. KubernetesPodOperator
21. Graph View : KubernetesPodOperator
22. Stackdriver Logging
23. Data Mart Cloud Composer BigQuery Cloud Pub/Sub
24. Graph View : BigQueryOperator & PubSubPublishOperator
25. Data Pipeline Cloud Composer BigQuery Cloud Storage Cloud Dataflow Cloud Functions Cloud Pub/Sub
26. Graph View : BashOperator & DataflowTemplateOperator
27. Demo
28. まとめ • プライベートなCloud Composer環境の構成で運用できる • GCPのマネージドサービスを連携するOperatorが用意されている ◦ GKEのマスターエンドポイントが有効であればkubectlコマンド 利用できる • DWHを構築するなどワークフローが必要なケースでは便利