Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Airflow ❤️ Kubernetes
Search
Tommy Nguyen
May 16, 2019
Technology
0
210
Airflow ❤️ Kubernetes
How Airflow is used @ 90 Seconds
Tommy Nguyen
May 16, 2019
Tweet
Share
More Decks by Tommy Nguyen
See All by Tommy Nguyen
Golden Images and How To Create Them
tuannvm
0
76
Terraform ❤️ Vungle
tuannvm
0
90
Istio The Simple Way
tuannvm
0
120
Other Decks in Technology
See All in Technology
GrafanaMeetup_AmazonManagedGrafanaのアクセス制御機能とマルチテナント環境下でのアクセス制御について
daitak
0
390
Grafana x PagerDuty Better Together
jacopen
1
250
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
7
1.3k
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
15
35k
開発生産性大幅アップ!Postman VS Code拡張機能
nagix
3
630
Cloud Service Mesh に触れ合う
phaya72
1
160
【基本】データベース設計
oracle4engineer
PRO
2
160
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
1k
How to Lead? Testimonial of a Lead Android Engineer
oleur
1
110
Rustで「プリズモイダル法」を利用して「土量計算」をガチでやる
nokonoko1203
1
280
よく聞くけど使ったことないソフトウェアNo.1 KafkaとSnowflake
foursue
4
500
Handling focus in 2024
tahia910
0
220
Featured
See All Featured
Dealing with People You Can't Stand - Big Design 2015
cassininazir
358
22k
Building Applications with DynamoDB
mza
88
5.6k
How to name files
jennybc
65
93k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Stop Working from a Prison Cell
hatefulcrawdad
267
19k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
Designing for Performance
lara
602
67k
No one is an island. Learnings from fostering a developers community.
thoeni
16
2.1k
Designing Experiences People Love
moore
136
23k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
188
16k
Building Effective Engineering Teams - LeadDev
addyosmani
32
1.9k
Gamification - CAS2011
davidbonilla
77
4.6k
Transcript
Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps
@ 90 Seconds https://tuannvm.com
None
Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s?
IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo
I. 90 Seconds Introduction
None
Cloud Video Creation Platform
II. What is airflow?
None
III. Airflow @ 90 Seconds, why?
We aim to be a platform, not a traditional video
company
None
Data
DevOps
Platform
None
None
IV. Why K8S? Not something else?
Because We are Kubernetes enthusiast
Simplify the installation
None
Utilize the resources
None
Scalability
None
The big picture
Previous Setup
None
V. But...
Where to store secrets?
How to do realtime update without restart?
How to scale horizontally?
VI. Solutions
Where to store secrets? K8S built-in secret? Give everyone access
to cluster?
None
To the rescue
None
None
None
https://github.com/hvac/hvac
None
None
How to do realtime update without restart? Git Sync
None
How to scale horizontally? Kubernetes Executor
None
VII. Lesson learned?
None
None
None
Bonus
Kubernetes Operator
Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753
TODO
GPU node pool
Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/
None
None
None
We are hiring! https://90seconds.com.sg/about/careers/singapore/
[email protected]