Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Airflow ❤️ Kubernetes
Search
Tommy Nguyen
May 16, 2019
Technology
0
230
Airflow ❤️ Kubernetes
How Airflow is used @ 90 Seconds
Tommy Nguyen
May 16, 2019
Tweet
Share
More Decks by Tommy Nguyen
See All by Tommy Nguyen
Golden Images and How To Create Them
tuannvm
0
85
Terraform ❤️ Vungle
tuannvm
0
97
Istio The Simple Way
tuannvm
0
130
Other Decks in Technology
See All in Technology
公開初日に Gemini CLI を試した話や FFmpeg と組み合わせてみた話など / Gemini CLI 初学者勉強会(#AI道場)
you
PRO
0
1.4k
Bill One 開発エンジニア 紹介資料
sansan33
PRO
4
13k
Contract One Engineering Unit 紹介資料
sansan33
PRO
0
6.9k
全部AI、全員Cursor、ドキュメント駆動開発 〜DevinやGeminiも添えて〜
rinchsan
10
5.2k
20250718_ITSurf_“Bet AI”を支える文化とコストマネジメント
helosshi
0
110
AI時代にも変わらぬ価値を発揮したい: インフラ・クラウドを切り口にユーザー価値と非機能要件に向き合ってエンジニアとしての地力を培う
netmarkjp
0
140
AWS 怖い話 WAF編 @fillz_noh #AWSStartup #AWSStartup_Kansai
fillznoh
0
130
(HackFes)米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
5
420
衛星運用をソフトウェアエンジニアに依頼したときにできあがるもの
sankichi92
1
1.1k
AIでテストプロセス自動化に挑戦する
sakatakazunori
1
560
Microsoft Defender XDRで疲弊しないためのインシデント対応
sophiakunii
2
330
Figma Dev Mode MCP Serverを用いたUI開発
zoothezoo
0
250
Featured
See All Featured
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Automating Front-end Workflow
addyosmani
1370
200k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Docker and Python
trallard
45
3.5k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
How to Ace a Technical Interview
jacobian
278
23k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Unsuck your backbone
ammeep
671
58k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.2k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Transcript
Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps
@ 90 Seconds https://tuannvm.com
None
Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s?
IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo
I. 90 Seconds Introduction
None
Cloud Video Creation Platform
II. What is airflow?
None
III. Airflow @ 90 Seconds, why?
We aim to be a platform, not a traditional video
company
None
Data
DevOps
Platform
None
None
IV. Why K8S? Not something else?
Because We are Kubernetes enthusiast
Simplify the installation
None
Utilize the resources
None
Scalability
None
The big picture
Previous Setup
None
V. But...
Where to store secrets?
How to do realtime update without restart?
How to scale horizontally?
VI. Solutions
Where to store secrets? K8S built-in secret? Give everyone access
to cluster?
None
To the rescue
None
None
None
https://github.com/hvac/hvac
None
None
How to do realtime update without restart? Git Sync
None
How to scale horizontally? Kubernetes Executor
None
VII. Lesson learned?
None
None
None
Bonus
Kubernetes Operator
Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753
TODO
GPU node pool
Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/
None
None
None
We are hiring! https://90seconds.com.sg/about/careers/singapore/ tommy@90seconds.tv