Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Airflow ❤️ Kubernetes

Avatar for Tommy Nguyen Tommy Nguyen
May 16, 2019
Technology
0
240

Airflow ❤️ Kubernetes

How Airflow is used @ 90 Seconds

Avatar for Tommy Nguyen

Tommy Nguyen

May 16, 2019
Tweet

More Decks by Tommy Nguyen

See All by Tommy Nguyen
Golden Images and How To Create Them
Avatar for Tommy Nguyen tuannvm
0
94
Terraform ❤️ Vungle
Avatar for Tommy Nguyen tuannvm
0
100
Istio The Simple Way
Avatar for Tommy Nguyen tuannvm
0
140

Other Decks in Technology

See All in Technology
Cortex Code CLI と一緒に進めるAgentic Data Engineering
Avatar for あれ __allllllllez__
0
590
事例から紐解くSHIFT流QA支援 ～大規模プロジェクトの品質管理支援、QA組織立ち上げ～ / 20260320 Nozomu Koketsu
Avatar for SHIFT EVOLVE shift_evolve
PRO
0
120
TypeScript 7.0の現在地と備え方
Avatar for uhyo uhyo
7
2k
Mitigating geopolitical risks with local-first software and atproto
Avatar for Martin Kleppmann ept
0
160
FastMCP OAuth Proxy with Cognito
Avatar for iganin hironobuiga
3
120
LINEヤフーにおけるAIOpsの現在地
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
4
1.9k
Zero Data Loss Autonomous Recovery Service サービス概要
Avatar for oracle4engineer oracle4engineer
PRO
3
13k
バクラク最古参プロダクトで重ねた技術投資を振り返る
Avatar for ypresto ypresto
0
210
Kiro Meetup #7 Kiro アップデート (2025/12/15〜2026/3/20)
Avatar for Katz Ueno katzueno
2
200
「お金で解決」が全てではない！大規模WebアプリのCI高速化 #phperkaigi
Avatar for すてにゃん stefafafan
4
2.1k
生成AIで速度と品質を両立する、QAエンジニア・開発者連携のAI協調型テストプロセス
Avatar for butatamasoba / Shota Kusaba shota_kusaba
0
350
Phase07_実務適用
Avatar for overflow ,Inc overflowinc
0
1k

Featured

See All Featured
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
260
Designing Experiences People Love
Avatar for Jonathan Moore moore
143
24k
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
220
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
273
27k
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
710
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.2k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
528
40k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
24k
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
Avatar for Kenny Baas-Schwegler baasie
0
490
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.4k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
300
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.8k

Transcript

  1. Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps

    @ 90 Seconds https://tuannvm.com
  2. None

  3. Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s?

    IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo

  4. I. 90 Seconds Introduction

  5. None

  6. Cloud Video Creation Platform

  7. II. What is airflow?

  8. None

  9. III. Airflow @ 90 Seconds, why?

  10. We aim to be a platform, not a traditional video

    company
  11. None

  12. Data

  13. DevOps

  14. Platform

  15. None
  16. None

  17. IV. Why K8S? Not something else?

  18. Because We are Kubernetes enthusiast

  19. Simplify the installation

  20. None

  21. Utilize the resources

  22. None

  23. Scalability

  24. None

  25. The big picture

  26. Previous Setup

  27. None

  28. V. But...

  29. Where to store secrets?

  30. How to do realtime update without restart?

  31. How to scale horizontally?

  32. VI. Solutions

  33. Where to store secrets? K8S built-in secret? Give everyone access

    to cluster?
  34. None

  35. To the rescue

  36. None
  37. None
  38. None

  39. https://github.com/hvac/hvac

  40. None
  41. None

  42. How to do realtime update without restart? Git Sync

  43. None

  44. How to scale horizontally? Kubernetes Executor

  45. None

  46. VII. Lesson learned?

  47. None
  48. None
  49. None

  50. Bonus

  51. Kubernetes Operator

  52. Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

  53. TODO

  54. GPU node pool

  55. Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/

  56. None
  57. None
  58. None

  59. We are hiring! https://90seconds.com.sg/about/careers/singapore/ [email protected]