Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Airflow ❤️ Kubernetes

Avatar for Tommy Nguyen Tommy Nguyen
May 16, 2019
Technology
250
0

Airflow ❤️ Kubernetes

How Airflow is used @ 90 Seconds

Avatar for Tommy Nguyen

Tommy Nguyen

May 16, 2019

More Decks by Tommy Nguyen

See All by Tommy Nguyen
Golden Images and How To Create Them
Avatar for Tommy Nguyen tuannvm
0
110
Terraform ❤️ Vungle
Avatar for Tommy Nguyen tuannvm
0
120
Istio The Simple Way
Avatar for Tommy Nguyen tuannvm
0
140

Other Decks in Technology

See All in Technology
AI駆動で「他アプリの音を加工できる」アプリを作った
Avatar for Masahiro satomasahiro2005
0
150
AI × アーキテクト 第1・2章｜生成AIの課題とアーキテクトの素養
Avatar for Daisuke Motoki daidaidai
0
170
Browser Spec: Testing Hotwire with Ruby
Avatar for Masayoshi Takahashi takahashim
1
120
触りたくないデータに、どう触るか AI時代のRBACアクセスガバナンス
Avatar for 株式会社IVRy（社員登壇資料） ivry_presentationmaterials
0
260
DWH の限界突破！ AI エージェント向け爆速リアルタイムデータ基盤 ClickHouse
Avatar for jozono jozono
1
470
ストレスなくAIと付き合いたい！こんなときどうしてる？
Avatar for Nami Kimura namiki64
3
490
「Docker Desktop不要の時代」は 来たのか？
Avatar for ueponx ueponx
0
770
Running Kotlin + Quarkus with Lambda
Avatar for Nealle nealle
1
3.7k
大型犬好きによるPugの紹介
Avatar for yamanoku yamanoku
1
160
ブラウザとサーバーから考えるAI時代のフロントエンド監視
Avatar for Shogo Fukami shogo4131
3
940
Metadata - Section 1: HTML Basics
Avatar for mohab soliman mohab1
0
120
分析の民主化を支えるメタデータ整備の実践
Avatar for KAKEHASHI kakehashi
PRO
4
1.4k

Featured

See All Featured
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
56k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
520
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
7.1k
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
2
950
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
184
10k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
330
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6.3k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.2k
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
3.2k
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
2
2.9k
A Guide to Academic Writing Using Generative AI - A Workshop
Avatar for Kenji Saito ks91
PRO
1
520

Transcript

  1. Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps

    @ 90 Seconds https://tuannvm.com
  2. None

  3. Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s?

    IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo

  4. I. 90 Seconds Introduction

  5. None

  6. Cloud Video Creation Platform

  7. II. What is airflow?

  8. None

  9. III. Airflow @ 90 Seconds, why?

  10. We aim to be a platform, not a traditional video

    company
  11. None

  12. Data

  13. DevOps

  14. Platform

  15. None
  16. None

  17. IV. Why K8S? Not something else?

  18. Because We are Kubernetes enthusiast

  19. Simplify the installation

  20. None

  21. Utilize the resources

  22. None

  23. Scalability

  24. None

  25. The big picture

  26. Previous Setup

  27. None

  28. V. But...

  29. Where to store secrets?

  30. How to do realtime update without restart?

  31. How to scale horizontally?

  32. VI. Solutions

  33. Where to store secrets? K8S built-in secret? Give everyone access

    to cluster?
  34. None

  35. To the rescue

  36. None
  37. None
  38. None

  39. https://github.com/hvac/hvac

  40. None
  41. None

  42. How to do realtime update without restart? Git Sync

  43. None

  44. How to scale horizontally? Kubernetes Executor

  45. None

  46. VII. Lesson learned?

  47. None
  48. None
  49. None

  50. Bonus

  51. Kubernetes Operator

  52. Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

  53. TODO

  54. GPU node pool

  55. Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/

  56. None
  57. None
  58. None

  59. We are hiring! https://90seconds.com.sg/about/careers/singapore/ [email protected]