Airflow ❤️ Kubernetes

How Airflow is used @ 90 Seconds

Transcript

1. Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps @ 90 Seconds https://tuannvm.com
3. Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s? IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo
4. I. 90 Seconds Introduction
6. Cloud Video Creation Platform
7. II. What is airflow?
9. III. Airflow @ 90 Seconds, why?
10. We aim to be a platform, not a traditional video company
12. Data
13. DevOps
14. Platform
17. IV. Why K8S? Not something else?
18. Because We are Kubernetes enthusiast
19. Simplify the installation
21. Utilize the resources
23. Scalability
25. The big picture
26. Previous Setup
28. V. But...
29. Where to store secrets?
30. How to do realtime update without restart?
31. How to scale horizontally?
32. VI. Solutions
33. Where to store secrets? K8S built-in secret? Give everyone access to cluster?
35. To the rescue
39. https://github.com/hvac/hvac
42. How to do realtime update without restart? Git Sync
44. How to scale horizontally? Kubernetes Executor
46. VII. Lesson learned?
50. Bonus
51. Kubernetes Operator
52. Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753
53. TODO
54. GPU node pool
55. Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/
59. We are hiring! https://90seconds.com.sg/about/careers/singapore/ tommy@90seconds.tv