Tweet
Tweet

Slide 1

Slide 1 text

Airflow ❤ Kubernetes Tommy Nguyen CKA & CKAD Senior DevOps @ 90 Seconds https://tuannvm.com

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Agenda I. 90s introduction II. Airflow? III. Airflow @ 90s? IV. Airflow on k8s? V. Difficulties VI. Solutions VII. Bonus VIII. Q & A IX. Demo

Slide 4

Slide 4 text

I. 90 Seconds Introduction

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Cloud Video Creation Platform

Slide 7

Slide 7 text

II. What is airflow?

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

III. Airflow @ 90 Seconds, why?

Slide 10

Slide 10 text

We aim to be a platform, not a traditional video company

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Data

Slide 13

Slide 13 text

DevOps

Slide 14

Slide 14 text

Platform

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

IV. Why K8S? Not something else?

Slide 18

Slide 18 text

Because We are Kubernetes enthusiast

Slide 19

Slide 19 text

Simplify the installation

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Utilize the resources

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Scalability

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

The big picture

Slide 26

Slide 26 text

Previous Setup

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

V. But...

Slide 29

Slide 29 text

Where to store secrets?

Slide 30

Slide 30 text

How to do realtime update without restart?

Slide 31

Slide 31 text

How to scale horizontally?

Slide 32

Slide 32 text

VI. Solutions

Slide 33

Slide 33 text

Where to store secrets? K8S built-in secret? Give everyone access to cluster?

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

To the rescue

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

https://github.com/hvac/hvac

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

How to do realtime update without restart? Git Sync

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

How to scale horizontally? Kubernetes Executor

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

VII. Lesson learned?

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Bonus

Slide 51

Slide 51 text

Kubernetes Operator

Slide 52

Slide 52 text

Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

Slide 53

Slide 53 text

TODO

Slide 54

Slide 54 text

GPU node pool

Slide 55

Slide 55 text

Reference - https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

We are hiring! https://90seconds.com.sg/about/careers/singapore/ [email protected]