Fiabiliser ses déploiements sur Kubernetes

Slide 1

Slide 1 text

1 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Bordeaux 33000 > France > www.pyxida.io Fiabiliser ses mises en production avec Kubernetes Meetup CNCF Bordeaux #5 Etienne Coutaud 4 Juin 2019

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

4 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Les déploiements sur Kubernetes apiVersion: apps/v1 kind: Deployment metadata: name: myapp labels: app: myapp spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: myapp image: myapp:v1.0.0 ports: - containerPort: 8080 [...]

Slide 5

Slide 5 text

5 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Ma mise en prod en v2.0.0 ? $> kubectl set image deployment/myapp myapp=myapp:v2.0.0 apiVersion: apps/v1 kind: Deployment metadata: name: myapp labels: app: myapp spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: myapp image: myapp:v2.0.0 ports: - containerPort: 8080 [...]

Slide 6

Slide 6 text

6 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (1/6) myapp.pyxida.io 100% -> v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0

Slide 7

Slide 7 text

7 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (2/6) myapp.pyxida.io 100% -> v1.0.0 myapp:v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 $> kubectl set image deployment/myapp myapp=myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 Check santé du pod

Slide 8

Slide 8 text

8 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (3/6) myapp.pyxida.io 67% -> v1.0.0 33% -> v2.0.0 myapp:v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 9

Slide 9 text

9 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (4/6) myapp.pyxida.io 33% -> v1.0.0 67% -> v2.0.0 myapp:v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 10

Slide 10 text

10 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (5/6) myapp.pyxida.io 100% -> v2.0.0 myapp:v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 11

Slide 11 text

11 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Le mécanisme du rolling update (6/6) myapp.pyxida.io 100% -> v2.0.0 myapp:v2.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 12

Slide 12 text

12 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Une bonne liveness et Readiness ﬁabilise ce mécanisme apiVersion: apps/v1 kind: Deployment metadata: name: myapp [...] spec: replicas: 3 [...] template: [...] spec: containers: - name: myapp image: myapp:v2.0.0 ports: - containerPort:8080 livenessProbe: httpGet: path: /healthz port: 8080 readinessProbe httpGet: path: /ready port: 8080 KO KO OK Kubernetes tente de redémarrer le pod OK OK KO Le pod sort du loadbalancing

Slide 13

Slide 13 text

13 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Généralités sur les probes Liveness et Readiness se conﬁgure de la même manière 3 façon de faire des probes ● La requête HTTP ● TCP ● Script/commande Voici un bout de code qui est une liveness :) http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) w.Write([]byte("ok")) })

Slide 14

Slide 14 text

14 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Si on résume ce mécanisme basique ● Montée de version en ZDD (Zero downtime deployment) ● Transparent pour les OPS, automatisé par Kubernetes ● Historique des replicasets nous permet de faire un rollout Ce mécanisme nous empêche cependant de pouvoir tester le comportement de notre application en amont dans les mêmes conditions

Slide 15

Slide 15 text

15 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Testons en faisant du blue/green deployment (1/3) myapp.pyxida.io 100% -> v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0

Slide 16

Slide 16 text

16 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Testons en faisant du blue/green deployment (2/3) myapp.pyxida.io 100% -> v1.0.0 0% -> v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 17

Slide 17 text

17 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Testons en faisant du blue/green deployment (3/3) myapp.pyxida.io 0% -> v1.0.0 100% -> v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 18

Slide 18 text

18 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Juste une histoire de labels (1/2) apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v1.0.0 template: metadata: labels: app: myapp version: v1.0.0 spec: containers: - name: myapp image: myapp:v1.0.0 ports: - containerPort: 8080 name: http [...] apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v2.0.0 template: metadata: labels: app: myapp version: v2.0.0 spec: containers: - name: myapp image: myapp:v2.0.0 ports: - containerPort: 8080 name: http [...] apiVersion: v1 kind: Service metadata: name: myapp spec: ports: - name: http port: 8080 protocol: TCP targetPort: http selector: app: myapp version: v1.0.0

Slide 19

Slide 19 text

19 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Juste une histoire de labels (2/2) apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v1.0.0 template: metadata: labels: app: myapp version: v1.0.0 spec: containers: - name: myapp image: myapp:v1.0.0 ports: - containerPort: 8080 name: http [...] apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v2.0.0 template: metadata: labels: app: myapp version: v2.0.0 spec: containers: - name: myapp image: myapp:v2.0.0 ports: - containerPort: 8080 name: http [...] apiVersion: v1 kind: Service metadata: name: myapp spec: ports: - name: http port: 8080 protocol: TCP targetPort: http selector: app: myapp version: v2.0.0

Slide 20

Slide 20 text

20 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Si on résume ce mécanisme ● Mise en production consiste uniquement en l’update d’un service pour faire pointer vers un nouveau label ● La nouvelle stack peut être montée et testée en amont ● Rollout simpliﬁé, il suffit de refaire pointer le service vers l’ancienne version Cependant : ● C’est raté pour le ZDD ● La nouvelle stack peut être tester sur la même infrastructure mais pas dans les mêmes conditions que la “vrai” production

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

23 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Sur Kubernetes - Méthode 1 : Avec les replicas myapp.pyxida.io 67% -> v1.0.0 33% -> v2.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0

Slide 24

Slide 24 text

24 Pyxida © 2018 - Reproduction interdite sans autorisation préalable apiVersion: v1 kind: Service metadata: name: myapp spec: ports: - name: http port: 8080 protocol: TCP targetPort: http selector: app: myapp Toujours une histoire de labels apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v1.0.0 template: metadata: labels: app: myapp version: v1.0.0 spec: containers: - name: myapp image: myapp:v1.0.0 ports: - containerPort: 8080 name: http [...] apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp version: v2.0.0 template: metadata: labels: app: myapp version: v2.0.0 spec: containers: - name: myapp image: myapp:v2.0.0 ports: - containerPort: 8080 name: http [...]

Slide 25

Slide 25 text

25 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Peu précis et gourmand en pods Version A Version B % voulu Nbr conteneurs % voulu Nbr conteneurs 99 99 1 1 85 6 15 1 70 2 30 1 50 1 1 1

Slide 26

Slide 26 text

26 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Si on résume ce mécanisme ● Mécanisme de canary release “by design” ● Permet un déploiement progressif Cependant : ● Pas très précis ● Les proportions ne sont plus bonne en cas d’autoscalabilité

Slide 27

Slide 27 text

Slide 28

Slide 28 text

28 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Un “vrai” canary release myapp.pyxida.io myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 86% 14% myapp-v1 myapp-v2

Slide 29

Slide 29 text

29 Pyxida © 2018 - Reproduction interdite sans autorisation préalable C’est là que cela se passe apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: traefik.ingress.kubernetes.io/service-weights: | myapp-v1: 86% myapp-v2: 14% name: my-app spec: rules: - http: paths: - backend: serviceName: myapp-v1 servicePort: 8080 path: / - backend: serviceName: myapp-v2 servicePort: 8080 path: /

Slide 30

Slide 30 text

30 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Si on résume ce mécanisme ● Les annotations facilitent la mise en place. ● Traeﬁk et Nginx supportent cette feature ● Contrôle ﬁn sur les pourcentages ● Plus de problème liés à la scalabilité Cependant : ● Quid des services internes qui ne sont pas exposé par une Ingress ?

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

33 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Notre archi avec Istio myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts: - myapp http: - route: - destination: host: myapp subset: v1.0.0 weight: 90 - destination: host: myapp subset: v2.0.0 weight: 10 myapp apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: myapp spec: host: myapp subsets: - name: v1.0.0 labels: version: v1.0.0 - name: v1.0.0 labels: version: v1.0.0

Slide 34

Slide 34 text

34 Pyxida © 2018 - Reproduction interdite sans autorisation préalable apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: myapp spec: host: myapp subsets: - name: v1.0.0 labels: version: v1.0.0 - name: v2.0.0 labels: version: v2.0.0 Notre archi avec Istio myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts: - myapp http: - route: - destination: host: myapp subset: v1.0.0 weight: 90 - destination: host: myapp subset: v2.0.0 weight: 10 myapp

Slide 35

Slide 35 text

35 Pyxida © 2018 - Reproduction interdite sans autorisation préalable apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: myapp spec: host: myapp subsets: - name: v1.0.0 labels: version: v1.0.0 - name: v2.0.0 labels: version: v2.0.0 Notre archi avec Istio myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts: - myapp http: - route: - destination: host: myapp subset: v1.0.0 weight: 90 - destination: host: myapp subset: v2.0.0 weight: 10 myapp Labels des pods

Slide 36

Slide 36 text

36 Pyxida © 2018 - Reproduction interdite sans autorisation préalable apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: myapp spec: host: myapp subsets: - name: v1.0.0 labels: version: v1.0.0 - name: v2.0.0 labels: version: v2.0.0 Notre archi avec Istio myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts: - myapp http: - route: - destination: host: myapp subset: v1.0.0 weight: 90 - destination: host: myapp subset: v2.0.0 weight: 10 myapp

Slide 37

Slide 37 text

37 Pyxida © 2018 - Reproduction interdite sans autorisation préalable apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: myapp spec: host: myapp subsets: - name: v1.0.0 labels: version: v1.0.0 - name: v2.0.0 labels: version: v2.0.0 Notre archi avec Istio myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts: - myapp http: - route: - destination: host: myapp subset: v1.0.0 weight: 90 - destination: host: myapp subset: v2.0.0 weight: 10 myapp 90% 10%

Slide 38

Slide 38 text

38 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Un service mesh est donc la meilleure solution ? ● Gestion fine du trafic ● Géré par des ressources externes à la base de l’architecture ● Gestion au niveau du service donc feature disponible à tout niveau Cependant : ● Complexifie l’architecture ● Plus compliqué à debugger ● Mais surtout ...

Slide 39

Slide 39 text

Slide 40

Slide 40 text

40 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Et si on le faisait nous même ? myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v1.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp:v2.0.0 myapp-v1 myapp-v2 myapp backend balance roundrobin server srv1 myapp-v1:8080 weight 90 server srv2 myapp-v2:8080 weight 10 90% 10%

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

44 Pyxida © 2018 - Reproduction interdite sans autorisation préalable Bordeaux 33000 > France > www.pyxida.io Take Away ! ● Plein de façons de gérer ses déploiements sur k8s ● Votre méthode de MEP reﬂète votre organisation ● Sans test, le blue/green ne sert à rien ● Sans métrique, oubliez le canary ● Les services mesh apportent des features … ● mais aussi de la complexité ● La meilleure des MEP est celle que vous avez décidée