Moving from Mesos to Kubernetes without anyone noticing

Slide 1

Slide 1 text

Moving from Mesos to Kubernetes without anyone noticing* Anubhav Mishra

Slide 2

Slide 2 text

Anubhav Mishra @anubhavm

Slide 3

Slide 3 text

Anubhav Mishra @anubhavm

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Anubhav Mishra @anubhavm

Slide 6

Slide 6 text

Anubhav Mishra @anubhavm Atlantis

Slide 7

Slide 7 text

Anubhav Mishra @anubhavm Atlantis

Slide 8

Slide 8 text

Anubhav Mishra @anubhavm Atlantis

Slide 9

Slide 9 text

Anubhav Mishra @anubhavm Atlantis

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Agenda ● Hootsuite’s Journey from Mesos to Kubernetes ● Microservices pipeline ○ Mesos and Marathon ○ Kubernetes ● Migration without major disruption ● Live demo! ● Lessons learned/Conclusion

Slide 13

Slide 13 text

Hootsuite Now

Slide 14

Slide 14 text

Numbers ● 120+ developers ● 60+ microservices ● 2 cluster schedulers ● 1500+ servers on AWS

Slide 15

Slide 15 text

2014

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

I want to build a microservice

Slide 18

Slide 18 text

I want to build a microservice

Slide 19

Slide 19 text

I want to build a microservice Oh! A “microservice”? Hmm.. seems to be the new thing huh. Yep, just create a JIRA ticket.

Slide 20

Slide 20 text

Minutes later….

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Weeks later….

Slide 25

Slide 25 text

Here are your servers! Well, that took a while!

Slide 26

Slide 26 text

Ok! Now I only need Java, Sensu checks and a Jenkins pipeline top deploy to the servers

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

2016-2017

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

I want to build a microservice

Slide 31

Slide 31 text

5 minutes later….

Slide 32

Slide 32 text

I just deployed a microservice to production!

Slide 33

Slide 33 text

Microservice Pipeline ./project-generator Pipeline as Code Mesos Marathon

Slide 34

Slide 34 text

Project Skeleton

Slide 35

Slide 35 text

= Project Skeleton ./project-generator

Slide 36

Slide 36 text

Pipeline as Code

Slide 37

Slide 37 text

Pipeline as Code

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Packaging

Slide 40

Slide 40 text

Deployment Files replicas: 1 resources: cpu: 2 memory: 200M healthChecks: ...

Slide 41

Slide 41 text

Makefile ● make deploy-dev ● make deploy-staging ● make deploy-production

Slide 42

Slide 42 text

Mesos Marathon

Slide 43

Slide 43 text

API make deploy

Slide 44

Slide 44 text

API make deploy POST { "id":"service-1", "cpus": 0.1, "mem": 10.0, "instances": 1 }

Slide 45

Slide 45 text

API make deploy POST { "id":"service-1", "cpus": 0.1, "mem": 10.0, "instances": 1 } service-1

Slide 46

Slide 46 text

Routing service-1 10.0.10.1 service-2 10.0.10.2

Slide 47

Slide 47 text

Routing service-1 10.0.10.1 service-2 10.0.10.2 ?

Slide 48

Slide 48 text

Routing - Fat Middleware service-1 10.0.10.1 service-2 10.0.10.2

Slide 49

Slide 49 text

curl http://localhost:5040/service/service-1/endpoint { upstream service-1 { server 10.0.10.1:5041; .... } } localhost:5040 service-2 10.0.10.2

Slide 50

Slide 50 text

{ upstream service-1 { server 10.0.10.1:5041; .... } } service-2 10.0.10.2 curl https://10.0.10.1:5041/service/service-1/endpoint service-1 10.0.10.1

Slide 51

Slide 51 text

{ upstream service-1 { server 10.0.10.1:5041; .... } } service-2 10.0.10.2 service-1 10.0.10.1 localhost:8080

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Why Kubernetes?

Slide 57

Slide 57 text

4 months x

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Microservices on Mesos and Marathon ● Project Skeleton ○ Golang or Scala ● Pipeline as Code ○ Jenkinsfile ○ Makefile ● Docker images for packaging ● API on top of Marathon ● Dynamic service discovery ● Fat middleware using Consul and NGINX

Slide 61

Slide 61 text

Microservices on Kubernetes ● Project Skeleton ○ Golang or Scala ● Pipeline as Code ○ Jenkinsfile ○ Makefile ● Docker images for packaging ● API on top of Marathon ● Dynamic service discovery ● Fat middleware using Consul and NGINX ● Documentation for getting started ● ./mesos2k8s

Slide 62

Slide 62 text

./mesos2k8s

Slide 63

Slide 63 text

Deployment Files ● make deploy-k8s-dev ● make deploy-k8s-staging ● make deploy-k8s-production

Slide 64

Slide 64 text

Pipeline as Code

Slide 65

Slide 65 text

Pipeline as Code

Slide 66

Slide 66 text

Pipeline as Code

Slide 67

Slide 67 text

Packaging

Slide 68

Slide 68 text

Packaging

Slide 69

Slide 69 text

Routing in Mesos service-1 10.0.10.1 service-2 10.0.10.2

Slide 70

Slide 70 text

Routing in Mesos service-2 10.0.10.2

Slide 71

Slide 71 text

Routing to K8s service-2 10.0.10.2 service-1 10.0.17.1 172.10.0.10 10.0.30.10

Slide 72

Slide 72 text

service-2 curl http://localhost:5040/service/service-1/endpoint { upstream service-1 { } upstream bridge-1 { server 10.0.20.1:5041; .... } } localhost:5040

Slide 73

Slide 73 text

service-2 { upstream service-1 { } upstream bridge-1 { server 10.0.20.1:5041; .... } } bridge-1 (multi-dc aware)

Slide 74

Slide 74 text

service-2 curl http://localhost:5040/service/service-1/endpoint { upstream service-1 { } upstream bridge-1 { server 10.0.20.1:5041; .... } } curl https://10.0.20.1:5041/service/service-1/endpoint bridge-1 (multi-dc aware)

Slide 75

Slide 75 text

service-2 curl http://localhost:5040/service/service-1/endpoint { upstream service-1 { } upstream bridge-1 { server 10.0.20.1:5041; .... } } curl https://10.0.30.1:5041/service/service-1/endpoint bridge-1 (multi-dc aware)

Slide 76

Slide 76 text

service-2 10.0.10.2 service-1 10.0.17.1 172.10.0.10 10.0.30.10 curl https://10.0.30.1:5041/service/service-1/endpoint

Slide 77

Slide 77 text

service-2 10.0.10.2 service-1 10.0.17.1 172.10.0.10 10.0.30.10 curl https://10.0.30.1:5041/service/service-1/endpoint http://service-1.default.svc.cluster.local:8080

Slide 78

Slide 78 text

service-2 10.0.10.2 service-1 10.0.17.1 172.10.0.10 10.0.30.10 curl https://10.0.30.1:5041/service/service-1/endpoint

Slide 79

Slide 79 text

service-2 10.0.10.2 service-1 10.0.17.1 172.10.0.10 10.0.30.10 curl https://10.0.30.1:5041/service/service-1/endpoint Love getting those OK responses!

Slide 80

Slide 80 text

Rollback service-1 10.0.10.1 service-2 { upstream service-1 { server 10.0.10.1:5041; } upstream bridge-1 { server 10.0.20.1:5041; .... }

Slide 81

Slide 81 text

foo 10.0.10.4 service-1 172.10.0.10 10.0.30.10

Slide 82

Slide 82 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 apiVersion: v1 kind: Service metadata: name: foo labels: app: foo spec: ports: - port: 5040 protocol: TCP name: http selector: app: nginx-skyline-router

Slide 83

Slide 83 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 curl http://foo:5040/endpoint

Slide 84

Slide 84 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 curl http://foo:5040/endpoint

Slide 85

Slide 85 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 curl http://foo:5040/endpoint

Slide 86

Slide 86 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 # match on: # service.namespace.svc.cluster.local # service.namespace # service server_name REGEX …. location / { rewrite ^/(.*)$ /service/$service/$1 break; proxy_pass https://egress_bridge; }

Slide 87

Slide 87 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 curl http://bridge1:5041/service/foo/endpoint bridge-1 (multi-dc aware)

Slide 88

Slide 88 text

foo 10.0.10.4 foo 10.0.17.100 service-1 172.10.0.10 10.0.30.10 bridge-1 (multi-dc aware) curl http://10.0.10.4:5041/service/foo/endpoint

Slide 89

Slide 89 text

Project Skeleton

Slide 90

Slide 90 text

Ship it! ./project-generator

Slide 91

Slide 91 text

Microservice Pipeline ./project-generator Pipeline as Code Kuberenetes

Slide 92

Slide 92 text

Documentation

Slide 93

Slide 93 text

Live Demo

Slide 94

Slide 94 text

No content

Slide 95

Slide 95 text

Migration Results ● Moved 20 services ● Time: 1 ½ month ● People: 3

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

Things fail , Let’s talk about it…. ● “The bad config outage” ● “The classic security group fail”

Slide 98

Slide 98 text

Lessons Learned/Conclusion

Slide 99

Slide 99 text

Choose the least important service

Slide 100

Slide 100 text

Have a rollback plan

Slide 101

Slide 101 text

Write down what your deployment pipeline looks like

Slide 102

Slide 102 text

Documentation should be written for humans to read

Slide 103

Slide 103 text

Pragmatic

Slide 104

Slide 104 text

Minimizing disruption = Great Adoption

Slide 105

Slide 105 text

● Migrating Container Schedulers: http://code.hootsuite.com/migrating-container-orchestrators-mesos-kubernetes-n omad/ ● Abstracting Marathon Deployment Details from Microservices: http://code.hootsuite.com/abstracting-marathon-deployment-details-from-microse rvices/ ● Consul: https://www.consul.io/ Links

Slide 106

Slide 106 text

@anubhavm Anubhav Mishra Thank you!

Slide 107

Slide 107 text

Developer Advocate - HashiCorp @anubhavm Anubhav Mishra I am joining https://medium.com/@anubhavmishra/i-am-joining-hashicorp-5a38e0977867