$30 off During Our Annual Pro Sale. View Details »

Moving from Mesos to Kubernetes without anyone noticing

Anubhav Mishra
December 10, 2017

Moving from Mesos to Kubernetes without anyone noticing

At Hootsuite, we’ve been using Mesos and Marathon as our microservices platform for over two years but last year, we made the decision to bet on Kubernetes as its replacement. Eight months later, a small team of three operations engineers had migrated our first microservice from Mesos to Kubernetes. All without developers making any code changes. This was possible by architecting our applications with the proper set of abstractions. Fast-forward three months later and we have almost 20 microservices running on Kubernetes in production.

In this session, we’ll do a live demo of migrating a service from Mesos to Kubernetes, just like how we did it at Hootsuite! We will cover why architecting your infrastructure with the “right” abstractions helps you do these huge migrations with ease and how Kubernetes already contains these abstractions. We will explore how having a service mesh helps routing between two platforms while doing the migration. Also, how a mature CI/CD pipeline can help you deploy to two platforms with ease. To conclude we will explore the differences in running a service in Mesos and Kubernetes.

Anubhav Mishra

December 10, 2017
Tweet

More Decks by Anubhav Mishra

Other Decks in Technology

Transcript

  1. Moving from
    Mesos to Kubernetes without anyone
    noticing*
    Anubhav Mishra

    View Slide

  2. Anubhav Mishra
    @anubhavm

    View Slide

  3. Anubhav Mishra
    @anubhavm

    View Slide

  4. View Slide

  5. Anubhav Mishra
    @anubhavm

    View Slide

  6. Anubhav Mishra
    @anubhavm
    Atlantis

    View Slide

  7. Anubhav Mishra
    @anubhavm
    Atlantis

    View Slide

  8. Anubhav Mishra
    @anubhavm
    Atlantis

    View Slide

  9. Anubhav Mishra
    @anubhavm
    Atlantis

    View Slide

  10. vs

    View Slide

  11. vs

    View Slide

  12. Agenda
    ● Hootsuite’s Journey from Mesos to Kubernetes
    ● Microservices pipeline
    ○ Mesos and Marathon
    ○ Kubernetes
    ● Migration without major disruption
    ● Live demo!
    ● Lessons learned/Conclusion

    View Slide

  13. Hootsuite Now

    View Slide

  14. Numbers
    ● 120+ developers
    ● 60+ microservices
    ● 2 cluster schedulers
    ● 1500+ servers on AWS

    View Slide

  15. 2014

    View Slide

  16. View Slide

  17. I want to build
    a microservice

    View Slide

  18. I want to build
    a microservice

    View Slide

  19. I want to build
    a microservice
    Oh! A “microservice”?
    Hmm.. seems to be the
    new thing huh. Yep, just
    create a JIRA ticket.

    View Slide

  20. Minutes later….

    View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. Weeks later….

    View Slide

  25. Here are your servers!
    Well, that took
    a while!

    View Slide

  26. Ok! Now I only need
    Java, Sensu checks
    and a Jenkins
    pipeline top deploy
    to the servers

    View Slide

  27. View Slide

  28. 2016-2017

    View Slide

  29. View Slide

  30. I want to build
    a microservice

    View Slide

  31. 5 minutes later….

    View Slide

  32. I just deployed
    a microservice
    to production!

    View Slide

  33. Microservice Pipeline
    ./project-generator
    Pipeline as Code
    Mesos Marathon

    View Slide

  34. Project Skeleton

    View Slide

  35. =
    Project Skeleton
    ./project-generator

    View Slide

  36. Pipeline as Code

    View Slide

  37. Pipeline as Code

    View Slide

  38. View Slide

  39. Packaging

    View Slide

  40. Deployment Files
    replicas: 1
    resources:
    cpu: 2
    memory: 200M
    healthChecks:
    ...

    View Slide

  41. Makefile
    ● make deploy-dev
    ● make deploy-staging
    ● make deploy-production

    View Slide

  42. Mesos Marathon

    View Slide

  43. API
    make deploy

    View Slide

  44. API
    make deploy
    POST
    {
    "id":"service-1",
    "cpus": 0.1,
    "mem": 10.0,
    "instances": 1
    }

    View Slide

  45. API
    make deploy
    POST
    {
    "id":"service-1",
    "cpus": 0.1,
    "mem": 10.0,
    "instances": 1
    }
    service-1

    View Slide

  46. Routing
    service-1
    10.0.10.1
    service-2
    10.0.10.2

    View Slide

  47. Routing
    service-1
    10.0.10.1
    service-2
    10.0.10.2
    ?

    View Slide

  48. Routing - Fat Middleware
    service-1
    10.0.10.1
    service-2
    10.0.10.2

    View Slide

  49. curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    localhost:5040
    service-2
    10.0.10.2

    View Slide

  50. {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    service-2
    10.0.10.2
    curl https://10.0.10.1:5041/service/service-1/endpoint
    service-1
    10.0.10.1

    View Slide

  51. {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    service-2
    10.0.10.2
    service-1
    10.0.10.1
    localhost:8080

    View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. Why Kubernetes?

    View Slide

  57. 4 months
    x

    View Slide

  58. View Slide

  59. ?

    View Slide

  60. Microservices on Mesos and Marathon
    ● Project Skeleton
    ○ Golang or Scala
    ● Pipeline as Code
    ○ Jenkinsfile
    ○ Makefile
    ● Docker images for packaging
    ● API on top of Marathon
    ● Dynamic service discovery
    ● Fat middleware using Consul and NGINX

    View Slide

  61. Microservices on Kubernetes
    ● Project Skeleton
    ○ Golang or Scala
    ● Pipeline as Code
    ○ Jenkinsfile
    ○ Makefile
    ● Docker images for packaging
    ● API on top of Marathon
    ● Dynamic service discovery
    ● Fat middleware using Consul and NGINX
    ● Documentation for getting started
    ● ./mesos2k8s

    View Slide

  62. ./mesos2k8s

    View Slide

  63. Deployment Files
    ● make deploy-k8s-dev
    ● make deploy-k8s-staging
    ● make deploy-k8s-production

    View Slide

  64. Pipeline as Code

    View Slide

  65. Pipeline as Code

    View Slide

  66. Pipeline as Code

    View Slide

  67. Packaging

    View Slide

  68. Packaging

    View Slide

  69. Routing in Mesos
    service-1
    10.0.10.1
    service-2
    10.0.10.2

    View Slide

  70. Routing in Mesos
    service-2
    10.0.10.2

    View Slide

  71. Routing to K8s
    service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10

    View Slide

  72. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    localhost:5040

    View Slide

  73. service-2
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    bridge-1
    (multi-dc aware)

    View Slide

  74. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    curl https://10.0.20.1:5041/service/service-1/endpoint
    bridge-1
    (multi-dc aware)

    View Slide

  75. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    curl https://10.0.30.1:5041/service/service-1/endpoint
    bridge-1
    (multi-dc aware)

    View Slide

  76. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint

    View Slide

  77. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    http://service-1.default.svc.cluster.local:8080

    View Slide

  78. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint

    View Slide

  79. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    Love getting those
    OK responses!

    View Slide

  80. Rollback
    service-1
    10.0.10.1
    service-2
    {
    upstream service-1 {
    server 10.0.10.1:5041;
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }

    View Slide

  81. foo
    10.0.10.4
    service-1
    172.10.0.10
    10.0.30.10

    View Slide

  82. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    apiVersion: v1
    kind: Service
    metadata:
    name: foo
    labels:
    app: foo
    spec:
    ports:
    - port: 5040
    protocol: TCP
    name: http
    selector:
    app: nginx-skyline-router

    View Slide

  83. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint

    View Slide

  84. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint

    View Slide

  85. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint

    View Slide

  86. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    # match on:
    # service.namespace.svc.cluster.local
    # service.namespace
    # service
    server_name REGEX
    ….
    location / {
    rewrite ^/(.*)$ /service/$service/$1 break;
    proxy_pass https://egress_bridge;
    }

    View Slide

  87. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://bridge1:5041/service/foo/endpoint
    bridge-1
    (multi-dc aware)

    View Slide

  88. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    bridge-1
    (multi-dc aware)
    curl http://10.0.10.4:5041/service/foo/endpoint

    View Slide

  89. Project Skeleton

    View Slide

  90. Ship it!
    ./project-generator

    View Slide

  91. Microservice Pipeline
    ./project-generator
    Pipeline as Code
    Kuberenetes

    View Slide

  92. Documentation

    View Slide

  93. Live Demo

    View Slide

  94. View Slide

  95. Migration Results
    ● Moved 20 services
    ● Time: 1 ½ month
    ● People: 3

    View Slide

  96. View Slide

  97. Things fail , Let’s talk about it….
    ● “The bad config outage”
    ● “The classic security group fail”

    View Slide

  98. Lessons Learned/Conclusion

    View Slide

  99. Choose the least important
    service

    View Slide

  100. Have a rollback plan

    View Slide

  101. Write down what your
    deployment pipeline looks like

    View Slide

  102. Documentation should be
    written for humans to read

    View Slide

  103. Pragmatic

    View Slide

  104. Minimizing disruption = Great
    Adoption

    View Slide

  105. ● Migrating Container Schedulers:
    http://code.hootsuite.com/migrating-container-orchestrators-mesos-kubernetes-n
    omad/
    ● Abstracting Marathon Deployment Details from Microservices:
    http://code.hootsuite.com/abstracting-marathon-deployment-details-from-microse
    rvices/
    ● Consul: https://www.consul.io/
    Links

    View Slide

  106. @anubhavm
    Anubhav Mishra
    Thank you!

    View Slide

  107. Developer Advocate - HashiCorp
    @anubhavm
    Anubhav Mishra
    I am joining
    https://medium.com/@anubhavmishra/i-am-joining-hashicorp-5a38e0977867

    View Slide