Moving from Mesos to Kubernetes without anyone noticing

At Hootsuite, we’ve been using Mesos and Marathon as our microservices platform for over two years but last year, we made the decision to bet on Kubernetes as its replacement. Eight months later, a small team of three operations engineers had migrated our first microservice from Mesos to Kubernetes. All without developers making any code changes. This was possible by architecting our applications with the proper set of abstractions. Fast-forward three months later and we have almost 20 microservices running on Kubernetes in production.

In this session, we’ll do a live demo of migrating a service from Mesos to Kubernetes, just like how we did it at Hootsuite! We will cover why architecting your infrastructure with the “right” abstractions helps you do these huge migrations with ease and how Kubernetes already contains these abstractions. We will explore how having a service mesh helps routing between two platforms while doing the migration. Also, how a mature CI/CD pipeline can help you deploy to two platforms with ease. To conclude we will explore the differences in running a service in Mesos and Kubernetes.

Transcript

1. Moving from
   Mesos to Kubernetes without anyone
   noticing*
   Anubhav Mishra
   

   View Slide

2. Anubhav Mishra
   @anubhavm
   

   View Slide

3. Anubhav Mishra
   @anubhavm
   

   View Slide

4. View Slide

5. Anubhav Mishra
   @anubhavm
   

   View Slide

6. Anubhav Mishra
   @anubhavm
   Atlantis
   

   View Slide

7. Anubhav Mishra
   @anubhavm
   Atlantis
   

   View Slide

8. Anubhav Mishra
   @anubhavm
   Atlantis
   

   View Slide

9. Anubhav Mishra
   @anubhavm
   Atlantis
   

   View Slide

10. vs
    

    View Slide

11. vs
    

    View Slide

12. Agenda
    ● Hootsuite’s Journey from Mesos to Kubernetes
    ● Microservices pipeline
    ○ Mesos and Marathon
    ○ Kubernetes
    ● Migration without major disruption
    ● Live demo!
    ● Lessons learned/Conclusion
    

    View Slide

13. Hootsuite Now
    

    View Slide

14. Numbers
    ● 120+ developers
    ● 60+ microservices
    ● 2 cluster schedulers
    ● 1500+ servers on AWS
    

    View Slide

15. View Slide

16. View Slide

17. I want to build
    a microservice
    

    View Slide

18. I want to build
    a microservice
    

    View Slide

19. I want to build
    a microservice
    Oh! A “microservice”?
    Hmm.. seems to be the
    new thing huh. Yep, just
    create a JIRA ticket.
    

    View Slide

20. Minutes later….
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. Weeks later….
    

    View Slide

25. Here are your servers!
    Well, that took
    a while!
    

    View Slide

26. Ok! Now I only need
    Java, Sensu checks
    and a Jenkins
    pipeline top deploy
    to the servers
    

    View Slide

27. View Slide

28. 2016-2017
    

    View Slide

29. View Slide

30. I want to build
    a microservice
    

    View Slide

31. 5 minutes later….
    

    View Slide

32. I just deployed
    a microservice
    to production!
    

    View Slide

33. Microservice Pipeline
    ./project-generator
    Pipeline as Code
    Mesos Marathon
    

    View Slide

34. Project Skeleton
    

    View Slide

35. =
    Project Skeleton
    ./project-generator
    

    View Slide

36. Pipeline as Code
    

    View Slide

37. Pipeline as Code
    

    View Slide

38. View Slide

39. Packaging
    

    View Slide

40. Deployment Files
    replicas: 1
    resources:
    cpu: 2
    memory: 200M
    healthChecks:
    ...
    

    View Slide

41. Makefile
    ● make deploy-dev
    ● make deploy-staging
    ● make deploy-production
    

    View Slide

42. Mesos Marathon
    

    View Slide

43. API
    make deploy
    

    View Slide

44. API
    make deploy
    POST
    {
    "id":"service-1",
    "cpus": 0.1,
    "mem": 10.0,
    "instances": 1
    }
    

    View Slide

45. API
    make deploy
    POST
    {
    "id":"service-1",
    "cpus": 0.1,
    "mem": 10.0,
    "instances": 1
    }
    service-1
    

    View Slide

46. Routing
    service-1
    10.0.10.1
    service-2
    10.0.10.2
    

    View Slide

47. Routing
    service-1
    10.0.10.1
    service-2
    10.0.10.2
    ?
    

    View Slide

48. Routing - Fat Middleware
    service-1
    10.0.10.1
    service-2
    10.0.10.2
    

    View Slide

49. curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    localhost:5040
    service-2
    10.0.10.2
    

    View Slide

50. {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    service-2
    10.0.10.2
    curl https://10.0.10.1:5041/service/service-1/endpoint
    service-1
    10.0.10.1
    

    View Slide

51. {
    upstream service-1 {
    server 10.0.10.1:5041;
    ....
    }
    }
    service-2
    10.0.10.2
    service-1
    10.0.10.1
    localhost:8080
    

    View Slide

52. View Slide

53. View Slide

54. View Slide

55. View Slide

56. Why Kubernetes?
    

    View Slide

57. 4 months
    x
    

    View Slide

58. View Slide

59. ?
    

    View Slide

60. Microservices on Mesos and Marathon
    ● Project Skeleton
    ○ Golang or Scala
    ● Pipeline as Code
    ○ Jenkinsfile
    ○ Makefile
    ● Docker images for packaging
    ● API on top of Marathon
    ● Dynamic service discovery
    ● Fat middleware using Consul and NGINX
    

    View Slide

61. Microservices on Kubernetes
    ● Project Skeleton
    ○ Golang or Scala
    ● Pipeline as Code
    ○ Jenkinsfile
    ○ Makefile
    ● Docker images for packaging
    ● API on top of Marathon
    ● Dynamic service discovery
    ● Fat middleware using Consul and NGINX
    ● Documentation for getting started
    ● ./mesos2k8s
    

    View Slide

62. ./mesos2k8s
    

    View Slide

63. Deployment Files
    ● make deploy-k8s-dev
    ● make deploy-k8s-staging
    ● make deploy-k8s-production
    

    View Slide

64. Pipeline as Code
    

    View Slide

65. Pipeline as Code
    

    View Slide

66. Pipeline as Code
    

    View Slide

67. Packaging
    

    View Slide

68. Packaging
    

    View Slide

69. Routing in Mesos
    service-1
    10.0.10.1
    service-2
    10.0.10.2
    

    View Slide

70. Routing in Mesos
    service-2
    10.0.10.2
    

    View Slide

71. Routing to K8s
    service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    

    View Slide

72. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    localhost:5040
    

    View Slide

73. service-2
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    bridge-1
    (multi-dc aware)
    

    View Slide

74. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    curl https://10.0.20.1:5041/service/service-1/endpoint
    bridge-1
    (multi-dc aware)
    

    View Slide

75. service-2
    curl http://localhost:5040/service/service-1/endpoint
    {
    upstream service-1 {
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    }
    curl https://10.0.30.1:5041/service/service-1/endpoint
    bridge-1
    (multi-dc aware)
    

    View Slide

76. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    

    View Slide

77. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    http://service-1.default.svc.cluster.local:8080
    

    View Slide

78. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    

    View Slide

79. service-2
    10.0.10.2
    service-1
    10.0.17.1
    172.10.0.10
    10.0.30.10
    curl https://10.0.30.1:5041/service/service-1/endpoint
    Love getting those
    OK responses!
    

    View Slide

80. Rollback
    service-1
    10.0.10.1
    service-2
    {
    upstream service-1 {
    server 10.0.10.1:5041;
    }
    upstream bridge-1 {
    server 10.0.20.1:5041;
    ....
    }
    

    View Slide

81. foo
    10.0.10.4
    service-1
    172.10.0.10
    10.0.30.10
    

    View Slide

82. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    apiVersion: v1
    kind: Service
    metadata:
    name: foo
    labels:
    app: foo
    spec:
    ports:
    - port: 5040
    protocol: TCP
    name: http
    selector:
    app: nginx-skyline-router
    

    View Slide

83. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint
    

    View Slide

84. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint
    

    View Slide

85. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://foo:5040/endpoint
    

    View Slide

86. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    # match on:
    # service.namespace.svc.cluster.local
    # service.namespace
    # service
    server_name REGEX
    ….
    location / {
    rewrite ^/(.*)$ /service/$service/$1 break;
    proxy_pass https://egress_bridge;
    }
    

    View Slide

87. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    curl http://bridge1:5041/service/foo/endpoint
    bridge-1
    (multi-dc aware)
    

    View Slide

88. foo
    10.0.10.4
    foo
    10.0.17.100
    service-1
    172.10.0.10
    10.0.30.10
    bridge-1
    (multi-dc aware)
    curl http://10.0.10.4:5041/service/foo/endpoint
    

    View Slide

89. Project Skeleton
    

    View Slide

90. Ship it!
    ./project-generator
    

    View Slide

91. Microservice Pipeline
    ./project-generator
    Pipeline as Code
    Kuberenetes
    

    View Slide

92. Documentation
    

    View Slide

93. Live Demo
    

    View Slide

94. View Slide

95. Migration Results
    ● Moved 20 services
    ● Time: 1 ½ month
    ● People: 3
    

    View Slide

96. View Slide

97. Things fail , Let’s talk about it….
    ● “The bad config outage”
    ● “The classic security group fail”
    

    View Slide

98. Lessons Learned/Conclusion
    

    View Slide

99. Choose the least important
    service
    

    View Slide

100. Have a rollback plan
     

     View Slide

101. Write down what your
     deployment pipeline looks like
     

     View Slide

102. Documentation should be
     written for humans to read
     

     View Slide

103. Pragmatic
     

     View Slide

104. Minimizing disruption = Great
     Adoption
     

     View Slide

105. ● Migrating Container Schedulers:
     http://code.hootsuite.com/migrating-container-orchestrators-mesos-kubernetes-n
     omad/
     ● Abstracting Marathon Deployment Details from Microservices:
     http://code.hootsuite.com/abstracting-marathon-deployment-details-from-microse
     rvices/
     ● Consul: https://www.consul.io/
     Links
     

     View Slide

106. @anubhavm
     Anubhav Mishra
     Thank you!
     

     View Slide

107. Developer Advocate - HashiCorp
     @anubhavm
     Anubhav Mishra
     I am joining
     https://medium.com/@anubhavmishra/i-am-joining-hashicorp-5a38e0977867
     

     View Slide