Kubernetes at XING: what it takes to build a coherent development environment

This talk is a walk through the design of our XING's internal PaaS, the story of problems we encountered and solutions we found sprinkled with reflections on Kubernetes programming interface and real examples of making developers' lives easier with automation and closed feedback loops. Keywords: Kubernetes, PaaS, GitHub, Jenkins, Golang, Groovy, custom controllers, Prometheus, CLI, lessons learned.

Transcript

1. Kubernetes at XING What it takes to build a coherent development environment Egor Balyshev ContainerDays 2019
2. Why containers?
3. XING • Career / work oriented social network • 16 million users • German-speaking countries • 1,500 employees
4. XING, circa 2017 • Two data centers • VMware & Chef • 100+ apps • Ruby, Scala, Elixir, Perl • GitHub, Jenkins, Capistrano
5. Pain points • Slow resource provisioning • Mistakes in host configuration • Complicated application boilerplates • Lack of granular security policies
7. The big picture code runtime
8. The big picture code runtime
9. Building containers
10. Dockerizing is easy
11. Dockerfile isn’t cutting it • OS and library version management • Security patch rollouts • Certificate installation
12. Dockerfile
13. Dockerfile Complex boilerplate
14. docker-build • Base images: OS + frameworks + dependencies • Centralized configuration • Production and development targets
15. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
16. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
17. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
18. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
19. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
20. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
21. build.yaml type: rails-app passenger_version: 6.0.2 image: reg.xing.com/xing/rails-misc os_release: bionic build_packages: - libmysqlclient-dev runtime_packages: - libmysqlclient20 assets_compile_command: rake assets:precompile
22. docker-test • Wraps docker-compose • Pulls dependencies • Cleans everything up
23. test_compose.yaml services: mysql: image: reg.xing.com/deps/mysql memcached: image: reg.xing.com/deps/memcached main: image: reg.xing.com/xing/rails-misc:$TAG entrypoint: script/test.sh links: - mysql - memcached environment: RAILS_CACHE_STORE_MEMCACHE_HOSTS: memcached DB_HOST: mysql
24. test_compose.yaml services: mysql: image: reg.xing.com/deps/mysql memcached: image: reg.xing.com/deps/memcached main: image: reg.xing.com/xing/rails-misc:$TAG entrypoint: script/test.sh links: - mysql - memcached environment: RAILS_CACHE_STORE_MEMCACHE_HOSTS: memcached DB_HOST: mysql
25. test_compose.yaml services: mysql: image: reg.xing.com/deps/mysql memcached: image: reg.xing.com/deps/memcached main: image: reg.xing.com/xing/rails-misc:$TAG entrypoint: script/test.sh links: - mysql - memcached environment: RAILS_CACHE_STORE_MEMCACHE_HOSTS: memcached DB_HOST: mysql
26. test_compose.yaml services: mysql: image: reg.xing.com/deps/mysql memcached: image: reg.xing.com/deps/memcached main: image: reg.xing.com/xing/rails-misc:$TAG entrypoint: script/test.sh links: - mysql - memcached environment: RAILS_CACHE_STORE_MEMCACHE_HOSTS: memcached DB_HOST: mysql
27. test_compose.yaml services: mysql: image: reg.xing.com/deps/mysql memcached: image: reg.xing.com/deps/memcached main: image: reg.xing.com/xing/rails-misc:$TAG entrypoint: script/test.sh links: - mysql - memcached environment: RAILS_CACHE_STORE_MEMCACHE_HOSTS: memcached DB_HOST: mysql
28. The big picture code runtime
29. The big picture code runtime
30. Image registry • Docker Trusted Registry • Quay
31. The big picture code runtime containers
32. The big picture code runtime images
33. Running containers
34. Mesos + Marathon, 2017 • One-shot tasks • Guaranteed CPU/memory ☹ • Granular access control • Developer support
35. “Containers” is a huge change • Automate everything! • Resource utilization! • Lock the things down! • We’re playing with Mesos!
36. What orchestrator?
37. Value system
38. XING platform values 1. Engineers’ productivity 2. Security 3. Maintenance cost 4. Platform resilience 5. Hardware utilization
39. DC/OS K8S
40. Kubernetes • Complete feature set • Detailed documentation • Open source • Tremendous momentum
41. The big picture code HTTP images deployments
42. The big picture code HTTP images deployments
43. Deployment pipeline
44. Platform values 1. Engineers’ productivity 2. Security 3. Maintenance cost 4. Platform resilience 5. Hardware utilization
45. Start with the code
46. GitHub • Organizations → Jenkins folders • Branches → Pipelines • Events → Pipeline triggers
47. Jenkins • Folders plugin • Kubernetes CLI plugin • Docker-build and docker-test
48. The big picture code HTTP images deployments
49. The big picture code HTTP images deployments
50. kubectl isn’t cutting it • K8s manifest duplication • No resource creation order • The status is not reported back
51. kubernetes-deploy • Open source • Templates and partials • Secret encryption • Deployment stages • Status reports
52. Jenkins pipeline 1. Checkout the code 2. Docker-build 3. Docker-test 4. Push to the registry 5. Kubernetes-deploy
53. Jenkinsfile @Library('xing') _ xing.dockerImage = "reg.xing.com/xing/rails-misc" xing.namespace = "xing" xingDefaultPipeline(this)
54. Jenkinsfile @Library('xing') _ xing.dockerImage = "reg.xing.com/xing/rails-misc" xing.namespace = "xing" xingDefaultPipeline(this)
55. Jenkinsfile @Library('xing') _ xing.dockerImage = "reg.xing.com/xing/rails-misc" xing.namespace = "xing" xingDefaultPipeline(this)
56. Jenkinsfile @Library('xing') _ xing.dockerImage = "reg.xing.com/xing/rails-misc" xing.namespace = "xing" xingDefaultPipeline(this)
57. Jenkins: gets the job done,
58. Jenkins: gets the job done, but be prepared for the pain
59. The big picture code HTTP images manifests status images
60. The big picture code HTTP images manifests status images
61. The big picture code HTTP images manifests status images
62. The big picture code HTTP images manifests status images
63. Configuration Automation
64. Kubernetes (custom) controllers • Observe the current state • Change it to the desired state • Repeat
65. Platform values 1. Engineers’ productivity 2. Security 3. Maintenance cost 4. Platform resilience 5. Hardware utilization
66. Start with the code
67. GitHub • Organizations → Kubernetes namespaces, Jenkins folders • Team membership → Deployment permissions • Branches → Stages • Events → Pipeline triggers
68. The big picture code HTTP images manifests status images
69. The big picture code HTTP images manifests status images
70. The big picture code HTTP images manifests status images folders, credentials orgs
71. The big picture code HTTP images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams
72. The big picture code HTTP images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories
73. The big picture code HTTP images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories
74. Integration
75. The big picture code HTTP images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories
76. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP
77. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP
78. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses
79. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP env variables rules services ingresses
80. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses env variables
81. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses values ConfigMaps
82. Custom controllers • Observe the current state • Change it to the desired state • Repeat
83. Custom controllers
84. Custom controllers
85. Custom controllers
86. Custom controllers • Application config reloads • Graceful node restarts • Docker runtime fixes • Metrics collection setup • Cross-cluster service discovery
87. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
88. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
89. Supporting developers
90. Documentation is King
91. Documentation • Separate website • 30,000 words (
93. Docs structure • Components: what it is and what it does • Tasks: daily routines, < 15 min • Tutorials: migrations, tuning guides
94. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
95. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
96. Support channel • “Argonaut of the week” role • Support channel is the only duty • Helps a lot!
97. Support questions • Incomplete on-boarding • Incident resolution • Feature requests
98. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
99. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
100. Platform values 1. Engineers’ productivity 2. Security 3. Maintenance cost 4. Platform resilience 5. Hardware utilization
101. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
102. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
103. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
104. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
105. Command-line interface • Single binary • Bundled kubectl, kubernetes-deploy • Automatic updates
106. Olympify • Auto-detects application type • Writes image build config • Writes Jenkins pipeline • Writes Kubernetes manifests
107. Day-to-day commands • app deploy, app rollback, app restart • secret encode, secret import • run job, run command • authenticate
108. Rolling your own CLI • Bundled dependencies • Built-in help • Bash scripts
109. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
110. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
111. Prometheus (format) • Widely supported • Pull model • Telegraph adds versatility • Thanos looks promising
112. The big picture code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
113. Results & observations
114. XING, 2019 • 230 applications • Kubernetes Olympus is the platform • 100% of eligible apps migrated • 9 clusters, 100 namespaces, 7000 pods • Massive proliferation of “misc” apps
115. Know your values
116. Expectations
117. Reality code HTTPS images manifests status images folders, credentials orgs namespaces, rbac roles orgs, teams secrets repositories HTTP rules services ingresses config ConfigMaps
118. Thank you! egor.balyshev@gmail.com xing.com/profile/Egor_Balyshev tech.xing.com
119. Acknowledgements I used a bunch of icons from thenounproject.com: • build by priyanka from the Noun Project • cogs by kurakuricon from the Noun Project • Command Line by Focus from the Noun Project • terminal by Focus from the Noun Project • Router by Lero Keller from the Noun Project • website by lastspark from the Noun Project • customer service by Deemak Daksina from the Noun Project • documentation by Mahmure Alp from the Noun Project • pause by uzeir syarief from the Noun Project All product and company logos belong to their original copyright holders.