Migrating a monolith to Kubernetes

Last year, a small team at GitHub set out to migrate a large portion of the application that serves GitHub.com to Kubernetes. This application fits the classic definition of a monolith: a large codebase, developed over many years, containing contributions from hunderds of engineers (many of which have moved on to other things). In this presentation, we'll cover our motivations for this migration, the factors that led us to choose Kubernetes, the strategies we used to empower a small team to make a change that affected a large engineering organization, and reflect on what we learned in the process.

Transcript

1. Migrating a monolith to Kubernetes DevOps Enterprise Summit 2017 Jesse Newland
2. Hi!
3. I’m Jesse Newland
4. @jnewland
5. 16 years in web operations
6. 6 years at GitHub
7. Engineering / Management
10. Technical leadership from Austin, TX
11. Why am I here? Kubernetes? Monoliths? DevOps? ENTERPRISE?
12. My job is to affect change in a technical organization
13. Github is growing, maturing, & evolving
14. Our solutions often don’t scale to fit the needs of our growing the organization
15. On a journey of continuous improvement
16. We are more alike, my friends, than we are unalike. Maya Angelou
18. https://githubengineering.com/kubernetes-at-github/
19. ❤
20. Kubernetes is an open- source system for automating deployment, scaling, and management of containerized applications
21. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community
22. I’m not here to tell you that you should adopt Kubernetes
23. Or even to go to deep into the technical details of our migration
24. https://githubengineering.com/kubernetes-at-github/ @jnewland
25. Kubernetes is a technology
26. Kubernetes is a super dope technology
27. Not a panacea
28. Use what’s right for you
29. I’d like to share an anecdote from our ongoing journey
30. The only slide with bullets, I promise! • Why we migrated our monolith to Kubernetes • How did we approached a large cross-team project • Where we are today • What we learned in the process • Where we’re headed
31. Why?
32. Context
33. The monolith
34. Ruby on Rails
35. github.com/ github/ github
36. GitHub dot com the website
37. 10 years old
38. Extremely important to early velocity
39. Increasing complexity
40. Diffusion of responsibility
42. Incredibly high performance hardware
43. Incredibly reliable hardware
44. Incredibly low latency networking
45. Incredibly high throughput networking
47. Unit of compute == instance
48. Instance setup tightly coupled with configuration management
49. API-driven, testable, but brutal feedback loop
50. Human-managed provisioning and load balancing config
51. High level of effort required to get a service into production
53. Our customer base is growing
54. Our customers are growing
55. Our ecosystem is growing
56. Our organization is growing
57. We’re shipping new products
58. We’re improving existing products
59. Our customers expect increasing speed and reliability
60. We saw indications that our approach was struggling to deal with these forces
61. The engineering culture at GitHub was attempting to evolve to encourage individual teams to act as maintainers of their own services
62. SRE's tools and practices for running services had not yet evolved to match
63. Easier to add functionality to an existing service
64. Unsurprisingly, the monolith kept growing
65. Increasing CI duration
66. Increasing deploy duration
67. Inflexible infrastructure
68. Inefficient infrastructure
69. Private cloud lock-in
70. Developer and user experience trending downward
71. The planets aligned in way that made all of these problems visible all at one
72. Hack week
73. Given a week to ship something new and innovative, what might we expect engineers to do?
74. 1) spend ~1 day on Puppet, provisioning, and load balancing config
75. 2) reach out to SRE on Thursday and ask for our help?
76. 3) build hack week features as a PR against the monolith
77. Microcosm of the larger problems with our approach
78. Incentives not aligned with the outcomes we desired
80. Our on-ramp went in the wrong direction
81. High effort required
83. We decided to make an investment in our tools
84. We decided to make an investment in our processes
85. We decided to make an investment in our technology
86. To support the other ongoing changes in our organization, we decided that we would work to level the playing field
87. To support the decomposition of the monolith, we decided that we would work to provide a better experience for new services
88. To enable SRE to spend more time on interesting services, we decided to work to reduce the amount of time we needed to spend on boring services
89. To reduce the time we spent on boring services, we decided to work to make the service provisioning process entirely self-service
90. To bring the infrastructure- building feedback loop down, we decided to base this new future on a container orchestration platform
91. To leverage the experience of Google and the strength of the community, we decided to build this new approach with Kubernetes
92. How?
93. okay sorry, a few more bullets • Passion team • Prototype • Pick an impactful and visible target • Product vision and project plan • Pwork • Pause and regroup
94. Passion team
95. https://github.com/blog/2316-organize-your-experts-with-ad-hoc-teams
96. Intentionally curate a diverse set of skills
97. Intentionally curate a diverse set of experience
98. Intentionally curate a diverse set of knowledge
99. Intentionally curate a diverse set of perspectives
100. SRE + Developer Experience + Platform Engineering
101. Project scoped team
102. @github/kubernetes github/kube #kube
103. Prototype
104. A strategy for not crying under the bed during hack week
105. Prototype Goals
106. Kubernetes cluster, load balancing, deployment strategy, docs
107. Leverage hack week standard of quality
108. Validate our hypothesis that we could provide a new and better experience with minimal effort
109. Validate our hypothesis that if provided with another option, engineers would flock to it
110. Learn more about Kubernetes
111. Seek feedback from engineers that used the new approach
112. Internal marketing
113. Wild success
115. Handful of projects launched with very little SRE involvement
116. Positive feedback
117. Learned a ton about an engineer’s perspective
118. Several of these projects still exist, and are maintained by their creating teams
119. Pick a big target
120. We decided to migrate the monolith
121. Why?
122. Pros
123. We wanted to validate something larger following our positive experience with smaller scale apps during Hack Week
124. A well worn path
125. We were confident in the testing strategies available to us
126. We had an overlapping need for dynamic lab environments
127. And an overlapping need for more flexibility to handle peaks and valleys of demand
128. Cons
129. It might not work
130. We might make things worse
131. We decided to put together a project plan and see if it felt viable
132. Vision and plan
133. Tons of high-impact, visible work ahead
134. Communication was crucial
135. Key elements of communicating change at GitHub
136. Know your goal
137. …and lead with it
138. Don’t mince words
139. Write conversationally
140. Include the alternatives you’ve considered
141. Doing nothing is always an alternative
142. Consider the production impact
143. Give it a URL
144. Pull request
145. Repeat the message using different mediums
146. Communication had the desired impact
147. Executive support
148. Additional engineering resources
149. Project management resources
150. Now all we had to do was not be wrong
151. How’d it go?
154. One big container
155. 1.1gb image
156. 100s image build
157. it's fine
158. Review lab
159. 50 times per day
160. Staff opt-in
161. Controlled experiments
164. ~100% of github.com web requests served by application processes running Kubernetes
165. Most of the functionality we built to support the monolith is available to other services
166. ~20% of all services are running on Kubernetes clusters
167. What'd we learn?
168. Positive outcomes
169. Reduced level of effort for new service setup
170. New services regularly deployed with little-to- no SRE involvement
171. APIs to query the running state of our system
172. APIs to mutate the running state of our system
173. Cloud-native platform to build against
174. Open
175. Emerging as a standard
176. Reduce lock-in
177. Commoditize compute providers
178. More OSS friendly than configuration management and glue
179. provider automation, config management, packages, & operating system
180. container images, resources, & apis
183. Challenges
184. SRE
185. Operationalizing a new platform
186. Docker instability
187. Changing the expectations of application engineers
188. Application Engineering
189. Change
190. Learning curve
191. Shorter process lifetimes
192. What happens on process shutdown?
193. What happens during ungraceful shutdown?
194. Things I’d do again
195. Passion team
196. Prioritize communication
197. Network effect via highly visible work
198. Gradual rollout
199. Things I’d do differently
200. More consciously consider the handoff phase
201. Document this approach to help it feel more regular
202. More open source
203. What’s next?
204. Seek feedback from engineers
205. Seek feedback from SREs
206. Seek feedback from leadership
207. Relentlessly focus on automating work that scales with traffic or organizational size
208. Build services that leverage the platform
209. Focus SRE efforts on improvements that benefit all services
210. Focus SRE efforts on improvements that benefit everyone
211. Keep improving
212. Thanks!
213. @jnewland