Kubernetes in Context

The rise of containerisation, orchestration, and microservices, are often rightly described as the future of computing, however they aren’t without their problems. In this talk we’ll look at some of the real world complexities and challenges we’ve faced deploying and using Kubernetes at Monzo, along with the advantages this now gives us as we rapidly develop our product and grow our company.

Presented at Bristech's 50th meetup.

Transcript

1. Kubernetes in Context Matt Heath, Monzo
2. Hi, I’m Matt @mattheath
15. A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2017 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE
16. A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2017 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE PREPAID
 LAUNCH
17. Nov 2015 Nov 2017 CUSTOMER
 GROWTH
18. monoliths traditional dev
20. ?
21. Application
22. Application Database
23. Application Database
24. Application Databases
25. Application Databases Search
26. Application Databases Search Caching
27. Application Databases Search Caching CAT GIFS
28. Application
29. APPLICATION Application
33. Feb 2015 services Oct 2017 300
35. pod
36. pod container container container
37. pod transaction service
38. pod web server config volume
39. pod volume
40. pod
41. pod
42. pods replica set
43. replica set pods deployment
44. pods replica set deployment
45. pods replica set deployment change
46. deployment change old pods old replica set
47. deployment change old pods old replica set new replica set
48. deployment change old pods old replica set new replica set new pod
49. deployment change old pods old replica set new replica set new pod
50. deployment change old replica set new pods new replica set
51. deployment change new replica set new pods
52. deployment replica set pods
53. pods
54. pods service
55. pods service kubedns
56. pods service routing - static cluster IP managed replicas resource allocations rollout strategies liveness checks anti-affinity, etc kubedns
57. ?
58. server
59. server server
60. server server server server server server
61. server server server server server server Kubernetes
62. server server server server server kubelet
63. kubelet kubelet kubelet kubelet kubelet kubelet
64. kubelet kubelet kubelet kubelet kubelet kubelet Kubernetes Master
65. kubelet kubelet kubelet kubelet kubelet kubelet Kubernetes Master etcd
66. kubernetes
68. + = ❤
69. kubernetes service service service service service service service service service service
70. Service Service
71. Service Service Service Service
72. Service Service Service Service Kubernetes
 Service
73. Service Discovery Load Balancing Timeouts and Expirations Retries Rate Limiting Connection Pooling Circuit Breaking Failure Detection Metrics and Tracing Interrupts Context Propagation
74. Service Discovery Load Balancing Timeouts and Expirations Retries Rate Limiting Connection Pooling Circuit Breaking Failure Detection Metrics and Tracing Interrupts Context Propagation
75. Service Service Service Service ?
76. Service Service Service Service linkerd
77. Service linkerd
78. Service linkerd service
 discovery
79. Service Service Service Service linkerd service
 discovery
80. Service Service Service Service linkerd
81. Service Service Service Service linkerd
82. Service Service Service Service linkerd
83. Service Service Service Service linkerd
84. Service Service Service Service Service host A Service
85. Service Service Service Service Service linkerd host A Service
86. Service Service Service Service linkerd host B Service Service Service Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
87. Service Service Service Service linkerd host B Service Service Service Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
88. Service Service Service Service linkerd host B Service Service Service Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
89. Service Service Service Service linkerd host B Service Service Service Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
90. Service Service Service Service linkerd host B Service Service Service Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
91. Load Balancer
92. Load Balancer HTTP API & Routing Layer
93. Load Balancer HTTP API & Routing Layer API
 Service
95. /webhooks —-> Webhook API
96. Load Balancer HTTP API & Routing Layer Webhook
 API
97. Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing Layer Webhook
 API
98. Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing Layer Webhook
 API Database
99. Database Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing Layer Webhook
 API Database
100. External Provider Database Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing Layer Webhook
 API Database
101. Service A Service B Load Balancer HTTP API & Routing Layer API Service
102. API Service Service A Service B Load Balancer HTTP API & Routing Layer
103. API Service Service A Service B Load Balancer HTTP API & Routing Layer
104. API Service Service A Service B Load Balancer HTTP API & Routing Layer Service C Service D Service E
106. Amazon 
 eu-west-1
107. Amazon 
 eu-west-1
109. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2
110. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2
111. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2
112. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2 BGP BGP BGP BGP
113. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2 BGP BGP
114. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2
115. third parties colocation “connectivity” pod services HA VPN
 Appliance VPN client Kubernetes
116. third parties colocation “connectivity” pod service HA VPN
 Appliance Kubernetes Wireguard
 network
 interface service pod GUE 
 forwarder routing table
 routes specific
 IP range to local
 gue0 interface GUE 
 listener routing table
 routes packets
 to local wg0
 interface *lots of stuff omitted for brevity, this is a lot more complicated than the diagram implies
117. Amazon 
 eu-west-1 Yay abstraction!
118. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
119. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
120. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
121. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
122. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
124. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns