Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes in Context

67f4a8f2a209a38d7242829947b26ba3?s=47 mattheath
November 02, 2017

Kubernetes in Context

The rise of containerisation, orchestration, and microservices, are often rightly described as the future of computing, however they aren’t without their problems. In this talk we’ll look at some of the real world complexities and challenges we’ve faced deploying and using Kubernetes at Monzo, along with the advantages this now gives us as we rapidly develop our product and grow our company.

Presented at Bristech's 50th meetup.

67f4a8f2a209a38d7242829947b26ba3?s=128

mattheath

November 02, 2017
Tweet

Transcript

  1. Kubernetes in Context Matt Heath, Monzo

  2. Hi, I’m Matt @mattheath

  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. A UK banking licence is authorised by the PRA and

    regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2017 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE
  16. A UK banking licence is authorised by the PRA and

    regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2017 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE PREPAID
 LAUNCH
  17. Nov 2015 Nov 2017 CUSTOMER
 GROWTH

  18. monoliths traditional dev

  19. None
  20. ?

  21. Application

  22. Application Database

  23. Application Database

  24. Application Databases

  25. Application Databases Search

  26. Application Databases Search Caching

  27. Application Databases Search Caching CAT GIFS

  28. Application

  29. APPLICATION Application

  30. None
  31. None
  32. None
  33. Feb 2015 services Oct 2017 300

  34. None
  35. pod

  36. pod container container container

  37. pod transaction service

  38. pod web server config volume

  39. pod volume

  40. pod

  41. pod

  42. pods replica set

  43. replica set pods deployment

  44. pods replica set deployment

  45. pods replica set deployment change

  46. deployment change old pods old replica set

  47. deployment change old pods old replica set new replica set

  48. deployment change old pods old replica set new replica set

    new pod
  49. deployment change old pods old replica set new replica set

    new pod
  50. deployment change old replica set new pods new replica set

  51. deployment change new replica set new pods

  52. deployment replica set pods

  53. pods

  54. pods service

  55. pods service kubedns

  56. pods service routing - static cluster IP managed replicas resource

    allocations rollout strategies liveness checks anti-affinity, etc kubedns
  57. ?

  58. server

  59. server server

  60. server server server server server server

  61. server server server server server server Kubernetes

  62. server server server server server kubelet

  63. kubelet kubelet kubelet kubelet kubelet kubelet

  64. kubelet kubelet kubelet kubelet kubelet kubelet Kubernetes Master

  65. kubelet kubelet kubelet kubelet kubelet kubelet Kubernetes Master etcd

  66. kubernetes

  67. None
  68. + = ❤

  69. kubernetes service service service service service service service service service

    service
  70. Service Service

  71. Service Service Service Service

  72. Service Service Service Service Kubernetes
 Service

  73. Service Discovery Load Balancing Timeouts and Expirations Retries Rate Limiting

    Connection Pooling Circuit Breaking Failure Detection Metrics and Tracing Interrupts Context Propagation
  74. Service Discovery Load Balancing Timeouts and Expirations Retries Rate Limiting

    Connection Pooling Circuit Breaking Failure Detection Metrics and Tracing Interrupts Context Propagation
  75. Service Service Service Service ?

  76. Service Service Service Service linkerd

  77. Service linkerd

  78. Service linkerd service
 discovery

  79. Service Service Service Service linkerd service
 discovery

  80. Service Service Service Service linkerd

  81. Service Service Service Service linkerd

  82. Service Service Service Service linkerd

  83. Service Service Service Service linkerd

  84. Service Service Service Service Service host A Service

  85. Service Service Service Service Service linkerd host A Service

  86. Service Service Service Service linkerd host B Service Service Service

    Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
  87. Service Service Service Service linkerd host B Service Service Service

    Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
  88. Service Service Service Service linkerd host B Service Service Service

    Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
  89. Service Service Service Service linkerd host B Service Service Service

    Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
  90. Service Service Service Service linkerd host B Service Service Service

    Service Service Service Service linkerd host A Service Service Service Service Service Service linkerd host C Service
  91. Load Balancer

  92. Load Balancer HTTP API & Routing Layer

  93. Load Balancer HTTP API & Routing Layer API
 Service

  94. None
  95. /webhooks —-> Webhook API

  96. Load Balancer HTTP API & Routing Layer Webhook
 API

  97. Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing

    Layer Webhook
 API
  98. Auth
 Service Webhook
 Service Load Balancer HTTP API & Routing

    Layer Webhook
 API Database
  99. Database Auth
 Service Webhook
 Service Load Balancer HTTP API &

    Routing Layer Webhook
 API Database
  100. External Provider Database Auth
 Service Webhook
 Service Load Balancer HTTP

    API & Routing Layer Webhook
 API Database
  101. Service A Service B Load Balancer HTTP API & Routing

    Layer API Service
  102. API Service Service A Service B Load Balancer HTTP API

    & Routing Layer
  103. API Service Service A Service B Load Balancer HTTP API

    & Routing Layer
  104. API Service Service A Service B Load Balancer HTTP API

    & Routing Layer Service C Service D Service E
  105. None
  106. Amazon 
 eu-west-1

  107. Amazon 
 eu-west-1

  108. None
  109. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2

  110. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2

  111. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2

  112. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2 BGP

    BGP BGP BGP
  113. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2 BGP

    BGP
  114. Amazon 
 eu-west-1 colocation
 dc 1 colocation
 dc 2

  115. third parties colocation “connectivity” pod services HA VPN
 Appliance VPN

    client Kubernetes
  116. third parties colocation “connectivity” pod service HA VPN
 Appliance Kubernetes

    Wireguard
 network
 interface service pod GUE 
 forwarder routing table
 routes specific
 IP range to local
 gue0 interface GUE 
 listener routing table
 routes packets
 to local wg0
 interface *lots of stuff omitted for brevity, this is a lot more complicated than the diagram implies
  117. Amazon 
 eu-west-1 Yay abstraction!

  118. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  119. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  120. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  121. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  122. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  123. None
  124. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  125. None
  126. None
  127. None