Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes in Context

mattheath
November 02, 2017
Programming
0
210

Kubernetes in Context

The rise of containerisation, orchestration, and microservices, are often rightly described as the future of computing, however they aren’t without their problems. In this talk we’ll look at some of the real world complexities and challenges we’ve faced deploying and using Kubernetes at Monzo, along with the advantages this now gives us as we rapidly develop our product and grow our company.

Presented at Bristech's 50th meetup.

mattheath

November 02, 2017
Tweet

More Decks by mattheath

See All by mattheath
Breaking down problems
mattheath
2
260
Contexts in Context
mattheath
0
130
A guided journey of Cloud Native
mattheath
1
100
Modelling prototypes to critical systems with Cassandra
mattheath
0
130
Building a Cloud Native Bank
mattheath
1
150
Building a Cloud Native Bank
mattheath
0
220
Building reliable APIs
mattheath
0
190
Go and Microservices - NDC London 2018
mattheath
0
140
Architecting a Bank from scratch
mattheath
1
650

Other Decks in Programming

See All in Programming
改めて整理するWebアプリのビルド・デプロイの基本【コンテナ編】
os1ma
2
280
Cloudflare 概論 Cloudflare Meetup Kick Off! #CloudflareUG
maroon1st
0
530
DDD Demystified ~結局DDDとは何なのか？何でないのか？~ #phperkaigi_reject
77web
0
620
＼ 祝！／AWS 大阪リージョン 2 周年の歩み
track3jyo
PRO
0
160
CSC308 Lecture 26
javiergs
PRO
0
120
Money Forward XでのGo利用事例について
keitaro1020
3
320
新社会人でも活躍したい！〜新卒1年目で活躍するためのコツ〜
ogijun2018
1
290
アンドキュメンテッド ちょうぜつソフトウェア 設計入門 「オブジェクト指向に定義はない」のか？
tanakahisateru
4
970
Eloquentとクエリビルダを両方使った実装の失敗例 - 第150回 PHP勉強会 #phpstudy
kotomin_m
0
170
Learning PHP with PHP Parser
inouehi
1
170
名付けできない画面を作ってはならない - 名前を付けるとは何か
chatii
0
270
Flaky Tests - Fighting Nightmares
leichteckig
0
200

Featured

See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
12
1.6k
Docker and Python
trallard
30
2k
Large-scale JavaScript Application Architecture
addyosmani
499
110k
Teambox: Starting and Learning
jrom
124
7.9k
A designer walks into a library…
pauljervisheath
199
16k
How GitHub Uses GitHub to Build GitHub
holman
466
280k
Designing Experiences People Love
moore
130
22k
GraphQLとの向き合い方2022年版
quramy
22
10k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
31
20k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
270
12k
Making Projects Easy
brettharned
102
4.9k

Transcript

  1. Kubernetes in Context
    Matt Heath, Monzo

    View Slide

  2. Hi, I’m Matt
    @mattheath

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2017
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE

    View Slide

  16. A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2017
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    PREPAID

    LAUNCH

    View Slide

  17. Nov
    2015
    Nov
    2017
    CUSTOMER

    GROWTH

    View Slide

  18. monoliths
    traditional dev

    View Slide

  19. View Slide

  20. ?

    View Slide

  21. Application

    View Slide

  22. Application
    Database

    View Slide

  23. Application
    Database

    View Slide

  24. Application
    Databases

    View Slide

  25. Application
    Databases
    Search

    View Slide

  26. Application
    Databases
    Search
    Caching

    View Slide

  27. Application
    Databases
    Search
    Caching
    CAT GIFS

    View Slide

  28. Application

    View Slide

  29. APPLICATION
    Application

    View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. Feb
    2015
    services Oct
    2017
    300

    View Slide

  34. View Slide

  35. pod

    View Slide

  36. pod
    container
    container
    container

    View Slide

  37. pod
    transaction service

    View Slide

  38. pod
    web server
    config volume

    View Slide

  39. pod
    volume

    View Slide

  40. pod

    View Slide

  41. pod

    View Slide

  42. pods
    replica set

    View Slide

  43. replica set
    pods
    deployment

    View Slide

  44. pods
    replica set
    deployment

    View Slide

  45. pods
    replica set
    deployment
    change

    View Slide

  46. deployment
    change
    old pods
    old replica set

    View Slide

  47. deployment
    change
    old pods
    old replica set new replica set

    View Slide

  48. deployment
    change
    old pods
    old replica set new replica set
    new pod

    View Slide

  49. deployment
    change
    old pods
    old replica set new replica set
    new pod

    View Slide

  50. deployment
    change
    old replica set
    new pods
    new replica set

    View Slide

  51. deployment
    change
    new replica set
    new pods

    View Slide

  52. deployment
    replica set
    pods

    View Slide

  53. pods

    View Slide

  54. pods
    service

    View Slide

  55. pods
    service
    kubedns

    View Slide

  56. pods
    service routing - static cluster IP
    managed replicas
    resource allocations
    rollout strategies
    liveness checks
    anti-affinity, etc
    kubedns

    View Slide

  57. ?

    View Slide

  58. server

    View Slide

  59. server server

    View Slide

  60. server server server server server server

    View Slide

  61. server server server server server server
    Kubernetes

    View Slide

  62. server server server server server
    kubelet

    View Slide

  63. kubelet kubelet kubelet kubelet kubelet kubelet

    View Slide

  64. kubelet kubelet kubelet kubelet kubelet kubelet
    Kubernetes Master

    View Slide

  65. kubelet kubelet kubelet kubelet kubelet kubelet
    Kubernetes Master
    etcd

    View Slide

  66. kubernetes

    View Slide

  67. View Slide

  68. + =
    ❤

    View Slide

  69. kubernetes
    service service service service service service
    service
    service
    service service

    View Slide

  70. Service
    Service

    View Slide

  71. Service
    Service Service
    Service

    View Slide

  72. Service
    Service Service
    Service
    Kubernetes

    Service

    View Slide

  73. Service Discovery
    Load Balancing
    Timeouts and Expirations
    Retries
    Rate Limiting
    Connection Pooling
    Circuit Breaking
    Failure Detection
    Metrics and Tracing
    Interrupts
    Context Propagation

    View Slide

  74. Service Discovery
    Load Balancing
    Timeouts and Expirations
    Retries
    Rate Limiting
    Connection Pooling
    Circuit Breaking
    Failure Detection
    Metrics and Tracing
    Interrupts
    Context Propagation

    View Slide

  75. Service
    Service Service
    Service
    ?

    View Slide

  76. Service
    Service Service
    Service
    linkerd

    View Slide

  77. Service
    linkerd

    View Slide

  78. Service
    linkerd
    service

    discovery

    View Slide

  79. Service
    Service Service
    Service
    linkerd
    service

    discovery

    View Slide

  80. Service
    Service Service
    Service
    linkerd

    View Slide

  81. Service
    Service Service
    Service
    linkerd

    View Slide

  82. Service
    Service Service
    Service
    linkerd

    View Slide

  83. Service
    Service Service
    Service
    linkerd

    View Slide

  84. Service
    Service Service
    Service
    Service
    host A
    Service

    View Slide

  85. Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service

    View Slide

  86. Service
    Service Service
    Service
    linkerd
    host B
    Service Service
    Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service
    Service
    Service Service
    Service
    Service
    linkerd
    host C
    Service

    View Slide

  87. Service
    Service Service
    Service
    linkerd
    host B
    Service Service
    Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service
    Service
    Service Service
    Service
    Service
    linkerd
    host C
    Service

    View Slide

  88. Service
    Service Service
    Service
    linkerd
    host B
    Service Service
    Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service
    Service
    Service Service
    Service
    Service
    linkerd
    host C
    Service

    View Slide

  89. Service
    Service Service
    Service
    linkerd
    host B
    Service Service
    Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service
    Service
    Service Service
    Service
    Service
    linkerd
    host C
    Service

    View Slide

  90. Service
    Service Service
    Service
    linkerd
    host B
    Service Service
    Service
    Service Service
    Service
    Service
    linkerd
    host A
    Service
    Service
    Service Service
    Service
    Service
    linkerd
    host C
    Service

    View Slide

  91. Load Balancer

    View Slide

  92. Load Balancer
    HTTP API & Routing Layer

    View Slide

  93. Load Balancer
    HTTP API & Routing Layer
    API

    Service

    View Slide

  94. View Slide

  95. /webhooks —-> Webhook API

    View Slide

  96. Load Balancer
    HTTP API & Routing Layer
    Webhook

    API

    View Slide

  97. Auth

    Service
    Webhook

    Service
    Load Balancer
    HTTP API & Routing Layer
    Webhook

    API

    View Slide

  98. Auth

    Service
    Webhook

    Service
    Load Balancer
    HTTP API & Routing Layer
    Webhook

    API
    Database

    View Slide

  99. Database
    Auth

    Service
    Webhook

    Service
    Load Balancer
    HTTP API & Routing Layer
    Webhook

    API
    Database

    View Slide

  100. External
    Provider
    Database
    Auth

    Service
    Webhook

    Service
    Load Balancer
    HTTP API & Routing Layer
    Webhook

    API
    Database

    View Slide

  101. Service
    A
    Service
    B
    Load Balancer
    HTTP API & Routing Layer
    API
    Service

    View Slide

  102. API
    Service
    Service
    A
    Service
    B
    Load Balancer
    HTTP API & Routing Layer

    View Slide

  103. API
    Service
    Service
    A
    Service
    B
    Load Balancer
    HTTP API & Routing Layer

    View Slide

  104. API
    Service
    Service
    A
    Service
    B
    Load Balancer
    HTTP API & Routing Layer
    Service
    C
    Service
    D
    Service
    E

    View Slide

  105. View Slide

  106. Amazon 

    eu-west-1

    View Slide

  107. Amazon 

    eu-west-1

    View Slide

  108. View Slide

  109. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2

    View Slide

  110. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2

    View Slide

  111. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2

    View Slide

  112. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2
    BGP
    BGP
    BGP
    BGP

    View Slide

  113. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2

    BGP
    BGP

    View Slide

  114. Amazon 

    eu-west-1
    colocation

    dc 1
    colocation

    dc 2


    View Slide

  115. third parties
    colocation
    “connectivity” pod
    services
    HA VPN

    Appliance
    VPN client

    Kubernetes

    View Slide

  116. third parties
    colocation
    “connectivity” pod
    service
    HA VPN

    Appliance

    Kubernetes
    Wireguard

    network

    interface
    service pod
    GUE 

    forwarder
    routing table

    routes specific

    IP range to local

    gue0 interface
    GUE 

    listener
    routing table

    routes packets

    to local wg0

    interface
    *lots of stuff omitted for brevity, this is a lot more complicated than the diagram implies

    View Slide

  117. Amazon 

    eu-west-1
    Yay abstraction!

    View Slide

  118. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  119. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  120. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  121. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  122. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  123. View Slide

  124. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
    API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns

    View Slide

  125. View Slide

  126. View Slide

  127. View Slide