Building a Bank with Kubernetes – Kubernetes London Meetup, Autumn 2016

Presented at the Kubernetes London Meetup, Autumn 2016

In this talk, Oliver Beattie, Head of Engineering at Monzo, talks about how Monzo is building a modern bank.

* How Monzo has architected its backend as a collection of microservices
* How the communication between microservices is one of the most important parts of the system
* How Monzo interconnects cloud and overlay networks with physical hardware and third parties

Transcript

1. Building a Bank with Kubernetes
   

   View Slide

2. Oliver Beattie
   @obeattie
   Head of Engineering, Monzo
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. Application
    Database
    

    View Slide

11. Application
    Database Cache
    

    View Slide

12. Application
    Database Cache
    Load balancer
    Application
    Application
    

    View Slide

13. Application
    Database Cache
    Load balancer
    Cache
    Cache
    Application
    Application
    

    View Slide

14. Application
    Database Cache
    Load balancer
    Database
    Database
    Cache
    Cache
    Application
    Application
    

    View Slide

15. Application
    Database Cache
    Load balancer
    Application
    Application
    Database
    Database
    Cache
    Cache
    

    View Slide

16. Application
    Database Cache
    Load balancer
    Application
    Application
    Database
    Database
    Cache
    Cache
    

    View Slide

17. View Slide

18. View Slide

19. Efficient
    Resilient
    Extensible
    Secure
    

    View Slide

20. View Slide

21. Efficient
    Resilient
    Extensible
    Secure
    

    View Slide

22. View Slide

23. k8 -worker hosts
    POD #2
    CONTAINER
    POD #1
    CONTAINER
    POD #3
    CONTAINER
    POD #4
    CONTAINER
    1000MB memory; 100% utilised
    250MB 250MB
    250MB
    250MB
    

    View Slide

24. Efficient
    Resilient
    Extensible
    Secure
    

    View Slide

25. View Slide

26. View Slide

27. Load balancing
    Tracing
    Circuit breakers
    Retries
    Canarying
    Service discovery
    Logging
    Timeouts
    Expirations
    Security policies
    

    View Slide

28. linkerd
    

    View Slide

29. HOST A
    service.
    CONTAINER
    10.224.15.2
    LINKERD
    10.102.32.198
    10.224.15.3
    GET / HTTP/1.1
    Host: service.
    HOST A
    service.cruft
    CONTAINER
    10.224.16.2
    service.cruft
    CONTAINER
    10.224.16.1
    LINKERD
    10.102.34.192
    10.224.16.3
    HOST C
    service.
    CONTAINER
    10.224.17.2
    LINKERD
    10.102.36.187
    10.224.17.3
    service.cruft
    CONTAINER
    10.224.17.1
    service.cruft
    CONTAINER
    10.224.15.1
    

    View Slide

30. HOST A
    service.
    CONTAINER
    10.224.15.2
    service.cruft
    CONTAINER
    10.224.15.1
    LINKERD
    10.102.32.198
    10.224.15.3
    Host: service.
    → 10.224.17.3
    GET / HTTP/1.1
    Host: service.
    HOST A
    service.cruft
    CONTAINER
    10.224.16.2
    service.cruft
    CONTAINER
    10.224.16.1
    LINKERD
    10.102.34.192
    10.224.16.3
    HOST C
    service.
    CONTAINER
    10.224.17.2
    service.cruft
    CONTAINER
    10.224.17.1
    LINKERD
    10.102.36.187
    10.224.17.3
    

    View Slide

31. HOST A
    service.
    CONTAINER
    10.224.15.2
    service.cruft
    CONTAINER
    10.224.15.1
    LINKERD
    10.102.32.198
    10.224.15.3
    Host: service.
    → 10.224.17.3
    GET / HTTP/1.1
    Host: service.
    HOST A
    service.cruft
    CONTAINER
    10.224.16.2
    service.cruft
    CONTAINER
    10.224.16.1
    LINKERD
    10.102.34.192
    10.224.16.3
    HOST C
    service.
    CONTAINER
    10.224.17.2
    service.cruft
    CONTAINER
    10.224.17.1
    LINKERD
    10.102.36.187
    10.224.17.3
    

    View Slide

32. HOST A
    service.
    CONTAINER
    10.224.15.2
    service.cruft
    CONTAINER
    10.224.15.1
    LINKERD
    10.102.32.198
    10.224.15.3
    Host: service.
    → 10.224.17.3
    GET / HTTP/1.1
    Host: service.
    HOST A
    service.cruft
    CONTAINER
    10.224.16.2
    service.cruft
    CONTAINER
    10.224.16.1
    LINKERD
    10.102.34.192
    10.224.16.3
    HOST C
    service.
    CONTAINER
    10.224.17.2
    service.cruft
    CONTAINER
    10.224.17.1
    LINKERD
    10.102.36.187
    10.224.17.3
    Host: service.
    → 10.224.17.2
    

    View Slide

33. View Slide

34. AWS region
    eu-west-1
    
    

    View Slide

35. View Slide

36. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    

    View Slide

37. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    ⚡
    

    View Slide

38. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    ⚡
    
    

    View Slide

39. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    BGP
    BGP
    BGP
    BGP
    

    View Slide

40. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    BGP
    BGP
    BGP
    BGP
    ⚡ ⚡
    

    View Slide

41. Co-location
    uk-1
    Co-location
    uk-2
    AWS region
    eu-west-1
    BGP
    BGP
    ⚡ ⚡
    
    

    View Slide

42. Connectivity Pod
    BGP
    IPSec
    BGP BGP
    Openswan
    (IPSec VPN)
    GNU Zebra

    (BGP)
    Hardware VPN
    device
    Services
    Services
    Services
    AWS Co-location Third parties
    

    View Slide

43. Efficient
    Resilient
    Extensible
    Secure
    

    View Slide

44. Device isolation
    Process isolation
    Data encryption
    Filesystem isolation
    Privilege isolation
    Network isolation
    Resource isolation
    Principle of least privilege
    Log monitoring
    Secret management
    

    View Slide

45. Device isolation
    Process isolation
    Data encryption
    Filesystem isolation
    Privilege isolation
    Network isolation
    Resource isolation
    Principle of least privilege
    Log monitoring
    Secret management
    

    View Slide

46. k8s-master
    Availability Zone A Availability Zone B Availability Zone C
    
    admin
    user
    data
    k8s-worker
    dmz
    

    View Slide

47. k8s-master
    Availability Zone A Availability Zone B Availability Zone C
    
    k8s-worker
    dmz
    

    View Slide

48. zone: super-secure
    

    View Slide

49. apiVersion: v1
    kind: Namespace
    metadata:
    name: bees
    

    View Slide

50. apiVersion: v1
    kind: Namespace
    metadata:
    name: bees
    annotations:
    net.beta.kubernetes.io/network-policy: |
    {
    "ingress": {
    "isolation": "DefaultDeny"
    }
    }
    

    View Slide

51. apiVersion: extensions/v1beta1
    kind: NetworkPolicy
    metadata:
    name: super-secure-zone
    spec:
    podSelector:
    matchLabels:
    zone: super-secure
    ingress:
    - from:
    - podSelector:
    matchLabels:
    zone: super-secure
    ports:
    - protocol: tcp
    

    View Slide

52. @obeattie
    

    View Slide