Building a Cloud Native Bank

A brief look at how Monzo, a UK based digital bank, have leveraged the features of Amazon's public cloud to build a secure, cloud native, banking platform — using modern open source technologies and common software development practices.

Presented at the AWS User Group UK meetup

Transcript

1. Building a
   Cloud-Native Bank
   Matt Heath, Monzo
   

   View Slide

2. ! ⛅ #
   

   View Slide

3. Hi, I’m Matt
   

   View Slide

4. @mattheath
   

   View Slide

5. View Slide

6. View Slide

7. Nov
   2015
   Oct
   2018
   CUSTOMER

   GROWTH
   

   View Slide

8. $
   1,031,133
   

   View Slide

9. £4 billion spent
   %
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. #
    

    View Slide

19. &
    

    View Slide

20. View Slide

21. '
    

    View Slide

22. LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2017
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    

    View Slide

23. Henrik Kniberg
    

    View Slide

24. View Slide

25. ?
    

    View Slide

26. ☁
    

    View Slide

27. ⛈
    

    View Slide

28. ⛅
    

    View Slide

29. Henrik Kniberg
    

    View Slide

30. James
    Leah
    Matt
    

    View Slide

31. James
    Leah
    Matt
    *
    

    View Slide

32. James
    Leah
    Matt
    +
    

    View Slide

33. ,
    

    View Slide

34. View Slide

35. View Slide

36. -
    -
    -
    

    View Slide

37. -
    -
    -
    X
    X
    X
    

    View Slide

38. -
    

    View Slide

39. %
    -
    %
    

    View Slide

40. .
    

    View Slide

41. -
    ☕☕
    

    View Slide

42. ☕☕ -
    %
    

    View Slide

43. ☁
    

    View Slide

44. EXTENSIBLE
    SCALABLE
    RELIABLE
    SECURE

    CHEAP
    

    View Slide

45. ?
    

    View Slide

46. Application
    

    View Slide

47. Application
    

    View Slide

48. APPLICATION
    Application
    

    View Slide

49. View Slide

50. View Slide

51. View Slide

52. LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2017
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    

    View Slide

53. A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2017
    JAN

    2016
    Feb

    2015
    APR

    2017
    PREPAID

    CARD

    LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    

    View Slide

54. View Slide

55. View Slide

56. View Slide

57. View Slide

58. ?
    

    View Slide

59. EC2

    Instance
    

    View Slide

60. instance instance
    

    View Slide

61. instance instance instance instance instance instance
    

    View Slide

62. instance instance instance instance instance instance
    {
    {
    {
    eu-west-1a eu-west-1b eu-west-1c
    

    View Slide

63. instance instance instance instance instance instance
    Kubernetes
    

    View Slide

64. instance instance instance instance instance
    kubelet
    

    View Slide

65. kubelet kubelet kubelet kubelet kubelet kubelet
    

    View Slide

66. kubelet kubelet kubelet kubelet kubelet kubelet
    Kubernetes Master
    

    View Slide

67. kubelet kubelet kubelet kubelet kubelet kubelet
    Kubernetes Master
    etcd
    

    View Slide

68. kubelet kubelet kubelet kubelet kubelet kubelet
    Kubernetes Master
    etcd
    

    View Slide

69. etcd
    

    View Slide

70. etcd etcd
    etcd
    

    View Slide

71. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    

    View Slide

72. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    quorum = 5 or more nodes alive (n/2+1)
    

    View Slide

73. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    quorum maintained
    

    View Slide

74. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    quorum maintained '
    

    View Slide

75. etcd
    etcd
    etcd
    etcd
    etcd
    unhealthy machines terminated 0
    

    View Slide

76. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    new machines automatically reattach EBS, cluster recovers 1
    

    View Slide

77. etcd ASG 4 etcd ASG 7
    etcd ASG 1
    etcd ASG 5 etcd ASG 8
    etcd ASG 2
    etcd ASG 3 etcd ASG 6
    etcd ASG 0
    each etcd node is an autoscaling group of 1, with a fixed identity
    

    View Slide

78. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    quorum lost 2
    

    View Slide

79. etcd
    etcd
    etcd
    etcd
    unhealthy machines terminated 0
    

    View Slide

80. etcd
    etcd
    etcd
    etcd
    etcd ASG 7
    etcd ASG 1
    etcd ASG 8
    etcd ASG 2
    etcd ASG 6
    autoscaling groups start new instances
    

    View Slide

81. etcd etcd
    etcd
    etcd etcd
    etcd
    etcd etcd
    etcd
    unhealthy machines replaced automatically, EBS reattached, cluster recovers 3
    

    View Slide

82. Blog
    

    View Slide

83. kubernetes
    

    View Slide

84. kubernetes
    pod pod pod pod pod pod
    pod
    pod
    pod pod
    

    View Slide

85. pod
    container
    service
    

    View Slide

86. pod
    scratch container
    go service
    

    View Slide

87. kubernetes
    service service service service service service
    service
    service
    service service
    

    View Slide

88. kubernetes
    service service service service service service
    service
    service
    service service
    

    View Slide

89. kubernetes
    service service service service service service
    service
    service
    service service
    

    View Slide

90. kubernetes
    service service service service service service
    service
    service
    service service
    

    View Slide

91. kubernetes
    service service service service service service
    service
    service
    service service service
    service
    service
    

    View Slide

92. kubernetes
    service service service service service
    service service service
    service
    service
    

    View Slide

93. kubernetes
    service Prometheus service service Kafka linkerd
    Kafka
    service
    service Elasticsearch calico
    API
    service
    

    View Slide

94. View Slide

95. View Slide

96. Request
    Limit
    

    View Slide

97. CPU Throttling
    

    View Slide

98. View Slide

99. API
    

    View Slide

100. API
     

     View Slide

101. API Gateway
     API
     Service
     API
     Service
     API
     Service
     API
     Service
     API
     Service
     API
     Service
     API
     Service
     API
     Service
     

     View Slide

102. View Slide

103. View Slide

104. View Slide

105. API Gateway
     Accounts Cards Pots Feed Payments …
     

     View Slide

106. API Gateway
     Accounts Cards Pots Feed Payments …
     New

     Exciting

     API!!
     5
     

     View Slide

107. Cards Pots Payments …
     API Gateway
     Accounts
     New

     Exciting

     API!!
     0
     0
     0
     0
     0
     Feed
     

     View Slide

108. Cards Pots Payments …
     API Gateway
     Accounts
     New

     Exciting

     API!!
     0
     0
     0
     0
     0
     6
     Feed
     

     View Slide

109. API Gateway
     Accounts Cards Pots Payments …
     New

     Exciting

     API!!
     5
     6
     Feed
     

     View Slide

110. Service
     Service
     

     View Slide

111. Service
     Service Service
     Service
     

     View Slide

112. Service
     Service Service
     Service
     Kubernetes

     Service
     

     View Slide

113. Service Discovery
     Load Balancing
     Timeouts and Expirations
     Retries
     Rate Limiting
     Connection Pooling
     Circuit Breaking
     Failure Detection
     Metrics and Tracing
     Interrupts
     Context Propagation
     

     View Slide

114. Service Discovery
     Load Balancing
     Timeouts and Expirations
     Retries
     Rate Limiting
     Connection Pooling
     Circuit Breaking
     Failure Detection
     Metrics and Tracing
     Interrupts
     Context Propagation
     

     View Slide

115. Service
     Service Service
     Service
     ?
     

     View Slide

116. Service
     Service Service
     Service
     linkerd
     

     View Slide

117. Service
     linkerd
     

     View Slide

118. Service
     linkerd
     service

     discovery
     

     View Slide

119. Service
     Service Service
     Service
     linkerd
     service

     discovery
     

     View Slide

120. Service
     Service Service
     Service
     linkerd
     

     View Slide

121. Service
     Service Service
     Service
     linkerd
     

     View Slide

122. Service
     Service Service
     Service
     linkerd
     

     View Slide

123. Service
     Service Service
     Service
     linkerd
     

     View Slide

124. kind: DaemonSet
     metadata:
     name: linkerd
     spec:
     template:
     spec:
     containers:
     - name: linkerd
     image: ecr:linkerd_vXXX
     - "/etc/linkerd/linkerd.yaml"
     volumeMounts:
     - name: linkerd-config
     mountPath: /etc/linkerd
     readOnly: true
     ports:
     - name: http
     containerPort: 443
     hostPort: 4140
     env:
     - name: POD_IP
     valueFrom:
     fieldRef:
     fieldPath: status.podIP
     resources:
     limits:
     cpu: 4
     memory: 2Gi
     monzo.com/cpu-period: 2500
     requests:
     cpu: 1
     memory: 1Gi
     services send
     requests to local

     linkerd daemonset
     linkerd
     daemonset
     pods & services

     on other machines
     

     View Slide

125. View Slide

126. View Slide

127. View Slide

128. Event Driven

     Architecture
     

     View Slide

129. Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     API
     Service
     

     View Slide

130. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     

     View Slide

131. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     D
     Service
     E
     

     View Slide

132. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     

     View Slide

133. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     6
     

     View Slide

134. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     6
     7
     

     View Slide

135. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     6
     7
     7
     7
     7
     

     View Slide

136. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     6
     7
     7
     7
     7
     Service
     D
     

     View Slide

137. API
     Service
     Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     Service
     C
     Service
     D
     Service
     E
     6
     Service
     D
     

     View Slide

138. View Slide

139. API
     Service
     API
     Service
     Load Balancer
     Monzo API Gateway
     API
     Service
     

     View Slide

140. API
     Service
     API
     Service
     Load Balancer
     Monzo API Gateway
     API
     Service
     

     View Slide

141. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     

     View Slide

142. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     

     View Slide

143. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     AWS API
     

     View Slide

144. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     AWS API
     

     View Slide

145. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     AWS API
     

     View Slide

146. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     AWS API
     Reject traffic
     at ALB
     Update rules

     via API
     

     View Slide

147. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     AWS API
     Reject traffic
     at ALB
     Update rules

     via API
     

     View Slide

148. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     AWS Shield
     AWS WAF
     8
     

     View Slide

149. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     8
     

     View Slide

150. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     8
     service
     

     View Slide

151. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     8
     service
     &
     

     View Slide

152. View Slide

153. View Slide

154. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     8
     service
     VPC
     Segregation

     Network ACLs
     Sec Groups
     

     View Slide

155. API
     Service
     API
     Service
     Application Load Balancer (ALB)
     Monzo API Gateway
     API
     Service
     8
     service
     8
     

     View Slide

156. instance
     instance instance
     instance instance instance
     instance instance instance
     9
     

     View Slide

157. instance
     instance instance
     instance instance instance
     instance instance instance
     9
     IAM
     Auditing
     CloudTrail
     8
     

     View Slide

158. View Slide

159. :
     

     View Slide

160. View Slide

161. Ship it and iterate

     Make changes small

     Make changes often
     Technical debt as a tool
     

     View Slide

162. API Gateway
     Accounts Cards Pots Feed Payments …
     New

     Exciting

     API!!
     5
     

     View Slide

163. View Slide

164. Production load tests & Shadow Traffic
     

     View Slide

165. EXTENSIBLE
     SCALABLE
     RELIABLE
     SECURE

     CHEAP
     

     View Slide

166. Empower teams
     

     View Slide

167. monzo.com/careers
     

     View Slide