Building Microservice Architectures with Go - VoxxedDays Berlin 2016

Presented at VoxxedDays Berlin 2016

Traditionally applications have been built as monoliths; single applications which become larger and more complex over time, which limit our ability to react to change. An example of this is the banking industry where mergers and acquisitions between banks have lead to a patchwork of different systems & technologies that cost billions of dollars per year to maintain. As a result, the pace of innovation in the banking industry has slowed to a crawl.

At Mondo we're building a new kind of bank, a smart bank that belongs in the 21st century, and we’re building it almost entirely in Go. This talk will cover how we’re developing new core banking systems from scratch backed by a microservice platform written in Go, running across multiple data centres using open source frameworks and tools including Docker and Mesos. We'll look at why Go is perfectly suited to this, our architectural decisions, common pitfalls to avoid, and how microservice architectures can vastly increase both the velocity of development teams and the scalability and fault tolerance of our systems.

Transcript

1. @mattheath
   #VoxxedBerlin
   Platinum Sponsor
   Building Microservice
   Architectures with Go
   Matt Heath
   Mondo
   

   View Slide

2. @mattheath
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. monoliths
   traditional dev
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. ?
    

    View Slide

17. DATABASE
    APPLICATION
    

    View Slide

18. DATABASE
    APPLICATION
    

    View Slide

19. DATABASE
    DATABASES
    APPLICATION
    

    View Slide

20. DATABASE
    DATABASES
    APPLICATION
    SEARCH
    

    View Slide

21. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    

    View Slide

22. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    CAT GIFS
    

    View Slide

23. ALL
    HAIL
    THE
    MONOLITH
    

    View Slide

24. DO NOT WANT
    

    View Slide

25. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    CAT GIFS
    

    View Slide

26. APPLICATION
    

    View Slide

27. View Slide

28. View Slide

29. Microservices
    at Mondo?
    

    View Slide

30. Single Responsibility
    Principle
    

    View Slide

31. Bounded Context
    

    View Slide

32. Well defined
    Interfaces
    

    View Slide

33. Composability
    

    View Slide

34. Getting started
    

    View Slide

35. LOAD BALANCER
    

    View Slide

36. LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

37. TRANSPORT
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

38. SERVICE SERVICE SERVICE
    TRANSPORT
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

39. SERVICE SERVICE SERVICE
    TRANSPORT
    DATABASE DATABASE DATABASE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

40. SERVICE SERVICE SERVICE
    TRANSPORT
    DATABASE DATABASE DATABASE
    LOAD BALANCER
    API GATEWAY
    

    View Slide

41. APPLICATION
    LOAD BALANCER
    

    View Slide

42. API GATEWAY
    APPLICATION
    LOAD BALANCER
    

    View Slide

43. API GATEWAY
    APPLICATION
    LOAD BALANCER
    

    View Slide

44. API GATEWAY
    APPLICATION
    LOAD BALANCER
    SERVICE
    SERVICE
    SERVICES
    

    View Slide

45. View Slide

46. Simple
    Static typing
    Static linking
    

    View Slide

47. Comprehensive stdlib
    eg. Networking
    

    View Slide

48. Concurrency
    

    View Slide

49. Interfaces
    

    View Slide

50. Go Kit
    micro
    gRPC
    Kite
    

    View Slide

51. mondough/typhon
    mondough/mercury
    

    View Slide

52. LOAD BALANCER
    

    View Slide

53. LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

54. API
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

55. View Slide

56. /webhooks —-> Webhook API
    

    View Slide

57. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

58. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

59. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    DATABASE
    

    View Slide

60. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    DATABASE
    DATABASE
    

    View Slide

61. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    DATABASE
    DATABASE
    EXTERNAL
    PROVIDER
    

    View Slide

62. SERVICE
    

    View Slide

63. Logic
    Handlers
    Storage
    SERVICE
    

    View Slide

64. mercury
    Logic
    Handlers
    Storage
    SERVICE
    

    View Slide

65. type Handler func(request) (response, error)
    

    View Slide

66. type Transport interface {
    ...
    }
    

    View Slide

67. type Transport interface {
    Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    ...
    }
    

    View Slide

68. type Transport interface {
    Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    ...
    }
    

    View Slide

69. mercury
    Logic
    Handlers
    Storage
    SERVICE
    

    View Slide

70. mercury
    Logic
    Handlers
    Storage libraries
    SERVICE
    

    View Slide

71. mercury
    Logic
    Handlers
    Storage libraries
    SERVICE
    Deployment
    Service Discovery
    Configuration
    Monitoring
    Authentication
    Authorisation
    Storage
    Circuit Breaking
    

    View Slide

72. View Slide

73. View Slide

74. Small
    Images
    

    View Slide

75. Statically
    Compiled
    

    View Slide

76. Root CA
    Certs
    

    View Slide

77. View Slide

78. Making our
    service reliable
    

    View Slide

79. Event Driven
    Architecture
    

    View Slide

80. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

81. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

82. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

83. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    SERVICE
    C
    SERVICE
    D
    E
    

    View Slide

84. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    SERVICE
    C
    SERVICE
    D
    G
    E
    F
    

    View Slide

85. ?
    ? ?
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    ?
    ?
    ??
    ?
    ?
    ?
    ?
    ?
    ?
    ?
    

    View Slide

86. Topology
    Management
    

    View Slide

87. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

88. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

89. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

90. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    SLOW /
    ERRORS
    

    View Slide

91. Fanout &
    Cancellation
    

    View Slide

92. WEBHOOK
    API
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

93. WEBHOOK
    API
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

94. WEBHOOK
    API
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

95. WEBHOOK
    API
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

96. WEBHOOK
    API
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

97. Context
    Propagation
    

    View Slide

98. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

99. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

100. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

101. Testing
     

     View Slide

102. Load
     Failure
     Degradation
     

     View Slide

103. Monitoring
     

     View Slide

104. Monitor your
     Business Logic
     

     View Slide

105. ♥ customers
     

     View Slide

106. In-Band vs
     Out of Band
     

     View Slide

107. type Checker func() (error, map[string]string)
     

     View Slide

108. Distributed Tracing
     

     View Slide

109. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

110. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

111. 8096820c-3b7b-47ec-bce6-1c239252ab40
     

     View Slide

112. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

113. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

114. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

115. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

116. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

117. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     SEND
     RECV
     SEND
     RECV
     RECV
     SEND
     RECV
     SEND
     

     View Slide

118. api api.customer service.customer
     SEND
     RECV
     SEND
     RECV
     RECV
     SEND
     RECV
     SEND
     phosphor
     

     View Slide

119. View Slide

120. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

121. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

122. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

123. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

124. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

125. View Slide

126. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

127. View Slide

128. View Slide

129. View Slide

130. Small
     Simple
     Easy to learn
     

     View Slide

131. Concurrency
     Interfaces
     Networking
     

     View Slide

132. Downsides?
     

     View Slide

133. Starting with
     Microservices?
     

     View Slide

134. #voxxedberlin
     Thanks!
     @mattheath
     @getmondo
     

     View Slide

135. ATM: Thomas Hawk

     Bank of Commerce: ABQ Museum Archives
     IBM System/360: IBM
     Absorbed: Saxbald Photography
     Orbital Ion Cannon: www.rom.ac
     Go Gopher: Renee French
     Control Room: NASA
     ATM Failure: George Redgrave
     Credits
     

     View Slide