Building Microservice Architectures with Go - Microservices Meetup Berlin

Presented at Microservices Meetup Berlin
http://www.meetup.com/Microservices-Meetup-Berlin/events/228032860/

Traditionally applications have been built as monoliths; single applications which become larger and more complex over time, which limit our ability to react to change. An example of this is the banking industry where mergers and acquisitions between banks have lead to a patchwork of different systems & technologies that cost billions of dollars per year to maintain. As a result, the pace of innovation in the banking industry has slowed to a crawl.

At Mondo we're building a new kind of bank, a smart bank that belongs in the 21st century, and we’re building it almost entirely in Go. This talk will cover how we’re developing new core banking systems from scratch backed by a microservice platform written in Go, running across multiple data centres using open source frameworks and tools including Docker and Mesos. We'll look at why Go is perfectly suited to this, our architectural decisions, common pitfalls to avoid, and how microservice architectures can vastly increase both the velocity of development teams and the scalability and fault tolerance of our systems.

Transcript

1. Building Microservice
   Architectures with Go
   Matt Heath, Mondo
   

   View Slide

2. @mattheath
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. monoliths
   traditional dev
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. ?
    

    View Slide

17. DATABASE
    APPLICATION
    

    View Slide

18. DATABASE
    APPLICATION
    

    View Slide

19. DATABASE
    DATABASES
    APPLICATION
    

    View Slide

20. DATABASE
    DATABASES
    APPLICATION
    SEARCH
    

    View Slide

21. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    

    View Slide

22. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    CAT GIFS
    

    View Slide

23. ALL
    HAIL
    THE
    MONOLITH
    

    View Slide

24. DO NOT WANT
    

    View Slide

25. DATABASE
    DATABASES
    APPLICATION
    CACHE
    SEARCH
    CAT GIFS
    

    View Slide

26. APPLICATION
    

    View Slide

27. View Slide

28. View Slide

29. Microservices
    at Mondo?
    

    View Slide

30. Single Responsibility
    Principle
    

    View Slide

31. Bounded Context
    

    View Slide

32. Well defined
    Interfaces
    

    View Slide

33. Composability
    

    View Slide

34. Getting started
    

    View Slide

35. LOAD BALANCER
    

    View Slide

36. LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

37. TRANSPORT
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

38. SERVICE SERVICE SERVICE
    TRANSPORT
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

39. SERVICE SERVICE SERVICE
    TRANSPORT
    DATABASE DATABASE DATABASE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

40. SERVICE SERVICE SERVICE
    TRANSPORT
    DATABASE DATABASE DATABASE
    LOAD BALANCER
    API GATEWAY
    

    View Slide

41. APPLICATION
    LOAD BALANCER
    

    View Slide

42. API GATEWAY
    APPLICATION
    LOAD BALANCER
    

    View Slide

43. API GATEWAY
    APPLICATION
    LOAD BALANCER
    

    View Slide

44. API GATEWAY
    APPLICATION
    LOAD BALANCER
    SERVICE
    SERVICE
    SERVICES
    

    View Slide

45. API GATEWAY
    LOAD BALANCER
    SERVICE
    SERVICE
    SERVICES
    

    View Slide

46. View Slide

47. Simple
    Static typing
    Static linking
    

    View Slide

48. Comprehensive stdlib
    eg. Networking
    

    View Slide

49. Concurrency
    

    View Slide

50. Interfaces
    

    View Slide

51. Go Kit
    micro
    gRPC
    Kite
    

    View Slide

52. mondough/typhon
    mondough/mercury
    

    View Slide

53. LOAD BALANCER
    

    View Slide

54. LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

55. API
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

56. View Slide

57. /webhooks —-> Webhook API
    

    View Slide

58. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

59. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

60. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    DATABASE
    

    View Slide

61. WEBHOOK
    API
    AUTH
    SERVICE
    WEBHOOK
    SERVICE
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    DATABASE DATABASE
    

    View Slide

62. SERVICE
    

    View Slide

63. Logic
    Handlers
    Storage
    SERVICE
    

    View Slide

64. mercury
    Logic
    Handlers
    Storage
    SERVICE
    

    View Slide

65. mercury
    Logic
    Handlers
    Storage libraries
    SERVICE
    

    View Slide

66. func main() {
    }
    

    View Slide

67. func main() {
    }
    

    View Slide

68. func main() {
    }
    

    View Slide

69. func main() {
    srv := service.Init(service.Config{
    Name: "service.account",
    Description: "Store bank accounts",
    })
    srv.AddEndpoints(
    handler.Create,
    handler.Read,
    handler.List,
    )
    srv.Run()
    }
    

    View Slide

70. func main() {
    srv := service.Init(service.Config{
    Name: "service.account",
    Description: "Store bank accounts",
    })
    srv.AddEndpoints(
    handler.Create,
    handler.Read,
    handler.List,
    )
    srv.Run()
    }
    

    View Slide

71. func main() {
    srv := service.Init(service.Config{
    Name: "service.account",
    Description: "Store bank accounts",
    })
    srv.AddEndpoints(
    handler.Create,
    handler.Read,
    handler.List,
    )
    srv.Run()
    }
    

    View Slide

72. type Endpoint struct {
    Name string
    Handler Handler
    Request interface{}
    Response interface{}
    }
    

    View Slide

73. type Endpoint struct {
    Name string
    Handler Handler
    Request interface{}
    Response interface{}
    }
    

    View Slide

74. type Endpoint struct {
    Name string
    Handler Handler
    Request interface{}
    Response interface{}
    }
    

    View Slide

75. type Handler func(request) (response, error)
    

    View Slide

76. func main() {
    srv := service.Init(service.Config{
    Name: "service.account",
    Description: "boop",
    })
    srv.AddEndpoints(
    handler.Create,
    handler.Read,
    handler.List,
    )
    srv.Run()
    }
    

    View Slide

77. func main() {
    srv := service.Init(service.Config{
    Name: "service.account",
    Description: "boop",
    })
    srv.AddEndpoints(
    handler.Create,
    handler.Read,
    handler.List,
    )
    srv.Run()
    }
    

    View Slide

78. type Server interface {
    Name() string
    AddEndpoints(eps ...Endpoint)
    RemoveEndpoints(eps ...Endpoint)
    Endpoint(name string) (Endpoint, bool)
    Endpoints() []Endpoint
    Start(trans transport.Transport) error
    Run(trans transport.Transport)
    Stop()
    Middleware() []ServerMiddleware
    SetMiddleware([]ServerMiddleware)
    AddMiddleware(ServerMiddleware)
    }
    

    View Slide

79. type Transport interface {
    Tomb() *tomb.Tomb
    Ready() Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    }
    

    View Slide

80. type Transport interface {
    Tomb() *tomb.Tomb
    Ready() Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    }
    

    View Slide

81. type Transport interface {
    Tomb() *tomb.Tomb
    Ready() Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    }
    

    View Slide

82. type Transport interface {
    Tomb() *tomb.Tomb
    Ready() Listen(serviceName string, inbound chanStopListening(serviceName string) bool
    Respond(req request, resp response) error
    Send(req request, timeout time.Duration) (response, error)
    }
    

    View Slide

83. mercury
    Logic
    Handlers
    Storage libraries
    SERVICE
    Deployment
    Service Discovery
    Configuration
    Monitoring
    Authentication
    Authorisation
    Storage
    Circuit Breaking
    

    View Slide

84. View Slide

85. View Slide

86. Statically Compiled
    Small Images
    Root CA Certs
    

    View Slide

87. View Slide

88. Making our
    service reliable
    

    View Slide

89. Event Driven
    Architecture
    

    View Slide

90. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

91. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

92. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    

    View Slide

93. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    SERVICE
    C
    SERVICE
    D
    E
    

    View Slide

94. API
    SERVICE
    SERVICE
    A
    SERVICE
    B
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    SERVICE
    C
    SERVICE
    D
    G
    E
    F
    

    View Slide

95. ?
    ? ?
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    ?
    ?
    ??
    ?
    ?
    ?
    ?
    ?
    ?
    ?
    

    View Slide

96. Topology
    Management
    

    View Slide

97. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

98. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

99. WEBHOOK
    API
    LOAD BALANCER
    HTTP API & ROUTING LAYER
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    WEBHOOK
    SERVICE
    

    View Slide

100. WEBHOOK
     API
     LOAD BALANCER
     HTTP API & ROUTING LAYER
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     SLOW /
     ERRORS
     

     View Slide

101. View Slide

102. View Slide

103. View Slide

104. View Slide

105. View Slide

106. Fanout &
     Cancellation
     

     View Slide

107. WEBHOOK
     API
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     

     View Slide

108. WEBHOOK
     API
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     

     View Slide

109. WEBHOOK
     API
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     

     View Slide

110. WEBHOOK
     API
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     

     View Slide

111. WEBHOOK
     API
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     WEBHOOK
     SERVICE
     

     View Slide

112. Context
     Propagation
     

     View Slide

113. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

114. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

115. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

116. View Slide

117. View Slide

118. View Slide

119. View Slide

120. Testing
     

     View Slide

121. Load
     Failure
     Degradation
     

     View Slide

122. Monitoring
     

     View Slide

123. Monitor your
     Business Logic
     

     View Slide

124. ♥ customers
     

     View Slide

125. In-Band vs
     Out of Band
     

     View Slide

126. type Checker func() (error, map[string]string)
     

     View Slide

127. View Slide

128. Distributed Tracing
     

     View Slide

129. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

130. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

131. 8096820c-3b7b-47ec-bce6-1c239252ab40
     

     View Slide

132. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

133. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

134. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

135. package context
     type Context interface {
     Deadline() (deadline time.Time, ok bool)
     Done() Err() error
     Value(key interface{}) interface{}
     }
     

     View Slide

136. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     

     View Slide

137. api
     api
     api.customer
     api.customer
     service.customer
     service.customer
     SEND
     RECV
     SEND
     RECV
     RECV
     SEND
     RECV
     SEND
     

     View Slide

138. api api.customer service.customer
     SEND
     RECV
     SEND
     RECV
     RECV
     SEND
     RECV
     SEND
     phosphor
     

     View Slide

139. View Slide

140. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

141. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

142. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

143. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

144. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

145. View Slide

146. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
     

     View Slide

147. View Slide

148. View Slide

149. View Slide

150. Small
     Simple
     Easy to learn
     

     View Slide

151. Concurrency
     Interfaces
     Networking
     

     View Slide

152. Downsides?
     

     View Slide

153. Starting with
     Microservices?
     

     View Slide

154. Thanks!
     @mattheath
     @getmondo
     

     View Slide

155. ATM: Thomas Hawk

     Bank of Commerce: ABQ Museum Archives
     IBM System/360: IBM
     Absorbed: Saxbald Photography
     Orbital Ion Cannon: www.rom.ac
     Go Gopher: Renee French
     Control Room: NASA
     ATM Failure: George Redgrave
     Credits
     

     View Slide