Elastic scaling in a (micro)service oriented architecture

Splitting an application up into multiple independent services can be a good way to keep it scaling and ensure stability and developer productivity in larger, growing teams. But just splitting the codebase, creating APIs and deploying the code on some servers is not enough, somehow your services need to know where and how other services are accessible. Classical approaches like hardcoding everything in every service or having a central load-balancer can quickly lead to problems in terms of scalability and maintainability. In this talk I'll show how we at ResearchGate tackled this challenge. With the help of tools like Consul and haproxy we created a setup that allows us to quickly boot and shutdown services. This ensures that all servers are utilized optimally and load spikes can be reacted upon quickly and automatically.

Transcript

1. Elastic Scaling in a (Micro)service oriented Architecture @BastianHofmann
4. Microservices
6. Service Oriented Architecture
7. Monolith
8. http://blog.philipphauer.de/microservices-nutshell-pros-cons/ Monolith Microservices
9. Benefits
10. Stricter separation of concerns
11. Diverse technology stacks
12. Things that you don’t want to do in language X
13. Problems
14. Problems
15. Challenges
16. Performance
17. Latency
18. Stability
19. Reliability
20. Transparency
21. Monitoring
22. Learning Curves
23. Code Reuse
24. Maintenance
25. Elastic Scaling?
26. Cloud Solutions
28. Using the cloud is not always possible
29. … or even desirable
30. Only solve some of the problems
31. How can we solve them
32. •Deploying •Running •Releasing •Configuring •Discovering •Monitoring •Error Handling •Scaling Agenda
33. A lot of this is also useful for monoliths
36. Our use case
37. https://www.flickr.com/photos/npobre/2601582256/
38. Deployment
39. How to get the services on our servers?
40. Diverse technology stacks
41. The same for every service
42. One Click Deployment
43. •Ansible •Capistrano •Saltstack •Custom •….
45. CI
46. Automation
47. Build/Test/Release pipeline
49. Availability
50. Zero Downtime Deployments
51. Server Server Server Server
52. Server Server Server Server
53. Server Server Server Server
54. Server Server Server Server
55. Server Server Server Server
56. Server Server Server Server
57. Stability
58. Canary environments
59. Server Server Server Server
60. Server Server Server Server Test with low amount of traffic
61. Fast deployments
62. Fast rollbacks
63. https://www.flickr.com/photos/40987321@N02/5580348753/
64. Different libraries, packages, web servers, configurations, versions
65. Base boxes
66. Services installed in a sandbox
67. https://www.docker.com/
68. https://twitter.com/mfdii/status/697532387240996864
69. Running the service
70. How do I stop and start a service and ensure it keeps running?
71. Diverse technology stacks
72. The same for every service
73. •Supervisord •Upstart •S6 •Ruine •Monit •Circus •Restartd •Docker •…
74. Releases
75. How to synchronize changes over services?
76. APIs
77. API Versioning
78. GET /v23/foo/abr Host: myservice.local
79. GET /foo/abr Host: myservice.local X-Version: 23
80. GET /foo/abr?version=23 Host: myservice.local
81. GET /foo/abr Host: myservice.local Accept: application/vnd.company.v23+json
82. No backwards compatibility breaks
83. Feature Flags
84. public function hasAccess() { return featureFlag()->isActive( FeatureFlag::TEST_ONE ); }
87. Every service has its own implementation
88. Shared database
89. Headers
90. GET /foo/abr Host: myservice.local X-Flag-NewFeature: 1
91. Configuration Management
92. How do I synchronize configuration over services?
93. [ "db_user": "user", "db_pw": "pw", "serviceA": "serviceA.local:8018" ]
94. Config file on disk
95. Duplication
96. Inconsistencies
97. Consul https://www.consul.io/
98. •Consul •Zookeeper •etcd •…
99. Consul Server Consul Server Consul Server Consul Agent ver Consul Agent Server Consul Agent Server Co Ag Server
100. Key/Value Store
101. $kv->put('test/foo/bar', 'bazinga'); $kv->get('test/foo/bar', ['raw' => true]); $kv->delete('test/foo/bar');
102. Credentials
103. $kv->put('test/db/pw', 'secret_pw');
104. https://www.vaultproject.io/
105. Cycling of credentials
106. Service Discovery
107. How does one service know where another service is?
108. Hostname + Port
109. Server Service A Server Service B Service C Service C
110. Configuration
111. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
112. Consul https://www.consul.io/
113. Load balancing?
114. Round robin in the client
115. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
116. Service/Server down?
117. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
118. Health checks
119. GET /health HTTP/1.1 Host: serviceA.local HTTP/1.1 200 OK
120. Central load balancer
121. Server Service A Server Service B Service C Service C Load balancer
122. Scalability?
123. Load balancer
124. Elasticity?
125. Load balancer
126. Consul https://www.consul.io/
127. Consul Server Consul Server Consul Server Consul Agent ver Consul Agent Server Consul Agent Server Co Ag Server
128. Consul for Service Discovery
129. Consul Agent Server Service A Registration Health check
130. Consul API
131. DNS
132. admin@hashicorp: dig web-frontend.service.consul. ANY ; <<>> DiG 9.8.3-P1 <<>> web-frontend.service.consul. ANY ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29981 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;web-frontend.service.consul. IN ANY ;; ANSWER SECTION: web-frontend.service.consul. 0 IN A 10.0.3.83 web-frontend.service.consul. 0 IN A 10.0.1.109
133. Consul-Template https://github.com/hashicorp/consul-template
134. Server Service A Server Service B Service C Service C Load balancer Consul Template
135. Single Point of Failure
136. Server Service A Server Service B Service C Service C Load balancer Consul Template Load balancer Consul Template
137. Load balance directly in the client
138. https://github.com/eBay/fabio
139. Monitoring
140. How are my services behaving?
142. Central Log Management
143. Elasticsearch
 Kibana Logstash
144. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
146. Tracing IDs
147. web server http service http service http service http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
148. X-Trace-Id: bbr8ehb984tbab894
149. https://www.loggly.com/
150. https://getsentry.com/
151. Measure everything
152. Server metrics
153. Application metrics
154. StatsD + Graphite
155. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message statsd
156. https://www.librato.com
157. http://www.soasta.com/
158. http://www.monitor.us/
159. Profiling
160. newrelic.com
161. Handling failures
162. What do I do when something breaks?
163. Errors happen
164. Detecting regressions
165. Server outages
166. Database overloads
167. Bugs
168. Service A Service B 200 OK
169. Service A Service B 5xx
170. Service A Service B Timeout
171. Circuit Breakers
172. Service A Service B 200 OK Circuit Breaker Status: closed Error rate: 0
173. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold
174. Service A Service B Circuit Breaker Status: -> open Error rate: > threshold
175. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold Test if still failing
176. Service A Service B 200 OK Circuit Breaker Status: -> close Error rate: 0 Test if still failing
177. https://github.com/Netflix/Hystrix
178. https://github.com/odesk/phystrix
179. Gracefully handling exceptions
180. Component based fronted
188. Degrading Functionality
189. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution
190. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION Institution
191. Profile Publications Publication Publication Publication LeftColumn Image Menu Institution
192. Test it
193. http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
194. Scalability
195. How do I handle traffic spikes?
196. Elasticity
197. Service A Service B 200 OK Circuit Breaker
198. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
199. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
200. Throttling
201. Service A Service B Circuit Breaker Service C Circuit Breaker Only allow xx% of calls
203. Priority
204. Service A Service B Circuit Breaker Service C Circuit Breaker 100% of calls 10% of calls
205. Elasticity
206. Service A Service B Circuit Breaker Service C Circuit Breaker Service B
207. Complete Solutions
209. https://mesosphere.github.io/marathon/
210. https://www.flickr.com/photos/darkdwarf/19701555974/
211. http://speakerdeck.com/u/bastianhofmann
212. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de