Elastic scaling in a (micro)service oriented architecture

Splitting an application up into multiple independent services can be a good way to keep it scaling and ensure stability and developer productivity in larger, growing teams. But just splitting the codebase, creating APIs and deploying the code on some servers is not enough, somehow your services need to know where and how other services are accessible. Classical approaches like hardcoding everything in every service or having a central load-balancer can quickly lead to problems in terms of scalability and maintainability.
In this talk I'll show how we at ResearchGate tackled this challenge. With the help of tools like Consul and haproxy we created a setup that allows us to quickly boot and shutdown services. This ensures that all servers are utilized optimally and load spikes can be reacted upon quickly and automatically.

Transcript

1. Elastic Scaling in a (Micro)service oriented Architecture @BastianHofmann
4. Microservices
6. Service Oriented Architecture
7. Monolith
8. http://blog.philipphauer.de/microservices-nutshell-pros-cons/ Monolith Microservices
9. Benefits
10. Stricter separation of concerns
11. Diverse technology stacks
12. Problems
13. Problems
14. Challenges
15. Performance
16. Latency
17. Stability
18. Reliability
19. Transparency
20. Monitoring
21. Learning Curves
22. Code Reuse
23. Maintenance
24. Elastic Scaling?
25. Cloud Solutions
27. Using the cloud is not always possible
28. … or even desirable
29. Only solve some of the problems
30. How can we solve them
31. •Deploying •Running •Releasing •Configuring •Discovering •Monitoring •Error Handling •Scaling •Developing Agenda
32. A lot of this is also useful for monoliths
35. http://speakerdeck.com/u/bastianhofmann
36. https://www.flickr.com/photos/npobre/2601582256/
37. Deployment
38. How to get the services on our servers?
39. Diverse technology stacks
40. The same for every service
41. One Click Deployment
42. •Ansible •Capistrano •Saltstack •Custom •….
44. CI
45. Automation
46. Build/Test/Release pipeline
48. Availability
49. Zero Downtime Deployments
50. Server Server Server Server
51. Server Server Server Server
52. Server Server Server Server
53. Server Server Server Server
54. Server Server Server Server
55. Server Server Server Server
56. Stability
57. Canary environments
58. Server Server Server Server
59. Server Server Server Server Test with low amount of traffic
60. Fast deployments
61. Fast rollbacks
62. https://www.flickr.com/photos/40987321@N02/5580348753/
63. Different libraries, packages, web servers, configurations, versions
64. Base boxes
65. Services installed in a sandbox
66. https://www.docker.com/
67. https://twitter.com/mfdii/status/697532387240996864
68. Running the service
69. How do I stop and start a service and ensure it keeps running?
70. Diverse technology stacks
71. The same for every service
72. •Supervisord •Upstart •S6 •Ruine •Monit •Circus •Restartd •…
73. Releases
74. How to synchronize changes over services?
75. APIs
76. API Versioning
77. GET /v23/foo/abr Host: myservice.local
78. GET /foo/abr Host: myservice.local X-Version: 23
79. GET /foo/abr?version=23 Host: myservice.local
80. GET /foo/abr Host: myservice.local Accept: application/vnd.company.v23+json
81. No backwards compatibility breaks
82. Feature Flags
83. public function hasAccess() { return featureFlag()->isActive( FeatureFlag::TEST_ONE ); }
86. Every service has its own implementation
87. Shared database
88. Headers
89. GET /foo/abr Host: myservice.local X-Flag-NewFeature: 1
90. Configuration Management
91. How do I synchronize configuration over services?
92. [ "db_user": "user", "db_pw": "pw", "serviceA": "serviceA.local:8018" ]
93. Config file on disk
94. Duplication
95. Inconsistencies
96. Consul https://www.consul.io/
97. •Consul •Zookeeper •etcd •…
98. Consul Server Consul Server Consul Server Consul Agent ver Consul Agent Server Consul Agent Server Co Ag Server
99. Key/Value Store
100. $kv->put('test/foo/bar', 'bazinga'); $kv->get('test/foo/bar', ['raw' => true]); $kv->delete('test/foo/bar');
101. Credentials
102. $kv->put('test/db/pw', 'secret_pw');
103. https://www.vaultproject.io/
104. Cycling of credentials
105. Service Discovery
106. How does one service know where another service is?
107. Hostname + Port
108. Server Service A Server Service B Service C Service C
109. Configuration
110. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
111. Consul https://www.consul.io/
112. Load balancing?
113. Round robin in the client
114. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
115. Service/Server down?
116. $config = [ 'serviceA' => [ '192.168.0.1:8001', '192.168.0.2:8001', ], 'serviceB' => [ '192.168.0.1:8002', ], 'serviceC' => [ '192.168.0.2:8003', ] ];
117. Health checks
118. GET /health HTTP/1.1 Host: serviceA.local HTTP/1.1 200 OK
119. Central load balancer
120. Server Service A Server Service B Service C Service C Load balancer
121. Scalability?
122. Load balancer
123. Elasticity?
124. Consul https://www.consul.io/
125. Consul Server Consul Server Consul Server Consul Agent ver Consul Agent Server Consul Agent Server Co Ag Server
126. Consul for Service Discovery
127. Consul Agent Server Service A Registration Health check
128. Consul API
129. DNS
130. admin@hashicorp: dig web-frontend.service.consul. ANY ; <<>> DiG 9.8.3-P1 <<>> web-frontend.service.consul. ANY ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29981 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;web-frontend.service.consul. IN ANY ;; ANSWER SECTION: web-frontend.service.consul. 0 IN A 10.0.3.83 web-frontend.service.consul. 0 IN A 10.0.1.109
131. Consul-Template https://github.com/hashicorp/consul-template
132. Server Service A Server Service B Service C Service C Load balancer Consul Template
133. Single Point of Failure
134. Server Service A Server Service B Service C Service C Load balancer Consul Template Load balancer Consul Template
135. Load balance directly in the client
136. https://github.com/eBay/fabio
137. Monitoring
138. How are my services behaving?
140. Central Log Management
141. Elasticsearch
 Kibana Logstash
142. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
144. Tracing IDs
145. web server http service http service http service http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
146. X-Trace-Id: bbr8ehb984tbab894
147. https://www.loggly.com/
148. https://getsentry.com/
149. Measure everything
150. Server metrics
151. Application metrics
152. StatsD + Graphite
153. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message statsd
154. https://www.librato.com
155. http://www.soasta.com/
156. http://www.monitor.us/
157. Profiling
158. newrelic.com
159. Handling failures
160. What do I do when something breaks?
161. Errors happen
162. Detecting regressions
163. Server outages
164. Database overloads
165. Bugs
166. Service A Service B 200 OK
167. Service A Service B 5xx
168. Service A Service B Timeout
169. Circuit Breakers
170. Service A Service B 200 OK Circuit Breaker Status: closed Error rate: 0
171. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold
172. Service A Service B Circuit Breaker Status: -> open Error rate: > threshold
173. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold Test if still failing
174. Service A Service B 200 OK Circuit Breaker Status: -> close Error rate: 0 Test if still failing
175. https://github.com/Netflix/Hystrix
176. https://github.com/odesk/phystrix
177. Gracefully handling exceptions
178. Component based fronted
187. Degrading Functionality
188. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution
189. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION Institution
190. Profile Publications Publication Publication Publication LeftColumn Image Menu Institution
191. Test it
192. http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
193. Scalability
194. How do I handle traffic spikes?
195. Elasticity
196. Service A Service B 200 OK Circuit Breaker
197. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
198. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
199. Throttling
200. Service A Service B Circuit Breaker Service C Circuit Breaker Only allow xx% of calls
202. Priority
203. Service A Service B Circuit Breaker Service C Circuit Breaker 100% of calls 10% of calls
204. Elasticity
205. Service A Service B Circuit Breaker Service C Circuit Breaker Service B
206. Development Environment
207. How do I enusre a productive dev environment?
208. Diverse technology stacks
209. Diverse environments
210. Good documentation
211. Common setups
212. https://www.docker.com/
213. Global Code Search
214. https://github.com/etsy/hound
215. Large scale refactorings
216. Monorepo
217. https://qafoo.com/talks/15_08_froscon_monorepos.pdf
218. Complete Solutions
220. https://mesosphere.github.io/marathon/
221. https://www.flickr.com/photos/darkdwarf/19701555974/
222. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de