Node.JS API Scalability

You would not want your service to be unusable at precisely the wrong time — while everyone is watching — would you?

With adequate preparation, however, you can build a service that can preserve during traffic bursts that exceed your initially estimated capacity by orders of magnitude.

It is one thing when you create a sample web application using Node.JS (and maybe utilizing the cluster module to distribute the load), and it is totally something else when you need to horizontally scale your architecture to hundreds of thousands of concurrent connections, while trying to ensure redundancy and high availability.

Knowing how to scale is important, and more important than that is knowing “when” to scale. For this, you should constantly monitor your system. There are certain clues that you need to pay special attention to, which are precursors of the fact that the current architecture is not enough and you need to scale out. — You can either define elastic rules to automatically do the scaling for you or you can do it manually; however, it does not change the fact that you have to know what to look for before scaling up or down.

In this talk, I will try to peek into what it takes to create a real-life, scalable, highly-available, and highly-responsive Node.JS application and try to address the topics mentioned above as much as I can.

Transcript

1. NODE JS API your like a boss Scaling http://htmlmag.com/etkinlik/the-frontiers-mini Volkan Özçelik June, 20, 2015 http://volkan.io/
2. Resources volkan.io Slides & Source Code volkan.io @linkibol
3. About Me • JavaScript Lover & Performance Freak • Current: • Technical Lead @ Cisco • Before: • Mobile Frontend Engineer @ Jive Software • VP of Technology @ grou.ps • CTO @ cember.net (acquired by Xing)
 • Chase me: • @linkibol • volkan.io • github.com/v0lkan • speakerdeck.com/volkan • linkedin.com/in/volkanozcelik
4. How do I Architect a Scalable and Consistent API with a Manageable Complexity? In a Nutshell… volkan.io @linkibol
5. Scalability & Consistency (in theory) volkan.io @linkibol
6. Scalability & Consistency (in practice)
7. Agenda • Why APIs? • Complexity • Consistency • Microservices • Know Your Platform • Know Your Tools • Limitations • V8 Limitations • OS Limitations • Monitoring • Throughput • Concurrency • Event Loop • Debugging • Automation • Logging • Configuration • Security • Perf Before Scale • Memory Leaks • IO Optimization • Hot Code Paths • Scaling Your API Environment Operations Scaling Up volkan.io @linkibol
8. Why APIS? web mobile iot saas
9. High-Level Topology $:> Clients http proxy ssl termination load balancer authentication authorization token exchange rate limiting API Gateway API Server …
10. Complexity volkan.io @linkibol
11. – Dan Ward “Through carelessness, inattention, or miscalculation, we may inadvertently overfill [the design] to the detriment of the whole. Additions once led to improvement. Beyond a certain point, that is no longer the case. Additions begin to make things worse. ” volkan.io @linkibol
12. Keep It Simple
13. Keep It Simpler
14. Find a Balance
15. Consistency volkan.io @linkibol
16. Eventual Consistency Eventual consistency is a consistency model used in distributed computing to achieve high availability that guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. see also: “Event Sourcing” 
 ( http://martinfowler.com/eaaDev/EventSourcing.html )
17. 99% of the Time Eventual Consistency is Good Enough Transactional Consistency in a distributed system is hard to achieve and expensive to implement. volkan.io @linkibol
18. Be Optimistic Client app.js data store send update ACK ASAP persist ACK update state verify state actual state
19. Be Optimistic Client app.js data store send update ACK ASAP persist ACK update state verify state actual state
20. Be Optimistic Client app.js data store send update ACK ASAP persist ACK update state verify state actual state
21. Before Everything Think Functionally volkan.io @linkibol
22. Think Functionally • Delegate; Don’t Return • Avoid Side Effects • Avoid State • Easier to Scale Horizontally • Easier to Swap/Reboot Instances • Session Affinity is Not A Problem • Better Performance
23. Think Functionally • Understand Control Flow Patterns • Understand Promise Patterns (and anti-patterns) • Avoid this and avoid new as much as you can. * • You’ll thank me later. • Think in Streams • Translate, Transform, Reduce https://medium.com/javascript-scene/the-two-pillars-of-javascript-ee6f3281e7f3 *
24. Think Functionally bolognese is onion and oil fried until golden mixed with ground beef mixed with tomato simmered for 20 minutes. cheese sauce is milk and cheese added progressively to roux while frying it until the sauce thickens. Lasagna is grated cheese on cheese sauce on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce baked for 45 minutes. roux is flour and butter fried briefly. baked is put in an oven dish in a hot oven. fried is put in a pan on high and mixed frequently. simmered is put in a pan on low and mixed infrequently. TRANSFORMATION TRANSFORMATION REDUCTION TRANSLATION TRANSFORMATION TRANSLATION TRANSLATION 1. Cook ground beef, onion, and garlic over medium heat until well browned. 2. Stir in crushed tomatoes, tomato paste, tomato sauce, and water. 3. Season with sugar, basil, fennel seeds, Italian seasoning, stirring occasionally. 4. Bring a large pot of lightly salted water to a boil. 5. Cook noodles in boiling water for 8 to 10 minutes. 6. Drain noodles, and rinse with cold water. 7. In a mixing bowl, combine ricotta cheese with egg, remaining parsley. 8. Arrange 6 noodles lengthwise over meat sauce. 9. Spread with one half of the ricotta cheese mixture. 10. Top with a third of mozzarella cheese slices. 11. Spoon 2 cups meat sauce over mozzarella. 12. Repeat layers, and top with remaining mozzarella and Parmesan cheese. 13. Bake in preheated oven for 25 minutes. 14. Remove foil, and bake an additional 25 minutes. 15. Cool for 15 minutes before serving.
25. Think Functionally bolognese is onion and oil fried until golden mixed with ground beef mixed with tomato simmered for 20 minutes. cheese sauce is milk and cheese added progressively to roux while frying it until the sauce thickens. Lasagna is grated cheese on cheese sauce on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce baked for 45 minutes. roux is flour and butter fried briefly. baked is put in an oven dish in a hot oven. fried is put in a pan on high and mixed frequently. simmered is put in a pan on low and mixed infrequently. TRANSFORMATION TRANSFORMATION REDUCTION TRANSLATION TRANSFORMATION TRANSLATION TRANSLATION
26. Know Your Platform
27. Perf Advice Is Addictive volkan.io @linkibol
28. Learn JavaScript Before the Deep Dive • Understand EcmaScript 5 • scope, hoisting, closures, promises, `this`, all that jazz. • Keep an Eye on EcmaScript 6 • Consider Using EcmaScript 6 for new Projects
29. Fact Check • “JavaScript is like Java, but easier.” • “Node.JS is slow.” • “I can’t let the app restart; it will take too long.” • “Node.JS is Insecure.” (no! people are.) • “Node.JS does not support feature x.”
30. Node.JS Is Perfect For… • IO-Heavy Applications • Data-Intensive Realtime Apps • RESTful / API-Driven (Micro)services • Streams • Queued (Lazy) Writes • Processing data on-the-fly
31. Node.JS Is not For… • Serving Static Files • CPU-bound Applications • Creating a Monolithic Infrastructure volkan.io @linkibol
32. The Event Loop volkan.io @linkibol
33. Node.JS Event Model * http://nikhilm.github.io/uvbook * https://strongloop.com/strongblog/node-js-event-loop/
34. Never Block The Event Loop • First rule of Node.JS Programming: • Never Block the Event Loop
 • Second rule of Node.JS Programming: • Never Block the Freaking Event Loop!
 • Corollary: • Delegate long (>10ms) Tasks to a Worker • child_process (*) • native add-ons (**) * https://nodejs.org/api/child_process.html
 ** https://nodejs.org/api/addons.html
35. Know the Ecosystem • Do Not Ignore The Ecosystem • Follow Community News and Updates • Attend to Conferences (like this one) volkan.io @linkibol
36. Know the Ecosystem • npm (as of June 20, 2015) • 155,880 (and increasing) total packages • 70,624,534 downloads yesterday • 389,190,331 downloads in the last week • 1,609,312,413 downloads in the last month
37. Embrace Open Source and contribute back volkan.io @linkibol
38. Know Your Tools volkan.io @linkibol
39. Node.JS is not a Swiss army knife • Load Balancing ➡ haproxy ( http://www.haproxy.org/ ) • Web Server ➡ nginx ( http://nginx.org/ ) • SSL Termination ➡ stud ( https://github.com/bumptech/stud ) • GZIP Compression ➡ nginx / haproxy • Static Assets ➡ CDN / Varnish ( https://www.varnish-cache.org/ )
40. Limitations
41. v8 Limitations volkan.io @linkibol
42. v8 Limitations • ~2gb ➡ heap limit (--max-old-space-size) • will be more than enough • spawn more processes for more (child_process) • ~1gb ➡ max size of a Buffer
43. • Lazily produce or consume data in buffered chunks. • Evented and non-blocking. • Low memory footprint. • Automatically handle back-pressure. • Buffers allow you to work around the v8 heap memory limit. • Most core node.js content sources/sinks are streams already. Show Love to Streams
44. OS Limitations volkan.io @linkibol
45. Open File Limits • “Error: EMFILE, Too many open files” • ulimit -n 65535; volkan.io @linkibol
46. Open File Limits
47. Configuring the Load Balancer see https://bit.ly/nginx-rocks
48. Configuring the Load Balancer see https://bit.ly/nginx-rocks
49. Additional Tweaks http.globalAgent.maxSockets = SOME_BIG_NUMBER; (if you have many outgoing requests) volkan.io @linkibol
50. Additional Tweaks sysctl -w net.core.somaxconn=1024; (default is 128) volkan.io @linkibol
51. Even More Tweaks
52. Monitoring volkan.io @linkibol
53. You Can’t Optimize What You Don’t Measure volkan.io @linkibol
54. Things to Watch Out For • Your API Service may Become CPU-Bound • External API Calls Can Be a Bottleneck • Always Keep an Eye on the Event Loop • Implement Sanity Checks • Circuit Breaker • Have an Upper Bound for Concurrency
55. Things to Watch Out For • Is app running and functional? • Is app overloaded? • How many errors have been raised so far? • How many times do forks restored? • Are all forks alive and okay? • Is app performant (throughput, memory utilization, concurrency)?
56. Which Will (most of the time) Boil Down to… • Watching Response Times • Watching CPU Utilization + General Sys Resource Usage • Watching Number of Concurrent Connections
57. Throughput / Concurrency volkan.io @linkibol
58. Throughput vs Concurrency throughput concurrent connections
59. Throughput vs Concurrency throughput concurrent connections throughput concurrent connections
60. Throughput vs Concurrency throughput concurrency
61. Throughput vs Concurrency throughput concurrency near-linear rapid decrease degrade throughput concurrency
62. Throughput vs Concurrency throughput concurrency near-linear rapid decrease degrade throughput concurrency near-linear rapid decrease degrade throughput concurrency ideal path practical path
63. Event Loop Delay volkan.io @linkibol
64. Track the Event Loop
65. Track the Event Loop volkan.io @linkibol
66. Track the Event Loop
67. Track the Event Loop
68. Track the Event Loop
69. Circuit Breaker closed fail (under threshold) open fail (reached threshold) checking… timer (exponential backoff) fail success volkan.io @linkibol
70. Circuit Breaker • Can be used with any kind of metric. • Event-loop tracking is just a specific example. • Useful when you depend on other APIs that might fail. volkan.io @linkibol
71. Monitoring Tools volkan.io @linkibol
72. Monitoring as a Service • nodetime • https://nodetime.com/ • newrelic • http://newrelic.com/nodejs • strongloop • https://strongloop.com/node-js/performance-monitoring/
73. Memwatch • https://github.com/lloyd/node-memwatch • heap stats • heap diffing • leak detection (subscribe to “leak” events)
74. cluster-live https://github.com/tj/cluster-live
75. Flame Graphs http://www.brendangregg.com/flamegraphs.html
76. Flame Graphs • Can Be Created Post-Mortem (after a core dump) • Can Be Created at Runtime (using gcore *) • You Can Use dtrace + stackvis to generate them ** http://man7.org/linux/man-pages/man1/gcore.1.html * http://blog.nodejs.org/2012/04/25/profiling-node-js/ **
77. Error Handling volkan.io @linkibol
78. Error Handling • Use an error object; not a String (or better, use an error event) • https://github.com/davepacheco/node-verror • Error handling for Async Code Is Hard • Throwing does not make sense ( scope and stack trace is lost ) • Consider using domains ( https://nodejs.org/api/domain.html )
79. Error Handling • Throwing is for programmer errors. • Consider using an error event instead of throwing.
80. Handle Errors Gracefully • Know how exceptions and errors propagate in Node.JS. • Raise error events to throwing exceptions. • Restart on uncaught exceptions. • Utilize domains. https://www.joyent.com/developers/node/design/errors
81. Using Domains
82. Processes Die Accept it volkan.io @linkibol
83. Processes Die Accept it volkan.io @linkibol
84. Processes Die Accept it volkan.io @linkibol
85. Processes Die Accept it No system is %100 resilient. Keep things as simple as possible. Build something that’s good enough for your purpose. Solve for the problems that are actually on your plate. volkan.io @linkibol
86. Keep It Running •forever ( https://github.com/foreverjs/forever ) •pm2 ( https://github.com/Unitech/pm2 ) •upstart ( http://upstart.ubuntu.com/ ) •systemd ( https://www.wikiwand.com/en/Systemd )
87. Debugging volkan.io @linkibol
88. Debugging • Live Debugging • Post-Mortem Debugging volkan.io @linkibol
89. Live Debugging Given the Tornado, Where’s the Butterfly? volkan.io @linkibol
90. Post-Mortem Debugging volkan.io @linkibol
91. Node.JS Debugging Myths • Debugging and Profiling in Node.JS is Hard • Debugging and Profiling in Node.JS is Immature • You Cannot Debug or Profile a Live Production Node.JS App volkan.io @linkibol
92. Node.JS Debugging Tips • Attaching a debugger to prod is not practical. • Make as much state as possible observable. • You can take a core dump, and analyze it later. volkan.io @linkibol
93. Debugging Node.JS • You Can Expose Internal State via an API and/or a CLI/REPL • https://github.com/davepacheco/kang • https://nodejs.org/api/repl.html • Expose Additional Logging Info at Runtime (in systems that support it) • bunyan -p ( https://github.com/trentm/node-bunyan )
94. Debugging Node.JS • Interactive Debugging • Using node-debug • https://nodejs.org/api/debugger.html • Using Node Inspector • https://github.com/node-inspector/node-inspector • Using Cloud9 IDE • https://c9.io/
95. Debugging Node.JS • Log Libraries Specialized for Dumping Debug Info • Caterpillar ( https://github.com/bevry/caterpillar ) • Tracer ( https://github.com/baryon/tracer ) volkan.io @linkibol
96. Logging volkan.io @linkibol
97. What to Log • Authentication & Authorization • Session Management • Method Entry Points • Erros and Weird Events • Specific Events (startup, shutdown, slowdown etc.) • High-Risk Functionalities (payments, privileges, admins etc)
98. Centralized Logging • Send logs to a log aggregation server; • Using a library that supports it: • bunyan ( https://github.com/trentm/node-bunyan ); • winston ( https://github.com/winstonjs/winston ); • custom (using Streams).
99. Configuration volkan.io @linkibol
100. • No Hard-Coded IP Addresses in Config Files • Let DNS do What it Does Best • Update Configuration From a Central Service • saltstack ( http://saltstack.com/ ) • chef ( https://www.chef.io/ ) • puppet ( https://puppetlabs.com/ ) Configuration
101. Automation volkan.io @linkibol
102. Have a CI/CD Pipeline • go ( http://www.go.cd/ ) • gradle ( http://gradle.org/ ) • jenkins ( https://jenkins-ci.org/ ) volkan.io @linkibol
103. Code Quality volkan.io @linkibol
104. Code Quality • Group Common Logic Into Reusable Modules • Modules Should Do One Thing, and Do One Thing Well • Use Static File Analyzers • jshint ( https://github.com/gruntjs/grunt-contrib-jshint ) • jscs ( https://github.com/jscs-dev/grunt-jscs ) • grunt-complexity ( https://github.com/vigetlabs/grunt-complexity )
105. Perf Before Scale volkan.io @linkibol
106. Memory Leaks volkan.io @linkibol
107. Memory Leaks volkan.io @linkibol
108. Memory Leaks volkan.io @linkibol
109. Perf Before Scale • Configure Your System for High Performance • Cache at Every Layer • The fastest API response is no response at all. • Delegate Long-Running(*) Operations volkan.io @linkibol
110. Perf Before Scale • CPU-Intensive Computations? • child_process + External Libraries • Native Extensions volkan.io @linkibol
111. IO Optimization volkan.io @linkibol
112. IO Optimization • Don’t Immediately Write Small Packets • Reduce the Number of Outgoing Requests • Use Lesser Abstractions for Maximum Throughput volkan.io @linkibol
113. Know Your Bottlenecks • 99% of the time you will be IO-bound. • You’ll scale horizontally before hitting 100% CPU. • You’ll have to try really hard to have a CPU or memory bottleneck. • Node.JS serves really well as a highly concurrent networking app. • Node.JS is very sensitive to memory leaks and blocking code. volkan.io @linkibol
114. Torture Your System • Try Chaos Monkey • https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey • Randomly send `kill -9` to Processes • Randomly Knock a Server Offline • Intentionally Run Out of Disk Space • Take an entire data center down
115. Going Bare Bones volkan.io @linkibol
116. express.js vs Bare-Bones Node.JS express.js http
117. Diving Deeper
118. Going Bare Bones express (with 5 common middleware) bare bones http bare bones tcp 0 1150 2300 3450 4600 4,517 3,561 2,891 requests per second ab -c 100 -n 1000 Tested on MacBook Pro, 2.4 GHz Intel Core i5, 16 GB 1600 MHz DDR3, single core, w/o keep-alive
119. Is It Worth It? • Consider Going Bare-Bones for Maximum Throughput • Tradeoff: • harder to maintain • more complex code • error prone • lots of edge cases • harder to use additional tooling (i.e. no dtrace support)
120. v8 Optimizations volkan.io @linkibol
121. Taking Control of the Garbage Controller • Warning: You probably would not want to do that! • --expose-gc • --nouse-idle-notification volkan.io @linkibol
122. Types of Compilers in v8 • Generic Compiler • Optimizing Compiler • Can Be Two or More Orders of Magnitude Faster volkan.io @linkibol
123. Optimize Only Hot Code Paths Unless You Have Solid Evidence to Do Otherwise volkan.io @linkibol
124. Tooling node --trace_opt 
 --trace_deopt 
 --allow-natives-syntax test.js; • console.log(%HasFastProperties(obj)) • console.log(%GetOptimizationStatus(fn)) https://github.com/Nathanaela/v8-natives
125. v8 Optimization Killers • These will be bailed out (likely, forever): • Using debugger anywhere within the function • Using eval anywhere within the function • Using with anywhere within the function https://github.com/petkaantonov/bluebird/wiki/Optimization-killers
126. v8 Optimization Killers • These will be bailed out (for now): • Generators • Functions that contain a for-of statement • Functions that contain try-catch or try-finally • let assignments • const assignments • functions that contain object literals with
 __proto__, get, or set declarations.
127. Infinite Loops With Unclear Logic • while(true) { … } • for( ; ; ) { … } volkan.io @linkibol
128. Objects Object.prototype.baz = function() {}; • Do not define enumerable properties in the prototype chain. • Use Object.defineProperty to create non-enumerable properties.
129. Objects
130. for/each
131. Isolate try/catch
132. Isolate try/catch
133. Isolate try/catch
134. Isolate try/catch
135. arguments Object
136. Iteration Tips • Rule of thumb: Avoid for/in loops in hot code paths. • Always use Object.keys to iterate an object. • Rethink your architecture 
 if you need to iterate the parent prototype’s keys. volkan.io @linkibol
137. Inherited keys volkan.io @linkibol
138. Promises volkan.io @linkibol
139. Callback Hell Is Real
140. Promises Promises are not as slow as they once were. https://github.com/petkaantonov/bluebird
141. Promises • Use promises — *ahem* BlueBird — liberally. • Consider using continuation passing style 
 (i.e. callbacks) for very hot code paths. • Callbacks are always faster, and more memory-efficient. • Do not optimize prematurely; measure things first!
142. Security volkan.io @linkibol
143. Do Not Run Node.JS As Root useradd -mrU web;
 mkdir /opt/web-app;
 chown web /opt/web-app;
 cd /opt/web-app;
 su web;
 node app.js;
 firewall-cmd --permanent --zone=public --add-port=3000/tcp volkan.io @linkibol
144. Security • https://www.owasp.org/index.php/ OWASP_Node_js_Goat_Project • http://nodesecurity.io/ • https://www.owasp.org/index.php/ OWASP_Zed_Attack_Proxy_Project volkan.io @linkibol
145. Common Threats • XSS / CSRF • Input Validation Attack • DoS / ReDoS • Request Size * not different from any other web app.
146. Your API at Scale
147. Goals • Minimize Client Response Time • Maximize Resource Efficiency on the Server
 Hint: Leave 50% of the memory unused
 (for taking core dumps)
148. Microservices to the rescue volkan.io @linkibol
149. Microservices web.js auth.js routing.js persistence.js messaging.js logging.js process A web.js process A web.js process B routing.js process C persistence.js process D messaging.js process E logging.js process F Monolith Microservice multiple modules single process multiple modules multiple processes
150. Think Twice http://martinfowler.com/bliki/MicroservicePremium.html
151. Favor Microservices Over Monoliths • Prefer Composition over Inheritance • Publish Modules Instead of Inlining Functionality • Use an Internal npm
152. Scaling Your API volkan.io @linkibol
153. Scaling Your API app memory volkan.io @linkibol
154. Cluster volkan.io @linkibol
155. Cluster
156. Cluster
157. Cluster
158. Cluster
159. Cluster + Domains volkan.io @linkibol
160. Cluster + Domains
161. Cluster + Domains
162. Cluster + Domains
163. Cluster + Domains
164. Cluster + Domains
165. Cluster + Zero Downtime Rolling Deployments volkan.io @linkibol
166. Zero Downtime kill --USR2 <pid> volkan.io @linkibol
167. Zero Downtime see also: https://github.com/doxout/recluster
168. Zero Downtime see also: https://github.com/doxout/recluster
169. Zero Downtime see also: https://github.com/doxout/recluster
170. Build Redundancy Everywhere volkan.io @linkibol
171. Build Redundancy Everywhere
172. Build Redundancy Everywhere
173. Build Redundancy Everywhere
174. Build Redundancy Everywhere
175. Build Redundancy Everywhere
176. Cluster app memory app app app volkan.io @linkibol
177. IS BIGGER ALWAYS BETTER? ultra mega super box with bazillion cores regular box
178. Let’s Step Back app memory volkan.io @linkibol
179. Delegate Heavy Computation to Workers app memory worker worker worker child_process volkan.io @linkibol
180. Split Logical Components into Distinct Services API app memory compute app memory worker worker worker child_process volkan.io @linkibol
181. IPC With a Message Bus API app memory compute app memory worker worker worker child_process message bus * * rabbitmq, zeromq, resque etc. see also http://queues.io/
182. Message Bus Topologies P C Send/Listen P C1 C2 Worker Queue X C1 PubSub P C2
183. Cluster Processes API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process volkan.io @linkibol
184. How Do We Multiplex? API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process
185. How Do We Multiplex? API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process ?
186. Introduce a LB and a Broker API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus volkan.io @linkibol
187. Load Balancing Options • Elastic Load Balancing as a Service (amazon, rackspace…) • Hardware Load Balancer (Cisco ACE, Barracuda, etc…) • Software Load Balancer • nginx • haproxy • home grown
188. Load Balancer
189. Making the Load Balancer Highly Available • round-robin DNS • https://www.wikiwand.com/en/Round-robin_DNS • heartbeat • https://www.wikiwand.com/en/Heartbeat_(computing) • keepalived • http://keepalived.org/ * You can use these tools to make any component HA.
190. Making the Load Balancer HA load balancer load balancer load balancer keepalived DNS failover active * see also: https://www.wikiwand.com/en/Virtual_Router_Redundancy_Protocol
191. SSL Termination load balancer load balancer keepalived DNS failover active SSL Terminator SSL Terminator keepalived DNS failover active load balancer load balancer * * * https://github.com/bumptech/stud
192. Elephant in the Room How Do We Manage State? volkan.io @linkibol
193. Share State With Redis API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis volkan.io @linkibol
194. Share State With Redis API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis volkan.io @linkibol
195. Make Redis Redundant redis redis (master) redis (read replica) redis (read replica) redis (read replica) redis (master) redis (read replica) redis (read replica) redis (read replica) round-robin DNS This will also increase throughput as a side benefit. See http://redis.io/topics/replication and http://redis.io/topics/cluster-tutorial You can also use a managed “memory as a service” solution.
196. We Can Add More… API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … …
197. we are actually using Microservices here volkan.io @linkibol
198. Microservices API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … …
199. Microservices API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … … API μ-Service
200. Microservices API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … … API μ-Service Compute μ-Service
201. Microservices API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet volkan.io @linkibol
202. What If We Exhaust All the Bandwidth? 
 That Means You’ve Become Famous Scalability Will Be the Least of Your Concerns volkan.io @linkibol
203. Scaling Into Multiple Zones API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet Zone 1 volkan.io @linkibol
204. Scaling Into Multiple Zones API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet Zone 1 API μ-Service Compute μ-Service load balancer broker redis redis message bus Zone 2
205. Scaling Into Multiple Zones round-robin DNS API μ-Service Compute μ-Service load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet
206. Scaling Into Multiple Zones round-robin DNS API μ-Service Compute μ-Service load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet
207. Mirroring the Data Stores round-robin DNS API μ-Service Compute μ-Service load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror
208. Mirroring the Data Stores round-robin DNS API μ-Service Compute μ-Service load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror
209. We Can Add More… round-robin DNS API μ-Service Compute μ-Service load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror …
210. scale 2 ∞ & 㱺
211. Thank You Questions? @linkibol