Node.JS API Scalability

0368b95a18e594981083b2eb9b177b2d?s=47 volkan
June 20, 2015

Node.JS API Scalability

You would not want your service to be unusable at precisely the wrong time — while everyone is watching — would you?

With adequate preparation, however, you can build a service that can preserve during traffic bursts that exceed your initially estimated capacity by orders of magnitude.

It is one thing when you create a sample web application using Node.JS (and maybe utilizing the cluster module to distribute the load), and it is totally something else when you need to horizontally scale your architecture to hundreds of thousands of concurrent connections, while trying to ensure redundancy and high availability.

Knowing how to scale is important, and more important than that is knowing “when” to scale. For this, you should constantly monitor your system. There are certain clues that you need to pay special attention to, which are precursors of the fact that the current architecture is not enough and you need to scale out. — You can either define elastic rules to automatically do the scaling for you or you can do it manually; however, it does not change the fact that you have to know what to look for before scaling up or down.

In this talk, I will try to peek into what it takes to create a real-life, scalable, highly-available, and highly-responsive Node.JS application and try to address the topics mentioned above as much as I can.

0368b95a18e594981083b2eb9b177b2d?s=128

volkan

June 20, 2015
Tweet

Transcript

  1. NODE JS API your like a boss Scaling http://htmlmag.com/etkinlik/the-frontiers-mini Volkan

    Özçelik June, 20, 2015 http://volkan.io/
  2. Resources volkan.io Slides & Source Code volkan.io @linkibol

  3. About Me • JavaScript Lover & Performance Freak • Current:

    • Technical Lead @ Cisco • Before: • Mobile Frontend Engineer @ Jive Software • VP of Technology @ grou.ps • CTO @ cember.net (acquired by Xing)
 • Chase me: • @linkibol • volkan.io • github.com/v0lkan • speakerdeck.com/volkan • linkedin.com/in/volkanozcelik
  4. How do I Architect a Scalable and Consistent API with

    a Manageable Complexity? In a Nutshell… volkan.io @linkibol
  5. Scalability & Consistency (in theory) volkan.io @linkibol

  6. Scalability & Consistency (in practice)

  7. Agenda • Why APIs? • Complexity • Consistency • Microservices

    • Know Your Platform • Know Your Tools • Limitations • V8 Limitations • OS Limitations • Monitoring • Throughput • Concurrency • Event Loop • Debugging • Automation • Logging • Configuration • Security • Perf Before Scale • Memory Leaks • IO Optimization • Hot Code Paths • Scaling Your API Environment Operations Scaling Up volkan.io @linkibol
  8. Why APIS? web mobile iot saas

  9. High-Level Topology $:> Clients http proxy ssl termination load balancer

    authentication authorization token exchange rate limiting API Gateway API Server …
  10. Complexity volkan.io @linkibol

  11. – Dan Ward “Through carelessness, inattention, or miscalculation, we may

    inadvertently overfill [the design] to the detriment of the whole. Additions once led to improvement. Beyond a certain point, that is no longer the case. Additions begin to make things worse. ” volkan.io @linkibol
  12. Keep It Simple

  13. Keep It Simpler

  14. Find a Balance

  15. Consistency volkan.io @linkibol

  16. Eventual Consistency Eventual consistency is a consistency model used in

    distributed computing to achieve high availability that guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. see also: “Event Sourcing” 
 ( http://martinfowler.com/eaaDev/EventSourcing.html )
  17. 99% of the Time Eventual Consistency is Good Enough Transactional

    Consistency in a distributed system is hard to achieve and expensive to implement. volkan.io @linkibol
  18. Be Optimistic Client app.js data store send update ACK ASAP

    persist ACK update state verify state actual state
  19. Be Optimistic Client app.js data store send update ACK ASAP

    persist ACK update state verify state actual state
  20. Be Optimistic Client app.js data store send update ACK ASAP

    persist ACK update state verify state actual state
  21. Before Everything Think Functionally volkan.io @linkibol

  22. Think Functionally • Delegate; Don’t Return • Avoid Side Effects

    • Avoid State • Easier to Scale Horizontally • Easier to Swap/Reboot Instances • Session Affinity is Not A Problem • Better Performance
  23. Think Functionally • Understand Control Flow Patterns • Understand Promise

    Patterns (and anti-patterns) • Avoid this and avoid new as much as you can. * • You’ll thank me later. • Think in Streams • Translate, Transform, Reduce https://medium.com/javascript-scene/the-two-pillars-of-javascript-ee6f3281e7f3 *
  24. Think Functionally bolognese is onion and oil fried until golden

    mixed with ground beef mixed with tomato simmered for 20 minutes. cheese sauce is milk and cheese added progressively to roux while frying it until the sauce thickens. Lasagna is grated cheese on cheese sauce on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce baked for 45 minutes. roux is flour and butter fried briefly. baked is put in an oven dish in a hot oven. fried is put in a pan on high and mixed frequently. simmered is put in a pan on low and mixed infrequently. TRANSFORMATION TRANSFORMATION REDUCTION TRANSLATION TRANSFORMATION TRANSLATION TRANSLATION 1. Cook ground beef, onion, and garlic over medium heat until well browned. 2. Stir in crushed tomatoes, tomato paste, tomato sauce, and water. 3. Season with sugar, basil, fennel seeds, Italian seasoning, stirring occasionally. 4. Bring a large pot of lightly salted water to a boil. 5. Cook noodles in boiling water for 8 to 10 minutes. 6. Drain noodles, and rinse with cold water. 7. In a mixing bowl, combine ricotta cheese with egg, remaining parsley. 8. Arrange 6 noodles lengthwise over meat sauce. 9. Spread with one half of the ricotta cheese mixture. 10. Top with a third of mozzarella cheese slices. 11. Spoon 2 cups meat sauce over mozzarella. 12. Repeat layers, and top with remaining mozzarella and Parmesan cheese. 13. Bake in preheated oven for 25 minutes. 14. Remove foil, and bake an additional 25 minutes. 15. Cool for 15 minutes before serving.
  25. Think Functionally bolognese is onion and oil fried until golden

    mixed with ground beef mixed with tomato simmered for 20 minutes. cheese sauce is milk and cheese added progressively to roux while frying it until the sauce thickens. Lasagna is grated cheese on cheese sauce on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce on bolognese on flat pasta on cheese sauce baked for 45 minutes. roux is flour and butter fried briefly. baked is put in an oven dish in a hot oven. fried is put in a pan on high and mixed frequently. simmered is put in a pan on low and mixed infrequently. TRANSFORMATION TRANSFORMATION REDUCTION TRANSLATION TRANSFORMATION TRANSLATION TRANSLATION
  26. Know Your Platform

  27. Perf Advice Is Addictive volkan.io @linkibol

  28. Learn JavaScript Before the Deep Dive • Understand EcmaScript 5

    • scope, hoisting, closures, promises, `this`, all that jazz. • Keep an Eye on EcmaScript 6 • Consider Using EcmaScript 6 for new Projects
  29. Fact Check • “JavaScript is like Java, but easier.” •

    “Node.JS is slow.” • “I can’t let the app restart; it will take too long.” • “Node.JS is Insecure.” (no! people are.) • “Node.JS does not support feature x.”
  30. Node.JS Is Perfect For… • IO-Heavy Applications • Data-Intensive Realtime

    Apps • RESTful / API-Driven (Micro)services • Streams • Queued (Lazy) Writes • Processing data on-the-fly
  31. Node.JS Is not For… • Serving Static Files • CPU-bound

    Applications • Creating a Monolithic Infrastructure volkan.io @linkibol
  32. The Event Loop volkan.io @linkibol

  33. Node.JS Event Model * http://nikhilm.github.io/uvbook * https://strongloop.com/strongblog/node-js-event-loop/

  34. Never Block The Event Loop • First rule of Node.JS

    Programming: • Never Block the Event Loop
 • Second rule of Node.JS Programming: • Never Block the Freaking Event Loop!
 • Corollary: • Delegate long (>10ms) Tasks to a Worker • child_process (*) • native add-ons (**) * https://nodejs.org/api/child_process.html
 ** https://nodejs.org/api/addons.html
  35. Know the Ecosystem • Do Not Ignore The Ecosystem •

    Follow Community News and Updates • Attend to Conferences (like this one) volkan.io @linkibol
  36. Know the Ecosystem • npm (as of June 20, 2015)

    • 155,880 (and increasing) total packages • 70,624,534 downloads yesterday • 389,190,331 downloads in the last week • 1,609,312,413 downloads in the last month
  37. Embrace Open Source and contribute back volkan.io @linkibol

  38. Know Your Tools volkan.io @linkibol

  39. Node.JS is not a Swiss army knife • Load Balancing

    ➡ haproxy ( http://www.haproxy.org/ ) • Web Server ➡ nginx ( http://nginx.org/ ) • SSL Termination ➡ stud ( https://github.com/bumptech/stud ) • GZIP Compression ➡ nginx / haproxy • Static Assets ➡ CDN / Varnish ( https://www.varnish-cache.org/ )
  40. Limitations

  41. v8 Limitations volkan.io @linkibol

  42. v8 Limitations • ~2gb ➡ heap limit (--max-old-space-size) • will

    be more than enough • spawn more processes for more (child_process) • ~1gb ➡ max size of a Buffer
  43. • Lazily produce or consume data in buffered chunks. •

    Evented and non-blocking. • Low memory footprint. • Automatically handle back-pressure. • Buffers allow you to work around the v8 heap memory limit. • Most core node.js content sources/sinks are streams already. Show Love to Streams
  44. OS Limitations volkan.io @linkibol

  45. Open File Limits • “Error: EMFILE, Too many open files”

    • ulimit -n 65535; volkan.io @linkibol
  46. Open File Limits

  47. Configuring the Load Balancer see https://bit.ly/nginx-rocks

  48. Configuring the Load Balancer see https://bit.ly/nginx-rocks

  49. Additional Tweaks http.globalAgent.maxSockets = SOME_BIG_NUMBER; (if you have many outgoing

    requests) volkan.io @linkibol
  50. Additional Tweaks sysctl -w net.core.somaxconn=1024; (default is 128) volkan.io @linkibol

  51. Even More Tweaks

  52. Monitoring volkan.io @linkibol

  53. You Can’t Optimize What You Don’t Measure volkan.io @linkibol

  54. Things to Watch Out For • Your API Service may

    Become CPU-Bound • External API Calls Can Be a Bottleneck • Always Keep an Eye on the Event Loop • Implement Sanity Checks • Circuit Breaker • Have an Upper Bound for Concurrency
  55. Things to Watch Out For • Is app running and

    functional? • Is app overloaded? • How many errors have been raised so far? • How many times do forks restored? • Are all forks alive and okay? • Is app performant (throughput, memory utilization, concurrency)?
  56. Which Will (most of the time) Boil Down to… •

    Watching Response Times • Watching CPU Utilization + General Sys Resource Usage • Watching Number of Concurrent Connections
  57. Throughput / Concurrency volkan.io @linkibol

  58. Throughput vs Concurrency throughput concurrent connections

  59. Throughput vs Concurrency throughput concurrent connections throughput concurrent connections

  60. Throughput vs Concurrency throughput concurrency

  61. Throughput vs Concurrency throughput concurrency near-linear rapid decrease degrade throughput

    concurrency
  62. Throughput vs Concurrency throughput concurrency near-linear rapid decrease degrade throughput

    concurrency near-linear rapid decrease degrade throughput concurrency ideal path practical path
  63. Event Loop Delay volkan.io @linkibol

  64. Track the Event Loop

  65. Track the Event Loop volkan.io @linkibol

  66. Track the Event Loop

  67. Track the Event Loop

  68. Track the Event Loop

  69. Circuit Breaker closed fail (under threshold) open fail (reached threshold)

    checking… timer (exponential backoff) fail success volkan.io @linkibol
  70. Circuit Breaker • Can be used with any kind of

    metric. • Event-loop tracking is just a specific example. • Useful when you depend on other APIs that might fail. volkan.io @linkibol
  71. Monitoring Tools volkan.io @linkibol

  72. Monitoring as a Service • nodetime • https://nodetime.com/ • newrelic

    • http://newrelic.com/nodejs • strongloop • https://strongloop.com/node-js/performance-monitoring/
  73. Memwatch • https://github.com/lloyd/node-memwatch • heap stats • heap diffing •

    leak detection (subscribe to “leak” events)
  74. cluster-live https://github.com/tj/cluster-live

  75. Flame Graphs http://www.brendangregg.com/flamegraphs.html

  76. Flame Graphs • Can Be Created Post-Mortem (after a core

    dump) • Can Be Created at Runtime (using gcore *) • You Can Use dtrace + stackvis to generate them ** http://man7.org/linux/man-pages/man1/gcore.1.html * http://blog.nodejs.org/2012/04/25/profiling-node-js/ **
  77. Error Handling volkan.io @linkibol

  78. Error Handling • Use an error object; not a String

    (or better, use an error event) • https://github.com/davepacheco/node-verror • Error handling for Async Code Is Hard • Throwing does not make sense ( scope and stack trace is lost ) • Consider using domains ( https://nodejs.org/api/domain.html )
  79. Error Handling • Throwing is for programmer errors. • Consider

    using an error event instead of throwing.
  80. Handle Errors Gracefully • Know how exceptions and errors propagate

    in Node.JS. • Raise error events to throwing exceptions. • Restart on uncaught exceptions. • Utilize domains. https://www.joyent.com/developers/node/design/errors
  81. Using Domains

  82. Processes Die Accept it volkan.io @linkibol

  83. Processes Die Accept it volkan.io @linkibol

  84. Processes Die Accept it volkan.io @linkibol

  85. Processes Die Accept it No system is %100 resilient. Keep

    things as simple as possible. Build something that’s good enough for your purpose. Solve for the problems that are actually on your plate. volkan.io @linkibol
  86. Keep It Running •forever ( https://github.com/foreverjs/forever ) •pm2 ( https://github.com/Unitech/pm2

    ) •upstart ( http://upstart.ubuntu.com/ ) •systemd ( https://www.wikiwand.com/en/Systemd )
  87. Debugging volkan.io @linkibol

  88. Debugging • Live Debugging • Post-Mortem Debugging volkan.io @linkibol

  89. Live Debugging Given the Tornado, Where’s the Butterfly? volkan.io @linkibol

  90. Post-Mortem Debugging volkan.io @linkibol

  91. Node.JS Debugging Myths • Debugging and Profiling in Node.JS is

    Hard • Debugging and Profiling in Node.JS is Immature • You Cannot Debug or Profile a Live Production Node.JS App volkan.io @linkibol
  92. Node.JS Debugging Tips • Attaching a debugger to prod is

    not practical. • Make as much state as possible observable. • You can take a core dump, and analyze it later. volkan.io @linkibol
  93. Debugging Node.JS • You Can Expose Internal State via an

    API and/or a CLI/REPL • https://github.com/davepacheco/kang • https://nodejs.org/api/repl.html • Expose Additional Logging Info at Runtime (in systems that support it) • bunyan -p ( https://github.com/trentm/node-bunyan )
  94. Debugging Node.JS • Interactive Debugging • Using node-debug • https://nodejs.org/api/debugger.html

    • Using Node Inspector • https://github.com/node-inspector/node-inspector • Using Cloud9 IDE • https://c9.io/
  95. Debugging Node.JS • Log Libraries Specialized for Dumping Debug Info

    • Caterpillar ( https://github.com/bevry/caterpillar ) • Tracer ( https://github.com/baryon/tracer ) volkan.io @linkibol
  96. Logging volkan.io @linkibol

  97. What to Log • Authentication & Authorization • Session Management

    • Method Entry Points • Erros and Weird Events • Specific Events (startup, shutdown, slowdown etc.) • High-Risk Functionalities (payments, privileges, admins etc)
  98. Centralized Logging • Send logs to a log aggregation server;

    • Using a library that supports it: • bunyan ( https://github.com/trentm/node-bunyan ); • winston ( https://github.com/winstonjs/winston ); • custom (using Streams).
  99. Configuration volkan.io @linkibol

  100. • No Hard-Coded IP Addresses in Config Files • Let

    DNS do What it Does Best • Update Configuration From a Central Service • saltstack ( http://saltstack.com/ ) • chef ( https://www.chef.io/ ) • puppet ( https://puppetlabs.com/ ) Configuration
  101. Automation volkan.io @linkibol

  102. Have a CI/CD Pipeline • go ( http://www.go.cd/ ) •

    gradle ( http://gradle.org/ ) • jenkins ( https://jenkins-ci.org/ ) volkan.io @linkibol
  103. Code Quality volkan.io @linkibol

  104. Code Quality • Group Common Logic Into Reusable Modules •

    Modules Should Do One Thing, and Do One Thing Well • Use Static File Analyzers • jshint ( https://github.com/gruntjs/grunt-contrib-jshint ) • jscs ( https://github.com/jscs-dev/grunt-jscs ) • grunt-complexity ( https://github.com/vigetlabs/grunt-complexity )
  105. Perf Before Scale volkan.io @linkibol

  106. Memory Leaks volkan.io @linkibol

  107. Memory Leaks volkan.io @linkibol

  108. Memory Leaks volkan.io @linkibol

  109. Perf Before Scale • Configure Your System for High Performance

    • Cache at Every Layer • The fastest API response is no response at all. • Delegate Long-Running(*) Operations volkan.io @linkibol
  110. Perf Before Scale • CPU-Intensive Computations? • child_process + External

    Libraries • Native Extensions volkan.io @linkibol
  111. IO Optimization volkan.io @linkibol

  112. IO Optimization • Don’t Immediately Write Small Packets • Reduce

    the Number of Outgoing Requests • Use Lesser Abstractions for Maximum Throughput volkan.io @linkibol
  113. Know Your Bottlenecks • 99% of the time you will

    be IO-bound. • You’ll scale horizontally before hitting 100% CPU. • You’ll have to try really hard to have a CPU or memory bottleneck. • Node.JS serves really well as a highly concurrent networking app. • Node.JS is very sensitive to memory leaks and blocking code. volkan.io @linkibol
  114. Torture Your System • Try Chaos Monkey • https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey •

    Randomly send `kill -9` to Processes • Randomly Knock a Server Offline • Intentionally Run Out of Disk Space • Take an entire data center down
  115. Going Bare Bones volkan.io @linkibol

  116. express.js vs Bare-Bones Node.JS express.js http

  117. Diving Deeper

  118. Going Bare Bones express (with 5 common middleware) bare bones

    http bare bones tcp 0 1150 2300 3450 4600 4,517 3,561 2,891 requests per second ab -c 100 -n 1000 Tested on MacBook Pro, 2.4 GHz Intel Core i5, 16 GB 1600 MHz DDR3, single core, w/o keep-alive
  119. Is It Worth It? • Consider Going Bare-Bones for Maximum

    Throughput • Tradeoff: • harder to maintain • more complex code • error prone • lots of edge cases • harder to use additional tooling (i.e. no dtrace support)
  120. v8 Optimizations volkan.io @linkibol

  121. Taking Control of the Garbage Controller • Warning: You probably

    would not want to do that! • --expose-gc • --nouse-idle-notification volkan.io @linkibol
  122. Types of Compilers in v8 • Generic Compiler • Optimizing

    Compiler • Can Be Two or More Orders of Magnitude Faster volkan.io @linkibol
  123. Optimize Only Hot Code Paths Unless You Have Solid Evidence

    to Do Otherwise volkan.io @linkibol
  124. Tooling node --trace_opt 
 --trace_deopt 
 --allow-natives-syntax test.js; • console.log(%HasFastProperties(obj))

    • console.log(%GetOptimizationStatus(fn)) https://github.com/Nathanaela/v8-natives
  125. v8 Optimization Killers • These will be bailed out (likely,

    forever): • Using debugger anywhere within the function • Using eval anywhere within the function • Using with anywhere within the function https://github.com/petkaantonov/bluebird/wiki/Optimization-killers
  126. v8 Optimization Killers • These will be bailed out (for

    now): • Generators • Functions that contain a for-of statement • Functions that contain try-catch or try-finally • let assignments • const assignments • functions that contain object literals with
 __proto__, get, or set declarations.
  127. Infinite Loops With Unclear Logic • while(true) { … }

    • for( ; ; ) { … } volkan.io @linkibol
  128. Objects Object.prototype.baz = function() {}; • Do not define enumerable

    properties in the prototype chain. • Use Object.defineProperty to create non-enumerable properties.
  129. Objects

  130. for/each

  131. Isolate try/catch

  132. Isolate try/catch

  133. Isolate try/catch

  134. Isolate try/catch

  135. arguments Object

  136. Iteration Tips • Rule of thumb: Avoid for/in loops in

    hot code paths. • Always use Object.keys to iterate an object. • Rethink your architecture 
 if you need to iterate the parent prototype’s keys. volkan.io @linkibol
  137. Inherited keys volkan.io @linkibol

  138. Promises volkan.io @linkibol

  139. Callback Hell Is Real

  140. Promises Promises are not as slow as they once were.

    https://github.com/petkaantonov/bluebird
  141. Promises • Use promises — *ahem* BlueBird — liberally. •

    Consider using continuation passing style 
 (i.e. callbacks) for very hot code paths. • Callbacks are always faster, and more memory-efficient. • Do not optimize prematurely; measure things first!
  142. Security volkan.io @linkibol

  143. Do Not Run Node.JS As Root useradd -mrU web;
 mkdir

    /opt/web-app;
 chown web /opt/web-app;
 cd /opt/web-app;
 su web;
 node app.js;
 firewall-cmd --permanent --zone=public --add-port=3000/tcp volkan.io @linkibol
  144. Security • https://www.owasp.org/index.php/ OWASP_Node_js_Goat_Project • http://nodesecurity.io/ • https://www.owasp.org/index.php/ OWASP_Zed_Attack_Proxy_Project volkan.io

    @linkibol
  145. Common Threats • XSS / CSRF • Input Validation Attack

    • DoS / ReDoS • Request Size * not different from any other web app.
  146. Your API at Scale

  147. Goals • Minimize Client Response Time • Maximize Resource Efficiency

    on the Server
 Hint: Leave 50% of the memory unused
 (for taking core dumps)
  148. Microservices to the rescue volkan.io @linkibol

  149. Microservices web.js auth.js routing.js persistence.js messaging.js logging.js process A web.js

    process A web.js process B routing.js process C persistence.js process D messaging.js process E logging.js process F Monolith Microservice multiple modules single process multiple modules multiple processes
  150. Think Twice http://martinfowler.com/bliki/MicroservicePremium.html

  151. Favor Microservices Over Monoliths • Prefer Composition over Inheritance •

    Publish Modules Instead of Inlining Functionality • Use an Internal npm
  152. Scaling Your API volkan.io @linkibol

  153. Scaling Your API app memory volkan.io @linkibol

  154. Cluster volkan.io @linkibol

  155. Cluster

  156. Cluster

  157. Cluster

  158. Cluster

  159. Cluster + Domains volkan.io @linkibol

  160. Cluster + Domains

  161. Cluster + Domains

  162. Cluster + Domains

  163. Cluster + Domains

  164. Cluster + Domains

  165. Cluster + Zero Downtime Rolling Deployments volkan.io @linkibol

  166. Zero Downtime kill --USR2 <pid> volkan.io @linkibol

  167. Zero Downtime see also: https://github.com/doxout/recluster

  168. Zero Downtime see also: https://github.com/doxout/recluster

  169. Zero Downtime see also: https://github.com/doxout/recluster

  170. Build Redundancy Everywhere volkan.io @linkibol

  171. Build Redundancy Everywhere

  172. Build Redundancy Everywhere

  173. Build Redundancy Everywhere

  174. Build Redundancy Everywhere

  175. Build Redundancy Everywhere

  176. Cluster app memory app app app volkan.io @linkibol

  177. IS BIGGER ALWAYS BETTER? ultra mega super box with bazillion

    cores regular box
  178. Let’s Step Back app memory volkan.io @linkibol

  179. Delegate Heavy Computation to Workers app memory worker worker worker

    child_process volkan.io @linkibol
  180. Split Logical Components into Distinct Services API app memory compute

    app memory worker worker worker child_process volkan.io @linkibol
  181. IPC With a Message Bus API app memory compute app

    memory worker worker worker child_process message bus * * rabbitmq, zeromq, resque etc. see also http://queues.io/
  182. Message Bus Topologies P C Send/Listen P C1 C2 Worker

    Queue X C1 PubSub P C2
  183. Cluster Processes API app memory compute app memory worker worker

    worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process volkan.io @linkibol
  184. How Do We Multiplex? API app memory compute app memory

    worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process
  185. How Do We Multiplex? API app memory compute app memory

    worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app memory compute app memory worker worker worker child_process message bus API app c l u s t e r c l u s t e r compute app worker worker worker child_process ?
  186. Introduce a LB and a Broker API app compute app

    worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus volkan.io @linkibol
  187. Load Balancing Options • Elastic Load Balancing as a Service

    (amazon, rackspace…) • Hardware Load Balancer (Cisco ACE, Barracuda, etc…) • Software Load Balancer • nginx • haproxy • home grown
  188. Load Balancer

  189. Making the Load Balancer Highly Available • round-robin DNS •

    https://www.wikiwand.com/en/Round-robin_DNS • heartbeat • https://www.wikiwand.com/en/Heartbeat_(computing) • keepalived • http://keepalived.org/ * You can use these tools to make any component HA.
  190. Making the Load Balancer HA load balancer load balancer load

    balancer keepalived DNS failover active * see also: https://www.wikiwand.com/en/Virtual_Router_Redundancy_Protocol
  191. SSL Termination load balancer load balancer keepalived DNS failover active

    SSL Terminator SSL Terminator keepalived DNS failover active load balancer load balancer * * * https://github.com/bumptech/stud
  192. Elephant in the Room How Do We Manage State? volkan.io

    @linkibol
  193. Share State With Redis API app compute app worker worker

    worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis volkan.io @linkibol
  194. Share State With Redis API app compute app worker worker

    worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis volkan.io @linkibol
  195. Make Redis Redundant redis redis (master) redis (read replica) redis

    (read replica) redis (read replica) redis (master) redis (read replica) redis (read replica) redis (read replica) round-robin DNS This will also increase throughput as a side benefit. See http://redis.io/topics/replication and http://redis.io/topics/cluster-tutorial You can also use a managed “memory as a service” solution.
  196. We Can Add More… API app compute app worker worker

    worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … …
  197. we are actually using Microservices here volkan.io @linkibol

  198. Microservices API app compute app worker worker worker child_process API

    app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … …
  199. Microservices API app compute app worker worker worker child_process API

    app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … … API μ-Service
  200. Microservices API app compute app worker worker worker child_process API

    app c l u s t e r c l u s t e r compute app worker worker worker child_process API app compute app worker worker worker child_process API app c l u s t e r c l u s t e r compute app worker worker worker child_process broker load balancer Internet message bus redis redis … … API μ-Service Compute μ-Service
  201. Microservices API μ-Service Compute μ-Service load balancer broker redis redis

    message bus Internet volkan.io @linkibol
  202. What If We Exhaust All the Bandwidth? 
 That Means

    You’ve Become Famous Scalability Will Be the Least of Your Concerns volkan.io @linkibol
  203. Scaling Into Multiple Zones API μ-Service Compute μ-Service load balancer

    broker redis redis message bus Internet Zone 1 volkan.io @linkibol
  204. Scaling Into Multiple Zones API μ-Service Compute μ-Service load balancer

    broker redis redis message bus Internet Zone 1 API μ-Service Compute μ-Service load balancer broker redis redis message bus Zone 2
  205. Scaling Into Multiple Zones round-robin DNS API μ-Service Compute μ-Service

    load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet
  206. Scaling Into Multiple Zones round-robin DNS API μ-Service Compute μ-Service

    load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet
  207. Mirroring the Data Stores round-robin DNS API μ-Service Compute μ-Service

    load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror
  208. Mirroring the Data Stores round-robin DNS API μ-Service Compute μ-Service

    load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror
  209. We Can Add More… round-robin DNS API μ-Service Compute μ-Service

    load balancer broker redis redis message bus API μ-Service Compute μ-Service load balancer broker redis redis message bus Internet mirror mirror …
  210. scale 2 ∞ & 㱺

  211. Thank You Questions? @linkibol