Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Designing Production Systems

Cc91b2930b6a4bce3e16c4ead40d9679?s=47 pigol
September 20, 2017

Designing Production Systems

Deck of the talk I gave at a company meet up on what all aspects to be considered when designing large scale production systems.

Cc91b2930b6a4bce3e16c4ead40d9679?s=128

pigol

September 20, 2017
Tweet

More Decks by pigol

Other Decks in Programming

Transcript

  1. Designing Production Systems A Braindump Piyush Goel

  2. … with proper design, the features come cheaply. This approach

    is arduous, but continues to succeed. -- Dennis Ritchie Design
  3. Design … but for what?

  4. Features … currently at hand

  5. Features … currently at hand and future Ask Questions? What’s

    next for the feature
  6. Design ... for Users

  7. Design ... for Smart users

  8. Design ... for Dumb users Never make assumptions

  9. Design … for Failure

  10. Design … for Failure Never make assumptions

  11. Design … for Constraints

  12. Design … for Scalability

  13. Design … for Scalability

  14. Design … for Scalability Don’t over optimize.. But know when

    your design will fail.
  15. Design … for Testability

  16. Design … for Testability Unit Testing Integration Testing Manual Testing

    Automation Testing
  17. Design … for Deployment Single server, multi server, rolling restarts,

    database scripts.
  18. Design … for Monitorability

  19. Design … for Monitorability Emit meaningful metrics!

  20. Design … for Debuggability

  21. Design … for Debuggability? Understand & Embrace Logs

  22. Design … for Recovery

  23. Design … for Recovery Systems will Fail … Plan how

    to recover!
  24. Design … for Security Users will abuse your system ..

    Deliberately or by Mistake!
  25. How do I design?

  26. How do I design? What is my product stack?

  27. How do I design? What is my product stack? •

    Browser • Mobile • DNS Servers • Load Balancer • Nginx • Web Server • Internal ELB • Rails • Redis/Varnish • ElasticSearch • MySQL • Wordpress • Postfix
  28. How do I design? Use & Understand the modules!

  29. How do I design? Modules should be designed to handle

    failures in dependee modules!
  30. How do I design?

  31. How do I design? What is my product stack? •

    Browser • Mobile • DNS Servers • Load Balancer • Nginx • Web Server • Internal ELB • Rails • Redis/Varnish/SideKiq • ElasticSearch • MySQL • Wordpress • Postfix
  32. How do I design? Handle Failures Gracefully!

  33. How do I design? Don’t let UX deteriorate! Frontend and

    Design Team
  34. How do I design? Fail Individual Components!

  35. How do I design? Fail Individual Components! - Circuit Breaking

    - Bulk Heading - Timing Out - Retries - Asynchronous Flows
  36. How do I design? CPU, RAM & I/O (Disk, Network)

    Understand the basic components … duh!!
  37. How do I design? Understand the nuts and bolts! -

    TCP - HTTP - Headers/Proxies
  38. How do I design? Learn tools and how to read

    numbers - Htop - Chrome Dev Tools - Iotop - Vmstat & more… - Netstat - Lsof
  39. How do I design? Numbers every engineers should know. Talk

    by Jeffrey Dean, Google. https://www.youtube.com/watch?v=mod XC5IWTJI
  40. How do I design? Read Read Read!! - Blogs -

    YouTube videos - Books - Papers - Talk to your friends