Designing Production Systems

Deck of the talk I gave at a company meet up on what all aspects to be considered when designing large scale production systems.

Transcript

1. Designing Production Systems A Braindump Piyush Goel
2. … with proper design, the features come cheaply. This approach is arduous, but continues to succeed. -- Dennis Ritchie Design
3. Design … but for what?
4. Features … currently at hand
5. Features … currently at hand and future Ask Questions? What’s next for the feature
6. Design ... for Users
7. Design ... for Smart users
8. Design ... for Dumb users Never make assumptions
9. Design … for Failure
10. Design … for Failure Never make assumptions
11. Design … for Constraints
12. Design … for Scalability
13. Design … for Scalability
14. Design … for Scalability Don’t over optimize.. But know when your design will fail.
15. Design … for Testability
16. Design … for Testability Unit Testing Integration Testing Manual Testing Automation Testing
17. Design … for Deployment Single server, multi server, rolling restarts, database scripts.
18. Design … for Monitorability
19. Design … for Monitorability Emit meaningful metrics!
20. Design … for Debuggability
21. Design … for Debuggability? Understand & Embrace Logs
22. Design … for Recovery
23. Design … for Recovery Systems will Fail … Plan how to recover!
24. Design … for Security Users will abuse your system .. Deliberately or by Mistake!
25. How do I design?
26. How do I design? What is my product stack?
27. How do I design? What is my product stack? • Browser • Mobile • DNS Servers • Load Balancer • Nginx • Web Server • Internal ELB • Rails • Redis/Varnish • ElasticSearch • MySQL • Wordpress • Postfix
28. How do I design? Use & Understand the modules!
29. How do I design? Modules should be designed to handle failures in dependee modules!
30. How do I design?
31. How do I design? What is my product stack? • Browser • Mobile • DNS Servers • Load Balancer • Nginx • Web Server • Internal ELB • Rails • Redis/Varnish/SideKiq • ElasticSearch • MySQL • Wordpress • Postfix
32. How do I design? Handle Failures Gracefully!
33. How do I design? Don’t let UX deteriorate! Frontend and Design Team
34. How do I design? Fail Individual Components!
35. How do I design? Fail Individual Components! - Circuit Breaking - Bulk Heading - Timing Out - Retries - Asynchronous Flows
36. How do I design? CPU, RAM & I/O (Disk, Network) Understand the basic components … duh!!
37. How do I design? Understand the nuts and bolts! - TCP - HTTP - Headers/Proxies
38. How do I design? Learn tools and how to read numbers - Htop - Chrome Dev Tools - Iotop - Vmstat & more… - Netstat - Lsof
39. How do I design? Numbers every engineers should know. Talk by Jeffrey Dean, Google. https://www.youtube.com/watch?v=mod XC5IWTJI
40. How do I design? Read Read Read!! - Blogs - YouTube videos - Books - Papers - Talk to your friends