Clearly I Have Made Some Bad Decisions

Too often in the organization of this conference we have heard "but I don’t have scalability issues".

This talk discusses what scalability issues actually are, and details why we all inevitably have them. Avoiding them, or delaying solutions until they are unavoidable, leads to making many bad "temporary" decisions that cannot be fixed further down the line.

I will discuss the methodologies and best practices that are required in order to be scalable, and describe the common mistakes they will temper, and why they should be implemented immediately. Finally, I will briefly touch on how to deal with rectifying the bad decisions that we all inevitably make, no matter how forward-thinking we are.

Transcript

2. “I don’t have scaling problems”
3. Scaling is about change not about quantity
4. Problems don’t occur when things are normal
5. If things change, you will have scaling problems
6. Work takes time to do
7. Work takes time to do Email needs to be read
8. Work takes time to do Email needs to be read Code runs on a server
9. Mistake! “I don’t have scaling problems”
10. Mistake! “I don’t have scaling problems” Not a mistake we’re making if we’re here?
11. Mistakes will be made Problems will happen
12. Mistakes will be made Problems will happen But there are things we can do to be prepared
13. #1 Measure Everything
14. How do you know if something is wrong?
15. How do you know if something is wrong? not wrong?
16. # uptime 17:27:18 up 405 days, 2:36, 1 user, load average: 26.93, 10.46, 6.16 !?!?
17. # uptime 17:27:18 up 405 days, 2:36, 1 user, load average: 26.93, 10.46, 6.16
19. Read your log files
20. Read your log files (Exceptions aren’t always exceptional)
21. Measure in production (hat tip: Coda, “metrics, metrics everywhere”) That’s the only place where things are really happening
22. Measure in production (hat tip: Coda, “metrics, metrics everywhere”) That’s the only place where things are really happening But don’t let your metrics cause performance problems
23. PING web (192.168.19.1): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 Sometimes you can just tell things are wrong
24. #2 Infrastructure as code (and config management)
25. Don’t do this.
26. Chef or Puppet (or cfengine or bcfg2) Server config is code
27. Chef or Puppet (or cfengine or bcfg2) Server config is code Revision control
28. Chef or Puppet (or cfengine or bcfg2) Server config is code Revision control Feature branches
29. Chef or Puppet (or cfengine or bcfg2) Server config is code Revision control Feature branches Commenting and authorship
30. Chef or Puppet (or cfengine or bcfg2) Server config is code Revision control Feature branches Commenting and authorship Centralized (not in someone’s head)
31. Should I choose Chef or Puppet?
32. Should I choose Chef or Puppet? Yes (Seriously, this is non-negotiable.)
33. How do I switch my servers to start using config management? My advice: build new ones, throw the old ones away.
34. Clean Known state test clusters Build
35. Clean Known state test clusters Destroy Build
36. Clean Known state test clusters live machines Destroy Build Build
37. Clean Known state test clusters live machines Destroy Build Build Use
38. Clean Known state test clusters live machines! Destroy Build Build Use Destroy
39. One-button servers What about your code?
40. #3a Real deployment
41. Don’t do this. $ svn up U www/index.php U www/payments.php U www/settings-live.php U www/settings-dev.php A www/specials.php U . Updated to revision 9703.
42. Deployment is more than just putting code in place.
43. Deployment is more than just putting code in place. reproducible idempotent rollouts
44. Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number
45. Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configuration
46. Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configuration triggered non-manually across any number of servers
47. Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configuration triggered non-manually across any number of servers with full dependency management
48. Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configuration triggered non-manually across any number of servers with full dependency management and automated regression testing.
49. Etsy’s Deployinator Vlad the Deployer Fabric Capistrano OS Packages Roll your own
50. #3b Continuous deployment
51. Holy Grail trunk = live tests block commits feature flags? dark launches?
52. Cowboy vs Perfectionist
53. Fast iteration = fast test results One huge feature tested ... and rejected Ten new tiny features tested Two accepted
54. Failure is comfortable Blame out, responsibility in Consequences immediately visible
55. Okay, fine: Continuous Integration
56. Things still go wrong After all that
57. #4 Plan for failure
58. Take backups Test backups
59. Automate servers Test server crashes
60. Netflix’s Chaos Monkey And cousins: the Simian Army
61. Server failures predicted and foiled What about code? New features?
62. #5 Future Compatibility
63. ALTER TABLE `user` ADD COLUMN `twootr` VARCHAR(16); CREATE INDEX `twootr_idx` ON `user` (`twootr`); Don’t do this. (on live)
64. “Future compatible” schemas “Future compatible” code Normalized tables are performance heavy Don’t assume any columns?
65. Shiny new Yucky old ? Read Write Migrate
66. What about other bad decisions?
67. #6 Wing It
68. - Django - MySQL spof.yola.com Scheduled for reboot
69. - Django - MySQL spof.yola.com - MySQL Slave Replication
70. - Django - MySQL spof.yola.com - MySQL
71. - Django spof.yola.com - Django - MySQL - MySQL Slave replication
72. - Django spof.yola.com - Django - MySQL - MySQL LB Slave replication
73. - Django spof.yola.com - Django - MySQL - MySQL LB Slave replication Drop DNS TTL
74. - Django spof.yola.com - Django - MySQL - MySQL LB Slave replication
76. But it’s okay
77. Jonathan Hitchcock @vhata github.com/vhata