Designing systems to scale

Talk given at PHPNW12 in Manchester, England

Transcript

1. Designing Systems To Scale
2. I’m Michael
3. I’m @mheap I’m Michael
4. I’m absolutely terrified I’m @mheap I’m Michael
5. Designing Systems To Scale
6. Designing Systems To Scale
7. Designing Systems To Scale
8. Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. - Doug McIlroy
9. Do one thing, and do it well
11. V1
12. V1 • Codeigniter Application
13. V1 • Codeigniter Application • No (working) indices on the database
14. V1 • Codeigniter Application • No (working) indices on the database • Used the REST API and pulled data on demand
15. V1 • Codeigniter Application • No (working) indices on the database • Used the REST API and pulled data on demand • PHP for realtime parsing
16. V1 • Codeigniter Application • No (working) indices on the database • Used the REST API and pulled data on demand • PHP for realtime parsing • All stored in one git repo
17. 24 Users
18. V1 • Codeigniter Application • No (working) indices on the database • Used the REST API and pulled data on demand • PHP for realtime parsing • All stored in one git repo
19. V2
20. V2 • Decoupled
21. V2 • Decoupled • Made up of five “apps”
22. V2 • Decoupled • Made up of five “apps” • Easier to scale
23. V2 • Decoupled • Made up of five “apps” • Easier to scale • More fault tolerant
24. MySQL Twitter Redis API Bucket Parser API Website Mobile App
25. Do one thing, and do it well
26. ... is rule #1
27. If all you have is a hammer...
28. ... everything looks like a nail
29. Rule #2
30. Use the right tool for the job
31. LNMPNRM
32. LNMPNRM • Linux
33. LNMPNRM • Linux • nginx
34. LNMPNRM • Linux • nginx • MySQL
35. LNMPNRM • Linux • nginx • MySQL • PHP
36. LNMPNRM • Linux • nginx • MySQL • PHP • NodeJS
37. LNMPNRM • Linux • nginx • MySQL • PHP • NodeJS • Redis
38. LNMPNRM • Linux • nginx • MySQL • PHP • NodeJS • Redis • Mongo DB
39. LNMPNRMR? • Linux • nginx • MySQL • PHP • NodeJS • Redis • Mongo DB • Ruby
40. Rule #3
41. Stand on the shoulders of giants
42. Do one thing, and do it well Use the right tool for the job Stand on the shoulders of giants
43. Now for the technical stuff...
44. Service Oriented Architecture
45. Shared Nothing Architecture
46. ... Except your database
47. Make it stateless
48. Make it event driven
49. API Driven Design
50. Models talk to the Database
51. Models talk to the API
52. API = Data
53. Website
54. Mobile App
55. Raspberry Pi
56. API = Data
57. Things I’ve learned
58. Expose JSON
59. REST is good
60. REST is good Except when it’s not
61. Databases
62. We use MySQL
63. We use InnoDB
64. MySQL 5.6 EXPLAIN ALL THE THINGS!
66. pt-online-schema-change xtrabackup pt-archive pt-query-digest pt-query-advisor pt-show-grants pt-fingerprint
67. Things I’ve learned
68. Indexes go from left to right
69. User (email, first, last, dob) name_index(first, last) email_index(email) user_index(last, dob)
70. InnoDB uses primary key in indexes
71. Speed up SELECT Slow down INSERT Indices
72. One connection means One query
73. ORM’s are evil
74. Denormalisation is ok
75. Backups
76. mysqldump
77. InnoDB Hot Backup
78. Percona Xtrabackup
79. 1,000,001
80. Recovery
81. Choose your battles
82. Servers
83. Servers (in the cloud?)
84. Learn to set up a server
85. Leave it to the professionals
86. Virtualisation
87. Get a VPS with Amazon AWS
88. Get a VPS with Rackspace
89. Get a VPS with Azure
90. Get a VPS with $vpsCompany
91. Leeds Hack
92. Things I’ve learned
93. £££
94. Multiple small beats one large
95. Dedicated is good too
96. Host your own database
97. Documentation
98. Not just about writing your own
100. You should also write your own
102. Easy Can we still deploy?
103. Easy Can we still deploy? Harder What if we need a new server firing up?
104. Easy Can we still deploy? Harder What if we need a new server firing up? Even Harder What if $customComponent breaks?
105. Make yourself dispensable
106. Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live - John f. Woods
107. Things I’ve learned
108. Smart Defaults
109. Dependency Management
110. Just use Composer or rubygems/npm/pip... etc
111. Things I’ve learned
112. Lock your dependecies
113. v 1.4.2
114. v 1.4.2 Bug Fix
115. v 1.4.2 Feature release
116. v 1.4.2 All hell breaks loose
117. Lock your dependecies
118. Control your dependecies
120. Control your dependecies
121. Automation
122. Predictability
123. It saves time
124. Choose one [MRJC]ake
125. forever stop /home/user/parser/parser.js && APP_ENV=production forever start -a -l ~/parser-logs/ parser.log -o ~/parser-logs/out.log -e ~/parser-logs/ parser.err.log --minUptime 10000 --spinSleepTime 8000 ~/ parser/parser.js
126. forever stop /home/user/parser/parser.js && APP_ENV=production forever start -a -l ~/parser-logs/ parser.log -o ~/parser-logs/out.log -e ~/parser-logs/ parser.err.log --minUptime 10000 --spinSleepTime 8000 ~/ parser/parser.js parser_control restart
127. Automating scary things
128. Push data not pull
129. Things I’ve learned
130. Vagrant
131. Puppet/Chef
132. Devops!
133. Logging
134. Log *everything*
135. Aggregated logs
136. Monolog
137. Make it an option
138. Things I’ve learned
139. Nothing... yet
140. Outsourcing
141. Things like hosting repos
142. Things like sending emails
143. Things like *insert task*
144. Outsourcing validation
145. Things I’ve learned
146. You don’t have to do everything
147. Testing
148. Testing is for peace of mind
149. It is not a design process
150. Integration tests over unit tests
151. Unit tests are nice too
152. Things I’ve learned
153. Test only what you need
154. Behat + Selenium
155. Being Human
156. You’re your own worst enemy
157. Logging queries to MongoDB
158. Node.js’ process.nextTick
159. Something I haven’t even realised
160. Things I’ve learned
161. Learn to fail
162. People make mistakes
163. Believe in yourself
164. I’m Michael
165. I’m @mheap I’m Michael
166. I hope you enjoyed this talk I’m @mheap I’m Michael