Slide 1
Slide 1 text
Designing Systems To Scale
Slide 2
Slide 2 text
I’m Michael
Slide 3
Slide 3 text
I’m @mheap
I’m Michael
Slide 4
Slide 4 text
I’m absolutely terrified
I’m @mheap
I’m Michael
Slide 5
Slide 5 text
Designing Systems To Scale
Slide 6
Slide 6 text
Designing Systems To Scale
Slide 7
Slide 7 text
Designing Systems To Scale
Slide 8
Slide 8 text
Write programs that do one thing
and do it well.
Write programs to work together.
Write programs to handle text
streams, because that is a universal
interface.
- Doug McIlroy
Slide 9
Slide 9 text
Do one
thing, and
do it well
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
V1
Slide 12
Slide 12 text
V1
• Codeigniter Application
Slide 13
Slide 13 text
V1
• Codeigniter Application
• No (working) indices on the database
Slide 14
Slide 14 text
V1
• Codeigniter Application
• No (working) indices on the database
• Used the REST API and pulled data on demand
Slide 15
Slide 15 text
V1
• Codeigniter Application
• No (working) indices on the database
• Used the REST API and pulled data on demand
• PHP for realtime parsing
Slide 16
Slide 16 text
V1
• Codeigniter Application
• No (working) indices on the database
• Used the REST API and pulled data on demand
• PHP for realtime parsing
• All stored in one git repo
Slide 17
Slide 17 text
24 Users
Slide 18
Slide 18 text
V1
• Codeigniter Application
• No (working) indices on the database
• Used the REST API and pulled data on demand
• PHP for realtime parsing
• All stored in one git repo
Slide 19
Slide 19 text
V2
Slide 20
Slide 20 text
V2
• Decoupled
Slide 21
Slide 21 text
V2
• Decoupled
• Made up of five “apps”
Slide 22
Slide 22 text
V2
• Decoupled
• Made up of five “apps”
• Easier to scale
Slide 23
Slide 23 text
V2
• Decoupled
• Made up of five “apps”
• Easier to scale
• More fault tolerant
Slide 24
Slide 24 text
MySQL
Twitter
Redis
API
Bucket
Parser
API
Website Mobile App
Slide 25
Slide 25 text
Do one
thing, and
do it well
Slide 26
Slide 26 text
... is
rule #1
Slide 27
Slide 27 text
If all you
have is a
hammer...
Slide 28
Slide 28 text
... everything
looks like a
nail
Slide 29
Slide 29 text
Rule #2
Slide 30
Slide 30 text
Use the
right tool
for the job
Slide 31
Slide 31 text
LNMPNRM
Slide 32
Slide 32 text
LNMPNRM
• Linux
Slide 33
Slide 33 text
LNMPNRM
• Linux
• nginx
Slide 34
Slide 34 text
LNMPNRM
• Linux
• nginx
• MySQL
Slide 35
Slide 35 text
LNMPNRM
• Linux
• nginx
• MySQL
• PHP
Slide 36
Slide 36 text
LNMPNRM
• Linux
• nginx
• MySQL
• PHP
• NodeJS
Slide 37
Slide 37 text
LNMPNRM
• Linux
• nginx
• MySQL
• PHP
• NodeJS
• Redis
Slide 38
Slide 38 text
LNMPNRM
• Linux
• nginx
• MySQL
• PHP
• NodeJS
• Redis
• Mongo DB
Slide 39
Slide 39 text
LNMPNRMR?
• Linux
• nginx
• MySQL
• PHP
• NodeJS
• Redis
• Mongo DB
• Ruby
Slide 40
Slide 40 text
Rule #3
Slide 41
Slide 41 text
Stand on the
shoulders
of giants
Slide 42
Slide 42 text
Do one thing, and do it well
Use the right tool for the job
Stand on the shoulders of giants
Slide 43
Slide 43 text
Now for the
technical
stuff...
Slide 44
Slide 44 text
Service
Oriented
Architecture
Slide 45
Slide 45 text
Shared
Nothing
Architecture
Slide 46
Slide 46 text
... Except your
database
Slide 47
Slide 47 text
Make it
stateless
Slide 48
Slide 48 text
Make it
event driven
Slide 49
Slide 49 text
API Driven
Design
Slide 50
Slide 50 text
Models
talk to the
Database
Slide 51
Slide 51 text
Models
talk to the
API
Slide 52
Slide 52 text
API = Data
Slide 53
Slide 53 text
Website
Slide 54
Slide 54 text
Mobile App
Slide 55
Slide 55 text
Raspberry Pi
Slide 56
Slide 56 text
API = Data
Slide 57
Slide 57 text
Things
I’ve
learned
Slide 58
Slide 58 text
Expose
JSON
Slide 59
Slide 59 text
REST is
good
Slide 60
Slide 60 text
REST is
good
Except when it’s not
Slide 61
Slide 61 text
Databases
Slide 62
Slide 62 text
We use
MySQL
Slide 63
Slide 63 text
We use
InnoDB
Slide 64
Slide 64 text
MySQL 5.6
EXPLAIN ALL
THE THINGS!
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
pt-online-schema-change
xtrabackup
pt-archive
pt-query-digest
pt-query-advisor
pt-show-grants
pt-fingerprint
Slide 67
Slide 67 text
Things
I’ve
learned
Slide 68
Slide 68 text
Indexes go
from left to
right
Slide 69
Slide 69 text
User (email, first, last, dob)
name_index(first, last)
email_index(email)
user_index(last, dob)
Slide 70
Slide 70 text
InnoDB
uses
primary key
in indexes
Slide 71
Slide 71 text
Speed up SELECT
Slow down INSERT
Indices
Slide 72
Slide 72 text
One connection
means
One query
Slide 73
Slide 73 text
ORM’s
are evil
Slide 74
Slide 74 text
Denormalisation
is ok
Slide 75
Slide 75 text
Backups
Slide 76
Slide 76 text
mysqldump
Slide 77
Slide 77 text
InnoDB Hot
Backup
Slide 78
Slide 78 text
Percona
Xtrabackup
Slide 79
Slide 79 text
1,000,001
Slide 80
Slide 80 text
Recovery
Slide 81
Slide 81 text
Choose
your battles
Slide 82
Slide 82 text
Servers
Slide 83
Slide 83 text
Servers
(in the cloud?)
Slide 84
Slide 84 text
Learn to set up
a server
Slide 85
Slide 85 text
Leave it to the
professionals
Slide 86
Slide 86 text
Virtualisation
Slide 87
Slide 87 text
Get a VPS with
Amazon AWS
Slide 88
Slide 88 text
Get a VPS with
Rackspace
Slide 89
Slide 89 text
Get a VPS with
Azure
Slide 90
Slide 90 text
Get a VPS with
$vpsCompany
Slide 91
Slide 91 text
Leeds Hack
Slide 92
Slide 92 text
Things
I’ve
learned
Slide 93
Slide 93 text
£££
Slide 94
Slide 94 text
Multiple small
beats
one large
Slide 95
Slide 95 text
Dedicated
is good too
Slide 96
Slide 96 text
Host your own
database
Slide 97
Slide 97 text
Documentation
Slide 98
Slide 98 text
Not just about
writing your own
Slide 99
Slide 99 text
No content
Slide 100
Slide 100 text
You should also
write your own
Slide 101
Slide 101 text
No content
Slide 102
Slide 102 text
Easy
Can we still deploy?
Slide 103
Slide 103 text
Easy
Can we still deploy?
Harder
What if we need a new server firing up?
Slide 104
Slide 104 text
Easy
Can we still deploy?
Harder
What if we need a new server firing up?
Even Harder
What if $customComponent breaks?
Slide 105
Slide 105 text
Make yourself
dispensable
Slide 106
Slide 106 text
Always code as if the guy who ends
up maintaining your code will be a
violent psychopath who knows
where you live
- John f. Woods
Slide 107
Slide 107 text
Things
I’ve
learned
Slide 108
Slide 108 text
Smart
Defaults
Slide 109
Slide 109 text
Dependency
Management
Slide 110
Slide 110 text
Just use
Composer
or rubygems/npm/pip... etc
Slide 111
Slide 111 text
Things
I’ve
learned
Slide 112
Slide 112 text
Lock your
dependecies
Slide 113
Slide 113 text
v 1.4.2
Slide 114
Slide 114 text
v 1.4.2
Bug Fix
Slide 115
Slide 115 text
v 1.4.2
Feature release
Slide 116
Slide 116 text
v 1.4.2
All hell breaks loose
Slide 117
Slide 117 text
Lock your
dependecies
Slide 118
Slide 118 text
Control your
dependecies
Slide 119
Slide 119 text
No content
Slide 120
Slide 120 text
Control your
dependecies
Slide 121
Slide 121 text
Automation
Slide 122
Slide 122 text
Predictability
Slide 123
Slide 123 text
It saves time
Slide 124
Slide 124 text
Choose one
[MRJC]ake
Slide 125
Slide 125 text
forever stop /home/user/parser/parser.js &&
APP_ENV=production forever start -a -l ~/parser-logs/
parser.log -o ~/parser-logs/out.log -e ~/parser-logs/
parser.err.log --minUptime 10000 --spinSleepTime 8000 ~/
parser/parser.js
Slide 126
Slide 126 text
forever stop /home/user/parser/parser.js &&
APP_ENV=production forever start -a -l ~/parser-logs/
parser.log -o ~/parser-logs/out.log -e ~/parser-logs/
parser.err.log --minUptime 10000 --spinSleepTime 8000 ~/
parser/parser.js
parser_control restart
Slide 127
Slide 127 text
Automating
scary things
Slide 128
Slide 128 text
Push data
not pull
Slide 129
Slide 129 text
Things
I’ve
learned
Slide 130
Slide 130 text
Vagrant
Slide 131
Slide 131 text
Puppet/Chef
Slide 132
Slide 132 text
Devops!
Slide 133
Slide 133 text
Logging
Slide 134
Slide 134 text
Log
*everything*
Slide 135
Slide 135 text
Aggregated
logs
Slide 136
Slide 136 text
Monolog
Slide 137
Slide 137 text
Make it
an option
Slide 138
Slide 138 text
Things
I’ve
learned
Slide 139
Slide 139 text
Nothing... yet
Slide 140
Slide 140 text
Outsourcing
Slide 141
Slide 141 text
Things like
hosting repos
Slide 142
Slide 142 text
Things like
sending emails
Slide 143
Slide 143 text
Things like
*insert task*
Slide 144
Slide 144 text
Outsourcing
validation
Slide 145
Slide 145 text
Things
I’ve
learned
Slide 146
Slide 146 text
You don’t
have to do
everything
Slide 147
Slide 147 text
Testing
Slide 148
Slide 148 text
Testing is for
peace of mind
Slide 149
Slide 149 text
It is not a
design process
Slide 150
Slide 150 text
Integration tests
over unit tests
Slide 151
Slide 151 text
Unit tests are
nice too
Slide 152
Slide 152 text
Things
I’ve
learned
Slide 153
Slide 153 text
Test only
what you
need
Slide 154
Slide 154 text
Behat +
Selenium
Slide 155
Slide 155 text
Being Human
Slide 156
Slide 156 text
You’re your own
worst enemy
Slide 157
Slide 157 text
Logging queries
to MongoDB
Slide 158
Slide 158 text
Node.js’
process.nextTick
Slide 159
Slide 159 text
Something I
haven’t even
realised
Slide 160
Slide 160 text
Things
I’ve
learned
Slide 161
Slide 161 text
Learn to fail
Slide 162
Slide 162 text
People make
mistakes
Slide 163
Slide 163 text
Believe in
yourself
Slide 164
Slide 164 text
I’m Michael
Slide 165
Slide 165 text
I’m @mheap
I’m Michael
Slide 166
Slide 166 text
I hope you enjoyed this
talk
I’m @mheap
I’m Michael