Slide 1
Slide 1 text
@robb1e
how to lean on others to get stuff done
Software Engineer, Pivotal Labs
Robbie Clutton
Startup Architecture
Slide 2
Slide 2 text
@robb1e
how to lean on others to get stuff done
Software Engineer, Pivotal Labs
Robbie Clutton
Startup Architecture
Slide 3
Slide 3 text
@robb1e
with Rails applications
Software Engineer, Pivotal Labs
Robbie Clutton
Startup Architecture
Slide 4
Slide 4 text
@robb1e
Simple, small web application
Service oriented RESTful/pubsub/LMAX/P2P
distributed architecture
Gray area
Slide 5
Slide 5 text
@robb1e
Our codebase is 5 years
old and too hard to change
Slide 6
Slide 6 text
@robb1e
We’ve allowed our design
to evolve into a big
ball of mud
Slide 7
Slide 7 text
@robb1e
We’ll probably create
services at some point,
might as well start there
Slide 8
Slide 8 text
@robb1e
I’m going to design
everything up front based on
unvaldiated assumptions
Slide 9
Slide 9 text
@robb1e
Kent Beck
Make it work,
make it right,
make it fast
Slide 10
Slide 10 text
@robb1e
• Features that have hypotheses
• Hypotheses that can be easily validated
• Code that is always production ready
• Code that is easy to change
Goals
Slide 11
Slide 11 text
@robb1e
Creating sustainable small architectures
Software Engineer, Pivotal Labs
Robbie Clutton
Startup Architecture
Slide 12
Slide 12 text
@robb1e
Real stories from
colleagues and myself
Slide 13
Slide 13 text
@robb1e
Names have been
changed to protect
the innocent
Slide 14
Slide 14 text
@robb1e
Some stories are
pre-production, others
are in production
Slide 15
Slide 15 text
@robb1e
Crazy Egg
Story
Slide 16
Slide 16 text
@robb1e
10am deploy CrazyEgg
Slide 17
Slide 17 text
@robb1e
5pm review CrazyEgg
Slide 18
Slide 18 text
@robb1e
Users clicking headers
that are not links
Slide 19
Slide 19 text
@robb1e
You could feel the
users frustration
Slide 20
Slide 20 text
@robb1e
Simple user testing can pay dividends
Lesson
Slide 21
Slide 21 text
@robb1e
• CrazyEgg.com
• UserTesting.com
• SliverbackApp.com
• LeanLaunchLab.com
• Trello.com
Tools
Slide 22
Slide 22 text
@robb1e
Funnels, user testing,
hypotheses and validations
Story
Slide 23
Slide 23 text
@robb1e
Product with wizard like
pages which pre-selected
default services
Slide 24
Slide 24 text
@robb1e
Changes to the basket
updates price in real-time
Slide 25
Slide 25 text
@robb1e
Funnel showed massive
drop off at a certain step
Slide 26
Slide 26 text
@robb1e
In person user testing
to discover why the
drop off was occurring
Slide 27
Slide 27 text
@robb1e
Create hypothesis to
stop users leaving at
this junction
Slide 28
Slide 28 text
@robb1e
Implement change:
allow users to use
default or create own
Slide 29
Slide 29 text
@robb1e
Review funnel
after deployment
Slide 30
Slide 30 text
@robb1e
Learn what is blocking the users
Lesson
Slide 31
Slide 31 text
@robb1e
• KissMetrics.com
• StatsD (Etsy)
• Cube (Square)
Tools
Slide 32
Slide 32 text
@robb1e
Always be validating
Take away
Slide 33
Slide 33 text
@robb1e
You’re gonna need
a bigger boat
Story
Slide 34
Slide 34 text
@robb1e
Dave walks into a new job
START-UP 3.0
Slide 35
Slide 35 text
@robb1e
We need more
RAM for the
database
Product manager tells you
Slide 36
Slide 36 text
@robb1e
Product manager tells you
This report takes
20 minutes
to run.
Slide 37
Slide 37 text
@robb1e
Hmm, ok
Slide 38
Slide 38 text
@robb1e
There are no indexes
Slide 39
Slide 39 text
@robb1e
Slide 40
Slide 40 text
@robb1e
No primary or
foreign keys
Slide 41
Slide 41 text
@robb1e
Needed more RAM so
the whole database
could fit in memory
Slide 42
Slide 42 text
@robb1e
Dave cleans up a bit,
report now takes 10
seconds to run
Slide 43
Slide 43 text
@robb1e
Use tools to discover simple mistakes
Lesson
Slide 44
Slide 44 text
@robb1e
Passing tests don’t
imply production quality
Bonus
Slide 45
Slide 45 text
@robb1e
• Rails Best Practices
• SQL Explain
• NewRelic.com
Tools
Slide 46
Slide 46 text
@robb1e
Instrument, refactor, repeat
Story
Slide 47
Slide 47 text
@robb1e
Dave walks into a new job
START-UP 3.0
Slide 48
Slide 48 text
@robb1e
Client moving from
ColdFusion to Ruby
Slide 49
Slide 49 text
@robb1e
Yes, there are people
still using ColdFusion
Slide 50
Slide 50 text
@robb1e
Ruby is slow and
we’re going to
production next
week.
Product manager tells you
Slide 51
Slide 51 text
@robb1e
Product manager tells you
We’ve made a
terrible mistake...
Slide 52
Slide 52 text
@robb1e
You say...
Hold on a minute,
let’s take a look
Slide 53
Slide 53 text
@robb1e
Instrument to find slow
requests/queries
Slide 54
Slide 54 text
@robb1e
Refactor slowest query
until more performant
with green tests
Slide 55
Slide 55 text
@robb1e
Rinse and repeat until
performance has
improved enough
Slide 56
Slide 56 text
@robb1e
Paul Hammond, 2012
Every scaling story:
1. Find the biggest problem
2. Fix the biggest problem
3. Repeat
Slide 57
Slide 57 text
@robb1e
‘Friday afternoon’ performance
refactoring can build upon itself
Bonus
Slide 58
Slide 58 text
@robb1e
• NewRelic.com
• CodeClimate.com
• Emma, FindBugs
Tools
Slide 59
Slide 59 text
@robb1e
Use tools to discover improvements
Take away
Slide 60
Slide 60 text
@robb1e
Distributed cache
Story
Slide 61
Slide 61 text
@robb1e
Website was growing
and gaining visitors
Slide 62
Slide 62 text
@robb1e
Scaling strategy was to
add app servers
Slide 63
Slide 63 text
@robb1e
Each server had the web
app and a local cache
Slide 64
Slide 64 text
@robb1e
Spinning up a new server
meant more pressure
on the database
Slide 65
Slide 65 text
@robb1e
Using a distributed cache
bought the team time to
make improvements
Slide 66
Slide 66 text
@robb1e
Caching can buy significant
performance improvements
Lesson
Slide 67
Slide 67 text
@robb1e
• MemcacheD.org
• Varnish-Cache.org
• Squid-Cache.org
Tools
Slide 68
Slide 68 text
@robb1e
To cache, or not to cache?
Story
Slide 69
Slide 69 text
@robb1e
Sometimes code
speaks to you
Yo.
Slide 70
Slide 70 text
@robb1e
This part is slow,
let’s cache it.
Problem solved
Slide 71
Slide 71 text
@robb1e
But I’m going to
invalidate that
elsewhere
Slide 72
Slide 72 text
@robb1e
Collection of widgets
being rendered with
new and old design
Slide 73
Slide 73 text
@robb1e
Can’t replicate on
staging or locally
Slide 74
Slide 74 text
@robb1e
Clear ALL the cache
Slide 75
Slide 75 text
@robb1e
Changing the template had
not invalidated the entry
Slide 76
Slide 76 text
@robb1e
- Phil Karlton
"There are only two hard things in
Computer Science: cache
invalidation and naming things."
Slide 77
Slide 77 text
@robb1e
Caching can obsure
poorly written code
Bonus
Slide 78
Slide 78 text
@robb1e
Be careful what you cache
Take away
Slide 79
Slide 79 text
@robb1e
Non-essential work
during a request
Story
Slide 80
Slide 80 text
@robb1e
User registration
stopped working
Slide 81
Slide 81 text
@robb1e
Mailing list provider
was down
Slide 82
Slide 82 text
@robb1e
Exception bubbled up
and prevented
registering new user
Slide 83
Slide 83 text
@robb1e
Put mailing list
subscription in
background job
Slide 84
Slide 84 text
@robb1e
Shorten the request/response cycle
Lesson
Slide 85
Slide 85 text
@robb1e
When dealing with integrations, some
healthy paranoia is a good thing
Bonus
Slide 86
Slide 86 text
@robb1e
• Background workers
• Message Queues
• Threads
Tools
Slide 87
Slide 87 text
@robb1e
A tale of two websites
Story
Slide 88
Slide 88 text
@robb1e
www.guardian.co.uk
125 requests
1.2MB
HTML: 3.7s
Loaded: 8.4s
Slide 89
Slide 89 text
@robb1e
m.guardian.co.uk
44 requests
340KB
HTML: 1.68s
Loaded: 3.32s
Slide 90
Slide 90 text
@robb1e
That’s not the result of
better SQL or server
optimizations
Slide 91
Slide 91 text
@robb1e
Result of highly tuned
client-side Javascript
and CSS
Slide 92
Slide 92 text
@robb1e
No (large)
Javascript libraries
Slide 93
Slide 93 text
@robb1e
Not even jQuery
Slide 94
Slide 94 text
@robb1e
Conditional loading of
secondary content
Slide 95
Slide 95 text
@robb1e
- Steve Saunders, 2007
“Optimize front-end performance
first, that's where 80% or more of
the end-user response time is
spent”
Slide 96
Slide 96 text
@robb1e
• Firebug
• Chrome Developer Tools
• Compass
• YSlow
• YUI Compressor
Tools
Slide 97
Slide 97 text
@robb1e
Perceived performance is more
important than actual performance
Take away
Slide 98
Slide 98 text
@robb1e
Was that really the
best use of your time?
Story
Slide 99
Slide 99 text
@robb1e
During technical due
diligence for an acquisition
Slide 100
Slide 100 text
@robb1e
The company had built
their own message queue
Slide 101
Slide 101 text
@robb1e
No persistence
Slide 102
Slide 102 text
@robb1e
Didn’t use standard
protocol like AMPQ
Slide 103
Slide 103 text
@robb1e
Not explicitly sending a
terminating character
would eventually result
in the queue crashing
Slide 104
Slide 104 text
@robb1e
Almost all transactions
passed through this queue
Slide 105
Slide 105 text
@robb1e
Not buying a message
queue company
Slide 106
Slide 106 text
@robb1e
- Joel Spolsky, 2001
"If it's a core business
function - do it yourself,
no matter what."
Slide 107
Slide 107 text
@robb1e
Time is the most expensive out going
Bonus
Slide 108
Slide 108 text
@robb1e
Real-time vs near-time
Story
Slide 109
Slide 109 text
@robb1e
Trading system which
updates users’ screen
every 10 seconds
Slide 110
Slide 110 text
@robb1e
Lots of number crunching
and message queues
Slide 111
Slide 111 text
@robb1e
Did some in the
field research
Slide 112
Slide 112 text
@robb1e
Traders only checked
values every few minutes
Slide 113
Slide 113 text
@robb1e
This was not high
volatile trading
http://www.boldjack.com/wp-content/uploads/2012/01/wall_street4.jpg
Slide 114
Slide 114 text
@robb1e
Removed message
queues and moved
to publishing updates
to web server directly
Slide 115
Slide 115 text
@robb1e
Reduced complexity
of the product
Slide 116
Slide 116 text
@robb1e
Ron Jefferies, ~2005
Always implement things when
you actually need them, never
when you just foresee that you
need them
Slide 117
Slide 117 text
@robb1e
‘Real-time’ can mean different things
depending on who you talk too
Bonus
Slide 118
Slide 118 text
@robb1e
Buy vs build
Story
Slide 119
Slide 119 text
@robb1e
$50 a month is really
expensive for this
hosted service
Slide 120
Slide 120 text
@robb1e
We can build it ourselves
and get exactly the
features we need
Slide 121
Slide 121 text
@robb1e
Can you build the
widget service yourself
in that time?
Slide 122
Slide 122 text
@robb1e
Are you in the
widget business?
Slide 123
Slide 123 text
@robb1e
Francis Hwang, 2012
The biggest expense for a startup
is your time. Not your laptop, not
your hosting bill, not your office,
but the hours in your day.
Slide 124
Slide 124 text
@robb1e
Focus on your differentiators
Bonus
Slide 125
Slide 125 text
@robb1e
Over engineering is a form of waste
Take away
Slide 126
Slide 126 text
@robb1e
Horizontal Scalability
Story
Slide 127
Slide 127 text
@robb1e
Guardian Content API
is read only and
eventually consistent
Slide 128
Slide 128 text
@robb1e
Used by m., iPhone app,
parts of www. and more
Slide 129
Slide 129 text
@robb1e
Just a simple API over
an indexed data store
Slide 130
Slide 130 text
@robb1e
Each server has
it’s own data store
Slide 131
Slide 131 text
@robb1e
Each data store is a replica
of an internal master
Slide 132
Slide 132 text
@robb1e
Simple, elegant design can prevent
complex architecture creep
Lesson
Slide 133
Slide 133 text
@robb1e
• Solr
• Elastic Search
• MongoDB
Tools
Slide 134
Slide 134 text
@robb1e
Emergency mode
Story
Slide 135
Slide 135 text
@robb1e
Use of feature switches
at Guardian enable
‘super happy fun mode’
Slide 136
Slide 136 text
@robb1e
Turn features off when
site under increased load
Slide 137
Slide 137 text
@robb1e
Content is king and must
be readable at all times
Slide 138
Slide 138 text
@robb1e
Page pressing enables
zero downtime and
last fallback
Slide 139
Slide 139 text
@robb1e
Feature flags can offer resilience as
well as a way to roll out new features
Lesson
Slide 140
Slide 140 text
@robb1e
Complex should be lots of simple
Take away
Slide 141
Slide 141 text
@robb1e
Allow architecture to evolve
Spend your time wisely
Refactor continuously
Slide 142
Slide 142 text
@robb1e
Sandi Metz
“The wrong abstraction is far more
damaging than no abstraction at
all. Waiting trumps guessing every
time”
Slide 143
Slide 143 text
@robb1e
Q/A