Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling For Planned Events
Search
Brian Akins
September 25, 2014
Programming
3
130
Scaling For Planned Events
Theory, Practice, and Aftermath
Brian Akins
September 25, 2014
Tweet
Share
More Decks by Brian Akins
See All by Brian Akins
CoreOS Overview
bakins
2
160
Large Scale Data Service as a Service
bakins
1
400
Growing Pains with Chef
bakins
0
73
Other Decks in Programming
See All in Programming
GNU Makeの使い方 / How to use GNU Make
kaityo256
PRO
13
4.4k
“Seeing Like a Programmer”—Resiliency, Limits, and Moral Hazards in Software Engineering (LambdaConf 2024)
chriskrycho
0
430
Enjoy Creative Coding with Ruby (RubyKaigi2024)
chobishiba
0
760
slow types ってなんだろう?
karad
0
210
Embedding it into Ruby code
soutaro
2
340
TypeScript Custom GitHub Action Development Tips
peaceiris
5
590
TypeScriptとGraphQLで実現する 型安全なAPI実装 / TSKaigi 2024
hokaccha
5
2.8k
How to implement a RubyVM with PHP?
memory1994
PRO
2
550
Implementing Design Systems in Swift
seyfoyun
2
530
Findy - エンジニア向け会社紹介 / Findy Letter for Engineers
findyinc
2
74k
Dev ContainersとTestcontainers
bells17
3
100
TypeScript 関数型スタイルでバックエンド開発のリアル
naoya
49
16k
Featured
See All Featured
Designing the Hi-DPI Web
ddemaree
276
33k
Creatively Recalculating Your Daily Design Routine
revolveconf
211
11k
The Brand Is Dead. Long Live the Brand.
mthomps
49
30k
How to Ace a Technical Interview
jacobian
273
22k
KATA
mclloyd
16
12k
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
GraphQLの誤解/rethinking-graphql
sonatard
56
9.3k
Why Our Code Smells
bkeepers
PRO
331
56k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
15
1.6k
Done Done
chrislema
178
15k
YesSQL, Process and Tooling at Scale
rocio
165
13k
Raft: Consensus for Rubyists
vanstee
133
6.3k
Transcript
Scaling For Planned Events Theory, Practice, and Aftermath
About @bakins • OmniTI - Senior Principal Architect • Horrible
public speaker • Old school C guy • Operations and Development • Wife and four kids in Atlanta, Georgia
What is a “Planned Event” • Event on existing site
• Single or repeating • Immovable dates • Elections, sporting events, etc.
Ideally • Event handled via normal site operations • Business
goals clearly defined • Clearly defined schedule • Realistic traffic/capacity models
Reality • Competing business goals • Heroic scaling efforts often
needed • Last minute feature additions • Real money tied to sponsorships • Can’t be tested in production until the event
Cache Everything? • If possible… • Introduce caching layer in
proxy mode early • Is your site really “dynamic”? • Need buy-in from all parties: dev, biz, ops, etc
Feature Creep • Company X will give us $Y is
we do feature Z • Deals are usually signed late • Competing priorities • Testing schedule compressed - or eliminated
Degradation Planning • Decide before the event - you only
get one chance • What features are most “important”? • Most impactful • Playbook: if/when X, then Y • Feature flags • “Benevolent dictator”
Dependencies • Know your dependencies • Direct • Indirect •
Third party • Find the bottlenecks • Include these services in your planning
Event Operations • Physical and virtual “war rooms” • Business
liaison • Do a post-mortem, good or bad • Archive metrics
Examples/Experiences • Names have been changed to protect the innocent
and the guilty…
Easy One • Sports site using PHP-platform • Once a
year event • Poor cache-invalidation methods • increase TTL before event, etc
“Waiting Room” • “Legacy” video delivery services • Capacity reservations
• Do not return an error to user • “Distract” the user
“No one uses mobile…” • Presidential election • Growth of
mobile web usage • Third party failure
Unexpected ending • Last episode of popular TV show •
Fragile stack • “Controversial” ending to show • Two orders of magnitude more traffic on site than expected
None