Scaling For Planned Events

Theory, Practice, and Aftermath

1. Scaling For Planned Events Theory, Practice, and Aftermath
2. About @bakins • OmniTI - Senior Principal Architect • Horrible public speaker • Old school C guy • Operations and Development • Wife and four kids in Atlanta, Georgia
3. What is a “Planned Event” • Event on existing site • Single or repeating • Immovable dates • Elections, sporting events, etc.
4. Ideally • Event handled via normal site operations • Business goals clearly defined • Clearly defined schedule • Realistic traffic/capacity models
5. Reality • Competing business goals • Heroic scaling efforts often needed • Last minute feature additions • Real money tied to sponsorships • Can’t be tested in production until the event
6. Cache Everything? • If possible… • Introduce caching layer in proxy mode early • Is your site really “dynamic”? • Need buy-in from all parties: dev, biz, ops, etc
7. Feature Creep • Company X will give us $Y is we do feature Z • Deals are usually signed late • Competing priorities • Testing schedule compressed - or eliminated
8. Degradation Planning • Decide before the event - you only get one chance • What features are most “important”? • Most impactful • Playbook: if/when X, then Y • Feature flags • “Benevolent dictator”
9. Dependencies • Know your dependencies • Direct • Indirect • Third party • Find the bottlenecks • Include these services in your planning
10. Event Operations • Physical and virtual “war rooms” • Business liaison • Do a post-mortem, good or bad • Archive metrics
11. Examples/Experiences • Names have been changed to protect the innocent and the guilty…
12. Easy One • Sports site using PHP-platform • Once a year event • Poor cache-invalidation methods • increase TTL before event, etc
13. “Waiting Room” • “Legacy” video delivery services • Capacity reservations • Do not return an error to user • “Distract” the user
14. “No one uses mobile…” • Presidential election • Growth of mobile web usage • Third party failure
15. Unexpected ending • Last episode of popular TV show • Fragile stack • “Controversial” ending to show • Two orders of magnitude more traffic on site than expected