Load Testing with over 1 million Users [WebPerfDays BCN]

This is the story on how I came to load test a HTTP-based API system with over 1 million concurrent users for an interactive German TV show.

I held this presentation on the WebPerfDays 2014 in Barcelona.

Transcript

1. Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba
 stormforger.com, @StormForgerApp
 WebPerfDays Barcelona 2014
2. The year is 2014… • Still no global IPv6 rollout • But we finally have .technology, .domains, .xyz and .guru TLDs
 • TV Shows are getting interactive
3. Quizduell • alias "QuizReto", "QuizClash", … • Mobile Quiz Game/App • >30M players worldwide • >14M in Germany
4. Let’s make an interactive TV show out of it!
5. "Quizduell im Ersten" VS
6. Behind the Scenes • In-App Web View using AngularJS • HTTP & JSON API written in Go • Hosted on Google App Engine
7. Show Premiere • May, 12th 2014 • ~1.6M viewers • 200,000 pre-registered players
9. Nothing worked… :-/ • The very first round of “Team Germany” failed • Bad press; much speculation
 about hackers, [D]DoS etc.
10. Disaster Recovery
11. I was called in
12. During the next days… • large scale load testing to provide insights • profiling, debugging, refactoring, …
13. The Challenge?
14. • This is what makes the show “interactive" • API polling every 1-10 sec • “server-side DDoS orchestration” (synchronized state polling & you have to answer questions within 15 sec) TV Synchronicity
15. Y U NO LOAD TEST BEFORE!?
16. pre launch load tests:
 up to 85k rps (~250k Users)
17. pre launch load tests:
 up to 85k rps (~250k Users) New load tests:
 Up to 330k rps (~1M Users)
18. Remember… Call* before load testing with 1M Users! *even Google
19. May 21st 2014:
 First try with App…
21. SUCCESS!
22. Issues • Google DoS Protection • Understand Google App Engine’s tuning & scaling knobs • Runtime environment on App Engine is not transparent
23. The Actual Problem • Customer insisted on last minute changes to the backend, mostly real-time statistic related • no time to load test again prior to show premiere
24. And the moral of this story… Do continuous load testing!* *e.g. with stormforger.com :)
25. Thanks! Sebastian Cohnen (@tisba), stormforger.com Article: http://bit.ly/Loadtest1MUsers
27. Test Setup • 50 Load Generators (AWS EC2 Ireland) • 800 cores, 1.5 TB RAM, lot’s of bandwidth • 3.3 TB data moved in over 2B requests • 1M User, 330k rps peak