Load Testing with 1,000,000 Users!

I gave this talk in March 2017 at the Web Engineering Düsseldorf Meetup (https://www.meetup.com/Web-Engineering-Duesseldorf/events/237783925/). Read more about the story at https://stormforger.com/blog/2014/05/27/load-testing-an-interactive-tv-show-with-over-1-million-users. Checkout https://stormforger.com!

Abstract:

This is a battle story talk about how we helped to debug and load test the interactive German TV quiz show on ARD back in 2014. The show failed during the first 5 minutes on the first episode. The talk will outline some background info, architecture and what happened, why it failed and what we did in order to help fixing it in one week. In the end the broadcaster (ARD) demanded external load tests to verify that the optimized/fixed system could sustain 1M concurrently active players…

Transcript

1. stormforger.com - stormforger.com - stormforger.com stormforger.com - stormforger.com - stormforger.com
2. stormforger.com - stormforger.com - stormforger.com stormforger.com - stormforger.com - stormforger.com
3. EHLO webengdus • Sebastian Cohnen (@tisba) • 7+ years consulting & development • focus on performance and architecture • founder & CTO StormForger.com
5. I …
6. Performance Testing
7. … from 10 Users to …
8. Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba stormforger.com, @StormForgerApp at March 2017
9. The year was 2014… • Still no global IPv6 rollout • But we finally have .technology, .domains, .xyz and .guru TLDs
 • TV Shows are getting interactive
10. Quizduell • alias "QuizClash", "QuizReto", … • Mobile Quiz Game/App • >30M players worldwide • >14M in Germany
11. Let’s make an interactive
 TV show out of it!
12. "Quizduell im Ersten" VS
13. • In-App Web View using AngularJS • HTTP & JSON API written in Go • Hosted on Google App Engine • Build by Behind the Scenes
14. • May, 12th 2014 • ~1.6M viewers • 200,000 pre-registered players Show Premiere
18. Hacker Cyber Attack!
19. Not quite…
20. Nothing worked… • The very first round of “Team Germany” failed • Service overwhelmed, slow,
 unresponsive… • Bad press; much speculation
 about hackers, [D]DoS etc.
21. Disaster Recovery
22. I was called
23. During the next days… • large scale load testing to provide insights • lots of configuration testing! • modeled game agents using StormForger to play the show • profiling, debugging, refactoring, …
24. Load Gen Setup • 50 Load Generators (AWS EC2 Ireland) • 800 cores, 1.5 TB RAM, lot’s of bandwidth • 3.3 TB data moved in over 2B requests • 1M Active Users, 330k rps peak
25. The Challenge?
26. TV Synchronicity • This is what makes the show “interactive" • API polling every 1-10 sec • “server-side DDoS orchestration” (synchronized state polling & you have to answer questions within 15 sec)
27. Y U NO LOAD TEST BEFORE!?
28. pre launch load tests:
 up to 85k rps (~250k Users)
29. New load tests:
 Up to 330k rps, ~1M Users pre-launch load tests:
 Up to 85k rps (~250k Users)
30. *even Google Remember… ⚠ ⚠ Call* before load testing with 1,000,000 Users!
31. May 21st 2014: First try with App…
33. Issues • Google DoS Protection • Understand Google App Engine’s tuning & scaling knobs • Runtime environment on App Engine is not transparent
34. The Actual Problem • Customer insisted on last minute changes to the backend, mostly real-time statistic related • no time to load test again prior to show premiere
35. Perf Test early, Perf Test often! Can you do
 continuous load testing?
37. Thanks! stormforger.com