Load Testing with 1M Users

I gave this talk at several occasions (meet ups, conferences, …). There are some blog posts on that topic as well over at https://stormforger.com/blog/2014/05/27/load-testing-an-interactive-tv-show-with-over-1-million-users. Checkout https://stormforger.com if you are interested in doing load testing yourself.

Abstract:

This is a battle story talk about how we helped to debug and load test the interactive German TV quiz show on ARD back in 2014. The show failed during the first 5 minutes on the first episode. The talk will outline some background info, architecture and what happened, why it failed and what we did in order to help fixing it in one week. In the end the broadcaster (ARD) demanded external load tests to verify that the optimized/fixed system could sustain 1M concurrently active players…

Transcript

1. @tisba Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba stormforger.com, @StormForgerApp June 2019 at $MEETUP " "
2. @tisba stormforger.com - stormforger.com - stormforger.com stormforger.com - stormforger.com - stormforger.com
3. @tisba EHLO $MEETUP • Sebastian Cohnen (@tisba) • 9+ years consulting & development • focus on performance and architecture • founder & CTO StormForger.com
4. @tisba
5. @tisba I !…
6. Performance Testing
7. @tisba … from 10 Users to …
8. @tisba Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba stormforger.com, @StormForgerApp June 2019 at $MEETUP " "
9. @tisba The year is 2014… • Still no global IPv6 rollout • But we finally have .technology, .domains, .xyz and .guru TLDs
 • TV Shows are getting interactive
10. @tisba Quizduell • alias "QuizClash", "QuizReto", … • Mobile Quiz Game/App • >30M players worldwide • >14M in Germany
11. @tisba Let’s make an interactive
 TV show out of it!
12. @tisba "Quizduell im Ersten" VS
13. @tisba • In-App Web View using AngularJS • HTTP & JSON API written in Golang • Hosted on Google App Engine Behind the Scenes
14. @tisba • May, 12th 2014 • ~1.6M viewers • 200,000 pre-registered players Show Premiere
15. @tisba
16. @tisba
17. @tisba # NIEEEEMALS…
18. @tisba Hacker Cyber Attack!
19. @tisba Not quite…
20. @tisba $ Nothing worked… % • The very first round of “Team Germany” failed • Service overwhelmed, slow,
 unresponsive… • Bad press; much speculation
 about hackers, [D]DoS etc.
21. @tisba Disaster Recovery
22. @tisba ☎
23. @tisba During the next days… • modeled game agents using StormForger to play the show • large scale load testing to provide insights • lots of configuration testing! • profiling, debugging, refactoring, …
24. @tisba Load Gen Setup • 50 Load Generators (AWS EC2 Ireland) • 800 cores, 1.5 TB RAM, lot’s of bandwidth • 3.3 TB data moved in over 2B requests • 1M Active Users, 330k rps peak Remember 2014?
 No eu-central-1 yet! #
25. @tisba The Challenge?
26. @tisba TV Synchronicity • This is what makes the show “interactive" • API polling every 1-10 sec • “server-side DDoS orchestration” (synchronized state polling & you have to answer questions within 15 sec)
27. @tisba Y U NO LOAD TEST BEFORE!?
28. @tisba pre launch load tests:
 up to 85k rps (~250k Users)
29. @tisba New load tests:
 Up to 330k rps, ~1M Users pre-launch load tests:
 Up to 85k rps (~250k Users)
30. @tisba * even Google Remember… ⚠ ⚠ Call* before load testing with 1,000,000 Users!
31. @tisba May 21st 2014: First try with App…
32. @tisba
33. @tisba Issues • Google DoS Protection • Understand Google App Engine’s tuning & scaling knobs • Runtime environment on App Engine is not transparent
34. @tisba The Actual Problem • Customer insisted on last minute changes to the backend, mostly real-time statistic related • no time to load test again prior to show premiere #
35. @tisba Perf Test early, Perf Test often! Can you do
 continuous load testing?
36. @tisba
37. Sebastian Cohnen
 support@stormforger.com @tisba