Slide 1

Slide 1 text

@tisba Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba stormforger.com, @StormForgerApp June 2019 at $MEETUP " "

Slide 2

Slide 2 text

@tisba stormforger.com - stormforger.com - stormforger.com stormforger.com - stormforger.com - stormforger.com

Slide 3

Slide 3 text

@tisba EHLO $MEETUP • Sebastian Cohnen (@tisba) • 9+ years consulting & development • focus on performance and architecture • founder & CTO StormForger.com

Slide 4

Slide 4 text

@tisba

Slide 5

Slide 5 text

@tisba I !…

Slide 6

Slide 6 text

Performance Testing

Slide 7

Slide 7 text

@tisba … from 10 Users to …

Slide 8

Slide 8 text

@tisba Load Testing with 1,000,000 Users Sebastian Cohnen, @tisba stormforger.com, @StormForgerApp June 2019 at $MEETUP " "

Slide 9

Slide 9 text

@tisba The year is 2014… • Still no global IPv6 rollout • But we finally have .technology, .domains, .xyz and .guru TLDs
 • TV Shows are getting interactive

Slide 10

Slide 10 text

@tisba Quizduell • alias "QuizClash", "QuizReto", … • Mobile Quiz Game/App • >30M players worldwide • >14M in Germany

Slide 11

Slide 11 text

@tisba Let’s make an interactive
 TV show out of it!

Slide 12

Slide 12 text

@tisba "Quizduell im Ersten" VS

Slide 13

Slide 13 text

@tisba • In-App Web View using AngularJS • HTTP & JSON API written in Golang • Hosted on Google App Engine Behind the Scenes

Slide 14

Slide 14 text

@tisba • May, 12th 2014 • ~1.6M viewers • 200,000 pre-registered players Show Premiere

Slide 15

Slide 15 text

@tisba

Slide 16

Slide 16 text

@tisba

Slide 17

Slide 17 text

@tisba # NIEEEEMALS…

Slide 18

Slide 18 text

@tisba Hacker Cyber Attack!

Slide 19

Slide 19 text

@tisba Not quite…

Slide 20

Slide 20 text

@tisba $ Nothing worked… % • The very first round of “Team Germany” failed • Service overwhelmed, slow,
 unresponsive… • Bad press; much speculation
 about hackers, [D]DoS etc.

Slide 21

Slide 21 text

@tisba Disaster Recovery

Slide 22

Slide 22 text

@tisba ☎

Slide 23

Slide 23 text

@tisba During the next days… • modeled game agents using StormForger to play the show • large scale load testing to provide insights • lots of configuration testing! • profiling, debugging, refactoring, …

Slide 24

Slide 24 text

@tisba Load Gen Setup • 50 Load Generators (AWS EC2 Ireland) • 800 cores, 1.5 TB RAM, lot’s of bandwidth • 3.3 TB data moved in over 2B requests • 1M Active Users, 330k rps peak Remember 2014?
 No eu-central-1 yet! #

Slide 25

Slide 25 text

@tisba The Challenge?

Slide 26

Slide 26 text

@tisba TV Synchronicity • This is what makes the show “interactive" • API polling every 1-10 sec • “server-side DDoS orchestration” (synchronized state polling & you have to answer questions within 15 sec)

Slide 27

Slide 27 text

@tisba Y U NO LOAD TEST BEFORE!?

Slide 28

Slide 28 text

@tisba pre launch load tests:
 up to 85k rps (~250k Users)

Slide 29

Slide 29 text

@tisba New load tests:
 Up to 330k rps, ~1M Users pre-launch load tests:
 Up to 85k rps (~250k Users)

Slide 30

Slide 30 text

@tisba * even Google Remember… ⚠ ⚠ Call* before load testing with 1,000,000 Users!

Slide 31

Slide 31 text

@tisba May 21st 2014: First try with App…

Slide 32

Slide 32 text

@tisba

Slide 33

Slide 33 text

@tisba Issues • Google DoS Protection • Understand Google App Engine’s tuning & scaling knobs • Runtime environment on App Engine is not transparent

Slide 34

Slide 34 text

@tisba The Actual Problem • Customer insisted on last minute changes to the backend, mostly real-time statistic related • no time to load test again prior to show premiere #

Slide 35

Slide 35 text

@tisba Perf Test early, Perf Test often! Can you do
 continuous load testing?

Slide 36

Slide 36 text

@tisba

Slide 37

Slide 37 text

Sebastian Cohnen
 [email protected] @tisba