Scaling and load testing Battlelog in Battlefield 4

Talk given at Uppsala Tech Meetup #2
http://www.meetup.com/Uppsala-Tech-Meetup/events/220347895/

Handling 6 million simultaneous users on launch day. Looking at the architecture used and how load testing was done to achieve this.

Transcript

1. SCALING AND LOAD TESTING HÅKAN ROSENHORN CARL BYSTRÖM
2. AGENDA → ABOUT BATTLELOG → PREPARING FOR BATTLEFIELD 4 LAUNCH → ARCHITECTURE → LOAD TESTING → MINIMIZING RISK WHEN LIVE → POST-LAUNCH NUMBERS → QUESTIONS?
4. PREPARING FOR BATTLEFIELD 4 LAUNCH
5. USERS OVER TIME PLAYERS LAUNCH
6. ESTIMATIONS FOR LAUNCH → 6 MILLION SIMULTANEOUS USERS → 50 000 HTTP REQUESTS/SEC → < 300 MS RESPONSE TIME → LESS THAN 0.1% REQUEST ERRORS WIKIPEDIA IS THE 6TH LARGEST SITE SERVES 100 000 REQ/S DURING PEAKS
7. ARCHITECTURE OVERVIEW
8. BATTLELOG SERVERS THREE BUILDING BLOCKS APP DATABASE CACHE
9. APP SERVERS → BUILT IN PYTHON → SINGLE THREADED NON BLOCKING IO → ONE OS PROCESS PER CPU CORE → IDENTICAL SERVERS → BEHIND A SINGLE LOAD BALANCER → EASY TO ADD OR REMOVE SERVERS
10. DATABASE SERVERS → NO JOINS → NO MASTER/SLAVE → HOT AND COLD DATA → APP-LEVEL SHARDING → SERVERS ON BARE METAL → IN-MEMORY KEY-VALUE STORAGE → PERSISTENT → ONLY HOT DATA → APP-LEVEL SHARDING → SERVERS ON BARE METAL + CASSANDRA FOR AN INTERNAL SERVICE
11. CACHE SERVERS → MEMCACHED → FINE-GRAINED AND CONDITIONAL CACHING → APP-LEVEL SHARDING → ~300K REQ/S AT FULL LOAD → SERVERS ON BARE METAL
12. LOAD TESTING
13. OUR RECIPE 1. USE IDENTICAL SETUP TO PRODUCTION 2. SIMULATE USER BEHAVIOR 3. START WITH THE “HAPPY CASE” 4. LOAD TESTING INSIDE DEV TEAM 5. REAL-TIME MONITORING OF SERVERS 6. ENSURE FAST TURNAROUND TIMES FOR CODE 7. MINIMIZE SUBSYSTEMS THAT NEED TESTING
14. 1 IDENTICAL SETUP TO PRODUCTION
15. 2 SIMULATE USER BEHAVIOR → DESCRIBE USER BEHAVIOR WITH CODE → DISTRIBUTED → CAN SIMULATE MILLIONS OF USERS → ADJUST LOAD RUN-TIME → BUILT IN PYTHON → WE RUN LOCUST ON AMAZON’S CLOUD, EC2 → ~40 SERVERS → PAY PER HOUR → ENABLES RAPID SCALE UP/DOWN OF TESTS LOCUST AN OPEN SOURCE LOAD TESTING TOOL FROM ESN
16. 3 START WITH ”HAPPY CASE” → BEGIN TESTING WITH OPTIMAL CONDITIONS → REMOVE EXTERNAL SUBSYSTEMS → DISABLE NON-VITAL INTERNAL SERVICES → EMPTY AND UNPOPULATED DATABASES → GRADUALLY ADD MORE SUBSYSTEMS → POPULATE DATA
17. 3 START WITH ”HAPPY CASE” NUCLEUS USERS, CREDENTIALS AND ENTITLEMENTS ORIGIN FRIENDS USER PRESENCE AND FRIEND RELATIONS BLAZE GAME SERVERS AND PLAYER STATS MOCKLEUS READ-ONLY MOCK SERVER AMIGO READ-ONLY MOCK SERVER BLAZE NOT YET
18. 5 REAL-TIME MONITORING → INSTRUMENT CODE WITH KEY METRICS → CODE PROFILER EMBEDDED → LIVE TRACING OF NETWORK CALLS → DATABASE, CACHES, INTERNAL + EA SERVICES → REAL-TIME SYSTEM METRICS → MEASURE CPU, MEMORY AND I/O
19. METRICS GRAPHS SHOWING CHANGE OVER TIME
20. LIVE CALL TRACING SHOW TIME SPENT IN NETWORK CALLS
21. SYSTEM METRICS IN REAL-TIME FOR EACH SERVER
22. 6 FAST CODE CHANGES → ITERATE AND USE SAME BRANCH AS REST OF DEV TEAM → ENSURE QUICK BUILDS IN BUILD SERVER → IN OUR CASE, LESS THAN 10 MINUTES → WE USE TORRENTS TO DEPLOY BUILDS TO +100 SERVERS → TAKES LESS THAN 2 MINUTES FOR 800 MB BUILDS
23. 7 MINIMIZE NEW TESTING → DON’T RE-TEST PERFORMING SERVICES → KNOW THEIR PERFORMANCE CHARACTERISTICS → KNOW HOW TO SCALE THEM OUT → SERVICES WE DIDN’T RE-TEST FOR BF4 → BEACONPUSH – USER PRESENCE → SERAPHINE – GAME SERVER INDEX → MYSQL, MEMCACHED AND SO ON
24. MINIMIZING RISK WHEN LIVE
25. 7 INDISPENSABLE TOOLS KILL SWITCHES ZERO DOWNTIME UPDATES MAINTENANCE CONSOLE METRICS INSTANT CONFIG CHANGES FEATURE ROLLOUT CIRCUIT BREAKERS
26. KILL SWITCHES → OVER 50 KILL SWITCHES → 5 EXAMPLES → BATTLELOG IN-GAME → FORUM → ACTIVITY FEED → BLAZE CALLS / EVENTS → ORIGIN FRIENDS CALLS ABILITY TO TURN OFF FEATURES INSTANTLY
27. FEATURE ROLLOUT DECOUPLE DEPLOYMENT FROM FEATURE RELEASE
28. CIRCUIT BREAKERS LET NETWORK CONNECTIONS FAIL FAST
29. METRICS MEASURE EVERYTHING
30. ZERO DOWNTIME UPDATES SILENT CHANGES WITHOUT INTERRUPTIONS
31. MAINTENANCE CONSOLE RUN AND CHANGE CODE AT RUNTIME ~> EXECUTE_CODE_ON_RUNNING_SERVERS()
32. INSTANT CONFIG CHANGES MODIFY SETTINGS WITHOUT DOWNTIME
33. POST-LAUNCH NUMBERS
34. PEAKED AT ~15 000 REQ/S EXCLUDING STATIC CONTENT 0 MINUTES OF DOWNTIME SINCE LAUNCH 450 COMMITS CHANGES DEPLOYED TO PRODUCTION FIRST 7 DAYS SPREAD OVER 7 ZERO DOWNTIME UPDATES
35. QUESTIONS? HÅKAN ROSENHORN CARL BYSTRÖM