Game Days - with Amazon Web Services

AWS offers a variety of tools enabling users to create highly scalable, durable and resilient architectures and the user community has developed a broad range of best practices and frameworks to build rock-solid systems on top of AWS.

Many customers such as Netflix, Airbnb, Smugmug and others, already demonstrated those practices in production systems. While the theoretical concepts of building resilient architectures are well established, the practices of maintaining such systems are less understood, mostly because of the unpredictability of production environments under stress conditions.

To address this issue, some adopted the concept of Game Days, which consists of simulating unexpected failures to test the resilience, detect and fix flaws, and more importantly train the operation teams on emergency situations. This session covers the best practices learned from many AWS customers who implemented the Game Days practice and the different failure simulation techniques that can be used on AWS.

Transcript

1. Carlos Conde GAME DAYS CRASH-TEST YOUR APPLICATION … AND YOUR TEAM Chief EMEA Technical Evangelism – Amazon Web Services @caarlco
2. PLAY IN A SIMULATED ENVIRONMENT
6. SIMULATION ENVIRONMENT
7. SIMULATION ENVIRONMENT FOR CRISIS SITUATIONS
8. GOOD WEATHER DOESN’T MAKE GOOD SAILORS
9. PREPARE FOR EXTREME SITUATIONS
10. THINGS ARE UNRELIABLE UNTIL THEY ARE TESTED
11. HOW TO PLAY ? GAME PREPARATION
12. SUPERVISORS & PLAYERS
13. MAKE A PLAN ROLE PLAY & SCOPE DEFINITION
14. SET A DATE (DEADLINE) MAKE IT FUN
15. CREATE THE SIMULATION ENVIRONMENT
16. GAMEDAY ENVIRONMENT AWS ACCOUNT A PRODUCTION ENVIRONMENT AWS ACCOUNT B
17. CLOUDFORMATION TEMPLATE
18. USE AWS IAM CONTROL WHO CAN DO WHAT DURING THE GAME
19. ACTIVATE AWS CLOUDTRAIL GAMEPLAY RECORDING & AUDITING
21. SIMULATE ACTIVITY
22. PRE-RECORDED NETWORK ACTIVITY
23. (THROTTLED) LOAD TEST USE SPOT INSTANCES + TOOLS
25. USING AMAZON EC2 TO SIMULATE 2.4 MILLION PLAYERS
26. COMMUNICATION CHANNELS IRC, SHARED DOCS, VIBER, …
27. PLAY ! THE FUN PART
28. SIMULATE FAILURES
30. TERMINATE RESOURCES CHANGE SECURITY GROUPS CHANGE IAM ROLES DISABLE IAM USER CHANGE /ETC/HOSTS FILE AMAZON RDS FAIL-OVER TEST
32. RECORD & ANNOTATE PLAYERS ACTIONS
33. IF YOU DON’T MEASURE IT, IT DIDN’T HAPPEN
34. AFTER PARTY CLOSING THE GAME
35. DEBRIEF WHERE THE MAGIC HAPPENS
36. HOW LONG DOES IT TAKE TO DETECT AN EVENT ?
37. NEW HIRE ? GO TO THE SIMULATOR
38. KEY TEAM PLAYERS ? SIMULATE “FAILURE” SITUATION
39. VALIDATE ASSUMPTIONS PROVE YOUR ARCHITECTURE KNOWLEDGE OF PROCEDURES
40. COMMUNICATION CHANEL & CHAIN OF COMMAND
41. GOOD WEATHER DOESN’T MAKE GOOD SAILORS
42. TELL ME ABOUT YOUR GAMES!
43. Thank You! Carlos Conde Chief EMEA Technical Evangelism @caarlco