Halo 4: High Demand with Low Latency and High Availability

Halo 4 is a first-person shooter on the Xbox 360, with fast-paced, competitive gameplay. To complement the code on disc, a set of services were developed and deployed in Azure to store player statistics, display player presence information, deliver daily challenges, modify playlists, catch cheaters and more. These services need to support high demand, with low latency and high availability.
Prior Halo games had services built on physical machines and utilized a SQL database for storage For Halo 4, 343 Industries built and deployed a new set of services built on Azure, and Project Orleans, Distributed Virtual Actors for Programmability and Scalability, to solve our scale and performance problems. In this session, we will discuss some of the challenges faced while building these services, and how Azure and Project Orleans were utilized.

Transcript

1. Halo 4: High Demand with Low Latency & High Availability ▪SurgeConf 2014
2. Caitie McCaffrey Distributed Systems Engineer @CaitieM20 CaitieM.com
5. Halo:CE - 6.43 million Halo 2 - 8.49 million Halo 3 - 11.87 million Halo 3: ODST - 6.22 million Halo Reach - 9.52 million
6. Presence Statistics Title Files Cheat Detection User Generated Content
8. $220 million in sales ! 1 million players online Day One
9. $300 million in sales ! 4 million players online ! 31.4 million hours Week One
10. 11.6 million players ! 1.5 billion games ! 270 million hours Overall
11. Architectural Challenges
12. Load Patterns
13. Cloud Azure Worker Roles Azure Table Azure Blob Azure Service Bus
14. Latency & Concurrency
15. Stateless 3 Tier Architecture
16. Latency Issues
17. Add A Cache Add A Cache
18. Concurrency Issues Concurrency Issues
19. Data Locality 19
23. Actor Model A framework & basis for reasoning about concurrency 23
24. Orleans “Orleans is a runtime and programming model for building distributed systems, based on the actor model” eXtreme Computing Group at Microsoft Research
25. Orleans Programming Model
26. • .NET Framework • Actor Interfaces • Promises • Actor References • Turns • Persistence
28. • .NET Framework • Actor Interfaces • Promises • Actor References • Turns • Persistence
30. • .NET Framework • Actor Interfaces • Promises • Actor References • Turns • Persistence
32. Orleans is a Distributed Actor Runtime
33. • Silos Provide ! Messaging ! Hosting ! Execution
34. Virtual Actors “An Orleans actor always exists, virtually. It cannot be explicitly created or destroyed.”
35. V Virtual Actors Perpetual Existence Automatic Instantiation Location Transparency Automatic Scale Out
37. Halo 4 Statistics Service 37
39. Player Grain
40. Game Grain
43. Performance & Scalability 43
44. “Orleans applications run at very high CPU Utilization. We have run load tests with full saturation … at 90%+ CPU utilization without any instability.”
45. Performance & Scalability
46. Performance & Scalability
47. Failure Domains 47
48. At Least Once Messaging
49. Machine Failures
50. TOO!
52. TOO!
53. TOO!
54. Orleans is AP
55. preview: http://aka.ms/orleans 
 samples: https://orleans.codeplex.com/
 
 
 Get Orleans
56. OR Questions?