Building Game AI for Better User Experiences

This deck was presented at Google Cloud Next ’19 in SF.
Video available: https://youtu.be/T6cb7jN0SSs

In DeNA, AI is a big bet for next-generation mobile games. This session introduces our AI applications made for “Gyakuten Othellonia” and how they were built with GCP. “Gyakuten Othellonia” is a popular mobile game in Japan, but it is challenging for beginners to win a match because of its complicated rules and a huge number of card combinations. To help such beginners learn how to play competitively, and most importantly, enjoy the game, we started to provide AI applications that 1) support deck building, and 2) control the behavior of the bot that archives human-like gameplay. In this session, we first share our experiences on how we designed, trained, and optimized our models for our applications. Secondly, we talk about how we built a scalable and robust backend system with Cloud Machine Learning Engine. In our case, high scalability (10x traffic in a peak), low latency, and zero errors were key requirements to serve large in-game events without debasing the quality. Finally, we provide some KPIs to show how our AI applications achieved better user experiences for beginners.

Transcript

1. MLAI300: Building Game AI for Better User Experiences Jun Okumura, ML Engineer, DeNA Ikki Tanaka, Data Scientist, DeNA Henry Tappen, Product Manager, Google Cloud
2. Gaming AI Team Jun Ernesto Okumura AI System Dept., AI Team Lead / ML Engineer Speaker Ikki Tanaka AI System Dept., Data Scientist / ML Engineer Speaker Takeshi Okada AI System Dept., ML Engineer Yu Kono AI System Dept., AI Researcher
3. Gaming in DeNA Gyakuten Othellonia Inhouse Nintendo Alliance Titles • Super Mario Run • Fire Emblem Heroes • Animal Crossing: Pocket Camp Megido 72 Alliance • Final Fantasy Record Keeper • Uta Macross 3rd Party • Granblue Fantasy
4. Problem Setting and Approach
5. Gyakuten Othellonia • Strategic app game ◦ Based on Board Game (Othello / Reversi) ◦ Variety of Characters / Skills • Released in 2016 ◦ 24M downloads so far ◦ Region: Japan / Taiwan
6. Deck and Character • Deck: 16 characters (among more than 3,000 options) • Character: Each character has its own skills, and status Player Deck Character Detail Status: HP (hit-point) ATK (attack-point) Skill description: When this character can flip just one opponent piece, the attack point will double
7. Issues to be solved • High churn-rate • Hard to construct optimal decks / No training field • Need better on-boarding support for beginners
8. Our approach • Support beginners by two AI functions Deck Recommendation select appropriate characters for deck (Association Analysis) Othellonia Dojo (battle AI) support human-level AI for practicing (Supervised Learning w/ Deep Neural Network)
9. Available Data Most recent 9 months, ~billion log entries • Extract high-skilled player’s deck/battle logs • Data Augmentation (battle AI) ◦ Board rotation, hand piece permutation, … battle record BigQuery
10. Deck Recommendation • Released on 6th Nov. 2018 • A player select 1. deck archetype, 2. deck cost and 3. favorite character • Metrics: Usage, Acceptance rate, Win rate, ... Build my deck! Association Rules Recommendation API Server Deck logs
11. Othellonia Dojo (AI-bot) • Feature released last week • A player select 1. deck archetype and 2. AI level • Metrics: Usage, Win rate, ... Battle record Deep Neural Network Trained AI model Let’s battle! Battle API Server game status Inference result
12. Deck recommendation
13. User inputs Select leader Select base characters Select well-associated characters Recommended deck Algorithm of deck recommendation
14. How are the relationships extracted? Association analysis is the statistical method to extract the relationships inside large-scale data GOOD RELATION! BAD RELATION… GOOD RELATION!
15. Reasons for choosing Google Cloud Platform Powerful and flexible tools for ML and Data Science Scalable and robust infrastructure services App Engine Cloud Functions Cloud Pub/Sub Cloud Datastore Stackdriver Logging Compute Engine Kubernetes Engine BigQuery AI Platform Notebooks Cloud Storage AI Platform
16. System Architecture with GCP batch Deck Logs BigQuery Association Analysis Compute Engine Cloud Storage Association Rules Master Data of Game (sqlite) batch Cloud Pub/Sub Cloud Storage Cloud Datastore App Engine Cloud Functions Update Time Association Rules Master Data of Game (pickle) (cron) Recommendation API App Engine User ID Characters Obtained Options of Recommendation Recommended Deck
17. Deep-Learning AI bot (Othellonia Dojo)
18. 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep learning makes AI like a human player 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep Neural Network Action Probability Battle logs of top-tier players Update Network >5000 features >10m battles
19. Training phase AI models trained in the Google Compute Engine are stored in the Cloud Storage training PvP Match Training AI model BigQuery Preprocessing Compute Engine Training Compute Engine Cloud Storage Cloud Storage Battle Record Battle Record AI model AI model
20. Inference phase Deploy the trained AI model stored in the Cloud Storage to AI Platform Predictor serving Inference API App Engine Scores of Actions AI PvE Match GAE/Python3.7 Auto-scaling AI Platform Auto-scaling Cloud Storage Inference by AI Ai Platform Features Scores of Actions Battle Record AI model
21. Estimated RPS (requests per second) RPS Time Testing of the systems High and spiky traffic at the moment that a new event opens
22. How to remove this errors? Load and scalability testing Auto-scaling of AI Platform is very helpful for our case ➔ Scale quickly ➔ Average latency is less than 1 second Requests 1:30 PM 2:00 PM 2:39 PM Errors 1:30 PM 2:00 PM 2:39 PM
23. Our approach to eliminate the errors We set another backup-endpoint to receive the request that fails to be operated in the main-endpoint Inference API App Engine Inference by AI AI Platform Prediction request Successful response Prediction request Inference by AI AI Platform main backup 5 sec limit something bad...
24. No errors at all! Total Error Rates 0.00% 0.50% 1.00% 1.50% No Backup Backup (minNode=10) Backup (minNode=20) Result of testings
25. Monitoring Stackdriver can check the performances: • Requests • Latency (percentile) • CPU/Memory usages • Instances • ...
26. Results and Future
27. Good adoption from beginners Deck Recommendation Time # of uses (beginner class) Event to promote deck building Release
28. • Good adoption from beginners • Win rate of beginners increased by 5% Deck Recommendation Time win rate at battle (beginner class) Release
29. Latency • The algorithm is complex, but the latency is low • Thanks to GCP, especially auto-scaling, it is running error-free Sat 23 Latency 24ms 22ms 20ms 18ms 16ms Stable latency with most requests Sun 24 Mon 25 Tue 26 Wed 27 Thu 28 Mar 1 Sat 2 Sun 3 Mon 4 Tue 5 Wed 6 Fri 22
30. Othellonia Dojo (AI-bot) Preliminary Results • Good enough win rate as a practice partner • Dojo is used for beginners well • Latency is stable and fast Player Class (play skill) Win Rates of AI (w/ different levels) Beginner Expert AI is strong enough! It can beat even most skilled players. Beginners can choose various levels of AI as a practice partner
31. Best practices • Record logs that completely reproduce battles • Creating headless game simulator • Scalable system components (AI Platform, GAE) • CI/CD • Project structure & communication
32. Henry Tappen Product Manager, AI Platform
33. DeNA elegantly used many services to solve an important user problem. Challenge to GCP team: How can we make it even easier for DeNA and others to accomplish similar use cases?
34. Custom containers Run any Docker container in the serverless training environment. Package your own dependencies or load any framework. Greater scale on inference Choose the CPU, RAM, and GPUs backing the inference servers Python code in inference Execute Python code before & after inference requests. Modify inputs before sending them to the framework + model Inspiration: fewers servers for DeNA to maintain Inspiration: managed association rule training New features to simplify launches Inspiration: better latency + reliability of DeNA’s AI
35. • Scalable, manageable, and flexible training • Low-latency model serving AI can create compelling gaming experiences • Simplify complex decisions for beginners • Enable single-player & practice Result: better engagement, especially from new players Google Cloud is here to help
36. Your Feedback is Greatly Appreciated! Complete the session survey in mobile app 1-5 star rating system Open field for comments Rate icon in status bar
37. Appendix
38. how to make an AI “like human” • We have developed internal tool to visualize scores of AI inference damage history action log AI’s score for each move
39. how to make an AI “like human” • Visualization to check whether the play of AI resembles to that of human AI’s board heatmap Human player’s board heatmap Turn distribution of AI / Human