MLAI300: Building Game AI for Better User Experiences Jun Okumura, ML Engineer, DeNA Ikki Tanaka, Data Scientist, DeNA Henry Tappen, Product Manager, Google Cloud 

Gaming AI Team Jun Ernesto Okumura AI System Dept., AI Team Lead / ML Engineer Speaker Ikki Tanaka AI System Dept., Data Scientist / ML Engineer Speaker Takeshi Okada AI System Dept., ML Engineer Yu Kono AI System Dept., AI Researcher 

Gaming in DeNA Gyakuten Othellonia Inhouse Nintendo Alliance Titles ● Super Mario Run ● Fire Emblem Heroes ● Animal Crossing: Pocket Camp Megido 72 Alliance ● Final Fantasy Record Keeper ● Uta Macross 3rd Party ● Granblue Fantasy 

Problem Setting and Approach 

Gyakuten Othellonia ● Strategic app game ○ Based on Board Game (Othello / Reversi) ○ Variety of Characters / Skills ● Released in 2016 ○ 24M downloads so far ○ Region: Japan / Taiwan 

Deck and Character ● Deck: 16 characters (among more than 3,000 options) ● Character: Each character has its own skills, and status Player Deck Character Detail Status: HP (hit-point) ATK (attack-point) Skill description: When this character can flip just one opponent piece, the attack point will double 

Issues to be solved ● High churn-rate ● Hard to construct optimal decks / No training field ● Need better on-boarding support for beginners 

Our approach ● Support beginners by two AI functions Deck Recommendation select appropriate characters for deck (Association Analysis) Othellonia Dojo (battle AI) support human-level AI for practicing (Supervised Learning w/ Deep Neural Network) 

Available Data Most recent 9 months, ~billion log entries ● Extract high-skilled player’s deck/battle logs ● Data Augmentation (battle AI) ○ Board rotation, hand piece permutation, … battle record BigQuery 

Deck Recommendation ● Released on 6th Nov. 2018 ● A player select 1. deck archetype, 2. deck cost and 3. favorite character ● Metrics: Usage, Acceptance rate, Win rate, ... Build my deck! Association Rules Recommendation API Server Deck logs 

Othellonia Dojo (AI-bot) ● Feature released last week ● A player select 1. deck archetype and 2. AI level ● Metrics: Usage, Win rate, ... Battle record Deep Neural Network Trained AI model Let’s battle! Battle API Server game status Inference result 

Deck recommendation 

User inputs Select leader Select base characters Select well-associated characters Recommended deck Algorithm of deck recommendation 

How are the relationships extracted? Association analysis is the statistical method to extract the relationships inside large-scale data GOOD RELATION! BAD RELATION… GOOD RELATION! 

Reasons for choosing Google Cloud Platform Powerful and flexible tools for ML and Data Science Scalable and robust infrastructure services App Engine Cloud Functions Cloud Pub/Sub Cloud Datastore Stackdriver Logging Compute Engine Kubernetes Engine BigQuery AI Platform Notebooks Cloud Storage AI Platform 

System Architecture with GCP batch Deck Logs BigQuery Association Analysis Compute Engine Cloud Storage Association Rules Master Data of Game (sqlite) batch Cloud Pub/Sub Cloud Storage Cloud Datastore App Engine Cloud Functions Update Time Association Rules Master Data of Game (pickle) (cron) Recommendation API App Engine User ID Characters Obtained Options of Recommendation Recommended Deck 

Deep-Learning AI bot (Othellonia Dojo) 

1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep learning makes AI like a human player 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep Neural Network Action Probability Battle logs of top-tier players Update Network >5000 features >10m battles 

Training phase AI models trained in the Google Compute Engine are stored in the Cloud Storage training PvP Match Training AI model BigQuery Preprocessing Compute Engine Training Compute Engine Cloud Storage Cloud Storage Battle Record Battle Record AI model AI model 

Inference phase Deploy the trained AI model stored in the Cloud Storage to AI Platform Predictor serving Inference API App Engine Scores of Actions AI PvE Match GAE/Python3.7 Auto-scaling AI Platform Auto-scaling Cloud Storage Inference by AI Ai Platform Features Scores of Actions Battle Record AI model 

Estimated RPS (requests per second) RPS Time Testing of the systems High and spiky traffic at the moment that a new event opens 

How to remove this errors? Load and scalability testing Auto-scaling of AI Platform is very helpful for our case ➔ Scale quickly ➔ Average latency is less than 1 second Requests 1:30 PM 2:00 PM 2:39 PM Errors 1:30 PM 2:00 PM 2:39 PM 

Our approach to eliminate the errors We set another backup-endpoint to receive the request that fails to be operated in the main-endpoint Inference API App Engine Inference by AI AI Platform Prediction request Successful response Prediction request Inference by AI AI Platform main backup 5 sec limit something bad... 

No errors at all! Total Error Rates 0.00% 0.50% 1.00% 1.50% No Backup Backup (minNode=10) Backup (minNode=20) Result of testings 

Monitoring Stackdriver can check the performances: ● Requests ● Latency (percentile) ● CPU/Memory usages ● Instances ● ... 

Results and Future 

Good adoption from beginners Deck Recommendation Time # of uses (beginner class) Event to promote deck building Release 

● Good adoption from beginners ● Win rate of beginners increased by 5% Deck Recommendation Time win rate at battle (beginner class) Release 

Latency ● The algorithm is complex, but the latency is low ● Thanks to GCP, especially auto-scaling, it is running error-free Sat 23 Latency 24ms 22ms 20ms 18ms 16ms Stable latency with most requests Sun 24 Mon 25 Tue 26 Wed 27 Thu 28 Mar 1 Sat 2 Sun 3 Mon 4 Tue 5 Wed 6 Fri 22 

Othellonia Dojo (AI-bot) Preliminary Results ● Good enough win rate as a practice partner ● Dojo is used for beginners well ● Latency is stable and fast Player Class (play skill) Win Rates of AI (w/ different levels) Beginner Expert AI is strong enough! It can beat even most skilled players. Beginners can choose various levels of AI as a practice partner 

Best practices ● Record logs that completely reproduce battles ● Creating headless game simulator ● Scalable system components (AI Platform, GAE) ● CI/CD ● Project structure & communication 

Henry Tappen Product Manager, AI Platform 

DeNA elegantly used many services to solve an important user problem. Challenge to GCP team: How can we make it even easier for DeNA and others to accomplish similar use cases? 

Custom containers Run any Docker container in the serverless training environment. Package your own dependencies or load any framework. Greater scale on inference Choose the CPU, RAM, and GPUs backing the inference servers Python code in inference Execute Python code before & after inference requests. Modify inputs before sending them to the framework + model Inspiration: fewers servers for DeNA to maintain Inspiration: managed association rule training New features to simplify launches Inspiration: better latency + reliability of DeNA’s AI 

● Scalable, manageable, and flexible training ● Low-latency model serving AI can create compelling gaming experiences ● Simplify complex decisions for beginners ● Enable single-player & practice Result: better engagement, especially from new players Google Cloud is here to help 

Your Feedback is Greatly Appreciated! Complete the session survey in mobile app 1-5 star rating system Open field for comments Rate icon in status bar 

Appendix 

how to make an AI “like human” ● We have developed internal tool to visualize scores of AI inference damage history action log AI’s score for each move 

how to make an AI “like human” ● Visualization to check whether the play of AI resembles to that of human AI’s board heatmap Human player’s board heatmap Turn distribution of AI / Human