Building Game AI for Better User Experiences

Building Game AI for Better User Experiences

This deck was presented at Google Cloud Next ’19 in SF.
Video available:

In DeNA, AI is a big bet for next-generation mobile games. This session introduces our AI applications made for “Gyakuten Othellonia” and how they were built with GCP. “Gyakuten Othellonia” is a popular mobile game in Japan, but it is challenging for beginners to win a match because of its complicated rules and a huge number of card combinations. To help such beginners learn how to play competitively, and most importantly, enjoy the game, we started to provide AI applications that 1) support deck building, and 2) control the behavior of the bot that archives human-like gameplay. In this session, we first share our experiences on how we designed, trained, and optimized our models for our applications. Secondly, we talk about how we built a scalable and robust backend system with Cloud Machine Learning Engine. In our case, high scalability (10x traffic in a peak), low latency, and zero errors were key requirements to serve large in-game events without debasing the quality. Finally, we provide some KPIs to show how our AI applications achieved better user experiences for beginners.



April 09, 2019


  1. 1.

    MLAI300: Building Game AI for Better User Experiences Jun Okumura,

    ML Engineer, DeNA Ikki Tanaka, Data Scientist, DeNA Henry Tappen, Product Manager, Google Cloud
  2. 2.

    Gaming AI Team Jun Ernesto Okumura AI System Dept., AI

    Team Lead / ML Engineer Speaker Ikki Tanaka AI System Dept., Data Scientist / ML Engineer Speaker Takeshi Okada AI System Dept., ML Engineer Yu Kono AI System Dept., AI Researcher
  3. 3.

    Gaming in DeNA Gyakuten Othellonia Inhouse Nintendo Alliance Titles •

    Super Mario Run • Fire Emblem Heroes • Animal Crossing: Pocket Camp Megido 72 Alliance • Final Fantasy Record Keeper • Uta Macross 3rd Party • Granblue Fantasy
  4. 5.

    Gyakuten Othellonia • Strategic app game ◦ Based on Board

    Game (Othello / Reversi) ◦ Variety of Characters / Skills • Released in 2016 ◦ 24M downloads so far ◦ Region: Japan / Taiwan
  5. 6.

    Deck and Character • Deck: 16 characters (among more than

    3,000 options) • Character: Each character has its own skills, and status Player Deck Character Detail Status: HP (hit-point) ATK (attack-point) Skill description: When this character can flip just one opponent piece, the attack point will double
  6. 7.

    Issues to be solved • High churn-rate • Hard to

    construct optimal decks / No training field • Need better on-boarding support for beginners
  7. 8.

    Our approach • Support beginners by two AI functions Deck

    Recommendation select appropriate characters for deck (Association Analysis) Othellonia Dojo (battle AI) support human-level AI for practicing (Supervised Learning w/ Deep Neural Network)
  8. 9.

    Available Data Most recent 9 months, ~billion log entries •

    Extract high-skilled player’s deck/battle logs • Data Augmentation (battle AI) ◦ Board rotation, hand piece permutation, … battle record BigQuery
  9. 10.

    Deck Recommendation • Released on 6th Nov. 2018 • A

    player select 1. deck archetype, 2. deck cost and 3. favorite character • Metrics: Usage, Acceptance rate, Win rate, ... Build my deck! Association Rules Recommendation API Server Deck logs
  10. 11.

    Othellonia Dojo (AI-bot) • Feature released last week • A

    player select 1. deck archetype and 2. AI level • Metrics: Usage, Win rate, ... Battle record Deep Neural Network Trained AI model Let’s battle! Battle API Server game status Inference result
  11. 13.
  12. 14.

    How are the relationships extracted? Association analysis is the statistical

    method to extract the relationships inside large-scale data GOOD RELATION! BAD RELATION… GOOD RELATION!
  13. 15.

    Reasons for choosing Google Cloud Platform Powerful and flexible tools

    for ML and Data Science Scalable and robust infrastructure services App Engine Cloud Functions Cloud Pub/Sub Cloud Datastore Stackdriver Logging Compute Engine Kubernetes Engine BigQuery AI Platform Notebooks Cloud Storage AI Platform
  14. 16.

    System Architecture with GCP batch Deck Logs BigQuery Association Analysis

    Compute Engine Cloud Storage Association Rules Master Data of Game (sqlite) batch Cloud Pub/Sub Cloud Storage Cloud Datastore App Engine Cloud Functions Update Time Association Rules Master Data of Game (pickle) (cron) Recommendation API App Engine User ID Characters Obtained Options of Recommendation Recommended Deck
  15. 18.

    1 6 2 4 9 2 2 5 3 1

    8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep learning makes AI like a human player 1 6 2 4 9 2 2 5 3 1 8 3 0 1 5 0 9 2 8 7 2 7 3 4 1 4 3 2 3 0 3 5 1 9 8 3 Deep Neural Network Action Probability Battle logs of top-tier players Update Network >5000 features >10m battles
  16. 19.

    Training phase AI models trained in the Google Compute Engine

    are stored in the Cloud Storage training PvP Match Training AI model BigQuery Preprocessing Compute Engine Training Compute Engine Cloud Storage Cloud Storage Battle Record Battle Record AI model AI model
  17. 20.

    Inference phase Deploy the trained AI model stored in the

    Cloud Storage to AI Platform Predictor serving Inference API App Engine Scores of Actions AI PvE Match GAE/Python3.7 Auto-scaling AI Platform Auto-scaling Cloud Storage Inference by AI Ai Platform Features Scores of Actions Battle Record AI model
  18. 21.

    Estimated RPS (requests per second) RPS Time Testing of the

    systems High and spiky traffic at the moment that a new event opens
  19. 22.

    How to remove this errors? Load and scalability testing Auto-scaling

    of AI Platform is very helpful for our case ➔ Scale quickly ➔ Average latency is less than 1 second Requests 1:30 PM 2:00 PM 2:39 PM Errors 1:30 PM 2:00 PM 2:39 PM
  20. 23.

    Our approach to eliminate the errors We set another backup-endpoint

    to receive the request that fails to be operated in the main-endpoint Inference API App Engine Inference by AI AI Platform Prediction request Successful response Prediction request Inference by AI AI Platform main backup 5 sec limit something bad...
  21. 24.

    No errors at all! Total Error Rates 0.00% 0.50% 1.00%

    1.50% No Backup Backup (minNode=10) Backup (minNode=20) Result of testings
  22. 25.

    Monitoring Stackdriver can check the performances: • Requests • Latency

    (percentile) • CPU/Memory usages • Instances • ...
  23. 27.

    Good adoption from beginners Deck Recommendation Time # of uses

    (beginner class) Event to promote deck building Release
  24. 28.

    • Good adoption from beginners • Win rate of beginners

    increased by 5% Deck Recommendation Time win rate at battle (beginner class) Release
  25. 29.

    Latency • The algorithm is complex, but the latency is

    low • Thanks to GCP, especially auto-scaling, it is running error-free Sat 23 Latency 24ms 22ms 20ms 18ms 16ms Stable latency with most requests Sun 24 Mon 25 Tue 26 Wed 27 Thu 28 Mar 1 Sat 2 Sun 3 Mon 4 Tue 5 Wed 6 Fri 22
  26. 30.

    Othellonia Dojo (AI-bot) Preliminary Results • Good enough win rate

    as a practice partner • Dojo is used for beginners well • Latency is stable and fast Player Class (play skill) Win Rates of AI (w/ different levels) Beginner Expert AI is strong enough! It can beat even most skilled players. Beginners can choose various levels of AI as a practice partner
  27. 31.

    Best practices • Record logs that completely reproduce battles •

    Creating headless game simulator • Scalable system components (AI Platform, GAE) • CI/CD • Project structure & communication
  28. 33.

    DeNA elegantly used many services to solve an important user

    problem. Challenge to GCP team: How can we make it even easier for DeNA and others to accomplish similar use cases?
  29. 34.

    Custom containers Run any Docker container in the serverless training

    environment. Package your own dependencies or load any framework. Greater scale on inference Choose the CPU, RAM, and GPUs backing the inference servers Python code in inference Execute Python code before & after inference requests. Modify inputs before sending them to the framework + model Inspiration: fewers servers for DeNA to maintain Inspiration: managed association rule training New features to simplify launches Inspiration: better latency + reliability of DeNA’s AI
  30. 35.

    • Scalable, manageable, and flexible training • Low-latency model serving

    AI can create compelling gaming experiences • Simplify complex decisions for beginners • Enable single-player & practice Result: better engagement, especially from new players Google Cloud is here to help
  31. 36.

    Your Feedback is Greatly Appreciated! Complete the session survey in

    mobile app 1-5 star rating system Open field for comments Rate icon in status bar
  32. 37.
  33. 38.

    how to make an AI “like human” • We have

    developed internal tool to visualize scores of AI inference damage history action log AI’s score for each move
  34. 39.

    how to make an AI “like human” • Visualization to

    check whether the play of AI resembles to that of human AI’s board heatmap Human player’s board heatmap Turn distribution of AI / Human