Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scala and Play for game servers

h_kishi
February 26, 2017

Scala and Play for game servers

2017 ScalaMatsuri

h_kishi

February 26, 2017
Tweet

Transcript

  1. Scala and Play
    for game servers
    @h_kishi
    Hiromichi Kishi
    2017 ScalaMatsuri
    ήʔϜαʔόͷͨΊͷScala/Play

    View full-size slide

  2. Agenda
    1.  Introduction
    2.  Why Scala and Play?
    3.  System architecture
    4.  Details of implementation
    5.  Performance tuning
    ຊ೔ͷΞδΣϯμ

    View full-size slide

  3. Introduction
    ·ͣॳΊʹࣗݾ঺հΛ͍͖ͤͯͨͩ͞·͢ɻ

    View full-size slide

  4. About me
    •  Hiromichi Kishi (@h_kishi)
    •  Technical advisor in AXEL MARK INC.
    •  Game developer and senior server-side engineer
    ΞΫηϧϚʔΫגࣜձࣾͰٕज़ސ໰Λ͍ͯ͠·͢ɻ
    ήʔϜ։ൃऀͰɺγχΞαʔόαΠυΤϯδχΞͰ͢ɻ

    View full-size slide

  5. About AXEL MARK
    •  Mobile game business
    •  Mobile advertising business
    ΞΫηϧϚʔΫͰ͸ϞόΠϧήʔϜࣄۀͱ޿ࠂࣄۀ͕
    ओͳࣄۀྖҬͰ͢ɻ
    © AXEL Game Entertainment © ݪହٱɾूӳࣾʗNHKɾNEPɾͽ͑Ζ
    © DeNA Co., Ltd. All rights reserved. /
    ɹDeveloped by AXEL GameStudio Inc.

    View full-size slide

  6. Why Scala and Play?
    ͲͷΑ͏ͳཧ༝͔ΒฐࣾͰScala/PlayΛ
    ࠾༻͔ͨ͠ʹ͍ͭͯͰ͢ɻ

    View full-size slide

  7. Adoption of Scala and Play
    •  “World Cross Saga” which was adopted Scala and Play was released
    on April 2016
    •  We are using them in new game titles which are under development
    too
    ڈ೥ͷࣗࣾλΠτϧ͔ΒScala/PlayΛ࠾༻͓ͯ͠Γɺ
    ݱࡏ։ൃதͷλΠτϧͰ΋ར༻͍ͯ͠·͢ɻ
    © AXEL Game Entertainment

    View full-size slide

  8. Discontent of the existing way
    •  We were using PHP and in-house framework
    •  We run our games continuously in several years and the code
    quantity was increasing…
    •  Motivation for new titles:
    1.  Safe and easy modification for existing source code
    2.  Prevent runtime errors in production environment caused by rare case of
    exceptions
    େن໛ͳϞόΠϧήʔϜΛ௕ظӡ༻͢ΔͷʹɺLLͩͱ
    ܧଓతͳ։ൃʹݱ৔Ͱ͍͔ͭ͘໰୊Λײ͍ͯ͡·ͨ͠ɻ

    View full-size slide

  9. Deciding factors of Scala
    •  Static typing
    •  Easy to modify existing codes
    •  Tools to describe briefly
    •  (e.g. Type interface, pattern match implicit and etc..)
    •  Simplicity leads reducing bugs
    •  Null-safe
    •  Hope for better Java and functional language
    •  Enable to use Java assets
    ScalaΛબΜͩͷ͸ɺݴޠͷಛੑ͕զʑͷཁ݅ʹ
    ͍͋ͬͯͨͨΊͰ͢ɻ

    View full-size slide

  10. Deciding factors of Play
    •  High performance
    •  It has been used successfully in many services
    •  There are many plugins in OSS
    •  We accumulated experiences using HTTP-based API
    Play͸౰࣌ͷScalaͷϑϨʔϜϫʔΫͷதͰ΋
    طʹଟ͘ͷ࠾༻ࣄྫ͕͋ͬͨ͜ͱ͔Βબͼ·ͨ͠ɻ

    View full-size slide

  11. System architecture
    γεςϜߏ੒ʹ͍ͭͯ࿩͠·͢ɻ

    View full-size slide

  12. Component technologies
    •  Client:
    •  C# (Unity 5.x)
    •  Server (AWS):
    •  Scala(2.11.x)
    •  Play(2.4.x/2.5.x)
    •  MySQL (Aurora)
    •  Redis (ElastiCache)
    •  Google BigQuery (Store for action logs)
    •  Real-time server:
    •  Node.js/Socket.IO
    զʑͷήʔϜ։ൃͰ͸ओʹΫϥΠΞϯταΠυ͸C#ɺ
    αʔόαΠυ͸Scala/Play+MySQLΛ࢖͍ͬͯ·͢ɻ

    View full-size slide

  13. Basic server-side implementation
    •  Traditional MVC(+S) pattern
    •  Framework: Play Framework
    •  Library: Scalaz (Only using the alternative of toRight)
    •  Database Libraries:
    •  Scalikejdbc + Skinny ORM
    •  Using play-evolution for database migration
    •  Coding convention:
    •  sbt-scalariform
    PlayΛ࢖ͬͨMVC(+S)ύλʔϯͰ࣮૷Λ͍ͯ͠·͢ɻ
    DBͰ͸ScalikejdbcͱSkinny ORMΛ࢖͍ͬͯ·͢ɻ

    View full-size slide

  14. Testing
    •  Testing library: Specs2
    •  Fixture tool: Skinny Factory Girl
    •  Coverage report: sbt-scoverage
    •  Using H2 database with MySQL mode for functional tests:
    val  db  =  inMemoryDatabase(name  =  "default",  options  =  Map(MODE  -­‐>  "default"))  
    //  Play2.4  
    FakeApplication(addtionalConfiguration  =  db)  
    //  Play2.5  
    new  GuiceApplicationBuilder().configure(db).build()  
    Specs2Λ࢖ͬͯςετΛॻ͍͍ͯ·͢ɻ
    ػೳςετͰ͸H2σʔλϕʔεΛར༻͍ͯ͠·͢ɻ

    View full-size slide

  15. Scale of the system (peek)
    •  The total number of users: More than 4.5 million
    •  Requests per second: Several thousands
    •  Average response time per second: 50ms
    •  Maximum time is requests to Apple API
    •  Queries per second: Several ten thousands
    •  Not using sharding
    •  Aurora is great
    •  Log records per day: Several ten million
    Ϣʔβ਺450ສɺ਺ઍrpsɺ਺ສqpsɺϩά਺ेສߦ/day
    (͋·Γ۩ମతͳ਺஋͸ग़ͤͳ͍ͷͰ΅͔ͯ͠·͢)

    View full-size slide

  16. Logging infrastructure
    •  Player’s action logs are stored on Google BigQuery
    •  Querying for the action logs (update, insert, and delete) is heavy operation that
    would affect the performance of the master database if performed directly.
    •  Important logs like payment transactions are just stored on MySQL
    •  These logs are used on the admin tool for customer supports and
    analytics of KPI
    ϓϨΠϠʔͷߦಈϩά͸BigQuery্ʹอଘ͞Εɺ
    ؅ཧπʔϧ͔ΒCS΍KPI෼ੳʹར༻͞Ε·͢ɻ
    Game servers Amazon Kinesis AWS Lambda

    View full-size slide

  17. Deployment
    •  AWS CodeDeploy
    1.  sbt dist
    2.  Upload to S3
    3.  CodeDeploy agents pull and update the new build
    •  Blue-Green Deployment
    •  No downtime
    •  Possible to rollback
    AWS CodeDeployͰσϓϩΠ͍ͯ͠·͢ɻ

    View full-size slide

  18. System architecture image
    γεςϜߏ੒ਤͰ͢ɻ
    Elastic Load
    Balancing
    Game servers
    Trinity
    (In-house billing
    system)
    Aurora
    (master)
    Aurora
    (slave)
    Admin tool
    BigQuery
    Amazon
    ElastiCache
    Read/Write
    Read/Write
    Read/Write
    Repplicate Read/Write
    Transfer
    logs
    Lobby server
    Room servers
    Game servers
    (API servers)
    Real-time servers
    Read
    WebSockets
    HTTP

    View full-size slide

  19. Details of implementation
    ͔͜͜Β͸࣮૷ͷৄࡉʹ͍ͭͯ࿩͠·͢ɻ

    View full-size slide

  20. •  Play provides points to add a common process:
    1.  Filters
    2.  BodyParsers
    3.  ActionFunctions
    •  Because it is difficult to compose ActionFunctions flexibly, we
    integrated stackable-controller instead of it.
    •  https://github.com/t2v/stackable-controller
    Extension points in Play
    Play͸ڞ௨ॲཧΛڬΉՕॴΛ͍͔ͭ͘ఏڙ͍ͯ͠·͢ɻ
    ͜ΕΒͷͲΕ͔Λ࢖ͬͯڞ௨ॲཧΛ௥Ճ͍ͯ͠·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action
    Execution order in Play

    View full-size slide

  21. Cache
    1.  Cache on Redis
    •  Response data to prevent multiple same requests
    (CacheElement stores responses and returns a response
    from the cache if the request is same one = idempotence)
    •  Temporary game data like game stage data
    2.  Cache on memories of each game servers
    •  Master data (game parameters) which expires when the version is updated
    •  Implemented with java.util.concurrent.ConcurrentHashMap
    (thread-safe)
    Ωϟογϡ͸RedisͱΦϯϝϞϦͷೋͭΛ࠾༻͍ͯ͠·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action

    View full-size slide

  22. •  Falsifications of request parameters and response bodies are
    popular way to cheat on the network
    •  SSL/TSL is not sufficient to prevent player "cheating" the game
    •  To avoid falsifications, we should implement signatures (e.g. SHA)
    and/or encryption (e.g. RSA and AES) by own
    Cryptography
    SSL/TSL͚ͩͰ͸νʔτରࡦʹͳΒͳ͍ͷͰɺ
    ࣗલͰ҉߸Խ΍ॺ໊ͷ࣮૷Λ͢Δඞཁ͕͋Γ·͢ɻ

    View full-size slide

  23. Decryption (requests)
    •  To use a form feature in Play, encrypted parameters are decrypted in
    the customized BodyParser
    •  (Because it is not easy to access request body in Filters)
    •  Then, the parameters are encoded by the default form encode
    PlayͷϑΥʔϜػೳΛ࢖͏ͨΊɺ
    ϦΫΤετύϥϝʔλͷ෮߸Խ͸BodyParserͰߦ͍·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action

    View full-size slide

  24. Encryption (responses)
    •  The response data are encrypted in EncryptFilter
    •  (Because Result.body is used sometimes in ActionFunctions)
    •  We should compress data before encrypting it because encrypted
    byte array cannot be compressed effectively
    Ϩεϙϯεͷ҉߸Խ͸EncrypFilterͰߦ͍ͬͯ·͢ɻ
    ҉߸ԽલʹѹॖΛ͢Δඞཁ͕͋Γ·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action

    View full-size slide

  25. Message formats
    •  In general, Web API returns a response data as JSON
    •  Problems:
    1.  When the large JSON is responded, the client may freeze for a moment
    and use increase memory usage to parse the JSON
    2.  Lose type-information by converting from a static typed language to a
    JSON format
    3.  Documentations and validations are required between client and server
    engineers
    •  In the case of mobile game developments, JSON is not a good
    choice
    ϞόΠϧήʔϜ։ൃʹ͓͍ͯ͸ɺJSON͸ඞͣ͠΋
    ྑ͍ϝοηʔδϑΥʔϚοτͷબ୒ࢶͰ͸͋Γ·ͤΜɻ

    View full-size slide

  26. Thrift as a message format
    •  RPC framework made by Facebook
    (Now, under Apache project)
    •  We use Thrift as not RPC framework but a message format
    •  Thrift features:
    •  Data size is small and serialization is faster than JSON
    •  Binary format and it does not have type information in the data (Adopts IDL)
    •  Supports many languages
    (C++, C#, Java, Python and Ruby etc..)
    զʑͷήʔϜͰ͸ThriftΛϝοηʔδϑΥʔϚοτͱͯ͠
    ࠾༻͠·ͨ͠ɻ

    View full-size slide

  27. Example to create class files
    ThriftͷIDLͷྫͰ͢ɻ
    struct  LoginResponse  {  
       1:  bool  success;  
       2:  i64  userId;  
    }  
    1. Define data structure with Thrift IDL
    Java C#
    $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift  
    $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift  
    2. Run following commands to generate files:
    3. Use each files on the server and the client
    LoginResponse.thrift
    LoginResponse.java LoginResponse.cs

    View full-size slide

  28. thrift
    repository
    client
    repository
    server
    repository
    Refers latest C# files
    Refers latest Scala files
    client engineers
    server engineers
    Update thrift files and
    generate C#/Java classes
    Flow to update thrift files and classes
    ThriftϑΝΠϧΛར༻͢ΔϑϩʔͰ͢ɻ
    Use
    Use
    Java C#

    View full-size slide

  29. Thrift serialization
    Java
    object
    C#
    object
    Binary  data  
    Server-side Client-side
    serialize deserialize
    Over HTTP

    View full-size slide

  30. Example to use between server and client
    ThriftͷγϦΞϥΠζΛߦ͏ྫͰ͢ɻ
    def  login  =  Action  {  
       val  loginResponse  =  new  LoginResponse    
       loginResponse.success  =  true  
       val  protocol  =  new  TCompactProtocol.Factory()  
       val  serializer  =  new  TSerializer(protocol)  
       Ok(serializer.serialize(loginResponse))  
    }  
    var  www  =  new  WWW("http://localhost/login");  
    yield  return  www;  
    var  stream  =  new  MemoryStream(www.bytes);  
    var  tProtocol  =  new  TCompactProtocol(new  TStreamTransport(stream,  stream));  
    var  loginResponse  =  new  LoginResponse();  
    loginResponse.Read(tProtocol);  
    2. Deserialize on the client-size (C#)
    1. Serialize on the server-side (Scala)

    View full-size slide

  31. Good points using Thrift
    1.  Good performance and saving memory
    2.  It is easy to share the specification for responses between the
    server-side and the client-side
    3.  By adding comments, IDL files work API documents
    4.  Replace magic numbers into Enum
    αʔόͱΫϥΠΞϯτͰڞ௨ͷσʔλߏ଄Λ࣋ͭ͜ͱ͕
    Ͱ͖ͨ͜ͱͰɺଟ͘ͷϝϦοτΛڗडͰ͖·ͨ͠ɻ

    View full-size slide

  32. FYI: twitter/scrooge
    •  https://github.com/twitter/scrooge
    •  Thrift generator for Scala made by Twitter
    •  Use Thrift in Scala without Java
    •  Compatible original thrift data
    •  We did not adopt scrooge because it was necessary to downgrade
    the version of Scala/sbt when we tried it
    •  It may be solved now
    ༨ஊͰ͕͢ɺScalaͰThriftΛѻ͏ʹ͸
    Twitter੡ͷscroogeͱ͍͏ϥΠϒϥϦ΋͋Γ·͢ɻ

    View full-size slide

  33. Admin tool
    •  Admin tool is implemented with Play Framework too
    •  The admin tool is separated from main code repository and refers it
    as a sub-project
    •  Additional libraries:
    •  Authorization: play2-auth
    •  Template: twirl
    •  Database: bigquery4s
    ؅ཧπʔϧ͸ϝΠϯͷϨϙδτϦ͔Β෼཭͞Ε͍ͯͯɺ
    αϒϞδϡʔϧͱͯ͠ࢀর͢Δܗʹͳ͍ͬͯ·͢ɻ

    View full-size slide

  34. Performance tuning
    ύϑΥʔϚϯενϡʔχϯάʹ͍ͭͯ

    View full-size slide

  35. Highly synchronous application
    •  Although Play is an asynchronous framework, JDBC does not
    support asynchronous (non-blocking IO)
    •  We describes our codes to handle database synchronously
    •  In the case of synchronous application, there are points to notice for
    the performance tuning
    JDBC͕ಉظAPI͔͠αϙʔτ͍ͯ͠ͳ͍ͷͰɺ
    զʑͷΞϓϦέʔγϣϯ͸ಉظతʹॻ͔Ε͍ͯ·͢ɻ

    View full-size slide

  36. Configurations for Akka actors
    •  In the case of traditional synchronous IO based application, it uses large
    thread pools of Play(Akka) to handle blocking IO.
    •  Play 2.4.x:
    •  Play 2.5.x:
    •  The numbers depends on the server CPU performance
    ಉظతͳIOϕʔεͷΞϓϦέʔγϣϯͰ͸
    େ༰ྔͷεϨουϓʔϧ͕ඞཁͱͳΓ·͢ɻ
    ɾakka.actor.default-dispatcher.parallelism-factor
    ɾakka.actor.default-dispatcher.parallelism-min
    ɾakka.actor.default-dispatcher.parallelism-max
    The number of threads: parallelism-min <= CPUs * parallelism-factor <= parallelism-max
    ɾakka.actor.default-dispatcher.thread-pool-executor.fixed-pool-size
    ɹThe fixed pool size should be the maximum size of database connection pool plus extra

    View full-size slide

  37. Configuration for Linux Kernel
    •  net.ip4.ip_local_port_range:
    •  Port range a process can use
    •  fs.file-max:
    •  The number of files a process can open
    (= The number of unix domain socket)
    େྔͷεϨουΛѻ͏৔߹ɺ
    Linux kernel΋มߋ͓͖ͯ͠·͠ΐ͏ɻ

    View full-size slide

  38. Other tunings on implementations
    •  Some N+1 problems are detected by New Relic
    •  The library to access external APIs was implemented as non-blocking
    •  Applying Future and create a specialized ThreadPool
    •  Replace codes using Reflections into Scala Macros
    •  We published it as OSS
    •  Macrooom: Macro-based Object-Object Mapper
    •  https://github.com/h-kishi/macrooom
    ͦͷଞͷ࣮૷ϨϕϧͰߦͬͨνϡʔχϯάʹ͍ͭͯͰ͢ɻ

    View full-size slide

  39. We are hiring!
    If you want to use Scala for game servers, please contact us!
    ฐࣾͰ͸ήʔϜαʔόΛScalaͰॻ͖͍ͨํΛ
    ืू͍ͯ͠·͢ʂ

    View full-size slide

  40. Thank you for your attention!
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ

    View full-size slide