Scala and Play for game servers

2017 ScalaMatsuri

Transcript

1. Scala and Play
   for game servers
   @h_kishi
   Hiromichi Kishi
   2017 ScalaMatsuri
   ήʔϜαʔόͷͨΊͷScala/Play
   

   View Slide

2. Agenda
   1.  Introduction
   2.  Why Scala and Play?
   3.  System architecture
   4.  Details of implementation
   5.  Performance tuning
   ຊ೔ͷΞδΣϯμ
   

   View Slide

3. Introduction
   ·ͣॳΊʹࣗݾ঺հΛ͍͖ͤͯͨͩ͞·͢ɻ
   

   View Slide

4. About me
   •  Hiromichi Kishi (@h_kishi)
   •  Technical advisor in AXEL MARK INC.
   •  Game developer and senior server-side engineer
   ΞΫηϧϚʔΫגࣜձࣾͰٕज़ސ໰Λ͍ͯ͠·͢ɻ
   ήʔϜ։ൃऀͰɺγχΞαʔόαΠυΤϯδχΞͰ͢ɻ
   

   View Slide

5. About AXEL MARK
   •  Mobile game business
   •  Mobile advertising business
   ΞΫηϧϚʔΫͰ͸ϞόΠϧήʔϜࣄۀͱ޿ࠂࣄۀ͕
   ओͳࣄۀྖҬͰ͢ɻ
   © AXEL Game Entertainment © ݪହٱɾूӳࣾʗNHKɾNEPɾͽ͑Ζ
   © DeNA Co., Ltd. All rights reserved. /
   ɹDeveloped by AXEL GameStudio Inc.
   

   View Slide

6. Why Scala and Play?
   ͲͷΑ͏ͳཧ༝͔ΒฐࣾͰScala/PlayΛ
   ࠾༻͔ͨ͠ʹ͍ͭͯͰ͢ɻ
   

   View Slide

7. Adoption of Scala and Play
   •  “World Cross Saga” which was adopted Scala and Play was released
   on April 2016
   •  We are using them in new game titles which are under development
   too
   ڈ೥ͷࣗࣾλΠτϧ͔ΒScala/PlayΛ࠾༻͓ͯ͠Γɺ
   ݱࡏ։ൃதͷλΠτϧͰ΋ར༻͍ͯ͠·͢ɻ
   © AXEL Game Entertainment
   

   View Slide

8. Discontent of the existing way
   •  We were using PHP and in-house framework
   •  We run our games continuously in several years and the code
   quantity was increasing…
   •  Motivation for new titles:
   1.  Safe and easy modification for existing source code
   2.  Prevent runtime errors in production environment caused by rare case of
   exceptions
   େن໛ͳϞόΠϧήʔϜΛ௕ظӡ༻͢ΔͷʹɺLLͩͱ
   ܧଓతͳ։ൃʹݱ৔Ͱ͍͔ͭ͘໰୊Λײ͍ͯ͡·ͨ͠ɻ
   

   View Slide

9. Deciding factors of Scala
   •  Static typing
   •  Easy to modify existing codes
   •  Tools to describe briefly
   •  (e.g. Type interface, pattern match implicit and etc..)
   •  Simplicity leads reducing bugs
   •  Null-safe
   •  Hope for better Java and functional language
   •  Enable to use Java assets
   ScalaΛબΜͩͷ͸ɺݴޠͷಛੑ͕զʑͷཁ݅ʹ
   ͍͋ͬͯͨͨΊͰ͢ɻ
   

   View Slide

10. Deciding factors of Play
    •  High performance
    •  It has been used successfully in many services
    •  There are many plugins in OSS
    •  We accumulated experiences using HTTP-based API
    Play͸౰࣌ͷScalaͷϑϨʔϜϫʔΫͷதͰ΋
    طʹଟ͘ͷ࠾༻ࣄྫ͕͋ͬͨ͜ͱ͔Βબͼ·ͨ͠ɻ
    

    View Slide

11. System architecture
    γεςϜߏ੒ʹ͍ͭͯ࿩͠·͢ɻ
    

    View Slide

12. Component technologies
    •  Client:
    •  C# (Unity 5.x)
    •  Server (AWS):
    •  Scala(2.11.x)
    •  Play(2.4.x/2.5.x)
    •  MySQL (Aurora)
    •  Redis (ElastiCache)
    •  Google BigQuery (Store for action logs)
    •  Real-time server:
    •  Node.js/Socket.IO
    զʑͷήʔϜ։ൃͰ͸ओʹΫϥΠΞϯταΠυ͸C#ɺ
    αʔόαΠυ͸Scala/Play+MySQLΛ࢖͍ͬͯ·͢ɻ
    

    View Slide

13. Basic server-side implementation
    •  Traditional MVC(+S) pattern
    •  Framework: Play Framework
    •  Library: Scalaz (Only using the alternative of toRight)
    •  Database Libraries:
    •  Scalikejdbc + Skinny ORM
    •  Using play-evolution for database migration
    •  Coding convention:
    •  sbt-scalariform
    PlayΛ࢖ͬͨMVC(+S)ύλʔϯͰ࣮૷Λ͍ͯ͠·͢ɻ
    DBͰ͸ScalikejdbcͱSkinny ORMΛ࢖͍ͬͯ·͢ɻ
    

    View Slide

14. Testing
    •  Testing library: Specs2
    •  Fixture tool: Skinny Factory Girl
    •  Coverage report: sbt-scoverage
    •  Using H2 database with MySQL mode for functional tests:
    val  db  =  inMemoryDatabase(name  =  "default",  options  =  Map(MODE  -­‐>  "default"))  
    //  Play2.4  
    FakeApplication(addtionalConfiguration  =  db)  
    //  Play2.5  
    new  GuiceApplicationBuilder().configure(db).build()  
    Specs2Λ࢖ͬͯςετΛॻ͍͍ͯ·͢ɻ
    ػೳςετͰ͸H2σʔλϕʔεΛར༻͍ͯ͠·͢ɻ
    

    View Slide

15. Scale of the system (peek)
    •  The total number of users: More than 4.5 million
    •  Requests per second: Several thousands
    •  Average response time per second: 50ms
    •  Maximum time is requests to Apple API
    •  Queries per second: Several ten thousands
    •  Not using sharding
    •  Aurora is great
    •  Log records per day: Several ten million
    Ϣʔβ਺450ສɺ਺ઍrpsɺ਺ສqpsɺϩά਺ेສߦ/day
    (͋·Γ۩ମతͳ਺஋͸ग़ͤͳ͍ͷͰ΅͔ͯ͠·͢)
    

    View Slide

16. Logging infrastructure
    •  Player’s action logs are stored on Google BigQuery
    •  Querying for the action logs (update, insert, and delete) is heavy operation that
    would affect the performance of the master database if performed directly.
    •  Important logs like payment transactions are just stored on MySQL
    •  These logs are used on the admin tool for customer supports and
    analytics of KPI
    ϓϨΠϠʔͷߦಈϩά͸BigQuery্ʹอଘ͞Εɺ
    ؅ཧπʔϧ͔ΒCS΍KPI෼ੳʹར༻͞Ε·͢ɻ
    Game servers Amazon Kinesis AWS Lambda
    

    View Slide

17. Deployment
    •  AWS CodeDeploy
    1.  sbt dist
    2.  Upload to S3
    3.  CodeDeploy agents pull and update the new build
    •  Blue-Green Deployment
    •  No downtime
    •  Possible to rollback
    AWS CodeDeployͰσϓϩΠ͍ͯ͠·͢ɻ
    

    View Slide

18. System architecture image
    γεςϜߏ੒ਤͰ͢ɻ
    Elastic Load
    Balancing
    Game servers
    Trinity
    (In-house billing
    system)
    Aurora
    (master)
    Aurora
    (slave)
    Admin tool
    BigQuery
    Amazon
    ElastiCache
    Read/Write
    Read/Write
    Read/Write
    Repplicate Read/Write
    Transfer
    logs
    Lobby server
    Room servers
    Game servers
    (API servers)
    Real-time servers
    Read
    WebSockets
    HTTP
    

    View Slide

19. Details of implementation
    ͔͜͜Β͸࣮૷ͷৄࡉʹ͍ͭͯ࿩͠·͢ɻ
    

    View Slide

20. •  Play provides points to add a common process:
    1.  Filters
    2.  BodyParsers
    3.  ActionFunctions
    •  Because it is difficult to compose ActionFunctions flexibly, we
    integrated stackable-controller instead of it.
    •  https://github.com/t2v/stackable-controller
    Extension points in Play
    Play͸ڞ௨ॲཧΛڬΉՕॴΛ͍͔ͭ͘ఏڙ͍ͯ͠·͢ɻ
    ͜ΕΒͷͲΕ͔Λ࢖ͬͯڞ௨ॲཧΛ௥Ճ͍ͯ͠·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action
    Execution order in Play
    

    View Slide

21. Cache
    1.  Cache on Redis
    •  Response data to prevent multiple same requests
    (CacheElement stores responses and returns a response
    from the cache if the request is same one = idempotence)
    •  Temporary game data like game stage data
    2.  Cache on memories of each game servers
    •  Master data (game parameters) which expires when the version is updated
    •  Implemented with java.util.concurrent.ConcurrentHashMap
    (thread-safe)
    Ωϟογϡ͸RedisͱΦϯϝϞϦͷೋͭΛ࠾༻͍ͯ͠·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action
    

    View Slide

22. •  Falsifications of request parameters and response bodies are
    popular way to cheat on the network
    •  SSL/TSL is not sufficient to prevent player "cheating" the game
    •  To avoid falsifications, we should implement signatures (e.g. SHA)
    and/or encryption (e.g. RSA and AES) by own
    Cryptography
    SSL/TSL͚ͩͰ͸νʔτରࡦʹͳΒͳ͍ͷͰɺ
    ࣗલͰ҉߸Խ΍ॺ໊ͷ࣮૷Λ͢Δඞཁ͕͋Γ·͢ɻ
    

    View Slide

23. Decryption (requests)
    •  To use a form feature in Play, encrypted parameters are decrypted in
    the customized BodyParser
    •  (Because it is not easy to access request body in Filters)
    •  Then, the parameters are encoded by the default form encode
    PlayͷϑΥʔϜػೳΛ࢖͏ͨΊɺ
    ϦΫΤετύϥϝʔλͷ෮߸Խ͸BodyParserͰߦ͍·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action
    

    View Slide

24. Encryption (responses)
    •  The response data are encrypted in EncryptFilter
    •  (Because Result.body is used sometimes in ActionFunctions)
    •  We should compress data before encrypting it because encrypted
    byte array cannot be compressed effectively
    Ϩεϙϯεͷ҉߸Խ͸EncrypFilterͰߦ͍ͬͯ·͢ɻ
    ҉߸ԽલʹѹॖΛ͢Δඞཁ͕͋Γ·͢ɻ
    Filter
    BodyParser
    ActionFunction
    Action
    

    View Slide

25. Message formats
    •  In general, Web API returns a response data as JSON
    •  Problems:
    1.  When the large JSON is responded, the client may freeze for a moment
    and use increase memory usage to parse the JSON
    2.  Lose type-information by converting from a static typed language to a
    JSON format
    3.  Documentations and validations are required between client and server
    engineers
    •  In the case of mobile game developments, JSON is not a good
    choice
    ϞόΠϧήʔϜ։ൃʹ͓͍ͯ͸ɺJSON͸ඞͣ͠΋
    ྑ͍ϝοηʔδϑΥʔϚοτͷબ୒ࢶͰ͸͋Γ·ͤΜɻ
    

    View Slide

26. Thrift as a message format
    •  RPC framework made by Facebook
    (Now, under Apache project)
    •  We use Thrift as not RPC framework but a message format
    •  Thrift features:
    •  Data size is small and serialization is faster than JSON
    •  Binary format and it does not have type information in the data (Adopts IDL)
    •  Supports many languages
    (C++, C#, Java, Python and Ruby etc..)
    զʑͷήʔϜͰ͸ThriftΛϝοηʔδϑΥʔϚοτͱͯ͠
    ࠾༻͠·ͨ͠ɻ
    

    View Slide

27. Example to create class files
    ThriftͷIDLͷྫͰ͢ɻ
    struct  LoginResponse  {  
       1:  bool  success;  
       2:  i64  userId;  
    }  
    1. Define data structure with Thrift IDL
    Java C#
    $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift  
    $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift  
    2. Run following commands to generate files:
    3. Use each files on the server and the client
    LoginResponse.thrift
    LoginResponse.java LoginResponse.cs
    

    View Slide

28. thrift
    repository
    client
    repository
    server
    repository
    Refers latest C# files
    Refers latest Scala files
    client engineers
    server engineers
    Update thrift files and
    generate C#/Java classes
    Flow to update thrift files and classes
    ThriftϑΝΠϧΛར༻͢ΔϑϩʔͰ͢ɻ
    Use
    Use
    Java C#
    

    View Slide

29. Thrift serialization
    Java
    object
    C#
    object
    Binary  data  
    Server-side Client-side
    serialize deserialize
    Over HTTP
    

    View Slide

30. Example to use between server and client
    ThriftͷγϦΞϥΠζΛߦ͏ྫͰ͢ɻ
    def  login  =  Action  {  
       val  loginResponse  =  new  LoginResponse    
       loginResponse.success  =  true  
       val  protocol  =  new  TCompactProtocol.Factory()  
       val  serializer  =  new  TSerializer(protocol)  
       Ok(serializer.serialize(loginResponse))  
    }  
    var  www  =  new  WWW("http://localhost/login");  
    yield  return  www;  
    var  stream  =  new  MemoryStream(www.bytes);  
    var  tProtocol  =  new  TCompactProtocol(new  TStreamTransport(stream,  stream));  
    var  loginResponse  =  new  LoginResponse();  
    loginResponse.Read(tProtocol);  
    2. Deserialize on the client-size (C#)
    1. Serialize on the server-side (Scala)
    

    View Slide

31. Good points using Thrift
    1.  Good performance and saving memory
    2.  It is easy to share the specification for responses between the
    server-side and the client-side
    3.  By adding comments, IDL files work API documents
    4.  Replace magic numbers into Enum
    αʔόͱΫϥΠΞϯτͰڞ௨ͷσʔλߏ଄Λ࣋ͭ͜ͱ͕
    Ͱ͖ͨ͜ͱͰɺଟ͘ͷϝϦοτΛڗडͰ͖·ͨ͠ɻ
    

    View Slide

32. FYI: twitter/scrooge
    •  https://github.com/twitter/scrooge
    •  Thrift generator for Scala made by Twitter
    •  Use Thrift in Scala without Java
    •  Compatible original thrift data
    •  We did not adopt scrooge because it was necessary to downgrade
    the version of Scala/sbt when we tried it
    •  It may be solved now
    ༨ஊͰ͕͢ɺScalaͰThriftΛѻ͏ʹ͸
    Twitter੡ͷscroogeͱ͍͏ϥΠϒϥϦ΋͋Γ·͢ɻ
    

    View Slide

33. Admin tool
    •  Admin tool is implemented with Play Framework too
    •  The admin tool is separated from main code repository and refers it
    as a sub-project
    •  Additional libraries:
    •  Authorization: play2-auth
    •  Template: twirl
    •  Database: bigquery4s
    ؅ཧπʔϧ͸ϝΠϯͷϨϙδτϦ͔Β෼཭͞Ε͍ͯͯɺ
    αϒϞδϡʔϧͱͯ͠ࢀর͢Δܗʹͳ͍ͬͯ·͢ɻ
    

    View Slide

34. Performance tuning
    ύϑΥʔϚϯενϡʔχϯάʹ͍ͭͯ
    

    View Slide

35. Highly synchronous application
    •  Although Play is an asynchronous framework, JDBC does not
    support asynchronous (non-blocking IO)
    •  We describes our codes to handle database synchronously
    •  In the case of synchronous application, there are points to notice for
    the performance tuning
    JDBC͕ಉظAPI͔͠αϙʔτ͍ͯ͠ͳ͍ͷͰɺ
    զʑͷΞϓϦέʔγϣϯ͸ಉظతʹॻ͔Ε͍ͯ·͢ɻ
    

    View Slide

36. Configurations for Akka actors
    •  In the case of traditional synchronous IO based application, it uses large
    thread pools of Play(Akka) to handle blocking IO.
    •  Play 2.4.x:
    •  Play 2.5.x:
    •  The numbers depends on the server CPU performance
    ಉظతͳIOϕʔεͷΞϓϦέʔγϣϯͰ͸
    େ༰ྔͷεϨουϓʔϧ͕ඞཁͱͳΓ·͢ɻ
    ɾakka.actor.default-dispatcher.parallelism-factor
    ɾakka.actor.default-dispatcher.parallelism-min
    ɾakka.actor.default-dispatcher.parallelism-max
    The number of threads: parallelism-min <= CPUs * parallelism-factor <= parallelism-max
    ɾakka.actor.default-dispatcher.thread-pool-executor.fixed-pool-size
    ɹThe fixed pool size should be the maximum size of database connection pool plus extra
    

    View Slide

37. Configuration for Linux Kernel
    •  net.ip4.ip_local_port_range:
    •  Port range a process can use
    •  fs.file-max:
    •  The number of files a process can open
    (= The number of unix domain socket)
    େྔͷεϨουΛѻ͏৔߹ɺ
    Linux kernel΋มߋ͓͖ͯ͠·͠ΐ͏ɻ
    

    View Slide

38. Other tunings on implementations
    •  Some N+1 problems are detected by New Relic
    •  The library to access external APIs was implemented as non-blocking
    •  Applying Future and create a specialized ThreadPool
    •  Replace codes using Reflections into Scala Macros
    •  We published it as OSS
    •  Macrooom: Macro-based Object-Object Mapper
    •  https://github.com/h-kishi/macrooom
    ͦͷଞͷ࣮૷ϨϕϧͰߦͬͨνϡʔχϯάʹ͍ͭͯͰ͢ɻ
    

    View Slide

39. We are hiring!
    If you want to use Scala for game servers, please contact us!
    ฐࣾͰ͸ήʔϜαʔόΛScalaͰॻ͖͍ͨํΛ
    ืू͍ͯ͠·͢ʂ
    

    View Slide

40. Thank you for your attention!
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ
    

    View Slide