Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scala and Play for game servers

February 26, 2017

Scala and Play for game servers

2017 ScalaMatsuri


February 26, 2017

Other Decks in Programming


  1. Scala and Play for game servers @h_kishi Hiromichi Kishi 2017

    ScalaMatsuri ήʔϜαʔόͷͨΊͷScala/Play
  2. Agenda 1.  Introduction 2.  Why Scala and Play? 3.  System

    architecture 4.  Details of implementation 5.  Performance tuning ຊ೔ͷΞδΣϯμ
  3. About me •  Hiromichi Kishi (@h_kishi) •  Technical advisor in

    AXEL MARK INC. •  Game developer and senior server-side engineer ΞΫηϧϚʔΫגࣜձࣾͰٕज़ސ໰Λ͍ͯ͠·͢ɻ ήʔϜ։ൃऀͰɺγχΞαʔόαΠυΤϯδχΞͰ͢ɻ
  4. About AXEL MARK •  Mobile game business •  Mobile advertising

    business ΞΫηϧϚʔΫͰ͸ϞόΠϧήʔϜࣄۀͱ޿ࠂࣄۀ͕ ओͳࣄۀྖҬͰ͢ɻ © AXEL Game Entertainment © ݪହٱɾूӳࣾʗNHKɾNEPɾͽ͑Ζ © DeNA Co., Ltd. All rights reserved. / ɹDeveloped by AXEL GameStudio Inc.
  5. Adoption of Scala and Play •  “World Cross Saga” which

    was adopted Scala and Play was released on April 2016 •  We are using them in new game titles which are under development too ڈ೥ͷࣗࣾλΠτϧ͔ΒScala/PlayΛ࠾༻͓ͯ͠Γɺ ݱࡏ։ൃதͷλΠτϧͰ΋ར༻͍ͯ͠·͢ɻ © AXEL Game Entertainment
  6. Discontent of the existing way •  We were using PHP

    and in-house framework •  We run our games continuously in several years and the code quantity was increasing… •  Motivation for new titles: 1.  Safe and easy modification for existing source code 2.  Prevent runtime errors in production environment caused by rare case of exceptions େن໛ͳϞόΠϧήʔϜΛ௕ظӡ༻͢ΔͷʹɺLLͩͱ ܧଓతͳ։ൃʹݱ৔Ͱ͍͔ͭ͘໰୊Λײ͍ͯ͡·ͨ͠ɻ
  7. Deciding factors of Scala •  Static typing •  Easy to

    modify existing codes •  Tools to describe briefly •  (e.g. Type interface, pattern match implicit and etc..) •  Simplicity leads reducing bugs •  Null-safe •  Hope for better Java and functional language •  Enable to use Java assets ScalaΛબΜͩͷ͸ɺݴޠͷಛੑ͕զʑͷཁ݅ʹ ͍͋ͬͯͨͨΊͰ͢ɻ
  8. Deciding factors of Play •  High performance •  It has

    been used successfully in many services •  There are many plugins in OSS •  We accumulated experiences using HTTP-based API Play͸౰࣌ͷScalaͷϑϨʔϜϫʔΫͷதͰ΋ طʹଟ͘ͷ࠾༻ࣄྫ͕͋ͬͨ͜ͱ͔Βબͼ·ͨ͠ɻ
  9. Component technologies •  Client: •  C# (Unity 5.x) •  Server

    (AWS): •  Scala(2.11.x) •  Play(2.4.x/2.5.x) •  MySQL (Aurora) •  Redis (ElastiCache) •  Google BigQuery (Store for action logs) •  Real-time server: •  Node.js/Socket.IO զʑͷήʔϜ։ൃͰ͸ओʹΫϥΠΞϯταΠυ͸C#ɺ αʔόαΠυ͸Scala/Play+MySQLΛ࢖͍ͬͯ·͢ɻ
  10. Basic server-side implementation •  Traditional MVC(+S) pattern •  Framework: Play

    Framework •  Library: Scalaz (Only using the alternative of toRight) •  Database Libraries: •  Scalikejdbc + Skinny ORM •  Using play-evolution for database migration •  Coding convention: •  sbt-scalariform PlayΛ࢖ͬͨMVC(+S)ύλʔϯͰ࣮૷Λ͍ͯ͠·͢ɻ DBͰ͸ScalikejdbcͱSkinny ORMΛ࢖͍ͬͯ·͢ɻ
  11. Testing •  Testing library: Specs2 •  Fixture tool: Skinny Factory

    Girl •  Coverage report: sbt-scoverage •  Using H2 database with MySQL mode for functional tests: val  db  =  inMemoryDatabase(name  =  "default",  options  =  Map(MODE  -­‐>  "default"))   //  Play2.4   FakeApplication(addtionalConfiguration  =  db)   //  Play2.5   new  GuiceApplicationBuilder().configure(db).build()   Specs2Λ࢖ͬͯςετΛॻ͍͍ͯ·͢ɻ ػೳςετͰ͸H2σʔλϕʔεΛར༻͍ͯ͠·͢ɻ
  12. Scale of the system (peek) •  The total number of

    users: More than 4.5 million •  Requests per second: Several thousands •  Average response time per second: 50ms •  Maximum time is requests to Apple API •  Queries per second: Several ten thousands •  Not using sharding •  Aurora is great •  Log records per day: Several ten million Ϣʔβ਺450ສɺ਺ઍrpsɺ਺ສqpsɺϩά਺ेສߦ/day (͋·Γ۩ମతͳ਺஋͸ग़ͤͳ͍ͷͰ΅͔ͯ͠·͢)
  13. Logging infrastructure •  Player’s action logs are stored on Google

    BigQuery •  Querying for the action logs (update, insert, and delete) is heavy operation that would affect the performance of the master database if performed directly. •  Important logs like payment transactions are just stored on MySQL •  These logs are used on the admin tool for customer supports and analytics of KPI ϓϨΠϠʔͷߦಈϩά͸BigQuery্ʹอଘ͞Εɺ ؅ཧπʔϧ͔ΒCS΍KPI෼ੳʹར༻͞Ε·͢ɻ Game servers Amazon Kinesis AWS Lambda
  14. Deployment •  AWS CodeDeploy 1.  sbt dist 2.  Upload to

    S3 3.  CodeDeploy agents pull and update the new build •  Blue-Green Deployment •  No downtime •  Possible to rollback AWS CodeDeployͰσϓϩΠ͍ͯ͠·͢ɻ
  15. System architecture image γεςϜߏ੒ਤͰ͢ɻ Elastic Load Balancing Game servers Trinity

    (In-house billing system) Aurora (master) Aurora (slave) Admin tool BigQuery Amazon ElastiCache Read/Write Read/Write Read/Write Repplicate Read/Write Transfer logs Lobby server Room servers Game servers (API servers) Real-time servers Read WebSockets HTTP
  16. •  Play provides points to add a common process: 1. 

    Filters 2.  BodyParsers 3.  ActionFunctions •  Because it is difficult to compose ActionFunctions flexibly, we integrated stackable-controller instead of it. •  https://github.com/t2v/stackable-controller Extension points in Play Play͸ڞ௨ॲཧΛڬΉՕॴΛ͍͔ͭ͘ఏڙ͍ͯ͠·͢ɻ ͜ΕΒͷͲΕ͔Λ࢖ͬͯڞ௨ॲཧΛ௥Ճ͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action Execution order in Play
  17. Cache 1.  Cache on Redis •  Response data to prevent

    multiple same requests (CacheElement stores responses and returns a response from the cache if the request is same one = idempotence) •  Temporary game data like game stage data 2.  Cache on memories of each game servers •  Master data (game parameters) which expires when the version is updated •  Implemented with java.util.concurrent.ConcurrentHashMap (thread-safe) Ωϟογϡ͸RedisͱΦϯϝϞϦͷೋͭΛ࠾༻͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action
  18. •  Falsifications of request parameters and response bodies are popular

    way to cheat on the network •  SSL/TSL is not sufficient to prevent player "cheating" the game •  To avoid falsifications, we should implement signatures (e.g. SHA) and/or encryption (e.g. RSA and AES) by own Cryptography SSL/TSL͚ͩͰ͸νʔτରࡦʹͳΒͳ͍ͷͰɺ ࣗલͰ҉߸Խ΍ॺ໊ͷ࣮૷Λ͢Δඞཁ͕͋Γ·͢ɻ
  19. Decryption (requests) •  To use a form feature in Play,

    encrypted parameters are decrypted in the customized BodyParser •  (Because it is not easy to access request body in Filters) •  Then, the parameters are encoded by the default form encode PlayͷϑΥʔϜػೳΛ࢖͏ͨΊɺ ϦΫΤετύϥϝʔλͷ෮߸Խ͸BodyParserͰߦ͍·͢ɻ Filter BodyParser ActionFunction Action
  20. Encryption (responses) •  The response data are encrypted in EncryptFilter

    •  (Because Result.body is used sometimes in ActionFunctions) •  We should compress data before encrypting it because encrypted byte array cannot be compressed effectively Ϩεϙϯεͷ҉߸Խ͸EncrypFilterͰߦ͍ͬͯ·͢ɻ ҉߸ԽલʹѹॖΛ͢Δඞཁ͕͋Γ·͢ɻ Filter BodyParser ActionFunction Action
  21. Message formats •  In general, Web API returns a response

    data as JSON •  Problems: 1.  When the large JSON is responded, the client may freeze for a moment and use increase memory usage to parse the JSON 2.  Lose type-information by converting from a static typed language to a JSON format 3.  Documentations and validations are required between client and server engineers •  In the case of mobile game developments, JSON is not a good choice ϞόΠϧήʔϜ։ൃʹ͓͍ͯ͸ɺJSON͸ඞͣ͠΋ ྑ͍ϝοηʔδϑΥʔϚοτͷબ୒ࢶͰ͸͋Γ·ͤΜɻ
  22. Thrift as a message format •  RPC framework made by

    Facebook (Now, under Apache project) •  We use Thrift as not RPC framework but a message format •  Thrift features: •  Data size is small and serialization is faster than JSON •  Binary format and it does not have type information in the data (Adopts IDL) •  Supports many languages (C++, C#, Java, Python and Ruby etc..) զʑͷήʔϜͰ͸ThriftΛϝοηʔδϑΥʔϚοτͱͯ͠ ࠾༻͠·ͨ͠ɻ
  23. Example to create class files ThriftͷIDLͷྫͰ͢ɻ struct  LoginResponse  {  

       1:  bool  success;      2:  i64  userId;   }   1. Define data structure with Thrift IDL Java C# $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift   $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift   2. Run following commands to generate files: 3. Use each files on the server and the client LoginResponse.thrift LoginResponse.java LoginResponse.cs
  24. thrift repository client repository server repository Refers latest C# files

    Refers latest Scala files client engineers server engineers Update thrift files and generate C#/Java classes Flow to update thrift files and classes ThriftϑΝΠϧΛར༻͢ΔϑϩʔͰ͢ɻ Use Use Java C#
  25. Example to use between server and client ThriftͷγϦΞϥΠζΛߦ͏ྫͰ͢ɻ def  login

     =  Action  {      val  loginResponse  =  new  LoginResponse        loginResponse.success  =  true      val  protocol  =  new  TCompactProtocol.Factory()      val  serializer  =  new  TSerializer(protocol)      Ok(serializer.serialize(loginResponse))   }   var  www  =  new  WWW("http://localhost/login");   yield  return  www;   var  stream  =  new  MemoryStream(www.bytes);   var  tProtocol  =  new  TCompactProtocol(new  TStreamTransport(stream,  stream));   var  loginResponse  =  new  LoginResponse();   loginResponse.Read(tProtocol);   2. Deserialize on the client-size (C#) 1. Serialize on the server-side (Scala)
  26. Good points using Thrift 1.  Good performance and saving memory

    2.  It is easy to share the specification for responses between the server-side and the client-side 3.  By adding comments, IDL files work API documents 4.  Replace magic numbers into Enum αʔόͱΫϥΠΞϯτͰڞ௨ͷσʔλߏ଄Λ࣋ͭ͜ͱ͕ Ͱ͖ͨ͜ͱͰɺଟ͘ͷϝϦοτΛڗडͰ͖·ͨ͠ɻ
  27. FYI: twitter/scrooge •  https://github.com/twitter/scrooge •  Thrift generator for Scala made

    by Twitter •  Use Thrift in Scala without Java •  Compatible original thrift data •  We did not adopt scrooge because it was necessary to downgrade the version of Scala/sbt when we tried it •  It may be solved now ༨ஊͰ͕͢ɺScalaͰThriftΛѻ͏ʹ͸ Twitter੡ͷscroogeͱ͍͏ϥΠϒϥϦ΋͋Γ·͢ɻ
  28. Admin tool •  Admin tool is implemented with Play Framework

    too •  The admin tool is separated from main code repository and refers it as a sub-project •  Additional libraries: •  Authorization: play2-auth •  Template: twirl •  Database: bigquery4s ؅ཧπʔϧ͸ϝΠϯͷϨϙδτϦ͔Β෼཭͞Ε͍ͯͯɺ αϒϞδϡʔϧͱͯ͠ࢀর͢Δܗʹͳ͍ͬͯ·͢ɻ
  29. Highly synchronous application •  Although Play is an asynchronous framework,

    JDBC does not support asynchronous (non-blocking IO) •  We describes our codes to handle database synchronously •  In the case of synchronous application, there are points to notice for the performance tuning JDBC͕ಉظAPI͔͠αϙʔτ͍ͯ͠ͳ͍ͷͰɺ զʑͷΞϓϦέʔγϣϯ͸ಉظతʹॻ͔Ε͍ͯ·͢ɻ
  30. Configurations for Akka actors •  In the case of traditional

    synchronous IO based application, it uses large thread pools of Play(Akka) to handle blocking IO. •  Play 2.4.x: •  Play 2.5.x: •  The numbers depends on the server CPU performance ಉظతͳIOϕʔεͷΞϓϦέʔγϣϯͰ͸ େ༰ྔͷεϨουϓʔϧ͕ඞཁͱͳΓ·͢ɻ ɾakka.actor.default-dispatcher.parallelism-factor ɾakka.actor.default-dispatcher.parallelism-min ɾakka.actor.default-dispatcher.parallelism-max The number of threads: parallelism-min <= CPUs * parallelism-factor <= parallelism-max ɾakka.actor.default-dispatcher.thread-pool-executor.fixed-pool-size ɹThe fixed pool size should be the maximum size of database connection pool plus extra
  31. Configuration for Linux Kernel •  net.ip4.ip_local_port_range: •  Port range a

    process can use •  fs.file-max: •  The number of files a process can open (= The number of unix domain socket) େྔͷεϨουΛѻ͏৔߹ɺ Linux kernel΋มߋ͓͖ͯ͠·͠ΐ͏ɻ
  32. Other tunings on implementations •  Some N+1 problems are detected

    by New Relic •  The library to access external APIs was implemented as non-blocking •  Applying Future and create a specialized ThreadPool •  Replace codes using Reflections into Scala Macros •  We published it as OSS •  Macrooom: Macro-based Object-Object Mapper •  https://github.com/h-kishi/macrooom ͦͷଞͷ࣮૷ϨϕϧͰߦͬͨνϡʔχϯάʹ͍ͭͯͰ͢ɻ
  33. We are hiring! If you want to use Scala for

    game servers, please contact us! ฐࣾͰ͸ήʔϜαʔόΛScalaͰॻ͖͍ͨํΛ ืू͍ͯ͠·͢ʂ