Scala and Play for game servers

2017 ScalaMatsuri

1. Scala and Play for game servers @h_kishi Hiromichi Kishi 2017 ScalaMatsuri ήʔϜαʔόͷͨΊͷScala/Play
2. Agenda 1.  Introduction 2.  Why Scala and Play? 3.  System architecture 4.  Details of implementation 5.  Performance tuning ຊ೔ͷΞδΣϯμ
3. Introduction ·ͣॳΊʹࣗݾ঺հΛ͍͖ͤͯͨͩ͞·͢ɻ
4. About me •  Hiromichi Kishi (@h_kishi) •  Technical advisor in AXEL MARK INC. •  Game developer and senior server-side engineer ΞΫηϧϚʔΫגࣜձࣾͰٕज़ސ໰Λ͍ͯ͠·͢ɻ ήʔϜ։ൃऀͰɺγχΞαʔόαΠυΤϯδχΞͰ͢ɻ
5. About AXEL MARK •  Mobile game business •  Mobile advertising business ΞΫηϧϚʔΫͰ͸ϞόΠϧήʔϜࣄۀͱ޿ࠂࣄۀ͕ ओͳࣄۀྖҬͰ͢ɻ © AXEL Game Entertainment © ݪହٱɾूӳࣾʗNHKɾNEPɾͽ͑Ζ © DeNA Co., Ltd. All rights reserved. / ɹDeveloped by AXEL GameStudio Inc.
6. Why Scala and Play? ͲͷΑ͏ͳཧ༝͔ΒฐࣾͰScala/PlayΛ ࠾༻͔ͨ͠ʹ͍ͭͯͰ͢ɻ
7. Adoption of Scala and Play •  “World Cross Saga” which was adopted Scala and Play was released on April 2016 •  We are using them in new game titles which are under development too ڈ೥ͷࣗࣾλΠτϧ͔ΒScala/PlayΛ࠾༻͓ͯ͠Γɺ ݱࡏ։ൃதͷλΠτϧͰ΋ར༻͍ͯ͠·͢ɻ © AXEL Game Entertainment
8. Discontent of the existing way •  We were using PHP and in-house framework •  We run our games continuously in several years and the code quantity was increasing… •  Motivation for new titles: 1.  Safe and easy modification for existing source code 2.  Prevent runtime errors in production environment caused by rare case of exceptions େن໛ͳϞόΠϧήʔϜΛ௕ظӡ༻͢ΔͷʹɺLLͩͱ ܧଓతͳ։ൃʹݱ৔Ͱ͍͔ͭ͘໰୊Λײ͍ͯ͡·ͨ͠ɻ
9. Deciding factors of Scala •  Static typing •  Easy to modify existing codes •  Tools to describe briefly •  (e.g. Type interface, pattern match implicit and etc..) •  Simplicity leads reducing bugs •  Null-safe •  Hope for better Java and functional language •  Enable to use Java assets ScalaΛબΜͩͷ͸ɺݴޠͷಛੑ͕զʑͷཁ݅ʹ ͍͋ͬͯͨͨΊͰ͢ɻ
10. Deciding factors of Play •  High performance •  It has been used successfully in many services •  There are many plugins in OSS •  We accumulated experiences using HTTP-based API Play͸౰࣌ͷScalaͷϑϨʔϜϫʔΫͷதͰ΋ طʹଟ͘ͷ࠾༻ࣄྫ͕͋ͬͨ͜ͱ͔Βબͼ·ͨ͠ɻ
11. System architecture γεςϜߏ੒ʹ͍ͭͯ࿩͠·͢ɻ
12. Component technologies •  Client: •  C# (Unity 5.x) •  Server (AWS): •  Scala(2.11.x) •  Play(2.4.x/2.5.x) •  MySQL (Aurora) •  Redis (ElastiCache) •  Google BigQuery (Store for action logs) •  Real-time server: •  Node.js/Socket.IO զʑͷήʔϜ։ൃͰ͸ओʹΫϥΠΞϯταΠυ͸C#ɺ αʔόαΠυ͸Scala/Play+MySQLΛ࢖͍ͬͯ·͢ɻ
13. Basic server-side implementation •  Traditional MVC(+S) pattern •  Framework: Play Framework •  Library: Scalaz (Only using the alternative of toRight) •  Database Libraries: •  Scalikejdbc + Skinny ORM •  Using play-evolution for database migration •  Coding convention: •  sbt-scalariform PlayΛ࢖ͬͨMVC(+S)ύλʔϯͰ࣮૷Λ͍ͯ͠·͢ɻ DBͰ͸ScalikejdbcͱSkinny ORMΛ࢖͍ͬͯ·͢ɻ
14. Testing •  Testing library: Specs2 •  Fixture tool: Skinny Factory Girl •  Coverage report: sbt-scoverage •  Using H2 database with MySQL mode for functional tests: val  db  =  inMemoryDatabase(name  =  "default",  options  =  Map(MODE  -­‐>  "default"))   //  Play2.4   FakeApplication(addtionalConfiguration  =  db)   //  Play2.5   new  GuiceApplicationBuilder().configure(db).build()   Specs2Λ࢖ͬͯςετΛॻ͍͍ͯ·͢ɻ ػೳςετͰ͸H2σʔλϕʔεΛར༻͍ͯ͠·͢ɻ
15. Scale of the system (peek) •  The total number of users: More than 4.5 million •  Requests per second: Several thousands •  Average response time per second: 50ms •  Maximum time is requests to Apple API •  Queries per second: Several ten thousands •  Not using sharding •  Aurora is great •  Log records per day: Several ten million Ϣʔβ਺450ສɺ਺ઍrpsɺ਺ສqpsɺϩά਺ेສߦ/day (͋·Γ۩ମతͳ਺஋͸ग़ͤͳ͍ͷͰ΅͔ͯ͠·͢)
16. Logging infrastructure •  Player’s action logs are stored on Google BigQuery •  Querying for the action logs (update, insert, and delete) is heavy operation that would affect the performance of the master database if performed directly. •  Important logs like payment transactions are just stored on MySQL •  These logs are used on the admin tool for customer supports and analytics of KPI ϓϨΠϠʔͷߦಈϩά͸BigQuery্ʹอଘ͞Εɺ ؅ཧπʔϧ͔ΒCS΍KPI෼ੳʹར༻͞Ε·͢ɻ Game servers Amazon Kinesis AWS Lambda
17. Deployment •  AWS CodeDeploy 1.  sbt dist 2.  Upload to S3 3.  CodeDeploy agents pull and update the new build •  Blue-Green Deployment •  No downtime •  Possible to rollback AWS CodeDeployͰσϓϩΠ͍ͯ͠·͢ɻ
18. System architecture image γεςϜߏ੒ਤͰ͢ɻ Elastic Load Balancing Game servers Trinity (In-house billing system) Aurora (master) Aurora (slave) Admin tool BigQuery Amazon ElastiCache Read/Write Read/Write Read/Write Repplicate Read/Write Transfer logs Lobby server Room servers Game servers (API servers) Real-time servers Read WebSockets HTTP
19. Details of implementation ͔͜͜Β͸࣮૷ͷৄࡉʹ͍ͭͯ࿩͠·͢ɻ
20. •  Play provides points to add a common process: 1.  Filters 2.  BodyParsers 3.  ActionFunctions •  Because it is difficult to compose ActionFunctions flexibly, we integrated stackable-controller instead of it. •  https://github.com/t2v/stackable-controller Extension points in Play Play͸ڞ௨ॲཧΛڬΉՕॴΛ͍͔ͭ͘ఏڙ͍ͯ͠·͢ɻ ͜ΕΒͷͲΕ͔Λ࢖ͬͯڞ௨ॲཧΛ௥Ճ͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action Execution order in Play
21. Cache 1.  Cache on Redis •  Response data to prevent multiple same requests (CacheElement stores responses and returns a response from the cache if the request is same one = idempotence) •  Temporary game data like game stage data 2.  Cache on memories of each game servers •  Master data (game parameters) which expires when the version is updated •  Implemented with java.util.concurrent.ConcurrentHashMap (thread-safe) Ωϟογϡ͸RedisͱΦϯϝϞϦͷೋͭΛ࠾༻͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action
22. •  Falsifications of request parameters and response bodies are popular way to cheat on the network •  SSL/TSL is not sufficient to prevent player "cheating" the game •  To avoid falsifications, we should implement signatures (e.g. SHA) and/or encryption (e.g. RSA and AES) by own Cryptography SSL/TSL͚ͩͰ͸νʔτରࡦʹͳΒͳ͍ͷͰɺ ࣗલͰ҉߸Խ΍ॺ໊ͷ࣮૷Λ͢Δඞཁ͕͋Γ·͢ɻ
23. Decryption (requests) •  To use a form feature in Play, encrypted parameters are decrypted in the customized BodyParser •  (Because it is not easy to access request body in Filters) •  Then, the parameters are encoded by the default form encode PlayͷϑΥʔϜػೳΛ࢖͏ͨΊɺ ϦΫΤετύϥϝʔλͷ෮߸Խ͸BodyParserͰߦ͍·͢ɻ Filter BodyParser ActionFunction Action
24. Encryption (responses) •  The response data are encrypted in EncryptFilter •  (Because Result.body is used sometimes in ActionFunctions) •  We should compress data before encrypting it because encrypted byte array cannot be compressed effectively Ϩεϙϯεͷ҉߸Խ͸EncrypFilterͰߦ͍ͬͯ·͢ɻ ҉߸ԽલʹѹॖΛ͢Δඞཁ͕͋Γ·͢ɻ Filter BodyParser ActionFunction Action
25. Message formats •  In general, Web API returns a response data as JSON •  Problems: 1.  When the large JSON is responded, the client may freeze for a moment and use increase memory usage to parse the JSON 2.  Lose type-information by converting from a static typed language to a JSON format 3.  Documentations and validations are required between client and server engineers •  In the case of mobile game developments, JSON is not a good choice ϞόΠϧήʔϜ։ൃʹ͓͍ͯ͸ɺJSON͸ඞͣ͠΋ ྑ͍ϝοηʔδϑΥʔϚοτͷબ୒ࢶͰ͸͋Γ·ͤΜɻ
26. Thrift as a message format •  RPC framework made by Facebook (Now, under Apache project) •  We use Thrift as not RPC framework but a message format •  Thrift features: •  Data size is small and serialization is faster than JSON •  Binary format and it does not have type information in the data (Adopts IDL) •  Supports many languages (C++, C#, Java, Python and Ruby etc..) զʑͷήʔϜͰ͸ThriftΛϝοηʔδϑΥʔϚοτͱͯ͠ ࠾༻͠·ͨ͠ɻ
27. Example to create class files ThriftͷIDLͷྫͰ͢ɻ struct  LoginResponse  {      1:  bool  success;      2:  i64  userId;   }   1. Define data structure with Thrift IDL Java C# $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift   $  thrift  –gen  java  –o  dist/  src/LoginResponse.thrift   2. Run following commands to generate files: 3. Use each files on the server and the client LoginResponse.thrift LoginResponse.java LoginResponse.cs
28. thrift repository client repository server repository Refers latest C# files Refers latest Scala files client engineers server engineers Update thrift files and generate C#/Java classes Flow to update thrift files and classes ThriftϑΝΠϧΛར༻͢ΔϑϩʔͰ͢ɻ Use Use Java C#
29. Thrift serialization Java object C# object Binary  data   Server-side Client-side serialize deserialize Over HTTP
30. Example to use between server and client ThriftͷγϦΞϥΠζΛߦ͏ྫͰ͢ɻ def  login  =  Action  {      val  loginResponse  =  new  LoginResponse        loginResponse.success  =  true      val  protocol  =  new  TCompactProtocol.Factory()      val  serializer  =  new  TSerializer(protocol)      Ok(serializer.serialize(loginResponse))   }   var  www  =  new  WWW("http://localhost/login");   yield  return  www;   var  stream  =  new  MemoryStream(www.bytes);   var  tProtocol  =  new  TCompactProtocol(new  TStreamTransport(stream,  stream));   var  loginResponse  =  new  LoginResponse();   loginResponse.Read(tProtocol);   2. Deserialize on the client-size (C#) 1. Serialize on the server-side (Scala)
31. Good points using Thrift 1.  Good performance and saving memory 2.  It is easy to share the specification for responses between the server-side and the client-side 3.  By adding comments, IDL files work API documents 4.  Replace magic numbers into Enum αʔόͱΫϥΠΞϯτͰڞ௨ͷσʔλߏ଄Λ࣋ͭ͜ͱ͕ Ͱ͖ͨ͜ͱͰɺଟ͘ͷϝϦοτΛڗडͰ͖·ͨ͠ɻ
32. FYI: twitter/scrooge •  https://github.com/twitter/scrooge •  Thrift generator for Scala made by Twitter •  Use Thrift in Scala without Java •  Compatible original thrift data •  We did not adopt scrooge because it was necessary to downgrade the version of Scala/sbt when we tried it •  It may be solved now ༨ஊͰ͕͢ɺScalaͰThriftΛѻ͏ʹ͸ Twitter੡ͷscroogeͱ͍͏ϥΠϒϥϦ΋͋Γ·͢ɻ
33. Admin tool •  Admin tool is implemented with Play Framework too •  The admin tool is separated from main code repository and refers it as a sub-project •  Additional libraries: •  Authorization: play2-auth •  Template: twirl •  Database: bigquery4s ؅ཧπʔϧ͸ϝΠϯͷϨϙδτϦ͔Β෼཭͞Ε͍ͯͯɺ αϒϞδϡʔϧͱͯ͠ࢀর͢Δܗʹͳ͍ͬͯ·͢ɻ
34. Performance tuning ύϑΥʔϚϯενϡʔχϯάʹ͍ͭͯ
35. Highly synchronous application •  Although Play is an asynchronous framework, JDBC does not support asynchronous (non-blocking IO) •  We describes our codes to handle database synchronously •  In the case of synchronous application, there are points to notice for the performance tuning JDBC͕ಉظAPI͔͠αϙʔτ͍ͯ͠ͳ͍ͷͰɺ զʑͷΞϓϦέʔγϣϯ͸ಉظతʹॻ͔Ε͍ͯ·͢ɻ
36. Configurations for Akka actors •  In the case of traditional synchronous IO based application, it uses large thread pools of Play(Akka) to handle blocking IO. •  Play 2.4.x: •  Play 2.5.x: •  The numbers depends on the server CPU performance ಉظతͳIOϕʔεͷΞϓϦέʔγϣϯͰ͸ େ༰ྔͷεϨουϓʔϧ͕ඞཁͱͳΓ·͢ɻ ɾakka.actor.default-dispatcher.parallelism-factor ɾakka.actor.default-dispatcher.parallelism-min ɾakka.actor.default-dispatcher.parallelism-max The number of threads: parallelism-min <= CPUs * parallelism-factor <= parallelism-max ɾakka.actor.default-dispatcher.thread-pool-executor.fixed-pool-size ɹThe fixed pool size should be the maximum size of database connection pool plus extra
37. Configuration for Linux Kernel •  net.ip4.ip_local_port_range: •  Port range a process can use •  fs.file-max: •  The number of files a process can open (= The number of unix domain socket) େྔͷεϨουΛѻ͏৔߹ɺ Linux kernel΋มߋ͓͖ͯ͠·͠ΐ͏ɻ
38. Other tunings on implementations •  Some N+1 problems are detected by New Relic •  The library to access external APIs was implemented as non-blocking •  Applying Future and create a specialized ThreadPool •  Replace codes using Reflections into Scala Macros •  We published it as OSS •  Macrooom: Macro-based Object-Object Mapper •  https://github.com/h-kishi/macrooom ͦͷଞͷ࣮૷ϨϕϧͰߦͬͨνϡʔχϯάʹ͍ͭͯͰ͢ɻ
39. We are hiring! If you want to use Scala for game servers, please contact us! ฐࣾͰ͸ήʔϜαʔόΛScalaͰॻ͖͍ͨํΛ ืू͍ͯ͠·͢ʂ
40. Thank you for your attention! ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ