and in-house framework • We run our games continuously in several years and the code quantity was increasing… • Motivation for new titles: 1. Safe and easy modification for existing source code 2. Prevent runtime errors in production environment caused by rare case of exceptions େنͳϞόΠϧήʔϜΛظӡ༻͢ΔͷʹɺLLͩͱ ܧଓతͳ։ൃʹݱͰ͍͔ͭ͘Λײ͍ͯ͡·ͨ͠ɻ
modify existing codes • Tools to describe briefly • (e.g. Type interface, pattern match implicit and etc..) • Simplicity leads reducing bugs • Null-safe • Hope for better Java and functional language • Enable to use Java assets ScalaΛબΜͩͷɺݴޠͷಛੑ͕զʑͷཁ݅ʹ ͍͋ͬͯͨͨΊͰ͢ɻ
been used successfully in many services • There are many plugins in OSS • We accumulated experiences using HTTP-based API Play࣌ͷScalaͷϑϨʔϜϫʔΫͷதͰ طʹଟ͘ͷ࠾༻ࣄྫ͕͋ͬͨ͜ͱ͔Βબͼ·ͨ͠ɻ
users: More than 4.5 million • Requests per second: Several thousands • Average response time per second: 50ms • Maximum time is requests to Apple API • Queries per second: Several ten thousands • Not using sharding • Aurora is great • Log records per day: Several ten million Ϣʔβ450ສɺઍrpsɺສqpsɺϩάेສߦ/day (͋·Γ۩ମతͳग़ͤͳ͍ͷͰ΅͔ͯ͠·͢)
BigQuery • Querying for the action logs (update, insert, and delete) is heavy operation that would affect the performance of the master database if performed directly. • Important logs like payment transactions are just stored on MySQL • These logs are used on the admin tool for customer supports and analytics of KPI ϓϨΠϠʔͷߦಈϩάBigQuery্ʹอଘ͞Εɺ ཧπʔϧ͔ΒCSKPIੳʹར༻͞Ε·͢ɻ Game servers Amazon Kinesis AWS Lambda
Filters 2. BodyParsers 3. ActionFunctions • Because it is difficult to compose ActionFunctions flexibly, we integrated stackable-controller instead of it. • https://github.com/t2v/stackable-controller Extension points in Play Playڞ௨ॲཧΛڬΉՕॴΛ͍͔ͭ͘ఏڙ͍ͯ͠·͢ɻ ͜ΕΒͷͲΕ͔Λͬͯڞ௨ॲཧΛՃ͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action Execution order in Play
multiple same requests (CacheElement stores responses and returns a response from the cache if the request is same one = idempotence) • Temporary game data like game stage data 2. Cache on memories of each game servers • Master data (game parameters) which expires when the version is updated • Implemented with java.util.concurrent.ConcurrentHashMap (thread-safe) ΩϟογϡRedisͱΦϯϝϞϦͷೋͭΛ࠾༻͍ͯ͠·͢ɻ Filter BodyParser ActionFunction Action
way to cheat on the network • SSL/TSL is not sufficient to prevent player "cheating" the game • To avoid falsifications, we should implement signatures (e.g. SHA) and/or encryption (e.g. RSA and AES) by own Cryptography SSL/TSL͚ͩͰνʔτରࡦʹͳΒͳ͍ͷͰɺ ࣗલͰ҉߸Խॺ໊ͷ࣮Λ͢Δඞཁ͕͋Γ·͢ɻ
encrypted parameters are decrypted in the customized BodyParser • (Because it is not easy to access request body in Filters) • Then, the parameters are encoded by the default form encode PlayͷϑΥʔϜػೳΛ͏ͨΊɺ ϦΫΤετύϥϝʔλͷ෮߸ԽBodyParserͰߦ͍·͢ɻ Filter BodyParser ActionFunction Action
• (Because Result.body is used sometimes in ActionFunctions) • We should compress data before encrypting it because encrypted byte array cannot be compressed effectively Ϩεϙϯεͷ҉߸ԽEncrypFilterͰߦ͍ͬͯ·͢ɻ ҉߸ԽલʹѹॖΛ͢Δඞཁ͕͋Γ·͢ɻ Filter BodyParser ActionFunction Action
data as JSON • Problems: 1. When the large JSON is responded, the client may freeze for a moment and use increase memory usage to parse the JSON 2. Lose type-information by converting from a static typed language to a JSON format 3. Documentations and validations are required between client and server engineers • In the case of mobile game developments, JSON is not a good choice ϞόΠϧήʔϜ։ൃʹ͓͍ͯɺJSONඞͣ͠ ྑ͍ϝοηʔδϑΥʔϚοτͷબࢶͰ͋Γ·ͤΜɻ
Facebook (Now, under Apache project) • We use Thrift as not RPC framework but a message format • Thrift features: • Data size is small and serialization is faster than JSON • Binary format and it does not have type information in the data (Adopts IDL) • Supports many languages (C++, C#, Java, Python and Ruby etc..) զʑͷήʔϜͰThriftΛϝοηʔδϑΥʔϚοτͱͯ͠ ࠾༻͠·ͨ͠ɻ
1: bool success; 2: i64 userId; } 1. Define data structure with Thrift IDL Java C# $ thrift –gen java –o dist/ src/LoginResponse.thrift $ thrift –gen java –o dist/ src/LoginResponse.thrift 2. Run following commands to generate files: 3. Use each files on the server and the client LoginResponse.thrift LoginResponse.java LoginResponse.cs
Refers latest Scala files client engineers server engineers Update thrift files and generate C#/Java classes Flow to update thrift files and classes ThriftϑΝΠϧΛར༻͢ΔϑϩʔͰ͢ɻ Use Use Java C#
= Action { val loginResponse = new LoginResponse loginResponse.success = true val protocol = new TCompactProtocol.Factory() val serializer = new TSerializer(protocol) Ok(serializer.serialize(loginResponse)) } var www = new WWW("http://localhost/login"); yield return www; var stream = new MemoryStream(www.bytes); var tProtocol = new TCompactProtocol(new TStreamTransport(stream, stream)); var loginResponse = new LoginResponse(); loginResponse.Read(tProtocol); 2. Deserialize on the client-size (C#) 1. Serialize on the server-side (Scala)
2. It is easy to share the specification for responses between the server-side and the client-side 3. By adding comments, IDL files work API documents 4. Replace magic numbers into Enum αʔόͱΫϥΠΞϯτͰڞ௨ͷσʔλߏΛ࣋ͭ͜ͱ͕ Ͱ͖ͨ͜ͱͰɺଟ͘ͷϝϦοτΛڗडͰ͖·ͨ͠ɻ
by Twitter • Use Thrift in Scala without Java • Compatible original thrift data • We did not adopt scrooge because it was necessary to downgrade the version of Scala/sbt when we tried it • It may be solved now ༨ஊͰ͕͢ɺScalaͰThriftΛѻ͏ʹ Twitterͷscroogeͱ͍͏ϥΠϒϥϦ͋Γ·͢ɻ
too • The admin tool is separated from main code repository and refers it as a sub-project • Additional libraries: • Authorization: play2-auth • Template: twirl • Database: bigquery4s ཧπʔϧϝΠϯͷϨϙδτϦ͔Β͞Ε͍ͯͯɺ αϒϞδϡʔϧͱͯ͠ࢀর͢Δܗʹͳ͍ͬͯ·͢ɻ
JDBC does not support asynchronous (non-blocking IO) • We describes our codes to handle database synchronously • In the case of synchronous application, there are points to notice for the performance tuning JDBC͕ಉظAPI͔͠αϙʔτ͍ͯ͠ͳ͍ͷͰɺ զʑͷΞϓϦέʔγϣϯಉظతʹॻ͔Ε͍ͯ·͢ɻ
synchronous IO based application, it uses large thread pools of Play(Akka) to handle blocking IO. • Play 2.4.x: • Play 2.5.x: • The numbers depends on the server CPU performance ಉظతͳIOϕʔεͷΞϓϦέʔγϣϯͰ େ༰ྔͷεϨουϓʔϧ͕ඞཁͱͳΓ·͢ɻ ɾakka.actor.default-dispatcher.parallelism-factor ɾakka.actor.default-dispatcher.parallelism-min ɾakka.actor.default-dispatcher.parallelism-max The number of threads: parallelism-min <= CPUs * parallelism-factor <= parallelism-max ɾakka.actor.default-dispatcher.thread-pool-executor.fixed-pool-size ɹThe fixed pool size should be the maximum size of database connection pool plus extra
by New Relic • The library to access external APIs was implemented as non-blocking • Applying Future and create a specialized ThreadPool • Replace codes using Reflections into Scala Macros • We published it as OSS • Macrooom: Macro-based Object-Object Mapper • https://github.com/h-kishi/macrooom ͦͷଞͷ࣮ϨϕϧͰߦͬͨνϡʔχϯάʹ͍ͭͯͰ͢ɻ
h_kishi
yusuke57
handstandsam
moomooya
erukiti
ajstarks
freddi
mpilquist
tsuyosh
tombo
shikugawa
920oj
susanpotter
edds
3n
samlambert
cromwellryan
jlugia
bkeepers
destraynor
dougneiner
chriscoyier
shpigford
pedronauck
