[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫

ScalaMatsuriでの講演資料です。
http://scalamatsuri.org/

＜セッション内容＞
グリーでは今年6月にGREEチャットというサービスをリリースしました。
本発表では、GREEチャットのバックエンドをScalaで開発したときの利点や苦労した点についてお話させて頂きます。
トピックは以下の通りです。

-弊社で初めてScalaのプロダクトを導入する際の苦労
-数十万DAUでもリアルタイム性を維持するための工夫
-AkkaやFinagleなどのOSSライブラリを使った非同期処理、並列処理

グリー株式会社 長谷川 貴之&尾崎 俊

Transcript

1. Copyright © GREE, Inc. All Rights Reserved. The Trial and Error in Releasing GREE Chat Shun Ozaki, Takayuki Hasegawa Scala Matsuri2014 B-6 GREE's First Scala Product
2. Copyright © GREE, Inc. All Rights Reserved. • Shun Ozaki • @wozaki • Joined in April, 2013 • Android Application • Takayuki Hasegawa • @hase1031 • Joined in April, 2013 • NLP, Machine Learning Self Introduction 1/56
3. Copyright © GREE, Inc. All Rights Reserved. • Released at June 2 GREE Chat 2/56
4. Copyright © GREE, Inc. All Rights Reserved. How to build our system for hundreds of thousands daily users Agenda 3/56
5. Copyright © GREE, Inc. All Rights Reserved. Share our knowledge we got through the development of GREE Chat • Why Scala? • Team development • Decision of frameworks • Obstacles & workarounds Goal 4/56
6. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
7. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
8. Copyright © GREE, Inc. All Rights Reserved. Decide the Language based on the Requirement
9. Copyright © GREE, Inc. All Rights Reserved. • )VOESFET
   PG
   UIPVTBOET daily users • 3FBMUJNF response Ø Connect with streaming • Run on B
   TNBMM
   OVNCFS of servers Ø Utilize server resource effectively • Maintain for over 
   ZFBST Requirements 8/56
10. Copyright © GREE, Inc. All Rights Reserved. ◎ GREE uses PHP so heavily • Many libraries and know-how in GREE △ Streaming • #connections = #processes △ Single thread, multi process • Overhead of spawning an OS process △ Maintainability • Dynamic typing 9/56
11. Copyright © GREE, Inc. All Rights Reserved. ◎Compatible with concurrent programming ◎One process, many connections ◎High maintainability • Static typing • Functional programming (no side effects) ◎Stimulate new technology learning in GREE • Open to new languages & technologies! Scala! 10/56
12. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
13. Copyright © GREE, Inc. All Rights Reserved. After adopting Scala …
14. Copyright © GREE, Inc. All Rights Reserved. Only two Scala programmers in a team of seven Shortage of Scala experts 2 ／ 7 13/56
15. Copyright © GREE, Inc. All Rights Reserved. Only two Scala programmers in a team of seven We must build up our skill! Shortage of Scala experts 2 ／ 7 14/56
16. Copyright © GREE, Inc. All Rights Reserved. 1. Self study 2. Study club 3. Pair programming Learning of Scala 15/56
17. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Understand the basic syntax
18. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Books 17/56
19. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Documents by Twitter https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 18/56
20. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Source code from OSS 19/56
21. Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Share what we learned by ourselves Introduction to functional programming
22. Copyright © GREE, Inc. All Rights Reserved. • Each engineer solves the problems provided by Scala experts • e.g., Binary Tree, Fibonacci Sequence 2. Study Club 21/56
23. Copyright © GREE, Inc. All Rights Reserved. • Example of factorial generation algorithm 2. Study Club Discussion with Members Use var Use recursion 22/56
24. Copyright © GREE, Inc. All Rights Reserved. Use var Use recursion • Example of factorial generation algorithm 2. Study Club Discussion with Members No side effect 23/56
25. Copyright © GREE, Inc. All Rights Reserved. 3. Pair Programming Practice by using the knowledge obtained from study club
26. Copyright © GREE, Inc. All Rights Reserved. • Effective learning • Learn about symbols (e.g., +, @), which cannot be searched easily on the Internet • Supplement the knowledge not taught at study club • Complex syntaxes (e.g., implicit, currying) • Development tools (e.g., sbt, IntelliJ) 3. Pair Programming 25/56
27. Copyright © GREE, Inc. All Rights Reserved. • Reasons of selecting Scala • Compatible with concurrent programming • Maintainability because of static typing • Learning of Scala • Self Study, Study Club, Pair Programming • Big burdens on Scala experts • Needed for teaching team members • Maintain the quality of codes Summary: Adopting Scala 26/56
28. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
29. Copyright © GREE, Inc. All Rights Reserved. • Servers separated by Queues • API Server: Process users’ requests • EventBus Server: Process events in Queue • Stream Server: Supply event to connected users Architecture of Backend (simplified) 28/56
30. Copyright © GREE, Inc. All Rights Reserved. Role of Each Server and Framework
31. Copyright © GREE, Inc. All Rights Reserved. • Enqueue events • e.g., Send message Join/leave conversation • Techniques to process a lot of requests • Delegate heavy tasks (e.g., Disk I/O) to others • Logic is written to run asynchronously • scala.concurrent.Future API Server Process users’ requests 30/56
32. Copyright © GREE, Inc. All Rights Reserved. • RPC system for JVM based on Netty • Support us to write asynchronous logic • Used by large scale web services • e.g., Twitter, Tumblr, Foursquare, Pinterest • Equipped with various clients • Redis, Memcached • With retry policy, connection pools 31/56
33. Copyright © GREE, Inc. All Rights Reserved. • Examples of tasks • Logging • Inquire GREE internal system • Store events in DB • etc. • Problems of concurrent processing • Deadlock, race condition, … EventBus Server Process events in Queue 32/56
34. Copyright © GREE, Inc. All Rights Reserved. Framework to write concurrent, distributed logic more easily • No need to handle shared resources • Race condition, dead-lock • Logic separation • Parent-Child • Fault tolerance 33/56
35. Copyright © GREE, Inc. All Rights Reserved. Example of Akka 34/56
36. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Match by Event type 35/56
37. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Send copy to child 36/56
38. Copyright © GREE, Inc. All Rights Reserved. Example of Akka log 37/56
39. Copyright © GREE, Inc. All Rights Reserved. • Connect by streaming • Increase connections as much as possible • Keep users data in the memory to reduce I/O • Asynchronous I/O by Finagle, Akka • Task division by Akka • Supply events to users • Update users data • Send KeepAlive to users Stream Server Supply events to connected users 38/56
40. Copyright © GREE, Inc. All Rights Reserved. • Each server is separated by Queue • API, EventBus, Stream Server • Framework is selected by considering asynchronous processing • Finagle, Akka • Task is divided by Akka • Easy to write concurrent logic Summary: Architecture 39/56
41. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
42. Copyright © GREE, Inc. All Rights Reserved. 1.JVM • Full GC 2. Self-created Scala library • Sharding • ID Architecture Obstacles 41/56
43. Copyright © GREE, Inc. All Rights Reserved. Full GC Problems
44. Copyright © GREE, Inc. All Rights Reserved. • Scala runs on JVM • We want to prevent “stop the world” • GC in young generation is faster than full GC • Try not to let it store object in old generation Full GC Old generation Young generation 43/56
45. Copyright © GREE, Inc. All Rights Reserved. • Use method that has side effects • Reuse object everywhere • Forget to release used resources Causes of Uncollected Reference Problem 44/56
46. Copyright © GREE, Inc. All Rights Reserved. • Use method that has side effects • Reuse object everywhere • Forget to release used resources Difficult to recognize by developers Causes of Uncollected Reference Problem 45/56
47. Copyright © GREE, Inc. All Rights Reserved. • Use val variable (final variable) • Don’t use mutable variable • Don’t leave references to objects for too long Short-Lived Object Create new objects instead of updating old ones 46/56
48. Copyright © GREE, Inc. All Rights Reserved. Sharding: Scalable Storage 47/56
49. Copyright © GREE, Inc. All Rights Reserved. Scale up? 48/56
50. Copyright © GREE, Inc. All Rights Reserved. Scale out! 49/56
51. Copyright © GREE, Inc. All Rights Reserved. • Sharding is necessary! • Considering capacity and #accesses • We have Cascade, a sharding library • For PHP, not Scala • So, we created Aurora and make it OSS • Aurora and application were developed simultaneously • So difficult to integrate them during development • Code modification must be done in many parts Horizontal Partitioning 50/56
52. Copyright © GREE, Inc. All Rights Reserved. Ref︓MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ ID Architecture: UUID and InnoDB
53. Copyright © GREE, Inc. All Rights Reserved. Ref︓MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ UUID AUTO_INCREMENT #record #time
54. Copyright © GREE, Inc. All Rights Reserved. • At first, we used UUID as primary key • But this caused MySQL to insert randomly • Created a library to generate ID instead • Referred to Snowflake by Twitter • Sequential insertion based on sorted created time Drop UUID, Take Time-Based 53/56
55. Copyright © GREE, Inc. All Rights Reserved. • JVM • Use short-lived objects to avoid Full GC • Self-created Scala library • Sharding • Made Scala library OSS • Integration of library to application was difficult • ID Architecture • Adopt time-based ID for sequential insertion Summary: Obstacles & Workaround 54/56
56. Copyright © GREE, Inc. All Rights Reserved. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
57. Copyright © GREE, Inc. All Rights Reserved. • Reasons for selecting Scala • Compatible with concurrent programming • Plan Scala learning within the team • Learning cost is expensive • Architecture & Frameworks • Each server is separated by Queue • Finagle, Akka • Shortage of libraries • We must have made libraries by ourselves • Needed to pull request to OSS repository Summary 56/56
58. Copyright © GREE, Inc. All Rights Reserved. Thanks! @tomoyoshi_ogura, @kyo_ago, @takc923 @yoshie_777, @le_chang, @beketa, @j5ik2o and ScalaMatsuri staffs