Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開まて...

gree_tech
September 14, 2014

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開までの苦労と工夫

ScalaMatsuriでの講演資料です。
http://scalamatsuri.org/

<セッション内容>
グリーでは今年6月にGREEチャットというサービスをリリースしました。
本発表では、GREEチャットのバックエンドをScalaで開発したときの利点や苦労した点についてお話させて頂きます。
トピックは以下の通りです。

-弊社で初めてScalaのプロダクトを導入する際の苦労
-数十万DAUでもリアルタイム性を維持するための工夫
-AkkaやFinagleなどのOSSライブラリを使った非同期処理、並列処理

グリー株式会社 長谷川 貴之&尾崎 俊

gree_tech

September 14, 2014
Tweet

More Decks by gree_tech

Other Decks in Technology

Transcript

  1. Copyright © GREE, Inc. All Rights Reserved. The Trial and

    Error in Releasing GREE Chat Shun Ozaki, Takayuki Hasegawa Scala Matsuri2014 B-6 GREE's First Scala Product
  2. Copyright © GREE, Inc. All Rights Reserved. • Shun Ozaki

    • @wozaki • Joined in April, 2013 • Android Application • Takayuki Hasegawa • @hase1031 • Joined in April, 2013 • NLP, Machine Learning Self Introduction 1/56
  3. Copyright © GREE, Inc. All Rights Reserved. How to build

    our system for hundreds of thousands daily users Agenda 3/56
  4. Copyright © GREE, Inc. All Rights Reserved. Share our knowledge

    we got through the development of GREE Chat • Why Scala? • Team development • Decision of frameworks • Obstacles & workarounds Goal 4/56
  5. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  6. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  7. Copyright © GREE, Inc. All Rights Reserved. • )VOESFETPGUIPVTBOET daily

    users • 3FBMUJNF response Ø Connect with streaming • Run on BTNBMMOVNCFS of servers Ø Utilize server resource effectively • Maintain for over ZFBST Requirements 8/56
  8. Copyright © GREE, Inc. All Rights Reserved. ◎ GREE uses

    PHP so heavily • Many libraries and know-how in GREE △ Streaming • #connections = #processes △ Single thread, multi process • Overhead of spawning an OS process △ Maintainability • Dynamic typing 9/56
  9. Copyright © GREE, Inc. All Rights Reserved. ◎Compatible with concurrent

    programming ◎One process, many connections ◎High maintainability • Static typing • Functional programming (no side effects) ◎Stimulate new technology learning in GREE • Open to new languages & technologies! Scala! 10/56
  10. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  11. Copyright © GREE, Inc. All Rights Reserved. Only two Scala

    programmers in a team of seven Shortage of Scala experts 2 / 7 13/56
  12. Copyright © GREE, Inc. All Rights Reserved. Only two Scala

    programmers in a team of seven We must build up our skill! Shortage of Scala experts 2 / 7 14/56
  13. Copyright © GREE, Inc. All Rights Reserved. 1. Self study

    2. Study club 3. Pair programming Learning of Scala 15/56
  14. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study

    Documents by Twitter https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 18/56
  15. Copyright © GREE, Inc. All Rights Reserved. 2. Study Club

    Share what we learned by ourselves Introduction to functional programming
  16. Copyright © GREE, Inc. All Rights Reserved. • Each engineer

    solves the problems provided by Scala experts • e.g., Binary Tree, Fibonacci Sequence 2. Study Club 21/56
  17. Copyright © GREE, Inc. All Rights Reserved. • Example of

    factorial generation algorithm 2. Study Club Discussion with Members Use var Use recursion 22/56
  18. Copyright © GREE, Inc. All Rights Reserved. Use var Use

    recursion • Example of factorial generation algorithm 2. Study Club Discussion with Members No side effect 23/56
  19. Copyright © GREE, Inc. All Rights Reserved. 3. Pair Programming

    Practice by using the knowledge obtained from study club
  20. Copyright © GREE, Inc. All Rights Reserved. • Effective learning

    • Learn about symbols (e.g., +, @), which cannot be searched easily on the Internet • Supplement the knowledge not taught at study club • Complex syntaxes (e.g., implicit, currying) • Development tools (e.g., sbt, IntelliJ) 3. Pair Programming 25/56
  21. Copyright © GREE, Inc. All Rights Reserved. • Reasons of

    selecting Scala • Compatible with concurrent programming • Maintainability because of static typing • Learning of Scala • Self Study, Study Club, Pair Programming • Big burdens on Scala experts • Needed for teaching team members • Maintain the quality of codes Summary: Adopting Scala 26/56
  22. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  23. Copyright © GREE, Inc. All Rights Reserved. • Servers separated

    by Queues • API Server: Process users’ requests • EventBus Server: Process events in Queue • Stream Server: Supply event to connected users Architecture of Backend (simplified) 28/56
  24. Copyright © GREE, Inc. All Rights Reserved. • Enqueue events

    • e.g., Send message Join/leave conversation • Techniques to process a lot of requests • Delegate heavy tasks (e.g., Disk I/O) to others • Logic is written to run asynchronously • scala.concurrent.Future API Server Process users’ requests 30/56
  25. Copyright © GREE, Inc. All Rights Reserved. • RPC system

    for JVM based on Netty • Support us to write asynchronous logic • Used by large scale web services • e.g., Twitter, Tumblr, Foursquare, Pinterest • Equipped with various clients • Redis, Memcached • With retry policy, connection pools 31/56
  26. Copyright © GREE, Inc. All Rights Reserved. • Examples of

    tasks • Logging • Inquire GREE internal system • Store events in DB • etc. • Problems of concurrent processing • Deadlock, race condition, … EventBus Server Process events in Queue 32/56
  27. Copyright © GREE, Inc. All Rights Reserved. Framework to write

    concurrent, distributed logic more easily • No need to handle shared resources • Race condition, dead-lock • Logic separation • Parent-Child • Fault tolerance 33/56
  28. Copyright © GREE, Inc. All Rights Reserved. • Connect by

    streaming • Increase connections as much as possible • Keep users data in the memory to reduce I/O • Asynchronous I/O by Finagle, Akka • Task division by Akka • Supply events to users • Update users data • Send KeepAlive to users Stream Server Supply events to connected users 38/56
  29. Copyright © GREE, Inc. All Rights Reserved. • Each server

    is separated by Queue • API, EventBus, Stream Server • Framework is selected by considering asynchronous processing • Finagle, Akka • Task is divided by Akka • Easy to write concurrent logic Summary: Architecture 39/56
  30. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  31. Copyright © GREE, Inc. All Rights Reserved. 1.JVM • Full

    GC 2. Self-created Scala library • Sharding • ID Architecture Obstacles 41/56
  32. Copyright © GREE, Inc. All Rights Reserved. • Scala runs

    on JVM • We want to prevent “stop the world” • GC in young generation is faster than full GC • Try not to let it store object in old generation Full GC Old generation Young generation 43/56
  33. Copyright © GREE, Inc. All Rights Reserved. • Use method

    that has side effects • Reuse object everywhere • Forget to release used resources Causes of Uncollected Reference Problem 44/56
  34. Copyright © GREE, Inc. All Rights Reserved. • Use method

    that has side effects • Reuse object everywhere • Forget to release used resources Difficult to recognize by developers Causes of Uncollected Reference Problem 45/56
  35. Copyright © GREE, Inc. All Rights Reserved. • Use val

    variable (final variable) • Don’t use mutable variable • Don’t leave references to objects for too long Short-Lived Object Create new objects instead of updating old ones 46/56
  36. Copyright © GREE, Inc. All Rights Reserved. • Sharding is

    necessary! • Considering capacity and #accesses • We have Cascade, a sharding library • For PHP, not Scala • So, we created Aurora and make it OSS • Aurora and application were developed simultaneously • So difficult to integrate them during development • Code modification must be done in many parts Horizontal Partitioning 50/56
  37. Copyright © GREE, Inc. All Rights Reserved. Ref︓MySQL InnoDB Primary

    Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ ID Architecture: UUID and InnoDB
  38. Copyright © GREE, Inc. All Rights Reserved. Ref︓MySQL InnoDB Primary

    Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ UUID AUTO_INCREMENT #record #time
  39. Copyright © GREE, Inc. All Rights Reserved. • At first,

    we used UUID as primary key • But this caused MySQL to insert randomly • Created a library to generate ID instead • Referred to Snowflake by Twitter • Sequential insertion based on sorted created time Drop UUID, Take Time-Based 53/56
  40. Copyright © GREE, Inc. All Rights Reserved. • JVM •

    Use short-lived objects to avoid Full GC • Self-created Scala library • Sharding • Made Scala library OSS • Integration of library to application was difficult • ID Architecture • Adopt time-based ID for sequential insertion Summary: Obstacles & Workaround 54/56
  41. Copyright © GREE, Inc. All Rights Reserved. • Reason of

    Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Outline
  42. Copyright © GREE, Inc. All Rights Reserved. • Reasons for

    selecting Scala • Compatible with concurrent programming • Plan Scala learning within the team • Learning cost is expensive • Architecture & Frameworks • Each server is separated by Queue • Finagle, Akka • Shortage of libraries • We must have made libraries by ourselves • Needed to pull request to OSS repository Summary 56/56
  43. Copyright © GREE, Inc. All Rights Reserved. Thanks! @tomoyoshi_ogura, @kyo_ago,

    @takc923 @yoshie_777, @le_chang, @beketa, @j5ik2o and ScalaMatsuri staffs