国技と Scala (Japan's national sport and Scala)

B6b32d5d2a05b7cebaea86bdf3e95077?s=47 Takuya Fujimura
September 06, 2014

国技と Scala (Japan's national sport and Scala)

Presentation at ScalaMatsuri '14.

Streaming here (Japanese).
http://live.nicovideo.jp/watch/lv191315006
# Start at 9:00〜

B6b32d5d2a05b7cebaea86bdf3e95077?s=128

Takuya Fujimura

September 06, 2014
Tweet

Transcript

  1. ࠃٕ ͱ Scala Takuya Fujimura Dwango Mobile Co., LTD. @tlync

    Japan’s National Sport and Scala 2014 ScalaMatsuri
  2. Sponsored  by

  3. Sponsored  by http://info.dwango.co.jp/recruit/ Scala Ͱ࢓ࣄ͍ͨ͠? ͦΜͳਓ͸ͪ͜Βˣ ※ ࢲ·Ͱ͓੠͕͚͍ͩ͘͞ɻ੍͓͍͍͠౓͕… şƄŲƃşƄŲƃ

  4. About Me • name: • Takuya Fujimura (@tlync) • job:

    • Tech. Director, Programmer etc… • Dwango Mobile Co., LTD. • like: • Wine, Beer, Cat, Programming • lang: • Java/Scala/JavaScript (Scheme/Swift) • Scala Newbie (< 1year)
  5. ࠃٕ Japan’s National Suport ?

  6. 国技 ! Yeah. You know. Japan’s National Suport !

  7. 大相撲 Oh! SUMO https://www.flickr.com/photos/jondresner/3795899641/in/photolist-

  8. Introduction SUMO ?

  9. SUMO in Japan

  10. SUMO in Swiss https://www.flickr.com/photos/asirap/3920327342/in/photolist-aVv7j6-bo6LEH-6YqGJJ-6YqGAA-7292HS-oxPiiK

  11. SUMO & Scala ? https://www.flickr.com/photos/jondresner/3795899641/in/photolist-

  12. The SUMO association’s Official iOS/Android App. Launch a Service 2014/03

    http://sumo.dwango.jp/
  13. None
  14. None
  15. Today’s talk • Share our • design decisions • impressions

    • practices
  16. Architecture

  17. MySQL   slave HDFS   ac6ve PHP Play   API

    Apache Play   Push HDFS   standby MySQL   master PHP iOS Android Memcached Play   API Apache Play   Push Memcached Play Apache api  02 LB LB Play Apache assets  01 assets  02 api  01 Legacy
 Backoffice  tool Shared  digital  assets   repository SUMO  DB Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video SUMO   data
  18. Middleware • Apache 2.4 • MySQL 5.5 • Memcached 1.4

    • HDFS 2.0 ※ shared
  19. Framework/Libraries • Scala 2.10 • Framework … Play! 2.2.1 •

    Template Engine… twirl (w/ play) • DBAL … ScalalikeJDBC 1.7.4 • Testing … ScalaTest 2.0 • DI … Scaldi 0.2.2
  20. Technical Functionality • SUMO Content REST API • Video Progressive

    Download • Install based Authentication • iOS/Android In-App Purchase/Billing Validation • Push Notification (Amazon SNS) etc…
  21. Source Code • Scala … 12,539 lines • Templates …

    116 files
  22. Why we use Scala?

  23. Why we use Scala? 1. Type-Safety 2. Less Code (than

    PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills
  24. Why we use Scala? 1. Type-Safety 2. Less Code (than

    PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills
  25. Type-Safety

  26. Goodbye Runtime Error by language

  27. Familiar Story A Service needs to keep evolving. But It’s

    hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost
  28. Familiar Story A Service needs to keep evolving. But It’s

    hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost We need type-safety dev environment to keep evolving software
  29. Improving our programming skills

  30. Improving our programming skills • Scala is multi paradigm language

    • Scala has many concepts • Programmers can get many new perspectives OOP FP Trait Monad Object Typed Class Actor
  31. Scala makes good programmer e.g: Just PHP Programmer > Scala

    Programmer Skill Learning OOP! Immutable! map/fold! Trait! Monad! OOP? Immutable? map/fold? Trait? Monad?
  32. How we learned Scala? • Toy Project Try & Error

    in irresponsible project It was terrible code…. • Reading Circle Programming Scala 2nd Edition 1 hour / day in our office thx @daneko0123
  33. Design

  34. Design Concept Focus on “Operation Phase” Maintainability Reliability

  35. Why?

  36. This is SUMO asociation’s official app not only ourselves.

  37. Initial Dev Time Operation Time < 平成二十二年 一月場所 平成二十三年 三月場所

    平成二十四年 五月場所 平成二十五年 七月場所 平成二十六年 九月場所 平成二十七年 十一月場所
  38. Design Decisions • Full type-safety environment • Domain Driven Design

    • Hybrid App (Web/Native) • Microservices
  39. Full type-safety environment

  40. Full type-safety environment Scala Lang Play & twirl Web &

    View ScalikeJDBC Persistence
  41. Excellent!

  42. Full type-safety environment makes great safety & productivity.

  43. Except compilation time…

  44. Except compilation time… Cons < Safety & Productivity

  45. @rikishiProfile.kakuzuke.map { k => <a class="button" href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ</a>}
 </div>
 }


    </div>
 <h2>ϓϩϑΟʔϧৄࡉ</h2>
 <div class="profile-detail">
 <table class="profile-table">
 <tr>
 <th>࢛ވ໊</th>
 <td>@rikishiProfile.shikonaFull
 <br>
 (@rikishiProfile.shikonaYomiFull)
 </td>
 </tr>
 <tr>
 <th>൪෇</th>
 <td>@rikishiProfile.banzuke match {
 case Some(b) => {@b.fullName}
 case None => {൪෇֎}
 }
 </td>
 </tr>
 <tr>
 <th>ग़਎஍</th>
 <td>@rikishiProfile.birthplace</td>
 </tr>
 @rikishiProfile.heya.map { h =>
 <tr>
 Complex? Really?
  46. @rikishiProfile.kakuzuke.map { k => <a class="button" href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ</a>}
 </div>
 }


    </div>
 <h2>ϓϩϑΟʔϧৄࡉ</h2>
 <div class="profile-detail">
 <table class="profile-table">
 <tr>
 <th>࢛ވ໊</th>
 <td>@rikishiProfile.shikonaFull
 <br>
 (@rikishiProfile.shikonaYomiFull)
 </td>
 </tr>
 <tr>
 <th>൪෇</th>
 <td>@rikishiProfile.banzuke match {
 case Some(b) => {@b.fullName}
 case None => {൪෇֎}
 }
 </td>
 </tr>
 <tr>
 <th>ग़਎஍</th>
 <td>@rikishiProfile.birthplace</td>
 </tr>
 @rikishiProfile.heya.map { h =>
 <tr>
 Complex? Really? Hatena’s designer is doing it, isn’t it!?
  47. Design Decisions • Full Type-Safety environment • Domain Driven Design

    • Hybrid App (Web/Native) • Microservices
  48. Design Decisions • Domain Driven Design • Mental Model =

    Language = Communication = Codes • Make Understandability & Clarity & Extensibility for SUMO specific specifications as code • Hybrid App (Web/Native) • Change content without AppStore/Google Play • Microservices • Separate concerns like PUSH notification.
  49. ./domain ./domain ├── auth │ ├── AccessToken.scala │ ├── Device.scala

    │ ├── RequestToken.scala │ ├── User.scala │ └── UserRepository.scala ├── honbasho │ ├── FusenType.scala │ ├── JudgeType.scala │ ├── banzuke │ │ ├── Banzuke.scala │ │ └── BanzukeRepository.scala │ ├── basho │ │ ├── Basho.scala │ │ └── BashoRepository.scala │ ├── hoshitori │ │ ├── CareerRecord.scala │ │ ├── Hoshitori.scala │ │ ├── HoshitoriHyo.scala │ │ ├── HoshitoriHyoRepository.scala Domain?
  50. 
 class 
 object … 
 String kakuzuke 
 番付

  51. object Kakuzuke { ! ! ! ! ! ! 


    …
 
 def all: Seq[Kakuzuke] = Seq(Makuuchi, Juryo, Makushi Sandanme, Jonidan, Jonokuchi)
 }
 
 sealed abstract class Kakuzuke(val id: Int, val name: S object Kakuzuke {
 case object Makuuchi extends Kakuzuke(1, "ນ಺")
 case object Juryo extends Kakuzuke(2, "े྆")
 case object Makushita extends Kakuzuke(3, "ນԼ")
 case object Sandanme extends Kakuzuke(4, "ࡾஈ໨")
 case object Jonidan extends Kakuzuke(5, "ংೋஈ")
 case object Jonokuchi extends Kakuzuke(6, "ংϊޱ") 格付
  52. class RikishiProfile(
 id: Int,
 shikona: String,
 shikonaYomi: String,
 birthplace: String,


    thumbnail: Option[RikishiThumbnail],
 val shikonaFull: String,
 val shikonaYomiFull: String,
 val heya: Option[Heya],
 val realName: String,
 val birthDay: LocalDate,
 val age: Int,
 val height: Float,
 val weight: Float,
 val banzuke: Option[Banzuke],
 val kakuzuke: Option[Kakuzuke],
 val prizeWinningStats: PrizeWinningStats) extends RikishiBaseProfile(
 id,
 shikona,
 shikonaYomi,
 力士
  53. Hybrid App

  54. None
  55. Web Na&ve

  56. Web Na&ve /api/v1/basho/xxx/summary /api/v1/basho accecpt:  applica0on/json accecpt:text/html

  57. None
  58. Web Na&ve

  59. Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-­‐info accecpt:  applica0on/json accecpt:text/html

  60. Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-­‐info accecpt:  applica0on/json accecpt:text/html Usability & Maintainability

  61. Microservices

  62. Microservices Play   SUMO  REST  API Play   Push  No&fica&on

     API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001
  63. Microservices Play   SUMO  REST  API Play   Push  No&fica&on

     API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001 Generic   Push  No&fca&on  Service SUMO   Domain SUMO   Domain
  64. Microservices Play   SUMO  REST  API Play   Push  No&fica&on

     API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001 Generic   Push  No&fca&on  Service SUMO   Domain SUMO   Domain Isolate generic functionality
  65. Library Impressions

  66. Library Impressions • ScalikeJDBC … DB Access • twirl …

    Template Engine • Scaldi … DI • ScalaTest … Testing
  67. ScalikeJDBC • Good • Just SQL … Avoid unnecessary abstraction.

    Good for existing system. • Type-Safe … QueryDSL • Bad • Development resource? → NO!!!! • ConnectionPoolSetting is not flexible (… well, we can submit pull request) • Note • Slick is looks like cool. But we don’t want to translate complex SQL queries to Scala code.
  68. 大相撲 powered by ScalikeJDBC!

  69. Twirl • Excellent! • Type-Safety… Prevent many errors at compile

    time. • Don’t be afraid to change method/property signature !!! • Good • Flexibility … Just Scala • Bad • Flexibility … Just Scala, Unfriendly for designers • Slow Feedback Cycle … Slow compilation… We developed initial templates without twirl using Grunt etc. • Note • I think better of this than I did before
  70. Twirl • Excellent! • Type-Safety… Prevent many errors at compile

    time. • Don’t be afraid to change method/property signature !!! • Good • Flexibility … Just Scala • Bad • Flexibility … Just Scala, Unfriendly for designers • Slow Feedback Cycle … Slow compilation… We developed initial templates without twirl using Grunt etc. • Note • I think better of this than I did before Should be evaluated!?
  71. Scaldi • Good • minimum & simple • Bad •

    Verbose declaration ! • Dynamic … You won’t get an error until you run your application • Note • The cake pattern seems quite verbose for me. • Next, I’ll look for other solution. (like Play! 2.4 DI) class FooService(implicit inj: Injector) extends Injectable Fixed after presentation: use constructor injection! instead of this. It’s simple!
  72. ScalaTest • Good • Simple … vs Specs2. Specs2 provides

    several notations that seems not simple for me. • Bad • Limited structuring texts … vs RSpec. No context etc… • Note • Well… Either way
  73. Practices

  74. Practices • Exception Handling • Non-Stop Deployment • Static Code

    Analysis • Tuning Akka for Blocking App
  75. Exception Handling

  76. Scala provides several ways handling exceptions. Problem Exception Handling Try

    Option Either try catch (Validation/scalaz)
  77. Scala provides several ways handling exceptions. Problem Exception Handling Try

    Option Either try catch (Validation/scalaz) Which is Scala way?
  78. None
  79. old Scala 2.10 < Scala 2.10 < old

  80. None
  81. old Scala 2.10 < old old

  82. class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging

    with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Either[InvalidTokenException, AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Either with Exception?
  83. class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging

    with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Try[AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Nope. Use Try
  84. I don’t know Scala way… But our policy is below.

    try catch Option Try Either (Validation,Either etc../scalaz) No explanation needed Do not use generally (Java world syntax) Use this for exception handling! < Scala 2.10 Use error handling except exceptions. …
  85. Caution Keep learning new grammars … Scala is growing.

  86. Static Code Analysis

  87. Problem Static Code Analysis [IMO] In 2013, Many unmaintained static

    code analysis libraries. such as SCCT, sbt-scct…
  88. Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344

  89. Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344 ?

  90. Use SCoverage now Solution? Static Code Analysis https://github.com/scoverage/sbt-scoverage https://github.com/scoverage/scalac-scoverage-plugin

  91. In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance

  92. In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance

  93. Others Static Code Analysis • ScalaStyle • Examines your Scala

    code and indicates potential problems with it. • Scapegoat • Scala static code analyzer • WartRemover • Flexible Scala code linting tool • cpd4sbt • Integrating PMD's Copy/Paste Detector • Scalariform (not analysis) • Source code formatting
  94. Non-Stop Deployment

  95. Play! needs to stop when deployment basically. Stop service n

    sec = LB’s Heartbeat Problem Non-Stop Deployment
  96. 1. Use Load Balancer API 2. Implement Server Status API

    /w LB 3. Pray to implement graceful restart in the future ;p 4. Others… Do you have any ideas? Solution? Non-Stop Deployment
  97. 1. Use Load Balancer API 2. Implement Server Status API

    /w LB 3. Continue to pray to implement graceful restart in the future 4. Others… Do you have any idea? Solution? Non-Stop Deployment
  98. Deployment Flow Non-Stop Deployment 1. Remove Node from LB via

    LB API ↓ 2. Deploy to detached Node new app. ↓ 3. Waiting for start ↓ 4. Add Node to LB via LB API ↓ 5. Repeat… !
  99. @task @roles("app") def _deploy_main_job(branch, working_tree, dist_skip, cache): ! developer_mode =

    working_tree or dist_skip ! # on local prepare_remote() ! if not dist_skip: package(cache) ! compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer() time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() !
  100. compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer()

    time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() ! clean_old_releases() ! @task @roles("app") def disconnect_loadblancer(): local(“/usr/local/bin/lb {0} disable".format(env.host)) ! @task @roles("app") def reconnect_loadblancer(): local("/usr/local/bin/lb {0} enable".format(env.host))
  101. Graceful Restart? Non-Stop Deployment

  102. Tuning Thread Pool

  103. Caution Tuning Thread Pool The Play! default configuration is optimized

    for Non-Blocking I/O
  104. Caution Tuning Thread Pool https://www.playframework.com/documentation/2.3.x/ThreadPools

  105. • Pure asynchronous • for Non-Blocking app • Highly synchronous

    • for Blocking app • Many specific thread pools • Hybrid • Few specific thread pools • Hybrid Tuning Practices Tuning hread Pool
  106. e.g: Highly Synchronous Tuning Thread Pool

  107. e.g: Few specific thread pool Tuning Thread Pool

  108. Summary

  109. • SUMO app is powered by Scala/Play! • Type-Safety world

    makes great safety & productivity. • Scala is suited for long-maintained or middle-large applications. • I worry about binary compatibility Summary
  110. finally..

  111. ! Scala is already practical. From now on, let’s share

    our practices, codes… more to keep evolving Scala ecosystem & community.
  112. Let’s Enjoy Scala !!

  113. ! thank you • Scala developers • ScalaMatsuri staffs •

    Sumo project members & supporters • everybody