国技と Scala (Japan's national sport and Scala)

Presentation at ScalaMatsuri '14.

Streaming here (Japanese).
http://live.nicovideo.jp/watch/lv191315006
# Start at 9:00〜

Transcript

1. ࠃٕ ͱ Scala Takuya Fujimura Dwango Mobile Co., LTD. @tlync Japan’s National Sport and Scala 2014 ScalaMatsuri
2. Sponsored  by
3. Sponsored  by http://info.dwango.co.jp/recruit/ Scala Ͱ࢓ࣄ͍ͨ͠? ͦΜͳਓ͸ͪ͜Βˣ ※ ࢲ·Ͱ͓੠͕͚͍ͩ͘͞ɻ੍͓͍͍͠౓͕… şƄŲƃşƄŲƃ
4. About Me • name: • Takuya Fujimura (@tlync) • job: • Tech. Director, Programmer etc… • Dwango Mobile Co., LTD. • like: • Wine, Beer, Cat, Programming • lang: • Java/Scala/JavaScript (Scheme/Swift) • Scala Newbie (< 1year)
5. ࠃٕ Japan’s National Suport ?
6. 国技 ! Yeah. You know. Japan’s National Suport !
7. 大相撲 Oh! SUMO https://www.flickr.com/photos/jondresner/3795899641/in/photolist-
8. Introduction SUMO ?
9. SUMO in Japan
10. SUMO in Swiss https://www.flickr.com/photos/asirap/3920327342/in/photolist-aVv7j6-bo6LEH-6YqGJJ-6YqGAA-7292HS-oxPiiK
11. SUMO & Scala ? https://www.flickr.com/photos/jondresner/3795899641/in/photolist-
12. The SUMO association’s Official iOS/Android App. Launch a Service 2014/03 http://sumo.dwango.jp/
15. Today’s talk • Share our • design decisions • impressions • practices
16. Architecture
17. MySQL   slave HDFS   ac6ve PHP Play   API Apache Play   Push HDFS   standby MySQL   master PHP iOS Android Memcached Play   API Apache Play   Push Memcached Play Apache api  02 LB LB Play Apache assets  01 assets  02 api  01 Legacy
 Backoffice  tool Shared  digital  assets   repository SUMO  DB Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video Torikumi   Video SUMO   data
18. Middleware • Apache 2.4 • MySQL 5.5 • Memcached 1.4 • HDFS 2.0 ※ shared
19. Framework/Libraries • Scala 2.10 • Framework … Play! 2.2.1 • Template Engine… twirl (w/ play) • DBAL … ScalalikeJDBC 1.7.4 • Testing … ScalaTest 2.0 • DI … Scaldi 0.2.2
20. Technical Functionality • SUMO Content REST API • Video Progressive Download • Install based Authentication • iOS/Android In-App Purchase/Billing Validation • Push Notification (Amazon SNS) etc…
21. Source Code • Scala … 12,539 lines • Templates … 116 files
22. Why we use Scala?
23. Why we use Scala? 1. Type-Safety 2. Less Code (than PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills
24. Why we use Scala? 1. Type-Safety 2. Less Code (than PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills
25. Type-Safety
26. Goodbye Runtime Error by language
27. Familiar Story A Service needs to keep evolving. But It’s hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost
28. Familiar Story A Service needs to keep evolving. But It’s hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost We need type-safety dev environment to keep evolving software
29. Improving our programming skills
30. Improving our programming skills • Scala is multi paradigm language • Scala has many concepts • Programmers can get many new perspectives OOP FP Trait Monad Object Typed Class Actor
31. Scala makes good programmer e.g: Just PHP Programmer > Scala Programmer Skill Learning OOP! Immutable! map/fold! Trait! Monad! OOP? Immutable? map/fold? Trait? Monad?
32. How we learned Scala? • Toy Project Try & Error in irresponsible project It was terrible code…. • Reading Circle Programming Scala 2nd Edition 1 hour / day in our office thx @daneko0123
33. Design
34. Design Concept Focus on “Operation Phase” Maintainability Reliability
35. Why?
36. This is SUMO asociation’s official app not only ourselves.
37. Initial Dev Time Operation Time < 平成二十二年 一月場所 平成二十三年 三月場所 平成二十四年 五月場所 平成二十五年 七月場所 平成二十六年 九月場所 平成二十七年 十一月場所
38. Design Decisions • Full type-safety environment • Domain Driven Design • Hybrid App (Web/Native) • Microservices
39. Full type-safety environment
40. Full type-safety environment Scala Lang Play & twirl Web & View ScalikeJDBC Persistence
41. Excellent!
42. Full type-safety environment makes great safety & productivity.
43. Except compilation time…
44. Except compilation time… Cons < Safety & Productivity
45. @rikishiProfile.kakuzuke.map { k => <a class="button" href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ</a>}
 </div>
 }
 </div>
 <h2>ϓϩϑΟʔϧৄࡉ</h2>
 <div class="profile-detail">
 <table class="profile-table">
 <tr>
 <th>࢛ވ໊</th>
 <td>@rikishiProfile.shikonaFull
 <br>
 (@rikishiProfile.shikonaYomiFull)
 </td>
 </tr>
 <tr>
 <th>൪෇</th>
 <td>@rikishiProfile.banzuke match {
 case Some(b) => {@b.fullName}
 case None => {൪෇֎}
 }
 </td>
 </tr>
 <tr>
 <th>ग़਎஍</th>
 <td>@rikishiProfile.birthplace</td>
 </tr>
 @rikishiProfile.heya.map { h =>
 <tr>
 Complex? Really?
46. @rikishiProfile.kakuzuke.map { k => <a class="button" href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ</a>}
 </div>
 }
 </div>
 <h2>ϓϩϑΟʔϧৄࡉ</h2>
 <div class="profile-detail">
 <table class="profile-table">
 <tr>
 <th>࢛ވ໊</th>
 <td>@rikishiProfile.shikonaFull
 <br>
 (@rikishiProfile.shikonaYomiFull)
 </td>
 </tr>
 <tr>
 <th>൪෇</th>
 <td>@rikishiProfile.banzuke match {
 case Some(b) => {@b.fullName}
 case None => {൪෇֎}
 }
 </td>
 </tr>
 <tr>
 <th>ग़਎஍</th>
 <td>@rikishiProfile.birthplace</td>
 </tr>
 @rikishiProfile.heya.map { h =>
 <tr>
 Complex? Really? Hatena’s designer is doing it, isn’t it!?
47. Design Decisions • Full Type-Safety environment • Domain Driven Design • Hybrid App (Web/Native) • Microservices
48. Design Decisions • Domain Driven Design • Mental Model = Language = Communication = Codes • Make Understandability & Clarity & Extensibility for SUMO specific specifications as code • Hybrid App (Web/Native) • Change content without AppStore/Google Play • Microservices • Separate concerns like PUSH notification.
49. ./domain ./domain ├── auth │ ├── AccessToken.scala │ ├── Device.scala │ ├── RequestToken.scala │ ├── User.scala │ └── UserRepository.scala ├── honbasho │ ├── FusenType.scala │ ├── JudgeType.scala │ ├── banzuke │ │ ├── Banzuke.scala │ │ └── BanzukeRepository.scala │ ├── basho │ │ ├── Basho.scala │ │ └── BashoRepository.scala │ ├── hoshitori │ │ ├── CareerRecord.scala │ │ ├── Hoshitori.scala │ │ ├── HoshitoriHyo.scala │ │ ├── HoshitoriHyoRepository.scala Domain?
50. 
 class 
 object … 
 String kakuzuke 
 番付
51. object Kakuzuke { ! ! ! ! ! ! 
 …
 
 def all: Seq[Kakuzuke] = Seq(Makuuchi, Juryo, Makushi Sandanme, Jonidan, Jonokuchi)
 }
 
 sealed abstract class Kakuzuke(val id: Int, val name: S object Kakuzuke {
 case object Makuuchi extends Kakuzuke(1, "ນ಺")
 case object Juryo extends Kakuzuke(2, "े྆")
 case object Makushita extends Kakuzuke(3, "ນԼ")
 case object Sandanme extends Kakuzuke(4, "ࡾஈ໨")
 case object Jonidan extends Kakuzuke(5, "ংೋஈ")
 case object Jonokuchi extends Kakuzuke(6, "ংϊޱ") 格付
52. class RikishiProfile(
 id: Int,
 shikona: String,
 shikonaYomi: String,
 birthplace: String,
 thumbnail: Option[RikishiThumbnail],
 val shikonaFull: String,
 val shikonaYomiFull: String,
 val heya: Option[Heya],
 val realName: String,
 val birthDay: LocalDate,
 val age: Int,
 val height: Float,
 val weight: Float,
 val banzuke: Option[Banzuke],
 val kakuzuke: Option[Kakuzuke],
 val prizeWinningStats: PrizeWinningStats) extends RikishiBaseProfile(
 id,
 shikona,
 shikonaYomi,
 力士
53. Hybrid App
55. Web Na&ve
56. Web Na&ve /api/v1/basho/xxx/summary /api/v1/basho accecpt:  applica0on/json accecpt:text/html
58. Web Na&ve
59. Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-­‐info accecpt:  applica0on/json accecpt:text/html
60. Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-­‐info accecpt:  applica0on/json accecpt:text/html Usability & Maintainability
61. Microservices
62. Microservices Play   SUMO  REST  API Play   Push  No&fica&on  API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001
63. Microservices Play   SUMO  REST  API Play   Push  No&fica&on  API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001 Generic   Push  No&fca&on  Service SUMO   Domain SUMO   Domain
64. Microservices Play   SUMO  REST  API Play   Push  No&fica&on  API Old  PHP   Backoffice  Tool Amazon  SNS hEp:9000 hEp:9001 hEp:9001 Generic   Push  No&fca&on  Service SUMO   Domain SUMO   Domain Isolate generic functionality
65. Library Impressions
66. Library Impressions • ScalikeJDBC … DB Access • twirl … Template Engine • Scaldi … DI • ScalaTest … Testing
67. ScalikeJDBC • Good • Just SQL … Avoid unnecessary abstraction. Good for existing system. • Type-Safe … QueryDSL • Bad • Development resource? → NO!!!! • ConnectionPoolSetting is not flexible (… well, we can submit pull request) • Note • Slick is looks like cool. But we don’t want to translate complex SQL queries to Scala code.
68. 大相撲 powered by ScalikeJDBC!
69. Twirl • Excellent! • Type-Safety… Prevent many errors at compile time. • Don’t be afraid to change method/property signature !!! • Good • Flexibility … Just Scala • Bad • Flexibility … Just Scala, Unfriendly for designers • Slow Feedback Cycle … Slow compilation… We developed initial templates without twirl using Grunt etc. • Note • I think better of this than I did before
70. Twirl • Excellent! • Type-Safety… Prevent many errors at compile time. • Don’t be afraid to change method/property signature !!! • Good • Flexibility … Just Scala • Bad • Flexibility … Just Scala, Unfriendly for designers • Slow Feedback Cycle … Slow compilation… We developed initial templates without twirl using Grunt etc. • Note • I think better of this than I did before Should be evaluated!?
71. Scaldi • Good • minimum & simple • Bad • Verbose declaration ! • Dynamic … You won’t get an error until you run your application • Note • The cake pattern seems quite verbose for me. • Next, I’ll look for other solution. (like Play! 2.4 DI) class FooService(implicit inj: Injector) extends Injectable Fixed after presentation: use constructor injection! instead of this. It’s simple!
72. ScalaTest • Good • Simple … vs Specs2. Specs2 provides several notations that seems not simple for me. • Bad • Limited structuring texts … vs RSpec. No context etc… • Note • Well… Either way
73. Practices
74. Practices • Exception Handling • Non-Stop Deployment • Static Code Analysis • Tuning Akka for Blocking App
75. Exception Handling
76. Scala provides several ways handling exceptions. Problem Exception Handling Try Option Either try catch (Validation/scalaz)
77. Scala provides several ways handling exceptions. Problem Exception Handling Try Option Either try catch (Validation/scalaz) Which is Scala way?
79. old Scala 2.10 < Scala 2.10 < old
81. old Scala 2.10 < old old
82. class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Either[InvalidTokenException, AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Either with Exception?
83. class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Try[AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Nope. Use Try
84. I don’t know Scala way… But our policy is below. try catch Option Try Either (Validation,Either etc../scalaz) No explanation needed Do not use generally (Java world syntax) Use this for exception handling! < Scala 2.10 Use error handling except exceptions. …
85. Caution Keep learning new grammars … Scala is growing.
86. Static Code Analysis
87. Problem Static Code Analysis [IMO] In 2013, Many unmaintained static code analysis libraries. such as SCCT, sbt-scct…
88. Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344
89. Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344 ?
90. Use SCoverage now Solution? Static Code Analysis https://github.com/scoverage/sbt-scoverage https://github.com/scoverage/scalac-scoverage-plugin
91. In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance
92. In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance
93. Others Static Code Analysis • ScalaStyle • Examines your Scala code and indicates potential problems with it. • Scapegoat • Scala static code analyzer • WartRemover • Flexible Scala code linting tool • cpd4sbt • Integrating PMD's Copy/Paste Detector • Scalariform (not analysis) • Source code formatting
94. Non-Stop Deployment
95. Play! needs to stop when deployment basically. Stop service n sec = LB’s Heartbeat Problem Non-Stop Deployment
96. 1. Use Load Balancer API 2. Implement Server Status API /w LB 3. Pray to implement graceful restart in the future ;p 4. Others… Do you have any ideas? Solution? Non-Stop Deployment
97. 1. Use Load Balancer API 2. Implement Server Status API /w LB 3. Continue to pray to implement graceful restart in the future 4. Others… Do you have any idea? Solution? Non-Stop Deployment
98. Deployment Flow Non-Stop Deployment 1. Remove Node from LB via LB API ↓ 2. Deploy to detached Node new app. ↓ 3. Waiting for start ↓ 4. Add Node to LB via LB API ↓ 5. Repeat… !
99. @task @roles("app") def _deploy_main_job(branch, working_tree, dist_skip, cache): ! developer_mode = working_tree or dist_skip ! # on local prepare_remote() ! if not dist_skip: package(cache) ! compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer() time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() !
100. compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer() time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() ! clean_old_releases() ! @task @roles("app") def disconnect_loadblancer(): local(“/usr/local/bin/lb {0} disable".format(env.host)) ! @task @roles("app") def reconnect_loadblancer(): local("/usr/local/bin/lb {0} enable".format(env.host))
101. Graceful Restart? Non-Stop Deployment
102. Tuning Thread Pool
103. Caution Tuning Thread Pool The Play! default configuration is optimized for Non-Blocking I/O
104. Caution Tuning Thread Pool https://www.playframework.com/documentation/2.3.x/ThreadPools
105. • Pure asynchronous • for Non-Blocking app • Highly synchronous • for Blocking app • Many specific thread pools • Hybrid • Few specific thread pools • Hybrid Tuning Practices Tuning hread Pool
106. e.g: Highly Synchronous Tuning Thread Pool
107. e.g: Few specific thread pool Tuning Thread Pool
108. Summary
109. • SUMO app is powered by Scala/Play! • Type-Safety world makes great safety & productivity. • Scala is suited for long-maintained or middle-large applications. • I worry about binary compatibility Summary
110. finally..
111. ! Scala is already practical. From now on, let’s share our practices, codes… more to keep evolving Scala ecosystem & community.
112. Let’s Enjoy Scala !!
113. ! thank you • Scala developers • ScalaMatsuri staffs • Sumo project members & supporters • everybody