$30 off During Our Annual Pro Sale. View Details »

国技と Scala (Japan's national sport and Scala)

国技と Scala (Japan's national sport and Scala)

Presentation at ScalaMatsuri '14.

Streaming here (Japanese).
http://live.nicovideo.jp/watch/lv191315006
# Start at 9:00〜

Takuya Fujimura

September 06, 2014
Tweet

More Decks by Takuya Fujimura

Other Decks in Programming

Transcript

  1. ࠃٕ ͱ
    Scala
    Takuya Fujimura
    Dwango Mobile Co., LTD.
    @tlync
    Japan’s National Sport and Scala
    2014 ScalaMatsuri

    View Slide

  2. Sponsored  by

    View Slide

  3. Sponsored  by
    http://info.dwango.co.jp/recruit/
    Scala Ͱ࢓ࣄ͍ͨ͠? ͦΜͳਓ͸ͪ͜Βˣ
    ※ ࢲ·Ͱ͓੠͕͚͍ͩ͘͞ɻ੍͓͍͍͠౓͕… şƄŲƃşƄŲƃ

    View Slide

  4. About Me
    • name:
    • Takuya Fujimura (@tlync)
    • job:
    • Tech. Director, Programmer etc…
    • Dwango Mobile Co., LTD.
    • like:
    • Wine, Beer, Cat, Programming
    • lang:
    • Java/Scala/JavaScript (Scheme/Swift)
    • Scala Newbie (< 1year)

    View Slide

  5. ࠃٕ
    Japan’s National Suport ?

    View Slide

  6. 国技 !
    Yeah. You know. Japan’s National Suport !

    View Slide

  7. 大相撲
    Oh! SUMO
    https://www.flickr.com/photos/jondresner/3795899641/in/photolist-

    View Slide

  8. Introduction
    SUMO ?

    View Slide

  9. SUMO in
    Japan

    View Slide

  10. SUMO in
    Swiss
    https://www.flickr.com/photos/asirap/3920327342/in/photolist-aVv7j6-bo6LEH-6YqGJJ-6YqGAA-7292HS-oxPiiK

    View Slide

  11. SUMO & Scala ?
    https://www.flickr.com/photos/jondresner/3795899641/in/photolist-

    View Slide

  12. The SUMO association’s
    Official iOS/Android App.
    Launch a Service 2014/03
    http://sumo.dwango.jp/

    View Slide

  13. View Slide

  14. View Slide

  15. Today’s talk
    • Share our
    • design decisions
    • impressions
    • practices

    View Slide

  16. Architecture

    View Slide

  17. MySQL  
    slave
    HDFS  
    ac6ve
    PHP
    Play  
    API
    Apache
    Play  
    Push
    HDFS  
    standby
    MySQL  
    master
    PHP
    iOS
    Android
    Memcached
    Play  
    API
    Apache
    Play  
    Push
    Memcached
    Play
    Apache
    api  02
    LB
    LB
    Play
    Apache
    assets  01 assets  02
    api  01
    Legacy

    Backoffice  tool
    Shared  digital  assets  
    repository
    SUMO  DB
    Torikumi  
    Video
    Torikumi  
    Video
    Torikumi  
    Video
    Torikumi  
    Video
    Torikumi  
    Video
    Torikumi  
    Video
    Torikumi  
    Video
    SUMO  
    data

    View Slide

  18. Middleware
    • Apache 2.4
    • MySQL 5.5
    • Memcached 1.4
    • HDFS 2.0 ※ shared

    View Slide

  19. Framework/Libraries
    • Scala 2.10
    • Framework … Play! 2.2.1
    • Template Engine… twirl (w/ play)
    • DBAL … ScalalikeJDBC 1.7.4
    • Testing … ScalaTest 2.0
    • DI … Scaldi 0.2.2

    View Slide

  20. Technical Functionality
    • SUMO Content REST API
    • Video Progressive Download
    • Install based Authentication
    • iOS/Android In-App Purchase/Billing
    Validation
    • Push Notification (Amazon SNS) etc…

    View Slide

  21. Source Code
    • Scala … 12,539 lines
    • Templates … 116 files

    View Slide

  22. Why we use Scala?

    View Slide

  23. Why we use Scala?
    1. Type-Safety
    2. Less Code (than PHP, Java)
    3. Java Experience (Me)
    4. Improving our programming skills

    View Slide

  24. Why we use Scala?
    1. Type-Safety
    2. Less Code (than PHP, Java)
    3. Java Experience (Me)
    4. Improving our programming skills

    View Slide

  25. Type-Safety

    View Slide

  26. Goodbye Runtime Error by language

    View Slide

  27. Familiar Story
    A Service needs to keep evolving.
    But It’s hard to maintain middle, large software
    without static-typed.
    Test is not enough
    for error detection
    Tests is written
    by human
    vs Cost

    View Slide

  28. Familiar Story
    A Service needs to keep evolving.
    But It’s hard to maintain middle, large software
    without static-typed.
    Test is not enough
    for error detection
    Tests is written
    by human
    vs Cost
    We need type-safety dev environment
    to keep evolving software

    View Slide

  29. Improving our programming skills

    View Slide

  30. Improving our programming skills
    • Scala is multi paradigm
    language
    • Scala has many
    concepts
    • Programmers can get
    many new perspectives
    OOP FP
    Trait Monad
    Object
    Typed
    Class
    Actor

    View Slide

  31. Scala makes good programmer
    e.g: Just PHP
    Programmer
    >
    Scala
    Programmer
    Skill
    Learning
    OOP!
    Immutable!
    map/fold!
    Trait!
    Monad!
    OOP?
    Immutable?
    map/fold?
    Trait?
    Monad?

    View Slide

  32. How we learned Scala?
    • Toy Project
    Try & Error in irresponsible project
    It was terrible code….
    • Reading Circle
    Programming Scala 2nd Edition
    1 hour / day in our office
    thx @daneko0123

    View Slide

  33. Design

    View Slide

  34. Design Concept
    Focus on “Operation Phase”
    Maintainability Reliability

    View Slide

  35. Why?

    View Slide

  36. This is SUMO asociation’s official
    app not only ourselves.

    View Slide

  37. Initial Dev Time Operation Time
    <
    平成二十二年 一月場所
    平成二十三年 三月場所
    平成二十四年 五月場所
    平成二十五年 七月場所
    平成二十六年 九月場所
    平成二十七年 十一月場所

    View Slide

  38. Design Decisions
    • Full type-safety environment
    • Domain Driven Design
    • Hybrid App (Web/Native)
    • Microservices

    View Slide

  39. Full type-safety environment

    View Slide

  40. Full type-safety environment
    Scala
    Lang
    Play & twirl
    Web & View
    ScalikeJDBC
    Persistence

    View Slide

  41. Excellent!

    View Slide

  42. Full type-safety environment
    makes
    great safety & productivity.

    View Slide

  43. Except compilation time…

    View Slide

  44. Except compilation time…
    Cons < Safety & Productivity

    View Slide

  45. @rikishiProfile.kakuzuke.map { k => href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ}


    }


    ϓϩϑΟʔϧৄࡉ




    ࢛ވ໊

    @rikishiProfile.shikonaFull



    (@rikishiProfile.shikonaYomiFull)




    ൪෇

    @rikishiProfile.banzuke match {

    case Some(b) => {@b.fullName}

    case None => {൪෇֎}

    }




    ग़਎஍

    @rikishiProfile.birthplace


    @rikishiProfile.heya.map { h =>


    Complex? Really?

    View Slide

  46. @rikishiProfile.kakuzuke.map { k => href="@banzukehyo(bashoId, k.id)">൪෇දΛݟΔ}


    }


    ϓϩϑΟʔϧৄࡉ




    ࢛ވ໊

    @rikishiProfile.shikonaFull



    (@rikishiProfile.shikonaYomiFull)




    ൪෇

    @rikishiProfile.banzuke match {

    case Some(b) => {@b.fullName}

    case None => {൪෇֎}

    }




    ग़਎஍

    @rikishiProfile.birthplace


    @rikishiProfile.heya.map { h =>


    Complex? Really?
    Hatena’s designer is doing it, isn’t it!?

    View Slide

  47. Design Decisions
    • Full Type-Safety environment
    • Domain Driven Design
    • Hybrid App (Web/Native)
    • Microservices

    View Slide

  48. Design Decisions
    • Domain Driven Design
    • Mental Model = Language = Communication =
    Codes
    • Make Understandability & Clarity & Extensibility for
    SUMO specific specifications as code
    • Hybrid App (Web/Native)
    • Change content without AppStore/Google Play
    • Microservices
    • Separate concerns like PUSH notification.

    View Slide

  49. ./domain
    ./domain
    ├── auth
    │ ├── AccessToken.scala
    │ ├── Device.scala
    │ ├── RequestToken.scala
    │ ├── User.scala
    │ └── UserRepository.scala
    ├── honbasho
    │ ├── FusenType.scala
    │ ├── JudgeType.scala
    │ ├── banzuke
    │ │ ├── Banzuke.scala
    │ │ └── BanzukeRepository.scala
    │ ├── basho
    │ │ ├── Basho.scala
    │ │ └── BashoRepository.scala
    │ ├── hoshitori
    │ │ ├── CareerRecord.scala
    │ │ ├── Hoshitori.scala
    │ │ ├── HoshitoriHyo.scala
    │ │ ├── HoshitoriHyoRepository.scala
    Domain?

    View Slide


  50. class

    object


    String
    kakuzuke

    番付

    View Slide

  51. object Kakuzuke {
    !
    !
    !
    !
    !
    !

    …


    def all: Seq[Kakuzuke] = Seq(Makuuchi, Juryo, Makushi
    Sandanme, Jonidan, Jonokuchi)

    }


    sealed abstract class Kakuzuke(val id: Int, val name: S
    object Kakuzuke {

    case object Makuuchi extends Kakuzuke(1, "ນ಺")

    case object Juryo extends Kakuzuke(2, "े྆")

    case object Makushita extends Kakuzuke(3, "ນԼ")

    case object Sandanme extends Kakuzuke(4, "ࡾஈ໨")

    case object Jonidan extends Kakuzuke(5, "ংೋஈ")

    case object Jonokuchi extends Kakuzuke(6, "ংϊޱ")
    格付

    View Slide

  52. class RikishiProfile(

    id: Int,

    shikona: String,

    shikonaYomi: String,

    birthplace: String,

    thumbnail: Option[RikishiThumbnail],

    val shikonaFull: String,

    val shikonaYomiFull: String,

    val heya: Option[Heya],

    val realName: String,

    val birthDay: LocalDate,

    val age: Int,

    val height: Float,

    val weight: Float,

    val banzuke: Option[Banzuke],

    val kakuzuke: Option[Kakuzuke],

    val prizeWinningStats: PrizeWinningStats) extends
    RikishiBaseProfile(

    id,

    shikona,

    shikonaYomi,

    力士

    View Slide

  53. Hybrid App

    View Slide

  54. View Slide

  55. Web
    Na&ve

    View Slide

  56. Web
    Na&ve
    /api/v1/basho/xxx/summary
    /api/v1/basho
    accecpt:  applica0on/json
    accecpt:text/html

    View Slide

  57. View Slide

  58. Web
    Na&ve

    View Slide

  59. Web
    Na&ve
    /api/v1/torikumi/xxx
    /api/v1/torikumi/xxx/video-­‐info
    accecpt:  applica0on/json
    accecpt:text/html

    View Slide

  60. Web
    Na&ve
    /api/v1/torikumi/xxx
    /api/v1/torikumi/xxx/video-­‐info
    accecpt:  applica0on/json
    accecpt:text/html
    Usability & Maintainability

    View Slide

  61. Microservices

    View Slide

  62. Microservices
    Play  
    SUMO  REST  API
    Play  
    Push  No&fica&on  API
    Old  PHP  
    Backoffice  Tool
    Amazon  SNS
    hEp:9000 hEp:9001
    hEp:9001

    View Slide

  63. Microservices
    Play  
    SUMO  REST  API
    Play  
    Push  No&fica&on  API
    Old  PHP  
    Backoffice  Tool
    Amazon  SNS
    hEp:9000 hEp:9001
    hEp:9001
    Generic  
    Push  No&fca&on  Service
    SUMO  
    Domain
    SUMO  
    Domain

    View Slide

  64. Microservices
    Play  
    SUMO  REST  API
    Play  
    Push  No&fica&on  API
    Old  PHP  
    Backoffice  Tool
    Amazon  SNS
    hEp:9000 hEp:9001
    hEp:9001
    Generic  
    Push  No&fca&on  Service
    SUMO  
    Domain
    SUMO  
    Domain
    Isolate generic functionality

    View Slide

  65. Library Impressions

    View Slide

  66. Library Impressions
    • ScalikeJDBC … DB Access
    • twirl … Template Engine
    • Scaldi … DI
    • ScalaTest … Testing

    View Slide

  67. ScalikeJDBC
    • Good
    • Just SQL … Avoid unnecessary abstraction. Good for existing system.
    • Type-Safe … QueryDSL
    • Bad
    • Development resource? → NO!!!!
    • ConnectionPoolSetting is not flexible (… well, we can submit pull
    request)
    • Note
    • Slick is looks like cool. But we don’t want to translate complex SQL
    queries to Scala code.

    View Slide

  68. 大相撲
    powered by
    ScalikeJDBC!

    View Slide

  69. Twirl
    • Excellent!
    • Type-Safety… Prevent many errors at compile time.
    • Don’t be afraid to change method/property signature !!!
    • Good
    • Flexibility … Just Scala
    • Bad
    • Flexibility … Just Scala, Unfriendly for designers
    • Slow Feedback Cycle … Slow compilation… We developed initial
    templates without twirl using Grunt etc.
    • Note
    • I think better of this than I did before

    View Slide

  70. Twirl
    • Excellent!
    • Type-Safety… Prevent many errors at compile time.
    • Don’t be afraid to change method/property signature !!!
    • Good
    • Flexibility … Just Scala
    • Bad
    • Flexibility … Just Scala, Unfriendly for designers
    • Slow Feedback Cycle … Slow compilation… We developed initial
    templates without twirl using Grunt etc.
    • Note
    • I think better of this than I did before
    Should be evaluated!?

    View Slide

  71. Scaldi
    • Good
    • minimum & simple
    • Bad
    • Verbose declaration
    !
    • Dynamic … You won’t get an error until you run your application
    • Note
    • The cake pattern seems quite verbose for me.
    • Next, I’ll look for other solution. (like Play! 2.4 DI)
    class FooService(implicit inj: Injector) extends Injectable
    Fixed after presentation: use constructor injection!
    instead of this. It’s simple!

    View Slide

  72. ScalaTest
    • Good
    • Simple … vs Specs2. Specs2 provides several
    notations that seems not simple for me.
    • Bad
    • Limited structuring texts … vs RSpec. No context
    etc…
    • Note
    • Well… Either way

    View Slide

  73. Practices

    View Slide

  74. Practices
    • Exception Handling
    • Non-Stop Deployment
    • Static Code Analysis
    • Tuning Akka for Blocking App

    View Slide

  75. Exception Handling

    View Slide

  76. Scala provides several ways
    handling exceptions.
    Problem
    Exception Handling
    Try
    Option
    Either
    try catch
    (Validation/scalaz)

    View Slide

  77. Scala provides several ways
    handling exceptions.
    Problem
    Exception Handling
    Try
    Option
    Either
    try catch
    (Validation/scalaz)
    Which is Scala way?

    View Slide

  78. View Slide

  79. old
    Scala 2.10 <
    Scala 2.10 <
    old

    View Slide

  80. View Slide

  81. old
    Scala 2.10 <
    old
    old

    View Slide

  82. class AccessTokenService(private val secret: String)
    (implicit i: Injector) extends Logging with Injectable {
    !
    val userRepository = inject[UserRepository]
    val TermOfValidityMills = 60000 // 1min
    !
    def issueToken(reqToken: RequestToken):
    Either[InvalidTokenException, AccessToken] = {
    log.debug(s"issueToken - reqToken=$reqToken")
    !
    ...
    }
    }
    Either with Exception?

    View Slide

  83. class AccessTokenService(private val secret: String)
    (implicit i: Injector) extends Logging with Injectable {
    !
    val userRepository = inject[UserRepository]
    val TermOfValidityMills = 60000 // 1min
    !
    def issueToken(reqToken: RequestToken): Try[AccessToken]
    = {
    log.debug(s"issueToken - reqToken=$reqToken")
    !
    ...
    }
    }
    Nope. Use Try

    View Slide

  84. I don’t know Scala way…
    But our policy is below.
    try catch
    Option
    Try
    Either
    (Validation,Either etc../scalaz)
    No explanation needed
    Do not use generally (Java world syntax)
    Use this for exception handling! < Scala 2.10
    Use error handling except exceptions.

    View Slide

  85. Caution
    Keep learning new grammars …
    Scala is growing.

    View Slide

  86. Static Code Analysis

    View Slide

  87. Problem
    Static Code Analysis
    [IMO]
    In 2013,
    Many unmaintained static code analysis libraries.
    such as SCCT, sbt-scct…

    View Slide

  88. Problem
    Static Code Analysis
    http://d.hatena.ne.jp/xuwei/20130930/1380511344

    View Slide

  89. Problem
    Static Code Analysis
    http://d.hatena.ne.jp/xuwei/20130930/1380511344
    ?

    View Slide

  90. Use SCoverage now
    Solution?
    Static Code Analysis
    https://github.com/scoverage/sbt-scoverage
    https://github.com/scoverage/scalac-scoverage-plugin

    View Slide

  91. In ScalaDay 2014
    Static Code Analysis
    https://speakerdeck.com/roch/scala-code-quality-assurance

    View Slide

  92. In ScalaDay 2014
    Static Code Analysis
    https://speakerdeck.com/roch/scala-code-quality-assurance

    View Slide

  93. Others
    Static Code Analysis
    • ScalaStyle
    • Examines your Scala code and indicates potential problems with it.
    • Scapegoat
    • Scala static code analyzer
    • WartRemover
    • Flexible Scala code linting tool
    • cpd4sbt
    • Integrating PMD's Copy/Paste Detector
    • Scalariform (not analysis)
    • Source code formatting

    View Slide

  94. Non-Stop Deployment

    View Slide

  95. Play! needs to stop when
    deployment basically.
    Stop service n sec
    = LB’s Heartbeat
    Problem
    Non-Stop Deployment

    View Slide

  96. 1. Use Load Balancer API
    2. Implement Server Status API /w LB
    3. Pray to implement graceful restart in
    the future ;p
    4. Others… Do you have any ideas?
    Solution?
    Non-Stop Deployment

    View Slide

  97. 1. Use Load Balancer API
    2. Implement Server Status API /w LB
    3. Continue to pray to implement
    graceful restart in the future
    4. Others… Do you have any idea?
    Solution?
    Non-Stop Deployment

    View Slide

  98. Deployment Flow
    Non-Stop Deployment
    1. Remove Node from LB via LB API

    2. Deploy to detached Node new app.

    3. Waiting for start

    4. Add Node to LB via LB API

    5. Repeat…
    !

    View Slide

  99. @task
    @roles("app")
    def _deploy_main_job(branch, working_tree, dist_skip, cache):
    !
    developer_mode = working_tree or dist_skip
    !
    # on local
    prepare_remote()
    !
    if not dist_skip:
    package(cache)
    !
    compile_rc_script()
    transfer()
    !
    # on remote
    infrate_package()
    if env.enable_lb_switching:
    disconnect_loadblancer()
    time.sleep(5)
    stop_app()
    switch_current_app()
    start_app()
    if env.enable_lb_switching:
    reconnect_loadblancer()
    !

    View Slide

  100. compile_rc_script()
    transfer()
    !
    # on remote
    infrate_package()
    if env.enable_lb_switching:
    disconnect_loadblancer()
    time.sleep(5)
    stop_app()
    switch_current_app()
    start_app()
    if env.enable_lb_switching:
    reconnect_loadblancer()
    !
    clean_old_releases()
    !
    @task
    @roles("app")
    def disconnect_loadblancer():
    local(“/usr/local/bin/lb {0} disable".format(env.host))
    !
    @task
    @roles("app")
    def reconnect_loadblancer():
    local("/usr/local/bin/lb {0} enable".format(env.host))

    View Slide

  101. Graceful Restart?
    Non-Stop Deployment

    View Slide

  102. Tuning Thread Pool

    View Slide

  103. Caution
    Tuning Thread Pool
    The Play! default configuration is
    optimized for Non-Blocking I/O

    View Slide

  104. Caution
    Tuning Thread Pool
    https://www.playframework.com/documentation/2.3.x/ThreadPools

    View Slide

  105. • Pure asynchronous
    • for Non-Blocking app
    • Highly synchronous
    • for Blocking app
    • Many specific thread pools
    • Hybrid
    • Few specific thread pools
    • Hybrid
    Tuning Practices
    Tuning hread Pool

    View Slide

  106. e.g: Highly Synchronous
    Tuning Thread Pool

    View Slide

  107. e.g: Few specific thread pool
    Tuning Thread Pool

    View Slide

  108. Summary

    View Slide

  109. • SUMO app is powered by Scala/Play!
    • Type-Safety world makes great safety &
    productivity.
    • Scala is suited for long-maintained or
    middle-large applications.
    • I worry about binary compatibility
    Summary

    View Slide

  110. finally..

    View Slide

  111. !
    Scala is already practical.
    From now on,
    let’s share
    our practices, codes… more
    to keep evolving
    Scala ecosystem & community.

    View Slide

  112. Let’s Enjoy
    Scala !!

    View Slide

  113. !
    thank you
    • Scala developers
    • ScalaMatsuri staffs
    • Sumo project members & supporters
    • everybody

    View Slide