国技と Scala (Japan's national sport and Scala)

by Takuya Fujimura

Slide 1

Slide 1 text

ࠃٕ ͱ Scala Takuya Fujimura Dwango Mobile Co., LTD. @tlync Japan’s National Sport and Scala 2014 ScalaMatsuri

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Sponsored by http://info.dwango.co.jp/recruit/ Scala Ͱ࢓ࣄ͍ͨ͠? ͦΜͳਓ͸ͪ͜Βˣ ※ ࢲ·Ͱ͓੠͕͚͍ͩ͘͞ɻ੍͓͍͍͠౓͕… şƄŲƃşƄŲƃ

Slide 4

Slide 4 text

About Me • name: • Takuya Fujimura (@tlync) • job: • Tech. Director, Programmer etc… • Dwango Mobile Co., LTD. • like: • Wine, Beer, Cat, Programming • lang: • Java/Scala/JavaScript (Scheme/Swift) • Scala Newbie (< 1year)

Slide 5

Slide 5 text

ࠃٕ Japan’s National Suport ?

Slide 6

Slide 6 text

国技 ! Yeah. You know. Japan’s National Suport !

Slide 7

Slide 7 text

大相撲 Oh! SUMO https://www.ﬂickr.com/photos/jondresner/3795899641/in/photolist-

Slide 8

Slide 8 text

Introduction SUMO ?

Slide 9

Slide 9 text

SUMO in Japan

Slide 10

Slide 10 text

SUMO in Swiss https://www.ﬂickr.com/photos/asirap/3920327342/in/photolist-aVv7j6-bo6LEH-6YqGJJ-6YqGAA-7292HS-oxPiiK

Slide 11

Slide 11 text

SUMO & Scala ? https://www.ﬂickr.com/photos/jondresner/3795899641/in/photolist-

Slide 12

Slide 12 text

The SUMO association’s Official iOS/Android App. Launch a Service 2014/03 http://sumo.dwango.jp/

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Today’s talk • Share our • design decisions • impressions • practices

Slide 16

Slide 16 text

Architecture

Slide 17

Slide 17 text

MySQL slave HDFS ac6ve PHP Play API Apache Play Push HDFS standby MySQL master PHP iOS Android Memcached Play API Apache Play Push Memcached Play Apache api 02 LB LB Play Apache assets 01 assets 02 api 01 Legacy  Backoﬃce tool Shared digital assets repository SUMO DB Torikumi Video Torikumi Video Torikumi Video Torikumi Video Torikumi Video Torikumi Video Torikumi Video SUMO data

Slide 18

Slide 18 text

Middleware • Apache 2.4 • MySQL 5.5 • Memcached 1.4 • HDFS 2.0 ※ shared

Slide 19

Slide 19 text

Framework/Libraries • Scala 2.10 • Framework … Play! 2.2.1 • Template Engine… twirl (w/ play) • DBAL … ScalalikeJDBC 1.7.4 • Testing … ScalaTest 2.0 • DI … Scaldi 0.2.2

Slide 20

Slide 20 text

Technical Functionality • SUMO Content REST API • Video Progressive Download • Install based Authentication • iOS/Android In-App Purchase/Billing Validation • Push Notification (Amazon SNS) etc…

Slide 21

Slide 21 text

Source Code • Scala … 12,539 lines • Templates … 116 files

Slide 22

Slide 22 text

Why we use Scala?

Slide 23

Slide 23 text

Why we use Scala? 1. Type-Safety 2. Less Code (than PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills

Slide 24

Slide 24 text

Why we use Scala? 1. Type-Safety 2. Less Code (than PHP, Java) 3. Java Experience (Me) 4. Improving our programming skills

Slide 25

Slide 25 text

Type-Safety

Slide 26

Slide 26 text

Goodbye Runtime Error by language

Slide 27

Slide 27 text

Familiar Story A Service needs to keep evolving. But It’s hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost

Slide 28

Slide 28 text

Familiar Story A Service needs to keep evolving. But It’s hard to maintain middle, large software without static-typed. Test is not enough for error detection Tests is written by human vs Cost We need type-safety dev environment to keep evolving software

Slide 29

Slide 29 text

Improving our programming skills

Slide 30

Slide 30 text

Improving our programming skills • Scala is multi paradigm language • Scala has many concepts • Programmers can get many new perspectives OOP FP Trait Monad Object Typed Class Actor

Slide 31

Slide 31 text

Scala makes good programmer e.g: Just PHP Programmer > Scala Programmer Skill Learning OOP! Immutable! map/fold! Trait! Monad! OOP? Immutable? map/fold? Trait? Monad?

Slide 32

Slide 32 text

How we learned Scala? • Toy Project Try & Error in irresponsible project It was terrible code…. • Reading Circle Programming Scala 2nd Edition 1 hour / day in our office thx @daneko0123

Slide 33

Slide 33 text

Design

Slide 34

Slide 34 text

Design Concept Focus on “Operation Phase” Maintainability Reliability

Slide 35

Slide 35 text

Why?

Slide 36

Slide 36 text

This is SUMO asociation’s official app not only ourselves.

Slide 37

Slide 37 text

Initial Dev Time Operation Time < 平成二十二年一月場所平成二十三年三月場所平成二十四年五月場所平成二十五年七月場所平成二十六年九月場所平成二十七年十一月場所

Slide 38

Slide 38 text

Design Decisions • Full type-safety environment • Domain Driven Design • Hybrid App (Web/Native) • Microservices

Slide 39

Slide 39 text

Full type-safety environment

Slide 40

Slide 40 text

Full type-safety environment Scala Lang Play & twirl Web & View ScalikeJDBC Persistence

Slide 41

Slide 41 text

Excellent!

Slide 42

Slide 42 text

Full type-safety environment makes great safety & productivity.

Slide 43

Slide 43 text

Except compilation time…

Slide 44

Slide 44 text

Except compilation time… Cons < Safety & Productivity

Slide 45

Slide 45 text

@rikishiProfile.kakuzuke.map { k => ൪෇දΛݟΔ}    }   

ϓϩϑΟʔϧৄࡉ

      ࢛ވ໊  @rikishiProfile.shikonaFull 
  (@rikishiProfile.shikonaYomiFull)        ൪෇  @rikishiProfile.banzuke match {  case Some(b) => {@b.fullName}  case None => {൪෇֎}  }        ग़਎஍  @rikishiProfile.birthplace    @rikishiProfile.heya.map { h =>    Complex? Really?

Slide 46

Slide 46 text

@rikishiProfile.kakuzuke.map { k => ൪෇දΛݟΔ}    }   

ϓϩϑΟʔϧৄࡉ

Slide 47

Slide 47 text

Design Decisions • Full Type-Safety environment • Domain Driven Design • Hybrid App (Web/Native) • Microservices

Slide 48

Slide 48 text

Design Decisions • Domain Driven Design • Mental Model = Language = Communication = Codes • Make Understandability & Clarity & Extensibility for SUMO specific specifications as code • Hybrid App (Web/Native) • Change content without AppStore/Google Play • Microservices • Separate concerns like PUSH notification.

Slide 49

Slide 49 text

./domain ./domain ├── auth │ ├── AccessToken.scala │ ├── Device.scala │ ├── RequestToken.scala │ ├── User.scala │ └── UserRepository.scala ├── honbasho │ ├── FusenType.scala │ ├── JudgeType.scala │ ├── banzuke │ │ ├── Banzuke.scala │ │ └── BanzukeRepository.scala │ ├── basho │ │ ├── Basho.scala │ │ └── BashoRepository.scala │ ├── hoshitori │ │ ├── CareerRecord.scala │ │ ├── Hoshitori.scala │ │ ├── HoshitoriHyo.scala │ │ ├── HoshitoriHyoRepository.scala Domain?

Slide 50

Slide 50 text

  class   object …   String kakuzuke   番付

Slide 51

Slide 51 text

object Kakuzuke { ! ! ! ! ! !   …    def all: Seq[Kakuzuke] = Seq(Makuuchi, Juryo, Makushi Sandanme, Jonidan, Jonokuchi)  }    sealed abstract class Kakuzuke(val id: Int, val name: S object Kakuzuke {  case object Makuuchi extends Kakuzuke(1, "ນ಺")  case object Juryo extends Kakuzuke(2, "े྆")  case object Makushita extends Kakuzuke(3, "ນԼ")  case object Sandanme extends Kakuzuke(4, "ࡾஈ໨")  case object Jonidan extends Kakuzuke(5, "ংೋஈ")  case object Jonokuchi extends Kakuzuke(6, "ংϊޱ") 格付

Slide 52

Slide 52 text

class RikishiProfile(  id: Int,  shikona: String,  shikonaYomi: String,  birthplace: String,  thumbnail: Option[RikishiThumbnail],  val shikonaFull: String,  val shikonaYomiFull: String,  val heya: Option[Heya],  val realName: String,  val birthDay: LocalDate,  val age: Int,  val height: Float,  val weight: Float,  val banzuke: Option[Banzuke],  val kakuzuke: Option[Kakuzuke],  val prizeWinningStats: PrizeWinningStats) extends RikishiBaseProfile(  id,  shikona,  shikonaYomi,  力士

Slide 53

Slide 53 text

Hybrid App

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Web Na&ve

Slide 56

Slide 56 text

Web Na&ve /api/v1/basho/xxx/summary /api/v1/basho accecpt: applica0on/json accecpt:text/html

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Web Na&ve

Slide 59

Slide 59 text

Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-‐info accecpt: applica0on/json accecpt:text/html

Slide 60

Slide 60 text

Web Na&ve /api/v1/torikumi/xxx /api/v1/torikumi/xxx/video-‐info accecpt: applica0on/json accecpt:text/html Usability & Maintainability

Slide 61

Slide 61 text

Microservices

Slide 62

Slide 62 text

Microservices Play SUMO REST API Play Push No&ﬁca&on API Old PHP Backoﬃce Tool Amazon SNS hEp:9000 hEp:9001 hEp:9001

Slide 63

Slide 63 text

Microservices Play SUMO REST API Play Push No&ﬁca&on API Old PHP Backoﬃce Tool Amazon SNS hEp:9000 hEp:9001 hEp:9001 Generic Push No&fca&on Service SUMO Domain SUMO Domain

Slide 64

Slide 64 text

Microservices Play SUMO REST API Play Push No&ﬁca&on API Old PHP Backoﬃce Tool Amazon SNS hEp:9000 hEp:9001 hEp:9001 Generic Push No&fca&on Service SUMO Domain SUMO Domain Isolate generic functionality

Slide 65

Slide 65 text

Library Impressions

Slide 66

Slide 66 text

Library Impressions • ScalikeJDBC … DB Access • twirl … Template Engine • Scaldi … DI • ScalaTest … Testing

Slide 67

Slide 67 text

ScalikeJDBC • Good • Just SQL … Avoid unnecessary abstraction. Good for existing system. • Type-Safe … QueryDSL • Bad • Development resource? → NO!!!! • ConnectionPoolSetting is not flexible (… well, we can submit pull request) • Note • Slick is looks like cool. But we don’t want to translate complex SQL queries to Scala code.

Slide 68

Slide 68 text

大相撲 powered by ScalikeJDBC!

Slide 69

Slide 69 text

Twirl • Excellent! • Type-Safety… Prevent many errors at compile time. • Don’t be afraid to change method/property signature !!! • Good • Flexibility … Just Scala • Bad • Flexibility … Just Scala, Unfriendly for designers • Slow Feedback Cycle … Slow compilation… We developed initial templates without twirl using Grunt etc. • Note • I think better of this than I did before

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Scaldi • Good • minimum & simple • Bad • Verbose declaration ! • Dynamic … You won’t get an error until you run your application • Note • The cake pattern seems quite verbose for me. • Next, I’ll look for other solution. (like Play! 2.4 DI) class FooService(implicit inj: Injector) extends Injectable Fixed after presentation: use constructor injection! instead of this. It’s simple!

Slide 72

Slide 72 text

ScalaTest • Good • Simple … vs Specs2. Specs2 provides several notations that seems not simple for me. • Bad • Limited structuring texts … vs RSpec. No context etc… • Note • Well… Either way

Slide 73

Slide 73 text

Practices

Slide 74

Slide 74 text

Practices • Exception Handling • Non-Stop Deployment • Static Code Analysis • Tuning Akka for Blocking App

Slide 75

Slide 75 text

Exception Handling

Slide 76

Slide 76 text

Scala provides several ways handling exceptions. Problem Exception Handling Try Option Either try catch (Validation/scalaz)

Slide 77

Slide 77 text

Scala provides several ways handling exceptions. Problem Exception Handling Try Option Either try catch (Validation/scalaz) Which is Scala way?

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

old Scala 2.10 < Scala 2.10 < old

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

old Scala 2.10 < old old

Slide 82

Slide 82 text

class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Either[InvalidTokenException, AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Either with Exception?

Slide 83

Slide 83 text

class AccessTokenService(private val secret: String) (implicit i: Injector) extends Logging with Injectable { ! val userRepository = inject[UserRepository] val TermOfValidityMills = 60000 // 1min ! def issueToken(reqToken: RequestToken): Try[AccessToken] = { log.debug(s"issueToken - reqToken=$reqToken") ! ... } } Nope. Use Try

Slide 84

Slide 84 text

I don’t know Scala way… But our policy is below. try catch Option Try Either (Validation,Either etc../scalaz) No explanation needed Do not use generally (Java world syntax) Use this for exception handling! < Scala 2.10 Use error handling except exceptions. …

Slide 85

Slide 85 text

Caution Keep learning new grammars … Scala is growing.

Slide 86

Slide 86 text

Static Code Analysis

Slide 87

Slide 87 text

Problem Static Code Analysis [IMO] In 2013, Many unmaintained static code analysis libraries. such as SCCT, sbt-scct…

Slide 88

Slide 88 text

Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344

Slide 89

Slide 89 text

Problem Static Code Analysis http://d.hatena.ne.jp/xuwei/20130930/1380511344 ?

Slide 90

Slide 90 text

Use SCoverage now Solution? Static Code Analysis https://github.com/scoverage/sbt-scoverage https://github.com/scoverage/scalac-scoverage-plugin

Slide 91

Slide 91 text

In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance

Slide 92

Slide 92 text

In ScalaDay 2014 Static Code Analysis https://speakerdeck.com/roch/scala-code-quality-assurance

Slide 93

Slide 93 text

Others Static Code Analysis • ScalaStyle • Examines your Scala code and indicates potential problems with it. • Scapegoat • Scala static code analyzer • WartRemover • Flexible Scala code linting tool • cpd4sbt • Integrating PMD's Copy/Paste Detector • Scalariform (not analysis) • Source code formatting

Slide 94

Slide 94 text

Non-Stop Deployment

Slide 95

Slide 95 text

Play! needs to stop when deployment basically. Stop service n sec = LB’s Heartbeat Problem Non-Stop Deployment

Slide 96

Slide 96 text

1. Use Load Balancer API 2. Implement Server Status API /w LB 3. Pray to implement graceful restart in the future ;p 4. Others… Do you have any ideas? Solution? Non-Stop Deployment

Slide 97

Slide 97 text

1. Use Load Balancer API 2. Implement Server Status API /w LB 3. Continue to pray to implement graceful restart in the future 4. Others… Do you have any idea? Solution? Non-Stop Deployment

Slide 98

Slide 98 text

Deployment Flow Non-Stop Deployment 1. Remove Node from LB via LB API ↓ 2. Deploy to detached Node new app. ↓ 3. Waiting for start ↓ 4. Add Node to LB via LB API ↓ 5. Repeat… !

Slide 99

Slide 99 text

@task @roles("app") def _deploy_main_job(branch, working_tree, dist_skip, cache): ! developer_mode = working_tree or dist_skip ! # on local prepare_remote() ! if not dist_skip: package(cache) ! compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer() time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() !

Slide 100

Slide 100 text

compile_rc_script() transfer() ! # on remote infrate_package() if env.enable_lb_switching: disconnect_loadblancer() time.sleep(5) stop_app() switch_current_app() start_app() if env.enable_lb_switching: reconnect_loadblancer() ! clean_old_releases() ! @task @roles("app") def disconnect_loadblancer(): local(“/usr/local/bin/lb {0} disable".format(env.host)) ! @task @roles("app") def reconnect_loadblancer(): local("/usr/local/bin/lb {0} enable".format(env.host))

Slide 101

Slide 101 text

Graceful Restart? Non-Stop Deployment

Slide 102

Slide 102 text

Tuning Thread Pool

Slide 103

Slide 103 text

Caution Tuning Thread Pool The Play! default configuration is optimized for Non-Blocking I/O

Slide 104

Slide 104 text

Caution Tuning Thread Pool https://www.playframework.com/documentation/2.3.x/ThreadPools

Slide 105

Slide 105 text

• Pure asynchronous • for Non-Blocking app • Highly synchronous • for Blocking app • Many specific thread pools • Hybrid • Few specific thread pools • Hybrid Tuning Practices Tuning hread Pool

Slide 106

Slide 106 text

e.g: Highly Synchronous Tuning Thread Pool

Slide 107

Slide 107 text

e.g: Few specific thread pool Tuning Thread Pool

Slide 108

Slide 108 text

Summary

Slide 109

Slide 109 text

• SUMO app is powered by Scala/Play! • Type-Safety world makes great safety & productivity. • Scala is suited for long-maintained or middle-large applications. • I worry about binary compatibility Summary

Slide 110

Slide 110 text

finally..

Slide 111

Slide 111 text

! Scala is already practical. From now on, let’s share our practices, codes… more to keep evolving Scala ecosystem & community.

Slide 112

Slide 112 text

Let’s Enjoy Scala !!

Slide 113

Slide 113 text

! thank you • Scala developers • ScalaMatsuri staffs • Sumo project members & supporters • everybody