Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons Learned

Lessons Learned

How to launch a product using Scala.

Feb 25, 2017 at Scala Matsuri 2014 in Tokyo, Japan.

Yoshimasa Niwa

February 25, 2017

More Decks by Yoshimasa Niwa

Other Decks in Programming


  1. How to launch a new product using Scala Lessons Learned

    Yoshimasa Niwa Feb 25, 2017 in Tokyo αʔϏεΛθϩ͔Β࡞ͬͯେ੾ͩͬͨͭͷ͜ͱ
  2. How to launch a product using Scala Lessons Learned I

    was (am) not a Scala specialist. Launched user facing products using Scala. There are many lessons I’ve learned. αʔϏεΛθϩ͔Β࡞ͬͯେ੾ͩͬͨͭͷ͜ͱ
  3. Today’s talk At glance History, we’re learning from history. Architecture,

    that built from the lessons. Implementation, that we’re using. Executions, form what I’ve learned lessons. ຊ೔ͷ࿩ͷ֓ཁ
  4. So it called “Monorail.” Many pains in the large monolithic

    system • It’s really easy to break someone’s modules. • grep(1) is the only tool we can believe. • Deploy everything, revert everything. ڊେͳ୯ҰͷγεςϜͰ͸ਏ͔ͬͨɻ
  5. to decompose monolithic Ruby on Rails application Rewrite each module

    in Scala class User < ActiveRecode::Base
 def self.find(...)
 end User.find(1) #=> #<UserServiceUser: ...> ڊେͳ3VCZPO3BJMTΞϓϦέʔγϣϯΛ෼ղͯͦ͠ΕͧΕͷϞδϡʔϧΛ4DBMBͰ ॻ͖௚ͨ͠ɻ
  6. Example code System is complex def main(args: Array[String]) {

    match {
 case Seq("command1") =>
 case Seq("command2") =>
 case Seq("command3") =>
 ... // Continue 10,000 lines
 } ෳࡶͳίʔυͷྫ
  7. Divide-and-rule. System is complex def func1() {
 } def

    func2() {
 } def func3() {
 } def main(args: Array[String]) {
 args match {
 case Seq("command1") =>
 case Seq("command2") =>
 case Seq(“command3") =>
  8. Create new functions in the system. Create a new service

    ৽αʔϏεΛ࡞ΔʹγεςϜʹ৽͍ؔ͠਺Λ௥Ճ͢Δ def service(
 request: Request
 ): Response def newService(
 request: Request
 ): Response
  9. Response is not always available immediately. In real world, however.

    def service(request: Request): Response = ... // How slow is this?
 val reposes = service(request)
  10. Example Future[+A] def func(a: String): Future[String] = … def main(args:

    Array[String]) {
 // This call is not blocking.
 val future = func(args.head)
 // Do awesome things...
 // Wait until reply is defined
  11. Future works same as the other capsules. Compose Future[+A] •

    map[A,B](A => B): Future[B] • flatMap[A,B](A => Future[B]): Future[B] • join[A,B](Future[A], Future[B]):
 Future[(A,B)] • collect[A](Seq[Future[A]]): Future[Seq[A]] 'VUVSF͸ଞͷ4DBMBͷίϨΫγϣϯͱಉ༷ʹ߹੒Մೳɻ A Future[A] B Future[B] map A Future[A] B Future[B] flatMap
  12. Example Compose service calls val userRepo: Service[Long, User] = ...

    val profileRepo: Service[User, Profile] = ...
 val postRepo: Service[User, Seq[Post]] = ...
 userRepo(userId).flatMap { user =>
 ).map { (profile, posts) =>
 renderResponse(profile, posts)
 } αʔϏεͷ݁Ռ͸'VUVSF͔ͩΒɺݺͼग़͠Λ࣍ʑ߹੒Ͱ͖Δɻ
  13. class GifSearchHandler {
 def apply(request: SearchRequest):
 Future[SearchResponse] = for {

    user <- userRepo(request.user_id)
 results <- Future.collect(
 gifRepos.map { repo =>
 repo(request.query, user.lang)
 } yield merge(results) } GIF Button Example ྫ͑͹(*'#VUUPOͷݕࡧॲཧ͸͜Μͳײ͡
  14. Deal with complexity “Functional” Architecture Divide-and-rule Function as a service

    Compose response Future[+A] to make a logic asynchronously ʮؔ਺తͳʯΞʔΩςΫνϟ
  15. • Define functions • Resolve functions • Implement functions •

    Instantiate functions • Discover functions Things we need to call function γεςϜ͕ඞཁͱ͍ͯ͠Δ͜ͱ
  16. Thrift, an IDL to define models and services Scrooge, a

    Thrift code generator in Scala Thrift, Scrooge Define functions ఆٛʹ͸5ISJGUΛ࢖͏ɻ4DSPPHFͰ4DBMBͷίʔυੜ੒ɻ
  17. Resolve functions Mono repo Git, “special version” Faster clone Pants,

    a build system for mono repo. ੩తͳґଘؔ܎͸ϞϊϨϙͱ1BOUTϏϧυγεςϜͰղܾ
  18. Finagle, a RPC core library. Finatra, a framework using Finagle

    TwitterServer (HTTP server and Thrift server) Guice dependency injection Finagle, Finatra Implement functions ࣮૷ʹ͸'JOBHMFΛϕʔεʹ'JOBUSBΛ࢖͍ͬͯΔ
  19. Instantiate functions Mesos, Aurora Mesos, a cluster manager for data

    center. Aurora framework, a job scheduler. Define job in aurora DSL file. .FTPTͱ"VSPSBϑϨʔϜϫʔΫͰΠϯελϯεΛ؅ཧ
  20. Discover functions Zookeeper Zookeeper, a reliable directory for service discovery.

    Announced by Aurora Resolved by Finagle ;PPLFFQFSͰαʔϏεΛൃݟ͢Δɻ"VSPSBͰΠϯελϯεΛ໊લొ࿥ͯ͠ɺ'JOBHMF Ͱղܾ
  21. .scala .thrift .aurora .scala .thrift .aurora Git .scala .thrift .aurora

    Mesos Aurora Zookeeper /s/gifsearch Instance Instance
  22. Things we need to build services Implementations Define in Thrift,

    manage every things in a single Git repository. Finatra to implement services. Aurora to run services and discovery. αʔϏεΛ࡞Δͷʹඞཁͳ࣮૷
  23. We’re a team Engineers, product manager, and designers works in

    a team. ΤϯδχΞɺϓϩμΫτϚωʔδϟɺσβΠφ͕ɺͻͱͭͷνʔϜͰ࢓ࣄ͠·͢ɻ
  24. Engineers have different responsibilities than Product Managers Product vs. Engineering

    ϓϩμΫτରΤϯδχΞϦϯά ΤϯδχΞ͸ϓϩμΫτϚωʔδϟͱ͸ҧ͏੹೚͕͋Δ
  25. Integrate everything at first. Build app from top to bottom

    Bad assumption may increase tech-debt. Right assumption may reduce tech-debt. Reduces many bad assumptions. ·্͔ͣΒԼ·Ͱͱʹ͔͘ܨ͍Ͱಈ͔͢ɻ૝ఆΛͰߟ͑Δ͜ͱΛͰ͖Δ͚ͩݮΒ͢ɻ S S S S S Service Service Service Service Service
  26. Create many instances with modification on Aurora Run devel instances

    Work-in-progress instance is okay. Working mock is better than tech-doc to deal with product manager, sometimes. ։ൃ༻ΠϯελϯεΛ࣮ߦ͓ͯ͘͠ɻ
  27. Override services for debugging and testing Dtab A delegation table.

    Override finagle resolver to swap services. Dtab-local: header for HTTP request. %UBCΛ࢖໊ͬͯલղܾΛ্ॻ͖ͯ͠αʔϏεΛϥϯλΠϜʹೖΕସ͑Δ
  28. are user facing and used by human beings. iOS and

    Android clients ϞόΠϧΫϥΠΞϯτ͸ਓ͕͔ؒͭ͏΋ͷɻ༧ଌ֎ͷૢ࡞Λ͞ΕΔ͜ͱ΋͋Δɻ
  29. User interface is a representation of internal states. Manage internal

    state. Users changes the internal state. React to the internal state changes. Applying animation, if needed. ಺෦ঢ়ଶͱมߋͷྲྀΕΛ؅ཧ͢Δɻ6*͸಺෦ঢ়ଶΛදݱ͢Δ΋ͷɻ S Action State Change UI Update Action U User
  30. Truth is in front of you as a code. Read

    code ৴͡ΒΕΔ΋ͷ͸ιʔε͚ͩɻ
  31. Single “source” of truth. Mono repo helps it. Identify exact

    source code by deploy SHA-1. Browse entire source code that the system is using. ୯ҰιʔεϨϙδτϦͰਅ࣮ΛݟΔ commit 8270324cdf0d1e60a83dbefa1de29d89f6107587
 commit f40648b9105f7478139694acd0821c15d3315dc2
 commit fa0dc865887a02b49576bdc1de57f4ad802d4710
 commit ebf31743cbe3008ec311d88aea42b987b0288cd3
  32. Name tells what it is. Define right functions Define API

    in reasonable way. Naming is really really important. They never read tech-doc. ਖ਼͍ؔ͠਺ͷఆٛ͸υΩϡϝϯτΑΓେ੾ɻ outputImageProviderFromBufferWithPixelFormat:pixelsW ide:pixelsHigh:baseAddress:bytesPerRow:releaseCallba ck:releaseContext:colorSpace:shouldColorMatch:
  33. Manage dependencies. Downstream Estimate QPS and latency. Right timeout and

    retry policy. Right load banaler ApertureLoadBalancer for lower QPS 214ͱϨΠςϯγΛ؅ཧɻϥΠϜΞ΢τͱϦτϥΠͷௐ੔ɺϩʔυόϥϯαͷௐ੔ɻ S S S S S Service Service Service Service Service
  34. Manage dependencies Upstream Look at CancelledRequestException. Upstream timed out for

    downstream, it will cancel request and downstream will see this exception. $BODFMMFE3FRVFTU&YDFQUJPOʹ஫ҙɻ 6QTUSFBN͕ݫ͍͠λΠϜΞ΢τΛ࣋ͬͯΔՄೳੑ͋Γɻ S S Service Service Timeout! Cancel
 Exception Timeout
  35. How to launch a product using Scala Lessons Learned 1.

    Build app from top to bottom
 to deal with product managers. 2. Manage internal state
 to deal with designers. 3. Read code
 to deal with engineers. αʔϏεΛθϩ͔Β࡞ͬͯେ੾ͩͬͨͭͷ͜ͱ