Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalaでの並行・並列処理戦略/strategy-for-concurrency-and-p...

fuzyco
November 10, 2018

 Scalaでの並行・並列処理戦略/strategy-for-concurrency-and-parallel-by-scala

Scala関西サミット2018で発表したスライドです。

fuzyco

November 10, 2018
Tweet

More Decks by fuzyco

Other Decks in Technology

Transcript

  1. ScalaίϨΫγϣϯ  ɾList, Array, etc. scala> val list = (1

    to 10000).toList list: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, … scala> list.map(_ + 42) res0: List[Int] = List(43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, … scala> list.max res1: Int = 10000
  2. ScalaฒྻίϨΫγϣϯ  ίϨΫγϣϯͷԋࢉΛฒྻʹ࣮ߦ͢Δ͜ͱ͕Ͱ͖Δ scala> val parList = list.par parList: scala.collection.parallel.immutable.ParSeq[Int]

    = ParVector(1, 2, 3, 4, 5, …) scala> parList.seq res1: scala.collection.immutable.Seq[Int] = Vector(1, 2, 3, 4, 5, 6, … ) ม׵͕Մೳ scala> parList.foreach(elm => print(s"$elm ")) 376 377 501 813 876 2 … ฒྻʹॲཧ͕࣮ߦ͞ΕΔ
  3. ScalaฒྻίϨΫγϣϯ  import scala.util.Random val numbers = Random.shuffle(Vector.tabulate(10000000)(i => i))

    val seqtime = measure { numbers.max } log(s"Sequential time ${seqtime.value} ms”) val partime = measure { numbers.par.max } log(s"Parallel time ${partime.value} ms") Sequential time 535.850688 ms Parallel time 193.393754 ms parϝιουΛ࢖͚ͬͨͩͰ ฒྻʹॲཧ͕ߦΘΕΔ ϥϯμϜͳ஋ͷཁૉΛ10000000ݸ࣋ͭVectorͷ ࠷େͷཁૉΛٻΊΔϓϩάϥϜ
  4. ෼ׂ౷࣏  ໰୊Λ෼ׂͯ͠ɺͦΕͧΕಠཱͯ͠ࢉग़͞ΕͨղΛ ౷߹͍ͯ͘͠Ξϓϩʔν(ྫ: Ϛʔδιʔτ) ෦෼ ղ ղ ෦෼ ղ

    ෦෼ ղ ෦෼ ղ ฒྻԽՄೳ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊
  5. ෼ׂ౷࣏  ߹ܭ஋ શମͷ ߹ܭ஋ ߹ܭ஋ ߹ܭ஋ ߹ܭ஋ ֤CPUͰฒྻʹ࣮ߦ "SSBZ

    ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ sumϝιουݺͼग़͠ʹΑΔίϨΫγϣϯͷ߹ܭ஋ͷܭࢉ ArrayΛεϨου ͷ਺͚ͩ෼ׂ ͦΕͧΕͷ߹ܭ஋ Λ଍͠߹ΘͤΔ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ
  6. ฒྻίϨΫγϣϯͷ࢓૊Έ  Splitter ෼ׂ౷࣏Λ࣮ݱ͍ͯ͠ΔΦϒδΣΫτ trait IterableSplitter[T] extends Iterator[T] { def

    dup: IterableSplitter[T] def remaining: Int def split: Seq[IterableSplitter[T]] } ίϨΫγϣϯΛޮ཰Α͘෼ׂ͢Δ Combiner trait Combiner[-Elem, +To] extends Builder[Elem, To] with Sizing with Parallel { def combine[N <: Elem, NewTo >: To](other: Combiner[N, NewTo]): Combiner[N, NewTo] } ෼ׂ͞ΕͨλεΫͷԋࢉ݁ՌΛ݁߹͢Δ
  7. εϨουؒͷಉظॲཧ (σʔλڝ߹)  object ParUid extends App { private val

    uid = new AtomicLong(0L) val seqtime = measure { for(i <- 0 until 10000000) uid.incrementAndGet() } println(s"Sequential time ${seqtime.value} ms”) val partime = measure { for(i <- (0 until 10000000).par) uid.incrementAndGet() } println(s"Parallel time ${partime.value} ms") } ※measureؔ਺͸ScalaMeterͷॲཧ࣌ؒΛܭଌ͢Δؔ਺ uidΛAtomicม਺Ͱ҆શͳಉظΛ ߦͬͯΧ΢ϯτΞοϓ͍ͯ͠Δ ϧʔϓॲཧͰΧ΢ϯτΞοϓ͢ΔϓϩάϥϜ
  8. εϨουؒͷಉظॲཧ (σʔλڝ߹)  4FRVFOUJBMUJNFNT 1BSBMMFMUJNFNT object ParUid extends App {

    private val uid = new AtomicLong(0L) val seqtime = measure { for(i <- 0 until 10000000) uid.incrementAndGet() } println(s"Sequential time ${seqtime.value} ms”) val partime = measure { for(i <- (0 until 10000000).par) uid.incrementAndGet() } println(s"Parallel time $partime ms") } ฒྻίϨΫγϣϯΛ࢖ͬͨ΄͏͕ ͕͔͔͍࣌ؒͬͯΔ εϨουؒͰσʔλΛਖ਼͘͠ڞ༗͢ΔͨΊʹಉظΛߦ͏ͷͰɺ ֤εϨουͰॲཧ଴͕ͪൃੜͯ͠εϧʔϓοτ͕௿Լ͢Δ
  9. ฒྻίϨΫγϣϯ΁ͷม׵ίετ  ListͱVectorΛͦΕͧΕ ฒྻίϨΫγϣϯʹม׵͢ΔϓϩάϥϜ object ParNonParallelizableCollections extends App with Logger

    { val list = List.fill(1000000)("") val vector = Vector.fill(1000000)(“") println(s"list conversion time: ${measure(list.par).value} ms") println(s"vector conversion time: ${measure(vector.par).value} ms") }
  10. ฒྻίϨΫγϣϯ΁ͷม׵ίετ  MJTUDPOWFSTJPOUJNFNT WFDUPSDPOWFSTJPOUJNFNT ListͱVectorΛͦΕͧΕ ฒྻίϨΫγϣϯʹม׵͢ΔϓϩάϥϜ object ParNonParallelizableCollections extends App

    with Logger { val list = List.fill(1000000)("") val vector = Vector.fill(1000000)(“") println(s"list conversion time: ${measure(list.par).value} ms") println(s"vector conversion time: ${measure(vector.par).value} ms") } ListͷฒߦίϨΫγϣϯ΁ͷม׵ ʹ͔ͳΓ͕͔͔͍࣌ؒͬͯΔ
  11. ฒྻίϨΫγϣϯ΁ͷม׵ίετ  List Vector ParVector scala> (1 to 10000).toList.par res2:

    ParVector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, … શͯͷཁૉΛίϐʔ͢ΔͷͰܭࢉίετ͕͔͔Δ ฒྻίϨΫγϣϯͷ࣮૷ Λ࣌ͨͳ͍ίϨΫγϣϯ ฒྻίϨΫγϣϯͷ࣮૷ Λ࣋ͭίϨΫγϣϯ ฒྻίϨΫγϣϯ
  12. ঢ়ଶΛѻ͏ԋࢉ  scala> var sum = 0 sum: Int =

    0 scala> val parList = (1 to 1000).toList.par parList: scala.collection.parallel.immutable.ParSeq[Int] = ParVector(1, 2, 3, 4, 5, … scala> var sum = 0 sum: Int = 0 scala> parList.foreach(sum += _); sum res01: Int = 307792 scala> parList.foreach(sum += _); sum res02: Int = 417753
  13. ݁߹ଇ͕੒Γཱͨͳ͍ԋࢉ  ฒྻͰ࣮ߦ͞ΕΔͷͰɺ݁߹཯͕੒Γཱͨͳ͍ԋࢉ͸ႈ౳ͳ஋ʹͳΒͳ͍ ྫ: (1-2)-3 != 1-(2-3) scala> val parList

    = (1 to 1000).toList.par parList: scala.collection.parallel.immutable.ParSeq[Int] = ParVector(1, 2, 3, 4, 5, … scala> parList.reduce(_ - _) res0: Int = -169948 scala> parList.reduce(_ - _) res1: Int = 181648
  14.  ฒྻCollectionͰIOॲཧ val userIds: List[String] = List("id") def findUser(userId: String):

    User = { database.findUser(userId) } val users = userIds.par.map { userId => findUser(userId) } CPU1 IO଴ͪ CPU2 IO଴ͪ ฒྻίϨΫγϣϯ಺ͰIOό΢ϯυͳॲཧΛߦ͏ͱɺ IO଴ͪͷؒશͯͷCPUͷॲཧ͕ఀࢭͯ͠͠·͏ DB΁ͷ ΞΫηε
  15. Future  import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits.global object FutureSample extends App

    { Future { println(s"[ThreadName] In Future: ${Thread.currentThread.getName}") println(1) } println(s"[ThreadName] In Main: ${Thread.currentThread.getName}") println(2) } Future.applyͰғΉͱɺผεϨουͰඇಉظॲཧ͕૸Δ [ThreadName] In Main: main 2 [ThreadName] In Future: scala-execution-context-global-11 1 ͜͜͸ ผεϨου
  16. Future  MainεϨου ผεϨου println(1) println(2) object FutureSample extends App

    { Future { println(s"[ThreadName] In Future: ${Thread.currentThread.getName}") println(1) } println(s"[ThreadName] In Main: ${Thread.currentThread.getName}") println(2) } Future.applyͰғΉͱɺผεϨουͰඇಉظॲཧ͕૸Δ
  17. Futureͷ߹੒Մೳੑ  map, flatMap def createResponse(user: User, team: Team): String

    val userF = Future { database.findUser("user1") } val teamF = Future { database.findTeam("team1") } val response = userF.flatMap { user => teamF.map { team => createResponse(user, team) } } response.foreach(println)
  18. Futureͷ߹੒Մೳੑ  ෳ਺ͷεϨουͰߦΘΕ͍ͯΔ ॲཧಉ࢜ͷ଴ͪ߹Θͤɺ Τϥʔͷ࣌ͷॲཧΛॻ͘ඞཁ͕ͳ͍ MainεϨου val userF val teamF

    val response Future.apply Future.apply database.findTeam user team response.onComplete flatMap map database.findUser def createResponse(user: User, team: Team): String val userF = Future { database.findUser("user1") } val teamF = Future { database.findTeam("team1") } val response = userF.flatMap { user => teamF.map { team => createResponse(user, team) } } response.foreach(println)
  19. Future಺Ͱͷঢ়ଶ؅ཧ  def deposit(account: Account, amount: Int): Future[Unit] = Future

    { if(account.money - amount > 0) { if(amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") jack's money: -3000 ϒϩοΫॲཧ ඇಉظʹAccountΦϒδΣΫτʹΞΫηε͢ΔϓϩάϥϜ
  20. Future಺Ͱͷঢ়ଶ؅ཧ  jack's money: -3000 def deposit(account: Account, amount: Int):

    Future[Unit] = Future { if(account.money - amount > 0) { if(amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ॲཧ͕ϒϩοΫ͞Ε͍ͯΔؒʹɺ ผͷεϨου͔ΒAccountΦϒδΣΫτʹ ΞΫηε͞Εͯ͠·͏
  21. Future಺Ͱͷঢ়ଶ؅ཧ  Insufficient funds (5000 < 8000) jack's money: 5000

    def deposit(account: Account, amount: Int): Future[Unit] = Future { account.synchronized { if (account.money - amount > 0) { if (amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ϩοΫΛ༻͍ͯಉظ
  22. Future಺Ͱͷঢ়ଶ؅ཧ  Insufficient funds (5000 < 8000) jack's money: 5000

    def deposit(account: Account, amount: Int): Future[Unit] = Future { account.synchronized { if (account.money - amount > 0) { if (amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ϩοΫΛ༻͍ͯಉظ σουϩοΫͷةݥੑ ա৒ͳϩοΫʹΑΔύϑΥʔϚϯεͷ௿Լ
  23. ΞΫλʔϞσϧ  ΦϒδΣΫτࢦ޲ ΞΫλʔࢦ޲ • શͯ͸ΦϒδΣΫτ • ঢ়ଶͷΧϓηϧԽ • ৼΔ෣͍

    • ΦϒδΣΫτͷੜ੒ • ΦϒδΣΫτ΁ͷϝοηʔδ • શͯ͸ΞΫλʔ • ঢ়ଶͷΧϓηϧԽ • ৼΔ෣͍ • ΞΫλʔͷੜ੒ • ΞΫλʔ΁ͷϝοηʔδ ʮશͯͷ΋ͷ͸ΞΫλʔͰ͋Δʯͱ͍͏ࢥ૝ class AccountActor(val name: String, var money: Int) extends Actor class Account private (val name: String, var money: Int)
  24. Akka Actor  Dispatcher͕ ActorʹεϨουΛ ׂΓ౰ͯΔ ΞΫλʔ ͜͜ͷඇಉظॲཧ෦෼͸Akka Actor ಺෦ͰΑ͠ͳʹ΍ͬͯ͘ΕΔ

    ϓϩάϥϚʔ͸Actor͕࣮ߦ͢Δॲཧͱɺ ϝοηʔδͷૹ৴ͷ࣮૷ʹूதͰ͖Δ DispatcherʹׂΓ౰ͯΒΕͨ εϨουΛ࢖ͬͯॲཧΛ࣮ߦ ΞΫλʔ ΞΫλʔ ΞΫλʔ
  25. Akka Actor  class HelloActor extends Actor { def receive

    = { case "Hello" => println(s"World!") } } val actorSystem = ActorSystem("System") val helloActor = actorSystem.actorOf(Props[HelloActor]) helloActor ! "Hello" ΞΫλʔͷ࣮ߦ͢Δॲཧͷ࣮૷ ϝοηʔδͷૹ৴ ඇಉظϝοηʔδύογϯάͷ࣮૷ ΞΫλʔͷੜ੒
  26. ୯ҰΞΫλʔ಺Ͱͷ ϝοηʔδॲཧͷஞ࣍ੑ  ಺෦ঢ়ଶ͸อޢ͞ΕΔͨΊɺಉظॲཧΛϓϩάϥϚ͕΍Δඞཁ͕ͳ͍ money: 2000 Insufficient funds (2000 <

    5000) class AccountActor(val name: String, var money: Int) extends Actor { override def receive: Receive = { case deposit: Deposit => { if(money - deposit.amount > 0) { if(deposit.amount == 8000) Thread.sleep(1000) money -= deposit.amount println(s"money: $money") } else { println(s"Insufficient funds ($money < ${deposit.amount})") } } } } lazy val ourSystem = ActorSystem("OurSystem") val accountActor = ourSystem.actorOf(Props(classOf[AccountActor], "Jack", 10000)) accountActor ! Deposit(8000) accountActor ! Deposit(5000) ϝοηʔδ͸ඇಉظʹૹΒΕΔ͕ɺ ॲཧ͸ஞ࣍తʹߦΘΕΔ
  27. ExecutionContext  object Future { def apply[T](body: =>T)(implicit @deprecatedName('execctx) executor:

    ExecutionContext): Future[T] } applyʹ౉ͯ͠ɺ಺෦ͰExecutionContext͕ඇಉظॲཧΛ࣮ߦ͍ͯ͠Δ Future Akka Actor dispatcher͸ExecutionContextΛ࢖ͬͯɺActorʹϝοηʔδΛૹ͍ͬͯΔ trait ActorContext extends ActorRefFactory { implicit def dispatcher: ExecutionContextExecutor } val defaultTaskSupport: TaskSupport = new ExecutionContextTaskSupport ฒྻ࣮ߦ͢Δtasksupportͷத਎͸ExecutionContext ฒྻίϨΫγϣϯ
  28. ExecutionContextͷར༻ํ๏  • ExecutionContext.global • ࣗ࡞ExecutionContext େ͖͘෼͚ͯ2छྨ͋Δ Cannot find an

    implicit ExecutionContext. You might pass [error] an (implicit ec: ExecutionContext) parameter to your method [error] or import scala.concurrent.ExecutionContext.Implicits.global.
  29. ExecutionContext.global  • scala.concurrentύοέʔδʹఆٛͯ͋͠ΔExecutionContext • ͱΓ͋͑ͣ͜ΕΛ࢖͑͹Future͕ಈ͘ • σϑΥϧτͰ͸ίΞ਺෼ͷεϨουϓʔϧΛ࡞੒͢Δ • ࣮૷͸BlockContextΛܧঝͨ͠ExecutorService

    • ฒྻίϨΫγϣϯͰσϑΥϧτͰ࢖ΘΕΔ implicit val ec = scala.concurrent.ExecutionContext.global import scala.concurrent.ExecutionContext.Implicits.global
  30. ࣗ࡞ExecutionContext  val executorService = Executors.newFixedThreadPool(4) implicit val ec =

    ExecutionContext.fromExecutorService(executorService) Future { … } ExecutorService͔ΒExecutionContextΛ࡞੒͢Δ͜ͱ͕Ͱ͖Δ ExecutorService: java.util.concurrentύοέʔδʹఆٛ͞Ε͍ͯΔ εϨουϓʔϧΛѻ͏ΦϒδΣΫτ
  31. ࣗ࡞ExecutionContext  ฒྻίϨΫγϣϯ parVector.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(4)) Akka Actor

    fork-join—dispatcher { type = Dispatcher executor = "fork-join-executor" fork-join-executor { parallelism-min = 8 parallelism-factor = 3.0 parallelism-max = 64 } throughput = 100 } thread-pool-dispatcher { type = Dispatcher executor = "thread-pool-executor" thread-pool-executor { fixed-pool-size = 32 } throughput = 1 } val helloActor = actorSystem.actorOf(Props[HelloActor].withDispatcher(“fork-join-dispatcher”)) val greeterActor = actorSystem.actorOf(Props[GreetActor].withDispatcher("thread-pool-dispatcher")) application.conf application.conf
  32. εϨουϓʔϧͷ෼ׂ  εϨου εϨου εϨου εϨου ExecutionContext.global εϨου εϨου ࣗ࡞ExecutionContext

    ϒϩοΩϯάॲཧ ϊϯϒϩοΩϯάॲཧ ϒϩοΩϯάॲཧ͸ͦΕઐ༻ͷExecutionContextΛ࡞੒ͯ͠ɺ ڞ༗ͷεϨουϓʔϧΛ઎༗͠ͳ͍Α͏ʹͨ͠΄͏͕Α͍
  33. ϒϩοΩϯάॲཧ with global  blockingϝιου͕ݺ͹Εͨ࣌ʹɺ εϨουΛࣗಈͰ૿΍ͯ͘͠ΕΔʢॳظεϨου਺: 8ʣ implicit val ec

    = scala.concurrent.ExecutionContext.global val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for (f <- futures) Await.ready(f, Duration.Inf) } log(s"Total time = ${time.value}") ExecutionContext.global
  34. ϒϩοΩϯάॲཧ with global  Total time = 1048 implicit val

    ec = scala.concurrent.ExecutionContext.global val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for (f <- futures) Await.ready(f, Duration.Inf) } log(s"Total time = ${time.value}") blockingϝιου͕ݺ͹ΕΔͱ ExecutionContext.global͕ ࣗಈͰεϨουΛੜ੒͢Δ blockingϝιου͕ݺ͹Εͨ࣌ʹɺ εϨουΛࣗಈͰ૿΍ͯ͘͠ΕΔʢॳظεϨου਺: 8ʣ
  35. ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  blockingϝιου͕ݺ͹Εͯ΋ɺ εϨου͕૿͑ͳ͍ʢεϨου਺: 8ʣ val executorService =

    Executors.newWorkStealingPool(8) implicit val ec = ExecutionContext.fromExecutorService(executorService) val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for(f <- futures) Await.ready(f, Duration.Inf) } println(s"Total time = ${time.value}") fromExecutorServiceΛ࢖༻
  36. ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  Total time = 13071 ExecutionContext.global ࢖༻࣌ΑΓ΋

    10ഒͷ࣌ؒΛཁ͢Δ blockingϝιου͕ݺ͹Εͯ΋ɺ εϨου͕૿͑ͳ͍ʢεϨου਺: 8ʣ val executorService = Executors.newWorkStealingPool(8) implicit val ec = ExecutionContext.fromExecutorService(executorService) val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for(f <- futures) Await.ready(f, Duration.Inf) } println(s"Total time = ${time.value}") ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  fromExecutorServiceΛ࢖༻
  37. ·ͱΊ  ϥΠϒϥϦ ಛ௃ Ϣʔεέʔε ฒྻίϨΫγϣϯ ίϨΫγϣϯʹର͢Δ ฒྻॲཧ ݕࡧ/બ୒/ ιʔτ/ूܭ

    'VUVSF 4DBMBΒ͍͠߹੒Մೳͳ ඇಉظॲཧ ঢ়ଶΛѻΘͳ͍ॲཧ ؔ਺ "LLB"DUPS ΞΫλʔϞσϧʹجͮ͘ ඇಉظϝοηʔδύογϯά ঢ়ଶΛѻ͏ΦϒδΣΫτ ϨεϙϯγϒͳγεςϜ