Scalaでの並行・並列処理戦略/strategy-for-concurrency-and-parallel-by-scala

Ca4df28501e4c9cfbceb91f367afa784?s=47 fuzyco
November 10, 2018

 Scalaでの並行・並列処理戦略/strategy-for-concurrency-and-parallel-by-scala

Scala関西サミット2018で発表したスライドです。

Ca4df28501e4c9cfbceb91f367afa784?s=128

fuzyco

November 10, 2018
Tweet

Transcript

  1. 26.

    ScalaίϨΫγϣϯ  ɾList, Array, etc. scala> val list = (1

    to 10000).toList list: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, … scala> list.map(_ + 42) res0: List[Int] = List(43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, … scala> list.max res1: Int = 10000
  2. 27.

    ScalaฒྻίϨΫγϣϯ  ίϨΫγϣϯͷԋࢉΛฒྻʹ࣮ߦ͢Δ͜ͱ͕Ͱ͖Δ scala> val parList = list.par parList: scala.collection.parallel.immutable.ParSeq[Int]

    = ParVector(1, 2, 3, 4, 5, …) scala> parList.seq res1: scala.collection.immutable.Seq[Int] = Vector(1, 2, 3, 4, 5, 6, … ) ม׵͕Մೳ scala> parList.foreach(elm => print(s"$elm ")) 376 377 501 813 876 2 … ฒྻʹॲཧ͕࣮ߦ͞ΕΔ
  3. 28.

    ScalaฒྻίϨΫγϣϯ  import scala.util.Random val numbers = Random.shuffle(Vector.tabulate(10000000)(i => i))

    val seqtime = measure { numbers.max } log(s"Sequential time ${seqtime.value} ms”) val partime = measure { numbers.par.max } log(s"Parallel time ${partime.value} ms") Sequential time 535.850688 ms Parallel time 193.393754 ms parϝιουΛ࢖͚ͬͨͩͰ ฒྻʹॲཧ͕ߦΘΕΔ ϥϯμϜͳ஋ͷཁૉΛ10000000ݸ࣋ͭVectorͷ ࠷େͷཁૉΛٻΊΔϓϩάϥϜ
  4. 30.
  5. 31.

    ෼ׂ౷࣏  ໰୊Λ෼ׂͯ͠ɺͦΕͧΕಠཱͯ͠ࢉग़͞ΕͨղΛ ౷߹͍ͯ͘͠Ξϓϩʔν(ྫ: Ϛʔδιʔτ) ෦෼ ղ ղ ෦෼ ղ

    ෦෼ ղ ෦෼ ղ ฒྻԽՄೳ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊ ෦෼ ໰୊
  6. 32.

    ෼ׂ౷࣏  ߹ܭ஋ શମͷ ߹ܭ஋ ߹ܭ஋ ߹ܭ஋ ߹ܭ஋ ֤CPUͰฒྻʹ࣮ߦ "SSBZ

    ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ sumϝιουݺͼग़͠ʹΑΔίϨΫγϣϯͷ߹ܭ஋ͷܭࢉ ArrayΛεϨου ͷ਺͚ͩ෼ׂ ͦΕͧΕͷ߹ܭ஋ Λ଍͠߹ΘͤΔ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ ෼ׂ͞Εͨ഑ྻ ͷ߹ܭ஋ͷܭࢉ
  7. 34.

    ฒྻίϨΫγϣϯͷ࢓૊Έ  Splitter ෼ׂ౷࣏Λ࣮ݱ͍ͯ͠ΔΦϒδΣΫτ trait IterableSplitter[T] extends Iterator[T] { def

    dup: IterableSplitter[T] def remaining: Int def split: Seq[IterableSplitter[T]] } ίϨΫγϣϯΛޮ཰Α͘෼ׂ͢Δ Combiner trait Combiner[-Elem, +To] extends Builder[Elem, To] with Sizing with Parallel { def combine[N <: Elem, NewTo >: To](other: Combiner[N, NewTo]): Combiner[N, NewTo] } ෼ׂ͞ΕͨλεΫͷԋࢉ݁ՌΛ݁߹͢Δ
  8. 37.

    εϨουؒͷಉظॲཧ (σʔλڝ߹)  object ParUid extends App { private val

    uid = new AtomicLong(0L) val seqtime = measure { for(i <- 0 until 10000000) uid.incrementAndGet() } println(s"Sequential time ${seqtime.value} ms”) val partime = measure { for(i <- (0 until 10000000).par) uid.incrementAndGet() } println(s"Parallel time ${partime.value} ms") } ※measureؔ਺͸ScalaMeterͷॲཧ࣌ؒΛܭଌ͢Δؔ਺ uidΛAtomicม਺Ͱ҆શͳಉظΛ ߦͬͯΧ΢ϯτΞοϓ͍ͯ͠Δ ϧʔϓॲཧͰΧ΢ϯτΞοϓ͢ΔϓϩάϥϜ
  9. 38.

    εϨουؒͷಉظॲཧ (σʔλڝ߹)  4FRVFOUJBMUJNFNT 1BSBMMFMUJNFNT object ParUid extends App {

    private val uid = new AtomicLong(0L) val seqtime = measure { for(i <- 0 until 10000000) uid.incrementAndGet() } println(s"Sequential time ${seqtime.value} ms”) val partime = measure { for(i <- (0 until 10000000).par) uid.incrementAndGet() } println(s"Parallel time $partime ms") } ฒྻίϨΫγϣϯΛ࢖ͬͨ΄͏͕ ͕͔͔͍࣌ؒͬͯΔ εϨουؒͰσʔλΛਖ਼͘͠ڞ༗͢ΔͨΊʹಉظΛߦ͏ͷͰɺ ֤εϨουͰॲཧ଴͕ͪൃੜͯ͠εϧʔϓοτ͕௿Լ͢Δ
  10. 39.

    ฒྻίϨΫγϣϯ΁ͷม׵ίετ  ListͱVectorΛͦΕͧΕ ฒྻίϨΫγϣϯʹม׵͢ΔϓϩάϥϜ object ParNonParallelizableCollections extends App with Logger

    { val list = List.fill(1000000)("") val vector = Vector.fill(1000000)(“") println(s"list conversion time: ${measure(list.par).value} ms") println(s"vector conversion time: ${measure(vector.par).value} ms") }
  11. 40.

    ฒྻίϨΫγϣϯ΁ͷม׵ίετ  MJTUDPOWFSTJPOUJNFNT WFDUPSDPOWFSTJPOUJNFNT ListͱVectorΛͦΕͧΕ ฒྻίϨΫγϣϯʹม׵͢ΔϓϩάϥϜ object ParNonParallelizableCollections extends App

    with Logger { val list = List.fill(1000000)("") val vector = Vector.fill(1000000)(“") println(s"list conversion time: ${measure(list.par).value} ms") println(s"vector conversion time: ${measure(vector.par).value} ms") } ListͷฒߦίϨΫγϣϯ΁ͷม׵ ʹ͔ͳΓ͕͔͔͍࣌ؒͬͯΔ
  12. 42.

    ฒྻίϨΫγϣϯ΁ͷม׵ίετ  List Vector ParVector scala> (1 to 10000).toList.par res2:

    ParVector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, … શͯͷཁૉΛίϐʔ͢ΔͷͰܭࢉίετ͕͔͔Δ ฒྻίϨΫγϣϯͷ࣮૷ Λ࣌ͨͳ͍ίϨΫγϣϯ ฒྻίϨΫγϣϯͷ࣮૷ Λ࣋ͭίϨΫγϣϯ ฒྻίϨΫγϣϯ
  13. 45.

    ঢ়ଶΛѻ͏ԋࢉ  scala> var sum = 0 sum: Int =

    0 scala> val parList = (1 to 1000).toList.par parList: scala.collection.parallel.immutable.ParSeq[Int] = ParVector(1, 2, 3, 4, 5, … scala> var sum = 0 sum: Int = 0 scala> parList.foreach(sum += _); sum res01: Int = 307792 scala> parList.foreach(sum += _); sum res02: Int = 417753
  14. 47.

    ݁߹ଇ͕੒Γཱͨͳ͍ԋࢉ  ฒྻͰ࣮ߦ͞ΕΔͷͰɺ݁߹཯͕੒Γཱͨͳ͍ԋࢉ͸ႈ౳ͳ஋ʹͳΒͳ͍ ྫ: (1-2)-3 != 1-(2-3) scala> val parList

    = (1 to 1000).toList.par parList: scala.collection.parallel.immutable.ParSeq[Int] = ParVector(1, 2, 3, 4, 5, … scala> parList.reduce(_ - _) res0: Int = -169948 scala> parList.reduce(_ - _) res1: Int = 181648
  15. 49.

     ฒྻCollectionͰIOॲཧ val userIds: List[String] = List("id") def findUser(userId: String):

    User = { database.findUser(userId) } val users = userIds.par.map { userId => findUser(userId) } CPU1 IO଴ͪ CPU2 IO଴ͪ ฒྻίϨΫγϣϯ಺ͰIOό΢ϯυͳॲཧΛߦ͏ͱɺ IO଴ͪͷؒશͯͷCPUͷॲཧ͕ఀࢭͯ͠͠·͏ DB΁ͷ ΞΫηε
  16. 53.

    Future  import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits.global object FutureSample extends App

    { Future { println(s"[ThreadName] In Future: ${Thread.currentThread.getName}") println(1) } println(s"[ThreadName] In Main: ${Thread.currentThread.getName}") println(2) } Future.applyͰғΉͱɺผεϨουͰඇಉظॲཧ͕૸Δ [ThreadName] In Main: main 2 [ThreadName] In Future: scala-execution-context-global-11 1 ͜͜͸ ผεϨου
  17. 54.

    Future  MainεϨου ผεϨου println(1) println(2) object FutureSample extends App

    { Future { println(s"[ThreadName] In Future: ${Thread.currentThread.getName}") println(1) } println(s"[ThreadName] In Main: ${Thread.currentThread.getName}") println(2) } Future.applyͰғΉͱɺผεϨουͰඇಉظॲཧ͕૸Δ
  18. 57.

    Futureͷ߹੒Մೳੑ  map, flatMap def createResponse(user: User, team: Team): String

    val userF = Future { database.findUser("user1") } val teamF = Future { database.findTeam("team1") } val response = userF.flatMap { user => teamF.map { team => createResponse(user, team) } } response.foreach(println)
  19. 58.

    Futureͷ߹੒Մೳੑ  ෳ਺ͷεϨουͰߦΘΕ͍ͯΔ ॲཧಉ࢜ͷ଴ͪ߹Θͤɺ Τϥʔͷ࣌ͷॲཧΛॻ͘ඞཁ͕ͳ͍ MainεϨου val userF val teamF

    val response Future.apply Future.apply database.findTeam user team response.onComplete flatMap map database.findUser def createResponse(user: User, team: Team): String val userF = Future { database.findUser("user1") } val teamF = Future { database.findTeam("team1") } val response = userF.flatMap { user => teamF.map { team => createResponse(user, team) } } response.foreach(println)
  20. 59.

    Future಺Ͱͷঢ়ଶ؅ཧ  def deposit(account: Account, amount: Int): Future[Unit] = Future

    { if(account.money - amount > 0) { if(amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") jack's money: -3000 ϒϩοΫॲཧ ඇಉظʹAccountΦϒδΣΫτʹΞΫηε͢ΔϓϩάϥϜ
  21. 60.

    Future಺Ͱͷঢ়ଶ؅ཧ  jack's money: -3000 def deposit(account: Account, amount: Int):

    Future[Unit] = Future { if(account.money - amount > 0) { if(amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ॲཧ͕ϒϩοΫ͞Ε͍ͯΔؒʹɺ ผͷεϨου͔ΒAccountΦϒδΣΫτʹ ΞΫηε͞Εͯ͠·͏
  22. 61.

    Future಺Ͱͷঢ়ଶ؅ཧ  Insufficient funds (5000 < 8000) jack's money: 5000

    def deposit(account: Account, amount: Int): Future[Unit] = Future { account.synchronized { if (account.money - amount > 0) { if (amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ϩοΫΛ༻͍ͯಉظ
  23. 62.

    Future಺Ͱͷঢ়ଶ؅ཧ  Insufficient funds (5000 < 8000) jack's money: 5000

    def deposit(account: Account, amount: Int): Future[Unit] = Future { account.synchronized { if (account.money - amount > 0) { if (amount == 8000) Thread.sleep(1000) account.money -= amount } else { println(s"Insufficient funds (${account.money} < ${amount})") } } } val account = new Account("Jack", 10000) val result1 = deposit(account, 8000) val result2 = deposit(account, 5000) Await.ready(Future.sequence(List(result1, result2)), Duration.Inf) println(s"jack's money: ${account.money}") ϩοΫΛ༻͍ͯಉظ σουϩοΫͷةݥੑ ա৒ͳϩοΫʹΑΔύϑΥʔϚϯεͷ௿Լ
  24. 67.

    ΞΫλʔϞσϧ  ΦϒδΣΫτࢦ޲ ΞΫλʔࢦ޲ • શͯ͸ΦϒδΣΫτ • ঢ়ଶͷΧϓηϧԽ • ৼΔ෣͍

    • ΦϒδΣΫτͷੜ੒ • ΦϒδΣΫτ΁ͷϝοηʔδ • શͯ͸ΞΫλʔ • ঢ়ଶͷΧϓηϧԽ • ৼΔ෣͍ • ΞΫλʔͷੜ੒ • ΞΫλʔ΁ͷϝοηʔδ ʮશͯͷ΋ͷ͸ΞΫλʔͰ͋Δʯͱ͍͏ࢥ૝ class AccountActor(val name: String, var money: Int) extends Actor class Account private (val name: String, var money: Int)
  25. 75.

    Akka Actor  Dispatcher͕ ActorʹεϨουΛ ׂΓ౰ͯΔ ΞΫλʔ ͜͜ͷඇಉظॲཧ෦෼͸Akka Actor ಺෦ͰΑ͠ͳʹ΍ͬͯ͘ΕΔ

    ϓϩάϥϚʔ͸Actor͕࣮ߦ͢Δॲཧͱɺ ϝοηʔδͷૹ৴ͷ࣮૷ʹूதͰ͖Δ DispatcherʹׂΓ౰ͯΒΕͨ εϨουΛ࢖ͬͯॲཧΛ࣮ߦ ΞΫλʔ ΞΫλʔ ΞΫλʔ
  26. 76.

    Akka Actor  class HelloActor extends Actor { def receive

    = { case "Hello" => println(s"World!") } } val actorSystem = ActorSystem("System") val helloActor = actorSystem.actorOf(Props[HelloActor]) helloActor ! "Hello" ΞΫλʔͷ࣮ߦ͢Δॲཧͷ࣮૷ ϝοηʔδͷૹ৴ ඇಉظϝοηʔδύογϯάͷ࣮૷ ΞΫλʔͷੜ੒
  27. 77.

    ୯ҰΞΫλʔ಺Ͱͷ ϝοηʔδॲཧͷஞ࣍ੑ  ಺෦ঢ়ଶ͸อޢ͞ΕΔͨΊɺಉظॲཧΛϓϩάϥϚ͕΍Δඞཁ͕ͳ͍ money: 2000 Insufficient funds (2000 <

    5000) class AccountActor(val name: String, var money: Int) extends Actor { override def receive: Receive = { case deposit: Deposit => { if(money - deposit.amount > 0) { if(deposit.amount == 8000) Thread.sleep(1000) money -= deposit.amount println(s"money: $money") } else { println(s"Insufficient funds ($money < ${deposit.amount})") } } } } lazy val ourSystem = ActorSystem("OurSystem") val accountActor = ourSystem.actorOf(Props(classOf[AccountActor], "Jack", 10000)) accountActor ! Deposit(8000) accountActor ! Deposit(5000) ϝοηʔδ͸ඇಉظʹૹΒΕΔ͕ɺ ॲཧ͸ஞ࣍తʹߦΘΕΔ
  28. 84.

    ExecutionContext  object Future { def apply[T](body: =>T)(implicit @deprecatedName('execctx) executor:

    ExecutionContext): Future[T] } applyʹ౉ͯ͠ɺ಺෦ͰExecutionContext͕ඇಉظॲཧΛ࣮ߦ͍ͯ͠Δ Future Akka Actor dispatcher͸ExecutionContextΛ࢖ͬͯɺActorʹϝοηʔδΛૹ͍ͬͯΔ trait ActorContext extends ActorRefFactory { implicit def dispatcher: ExecutionContextExecutor } val defaultTaskSupport: TaskSupport = new ExecutionContextTaskSupport ฒྻ࣮ߦ͢Δtasksupportͷத਎͸ExecutionContext ฒྻίϨΫγϣϯ
  29. 86.

    ExecutionContextͷར༻ํ๏  • ExecutionContext.global • ࣗ࡞ExecutionContext େ͖͘෼͚ͯ2छྨ͋Δ Cannot find an

    implicit ExecutionContext. You might pass [error] an (implicit ec: ExecutionContext) parameter to your method [error] or import scala.concurrent.ExecutionContext.Implicits.global.
  30. 87.

    ExecutionContext.global  • scala.concurrentύοέʔδʹఆٛͯ͋͠ΔExecutionContext • ͱΓ͋͑ͣ͜ΕΛ࢖͑͹Future͕ಈ͘ • σϑΥϧτͰ͸ίΞ਺෼ͷεϨουϓʔϧΛ࡞੒͢Δ • ࣮૷͸BlockContextΛܧঝͨ͠ExecutorService

    • ฒྻίϨΫγϣϯͰσϑΥϧτͰ࢖ΘΕΔ implicit val ec = scala.concurrent.ExecutionContext.global import scala.concurrent.ExecutionContext.Implicits.global
  31. 88.

    ࣗ࡞ExecutionContext  val executorService = Executors.newFixedThreadPool(4) implicit val ec =

    ExecutionContext.fromExecutorService(executorService) Future { … } ExecutorService͔ΒExecutionContextΛ࡞੒͢Δ͜ͱ͕Ͱ͖Δ ExecutorService: java.util.concurrentύοέʔδʹఆٛ͞Ε͍ͯΔ εϨουϓʔϧΛѻ͏ΦϒδΣΫτ
  32. 89.

    ࣗ࡞ExecutionContext  ฒྻίϨΫγϣϯ parVector.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(4)) Akka Actor

    fork-join—dispatcher { type = Dispatcher executor = "fork-join-executor" fork-join-executor { parallelism-min = 8 parallelism-factor = 3.0 parallelism-max = 64 } throughput = 100 } thread-pool-dispatcher { type = Dispatcher executor = "thread-pool-executor" thread-pool-executor { fixed-pool-size = 32 } throughput = 1 } val helloActor = actorSystem.actorOf(Props[HelloActor].withDispatcher(“fork-join-dispatcher”)) val greeterActor = actorSystem.actorOf(Props[GreetActor].withDispatcher("thread-pool-dispatcher")) application.conf application.conf
  33. 92.

    εϨουϓʔϧͷ෼ׂ  εϨου εϨου εϨου εϨου ExecutionContext.global εϨου εϨου ࣗ࡞ExecutionContext

    ϒϩοΩϯάॲཧ ϊϯϒϩοΩϯάॲཧ ϒϩοΩϯάॲཧ͸ͦΕઐ༻ͷExecutionContextΛ࡞੒ͯ͠ɺ ڞ༗ͷεϨουϓʔϧΛ઎༗͠ͳ͍Α͏ʹͨ͠΄͏͕Α͍
  34. 95.

    ϒϩοΩϯάॲཧ with global  blockingϝιου͕ݺ͹Εͨ࣌ʹɺ εϨουΛࣗಈͰ૿΍ͯ͘͠ΕΔʢॳظεϨου਺: 8ʣ implicit val ec

    = scala.concurrent.ExecutionContext.global val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for (f <- futures) Await.ready(f, Duration.Inf) } log(s"Total time = ${time.value}") ExecutionContext.global
  35. 96.

    ϒϩοΩϯάॲཧ with global  Total time = 1048 implicit val

    ec = scala.concurrent.ExecutionContext.global val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for (f <- futures) Await.ready(f, Duration.Inf) } log(s"Total time = ${time.value}") blockingϝιου͕ݺ͹ΕΔͱ ExecutionContext.global͕ ࣗಈͰεϨουΛੜ੒͢Δ blockingϝιου͕ݺ͹Εͨ࣌ʹɺ εϨουΛࣗಈͰ૿΍ͯ͘͠ΕΔʢॳظεϨου਺: 8ʣ
  36. 97.

    ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  blockingϝιου͕ݺ͹Εͯ΋ɺ εϨου͕૿͑ͳ͍ʢεϨου਺: 8ʣ val executorService =

    Executors.newWorkStealingPool(8) implicit val ec = ExecutionContext.fromExecutorService(executorService) val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for(f <- futures) Await.ready(f, Duration.Inf) } println(s"Total time = ${time.value}") fromExecutorServiceΛ࢖༻
  37. 98.

    ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  Total time = 13071 ExecutionContext.global ࢖༻࣌ΑΓ΋

    10ഒͷ࣌ؒΛཁ͢Δ blockingϝιου͕ݺ͹Εͯ΋ɺ εϨου͕૿͑ͳ͍ʢεϨου਺: 8ʣ val executorService = Executors.newWorkStealingPool(8) implicit val ec = ExecutionContext.fromExecutorService(executorService) val time = measure { val futures = for(_ <- 0 until 100) yield Future { blocking { Thread.sleep(1000) } } for(f <- futures) Await.ready(f, Duration.Inf) } println(s"Total time = ${time.value}") ϒϩοΩϯάॲཧ with ࣗ࡞ExecutionContext  fromExecutorServiceΛ࢖༻
  38. 103.

    ·ͱΊ  ϥΠϒϥϦ ಛ௃ Ϣʔεέʔε ฒྻίϨΫγϣϯ ίϨΫγϣϯʹର͢Δ ฒྻॲཧ ݕࡧ/બ୒/ ιʔτ/ूܭ

    'VUVSF 4DBMBΒ͍͠߹੒Մೳͳ ඇಉظॲཧ ঢ়ଶΛѻΘͳ͍ॲཧ ؔ਺ "LLB"DUPS ΞΫλʔϞσϧʹجͮ͘ ඇಉظϝοηʔδύογϯά ঢ়ଶΛѻ͏ΦϒδΣΫτ ϨεϙϯγϒͳγεςϜ