akka-stream を始めるときに覚えておきたいこと

akka-stream 特有の単語や概念を分かりやすく説明します。

Transcript

1. akka-stream Λ࢝ΊΔͱ͖ʹ͓֮͑ͯ ͖͍ͨ͜ͱ Տ಺ ਸ Twitter: @kawachi ɹ ɹ Photo by Junkichi Egashira is licensed under CC BY 4.0 International 1
2. ετϦʔϜॲཧ σʔλΛஞ࣍ॲཧ͢Δ͜ͱ • ϝϞϦʹ৐Γ੾Βͳ͍σʔλͷॲཧ • 1TBͷϑΝΠϧΛॲཧ͍ͨ͠ • ετϦʔϜͱͯ͠දݱ͢Δͷ͕ࣗવ ͳॲཧʢ௨৴ʣͳͲ • νϟοτͷ bot Λ࡞Γ͍ͨ 2
3. ετϦʔϜॲཧͷ؆୯ͳྫ୊Λߟ͑ͯΈ·͠ΐ͏ɻ Q. ϑΝΠϧ͕nߦҎ্͋Δ͔൑ఆͤΑ ͨͩ͠ϑΝΠϧ͸ϝϞϦʹ৐Γ੾Βͳ͍΄Ͳେ͖͍ɻ ಛେϑΝΠϧ͕૬खͷՄೳੑ͕͋ΔͷͰɺ൑ఆͰ͖ͨͱ͜ΖͰ ಡΈࠐΈΛऴΘΔ͜ͱɻ 3
4. ͸͍ɻIterator ͰͰ͖·͢ɻ def q1(fileName: String, n: Int): Boolean = { val src = io.Source.fromFile(fileName) val lines: Iterator[String] = src.getLines() var count = 0 while (lines.hasNext && n < count) { lines.next() count += 1 } count == n } 4
5. ՝୊1: Ϧιʔε؅ཧͰؒҧ͍΍͍͢ def q1Again(fileName: String, n: Int): Boolean = { val src = io.Source.fromFile(fileName) try { val lines: Iterator[String] = src.getLines() var count = 0 while (lines.hasNext && n < count) { lines.next() count += 1 } count == n } finally { src.close() } } 5
6. ΋͏গ͠೉͍͠໰୊Λߟ͑ͯΈ·͠ΐ͏ɻ Q. ϑΝΠϧ͕nߦҎ্͋Δ͔ɺ͓Αͼ ۭߦ͕mߦҎ্͋Δ͔Λ൑ఆͤΑ 6
7. def hasMoreThan(lines: Iterator[String], n: Int) = { var count = 0 while (lines.hasNext && count < n) { lines.next() count += 1 } count == n } def q2(fileName: String, n: Int, m: Int) = { val src = io.Source.fromFile(fileName) try { val lines: Iterator[String] = src.getLines() (hasMoreThan(lines, n), hasMoreThan(lines.filter(_.isEmpty), m)) } finally { src.close() } } 7
8. ՝୊2: εςʔτϑϧͳͷͰؒҧ͑΍͍͢ val lines: Iterator[String] = src.getLines() (hasMoreThan(lines, n), hasMoreThan(lines.filter(_.isEmpty), m)) // ^^^^^ Good ^^^^^ !!! >_< 8
9. ՝୊3: εϨου͕ block ͞ΕΔ Iterator ͸ .hasNext ͱ .next() ͕ϕʔεɻ while (lines.hasNext) { lines.next() // <- !!! blocking !!! … } 9
10. actor Ͱ՝୊ΛղܾͰ͖ΔͩΖ͏͔? 10
11. • ՝୊1: Ϧιʔε؅ཧͰؒҧ͍΍͍͢ • → Ϧιʔε؅ཧΛϑΝΠϧಡΈࠐΈ Actor ʹہॴԽͰ͖Δɻͦͷ Actor ֎ Ͱ͸ؒҧ͑ͳ͍ɻ • ՝୊2: εςʔτϑϧͳͷͰؒҧ͑΍͍͢ • → Actor Λ࡞Δࡍͷ akka.actor.Props ͸ immutable ͳ஋ɻ࠶ར༻Մೳɻ • ՝୊3: εϨου͕ block ͞ΕΔ • → ϝοηʔδۦಈͳͷͰඇಉظ API Λࣗવʹѻ͑Δ ׬શղܾʂ 11
12. ৽ͨͳ՝୊ 12
13. ࠓճ঺հ͢Δ akka-stream Ͱ͸ Actor ͱൺ΂ͯ • ߏ੒ཁૉ͕ग़ྗ/ೖྗ͢Δཁૉʹܕ͕͍͍ͭͯΔ • όοϑΝᷓΕ͠ͳ͍ Iterator ͱൺ΂ͯ • Ϧιʔε؅ཧ͕ہॴԽ͞Εؒҧ͑ʹ͍͘ • ߏ੒ཁૉ͕ immutable ͳ஋ • ඇಉظ API ͱ૊Έ߹ΘͤΒΕΔ 13
14. akka-stream 14
15. ґଘੑ௥Ճ Play 2.5 ͔Β Play ͷ࣮૷ʹ akka-stream ͕࢖ΘΕ͍ͯΔ → ৽ͨʹґଘੑΛ௥Ճ͢Δඞཁͳ͠ Play ؀ڥͰͳ͍৔߹͸ґଘੑΛ௥Ճ libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.4.11" 15
16. ୯७ͳετϦʔϜ 16
17. ྫ: 1~5 Λྲྀͯ͠ * 2 ͯ͠ println() import akka.actor.ActorSystem import akka.stream.ActorMaterializer import akka.stream.scaladsl._ val src = Source(1 to 5) val flow = Flow[Int].map(_ * 2) val sink = Sink.foreach[Int](println) val stream = src.via(flow).to(sink) implicit val system = ActorSystem() implicit val materializer = ActorMaterializer() stream.run() 17
18. ετϦʔϜͷهड़ val src = Source(1 to 5) val flow = Flow[Int].map(_ * 2) val sink = Sink.foreach[Int](println) val stream = src.via(flow).to(sink) • هड़ ͱ࣮ߦ͕෼཭͞Ε͍ͯΔ • هड़͸ઃܭॻΛॻ͘࡞ۀ • ෭࡞༻ͳ͠ • src, flow, sink, stream ͷ࠶ར༻OK! 18
19. Source, Flow, Sink 19
20. Source, Flow, Sink ͷ߹੒ 20
21. RunnableGraph RunnableGraph ͸ run() ՄೳͳετϦ ʔϜͷهड़ɻ run() ͢Δʹ͸ೖग़ྗ͕શͯ઀ଓ͞Ε ͍ͯΔඞཁ͕͋Δɻ Source ΍ Flowɺ Sink ͸ run() Ͱ͖ͳ ͍ɻ 21
22. ετϦʔϜͷ࣮ߦ • ࣮ߦ։࢝͢Δ͜ͱΛ materialize (͋Δ͍͸ run)ͱݺͿ • ࣮ߦʹ͸ Materializer ͕ඞཁ • Ұ౓هड़ͨ͠ετϦʔϜ͸Կ౓Ͱ΋࣮ߦͰ͖Δ implicit val system = ActorSystem() implicit val materializer = ActorMaterializer()(system) stream.run()(materializer) 22
23. Materialized value 23
24. run() ͭ·Γ materialize ͷ໭Γ஋ɻ஋ ͱͯ͠ಘΒΕΔͷ͸ Materialized value ͷΈɻ ༻్ • ੍ޚϙΠϯτ • ֎෦͔ΒετϦʔϜΛ੍ޚ͢Δ • ؂ࢹϙΠϯτ • ࣮ߦঢ়ଶ΍ॲཧ݁Ռ 24
25. val stream: RunnableGraph[Future[IOResult]] = FileIO.fromPath(Paths.get("src_file")) .to(FileIO.toPath(Paths.get("dest_file"))) val result: Future[IOResult] = stream.run() // ↑ Materialized value result.onSuccess { case ioResult => println(s"${ioResult.count} byteॲཧ͠·ͨ͠") } 25
26. RunnableGraph ͷ(ຊ౰ͷ)ܕ RunnableGraph[+Mat] Mat ͕ materialized value ͷܕ run() ͢Δͱ Mat ܕͷ materialized value ͕໭Γ஋ͱͯ͠ฦ͞ ΕΔɻ 26
27. Source, Flow, Sink ͷ(ຊ౰ͷ)ܕ Source[+Out,+Mat] Flow[-In,+Out,+Mat] Sink[-In,+Mat] ߏ੒ཁૉ (Source, Flow, Sink) ͕ͦΕͧΕ Mat ܕͷ materialized value Λ࣋ͭɻ ͜ΕΒΛ૊Έ߹ͤͯ RunnableGraph Λ࡞ͬͯ run() ͢Δɻ run() ͷ໭Γ஋ͱͯ͠ಘΒΕΔͷ͸ RunnableGraph ͷ Mat ͚ͩɻ 27
30. ߹੒࣌ͷ materialized value • via(), to() ͸ࠨଆͷ materialzed value Λ߹੒ޙͷ materialized value ʹ࠾༻ • viaMat(), toMat() Ͱࣗ༝ʹͰ͖Δ import akka.stream.scaladsl.Keep source.viaMat(flow)(Keep.left) // ࠨͷ Mat (σϑΥϧτ) source.viaMat(flow)(Keep.right) // ӈͷ Mat source.viaMat(flow)(Keep.both) // ྆ Mat ͷλϓϧ source.viaMat(flow)(Keep.none) // NotUsed 30
31. Source, Flow, Sink 31
32. Source, Flow ڞ௨ͷجຊతͳૢ࡞ map(), filter(), filterNot(), take(), takeWhile(), drop(), dropWhile(), fold(), reduce() ඪ४ϥΠϒϥϦͷίϨΫγϣϯAPIʹࣅͨϝιου܈ Source ΍ Flow ͷग़ྗཁૉ͕ίϨΫγϣϯͷཁૉʹରԠ͢Δͱ ଊ͑Ε͹௚ײతʹૢ࡞Մೳ flatMap() ͸ͳ͍ɻmapConcat() ͋Δ͍͸ flatMapConcat() ͕૬౰ɻ 32
33. ετϦʔϜͳΒͰ͸ͷૢ࡞ • takeWithin(d: FiniteDuration) • dropWithin(d: FiniteDuration) • ࢦఆ࣌ؒ಺ʹ౸ணͨ͠΋ͷ͚ͩऔΓग़͢/ࣺͯΔ • groupedWithin(n: Int, d: FiniteDuration) • ࢦఆͨ࣌ؒ͠ʹ౸ணͨ͠ཁૉʢ࠷େn݅ʣΛ·ͱΊͯ Seq Ͱग़ྗ 33
34. Source ͷ࡞Γํ (1) • Source.apply[T](iterable: Iterable[T]): Source[T, NotUsed] Source.fromIterator[T](f: () 㱺 Iterator[T]): Source[T, NotUsed] • Iterable ΍ Iterator ͔Β Source Λ࡞Δ • Source.repeat[T](element: T): Source[T, NotUsed] • elementΛӬԕʹग़͢ 34
35. Source ͷ࡞Γํ (2) • Source.tick[T](initialDelay: FiniteDuration, interval: FiniteDuration, tick: T): Source[T, Cancellable] • Ұఆ࣌ؒຖʹ tick Λग़͢ • Source.actorRef[T](bufferSize: Int, overflowStrategy: OverflowStrategy): Source[T, ActorRef] • materialized value ͷ ActorRef ʹϝοηʔδΛ౤͛Δͱ Source ͔ Βग़ྗ͞ΕΔ ੍ޚϙΠϯτͱͯ͠ materialized value ͕࢖ΘΕΔྫ 35
37. Sink ͷ࡞Γํ • Sink.foreach[T](f: T 㱺 Unit): Sink[T, Future[Done]] • TܕͷཁૉΛडऔΔͨͼʹfΛ࣮ߦ͢Δ • Sink.fold[U, T](zero: U)(f: (U, T) 㱺 U): Sink[T, Future[U]] • Sink.reduce[T](f: (T, T) 㱺 T): Sink[T, Future[T]] • ৞ΈࠐΈ 37
38. Flow ͷ࡞Γํ • Flow.apply[T]: Flow[T,T,NotUsed] • ্ྲྀ͔Βड৴ͨ͠ཁૉΛͦͷ··Լྲྀ΁ૹ৴͢Δ • Flow[Int].map(_.toString) ͷΑ͏ʹม׵ͷελʔτϙΠϯτ ͱͯ͠࢖͏ • Flow.fromSinkAndSource[I,O](sink: Graph[SinkShape[I],_], source: Graph[SourceShape[O],_): Flow[I,O,NotUsed] • Sink ͱ Source Λ·ͱΊͯ Flow ʹ͢Δ 38
39. Async Boundary ඇಉظڥք 39
42. ඇಉظڥքΛೖΕΔ΂͖͔? ڥքΛೖΕΕ͹ෳ਺ͷCPUͰฒྻ࣮ߦ (pipeline࣮ߦ) Ͱ͖Δ v.s. ڥքΛ·͙ͨ΍ΓऔΓͷίετ .async ΛೖΕΔ͚ͩͰ pipeline ͕ߏஙͰ͖Δ 42
43. Materialize ࣌ʹɺ1 actor Ͱ࣮ߦ͢ΔΑ͏ʹࣗಈతʹ༥߹(fuse) ͞ΕΔɻ༥߹ʹ͸Φʔόϔου͕͋Δɻ͋Β͔͡Ί༥߹͢Δ͜ ͱͰ materialize ࣌ͷΦʔόʔϔουΛআڈͰ͖Δɻ val fusedStream = akka.stream.Fusing.aggressive(myStream) application.conf Ͱࣗಈతͳ༥߹ΛແޮԽʢσόοά༻ʣ akka.stream.materializer.auto-fusing=off 43
44. ͱ͜ΖͰ .async ͷ࣮૷͸ override def async: Repr[Out] = addAttributes(Attributes.asyncBoundary) ଐੑ (attribute) ͰετϦʔϜߏ੒ཁૉΛϚʔΫɻ ࣮ߦ࣌ʹ Materializer ͕ଐੑΛݟͯ༥߹͢Δ͔൑அɻ ଞʹ͸ͲΜͳଐੑ͕ʁ 44
45. import akka.event.Logging.WarningLevel import akka.stream.{ActorAttributes, Attributes} // ߏ੒ཁૉʹ໊લΛ෇͚Δ src.named("My source") // ཁૉΛϩάग़ྗ͢ΔࡍͷϩάϨϕϧ src.log("foo").withAttributes(Attributes.logLevels(WarningLevel)) // ActorMaterializer Ͱ࣮ߦ͢Δͱ͖ͷ dispatcher Λࢦఆ src.addAttributes(ActorAttributes.dispatcher("my-dispatcher")) // Τϥʔ࣌ͷ supervision strategy src.addAttributes(ActorAttributes.supervisionStrategy(decider)) 45
46. Back pressure എѹ੍ޚ 46
47. ϦΞΫςΟϒγεςϜ(ϦΞΫςΟϒϚχϑΣετ1ΑΓ) ϝοηʔδۦಈ (Message Driven): ϦΞΫςΟϒγεςϜ͸ ඇಉ ظͳϝοηʔδύογϯά ʹґͬͯίϯϙʔωϯτؒͷڥքΛ ཱ֬͢Δɻ… ·ͨɺγεςϜ಺ʹ ϝοηʔδΩϡʔ Λ࡞੒ͯ͠ ؂ࢹ͠ɺඞཁͳΒ όοΫϓϨογϟʔ Λద༻͢Δ͜ͱͰϑϩʔ ੍ޚ͕ՄೳʹͳΔɻ… 1 http://www.reactivemanifesto.org/ja 47
48. ඇಉظϝοηʔδΛҰํతʹ౤͛ΔͱϝοηʔδΩϡʔ͕ᷓΕΔ 48
49. back pressure 49
50. ඇಉظڥքΛ·͙ͨϝοηʔδͷड৴όοϑΝ akka.stream.materializer.max-input-buffer-size = 16 val materializer = ActorMaterializer( ActorMaterializerSettings(system) .withInputBuffer( initialSize = 64, maxSize = 64)) ੑೳνϡʔχϯά༻ 50
52. ໌ࣔతͳྲྀྔௐ੔ Source/Flow ݻఆαΠζͷόοϑΝΛઃ͚Δɻ def buffer(size: Int, overflowStrategy: OverflowStrategy): Repr[Out] Ұఆͷ଎౓ʹ཈͑Δ def throttle(elements: Int, per: FiniteDuration, maximumBurst: Int, mode: ThrottleMode): Repr[Out] 52
53. ֎෦࿈ܞ 53
54. mapAsync(), mapAsyncUnordered() def send(email: Email): Future[X] = ??? val src: Source[Email] = ??? // 4 ฒྻͰϝʔϧΛૹ৴ (໭Γ஋͸ Source[X]) src.mapAsync(4)(email) // ޙ͔Βདྷͨ΋ͷͰ΋ऴΘͬͨॱʹԼྲྀ΁ྲྀ͢ src.mapAsyncUnordered(4)(email) 54
55. Reactive streams Reactive stream ͷ PublisherɺSubscriber ͔Β Source, Sink Λ࡞Δ Source.fromPublisher(myPublisher) Sink.fromSubscriber(mySubscriber) 55
56. ઐ༻ͷ stage • Reactive kafka • Akka stream contrib • MQTT, AMQP ͳͲ ࣗ෼Ͱ GraphStage Λ࣮૷͢Δ (akka blog) Writing Akka Streams Connectors for existing APIs 56
57. akka-http akka-stream Λ࡞ͬͯ࡞ΒΕͨ HTTP toolkitɻ Server ΋ client ΋ॻ͚Δɻ 57
58. Graph ͱͯ͠ͷετϦʔϜ 58
59. γϯϓϧͳ෼ذɺ߹ྲྀ (merge, alsoTo) val mergedSrc = src1 .merge(src2) .merge(src3) val branchedSrc = src1 .alsoTo(sink1) .alsoTo(sink2) 59
60. ෳࡶͳ෼ذɺ߹ྲྀ͸ Graph DSL RunnableGraph.fromGraph(GraphDSL.create() { implicit builder => import GraphDSL.Implicits._ val in = Source(1 to 10) val out = Sink.ignore val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge ClosedShape }) 60
61. in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge ݟͨ໨͕ͦͬ͘Γʂ௚ײతʂ 61
62. GraphDSL Ͱ͙͢࢖͑ Δ෼ذͱ߹ྲྀ • ߹ྲྀ (fan-in) • Merge, MergePreferred • Zip, ZipWith • ෼ذ (fan-out) • Broadcast • Balance • Unzip, UnzipWith 62
63. ·ͱΊ • ετϦʔϜॲཧΛೃછΈͷ͋Δ API Ͱߏ੒Ͱ͖Δ • ϓϩάϥϜΛؒҧ͑ʹ͍͘Α͏ʹ޻෉͞Ε͍ͯΔ • ඇಉظ࣮ߦͱback pressure ʹΑΔϑϩʔ੍ޚ͕σϑΥϧτ Ͱ͍ͭͯ͘Δ • ෳࡶͳ෼ذɺ߹ྲྀ΋ graph DSL Ͱ௚ײతʹॻ͚Δ 63
64. Tips: akka.stream.javadsl Λิ׬͠ͳ͍ IntelliJ IDEA ͷิ׬͸ύοέʔδ͝ͱʹແޮԽͰ͖Δ 64
65. News: GearpumpMaterializer Apache Gearpump ͷ࣍ϦϦʔε(0.8.2)͔Β Gearpump ্Ͱ stream Λ materialize ͢Δ GearpumpMaterializer ؚ͕·ΕΔɻ ෼ࢄίϯϐϡʔςΟϯά + akka-stream ͷ໷໌͚ The Gearpump Materializer, Kam Kasravi 65
66. Thank you! Any questions? 66