akka-stream を始めるときに覚えておきたいこと

Slide 1

Slide 1 text

akka-stream Λ࢝ΊΔͱ͖ʹ͓֮͑ͯ ͖͍ͨ͜ͱ Տ಺ ਸ Twitter: @kawachi ɹ ɹ Photo by Junkichi Egashira is licensed under CC BY 4.0 International 1

Slide 2

Slide 2 text

ετϦʔϜॲཧ σʔλΛஞ࣍ॲཧ͢Δ͜ͱ • ϝϞϦʹ৐Γ੾Βͳ͍σʔλͷॲཧ • 1TBͷϑΝΠϧΛॲཧ͍ͨ͠ • ετϦʔϜͱͯ͠දݱ͢Δͷ͕ࣗવ ͳॲཧʢ௨৴ʣͳͲ • νϟοτͷ bot Λ࡞Γ͍ͨ 2

Slide 3

Slide 3 text

ετϦʔϜॲཧͷ؆୯ͳྫ୊Λߟ͑ͯΈ·͠ΐ͏ɻ Q. ϑΝΠϧ͕nߦҎ্͋Δ͔൑ఆͤΑ ͨͩ͠ϑΝΠϧ͸ϝϞϦʹ৐Γ੾Βͳ͍΄Ͳେ͖͍ɻ ಛେϑΝΠϧ͕૬खͷՄೳੑ͕͋ΔͷͰɺ൑ఆͰ͖ͨͱ͜ΖͰ ಡΈࠐΈΛऴΘΔ͜ͱɻ 3

Slide 4

Slide 4 text

͸͍ɻIterator ͰͰ͖·͢ɻ def q1(fileName: String, n: Int): Boolean = { val src = io.Source.fromFile(fileName) val lines: Iterator[String] = src.getLines() var count = 0 while (lines.hasNext && n < count) { lines.next() count += 1 } count == n } 4

Slide 5

Slide 5 text

՝୊1: Ϧιʔε؅ཧͰؒҧ͍΍͍͢ def q1Again(fileName: String, n: Int): Boolean = { val src = io.Source.fromFile(fileName) try { val lines: Iterator[String] = src.getLines() var count = 0 while (lines.hasNext && n < count) { lines.next() count += 1 } count == n } finally { src.close() } } 5

Slide 6

Slide 6 text

΋͏গ͠೉͍͠໰୊Λߟ͑ͯΈ·͠ΐ͏ɻ Q. ϑΝΠϧ͕nߦҎ্͋Δ͔ɺ͓Αͼ ۭߦ͕mߦҎ্͋Δ͔Λ൑ఆͤΑ 6

Slide 7

Slide 7 text

def hasMoreThan(lines: Iterator[String], n: Int) = { var count = 0 while (lines.hasNext && count < n) { lines.next() count += 1 } count == n } def q2(fileName: String, n: Int, m: Int) = { val src = io.Source.fromFile(fileName) try { val lines: Iterator[String] = src.getLines() (hasMoreThan(lines, n), hasMoreThan(lines.filter(_.isEmpty), m)) } finally { src.close() } } 7

Slide 8

Slide 8 text

՝୊2: εςʔτϑϧͳͷͰؒҧ͑΍͍͢ val lines: Iterator[String] = src.getLines() (hasMoreThan(lines, n), hasMoreThan(lines.filter(_.isEmpty), m)) // ^^^^^ Good ^^^^^ !!! >_< 8

Slide 9

Slide 9 text

՝୊3: εϨου͕ block ͞ΕΔ Iterator ͸ .hasNext ͱ .next() ͕ϕʔεɻ while (lines.hasNext) { lines.next() // <- !!! blocking !!! … } 9

Slide 10

Slide 10 text

actor Ͱ՝୊ΛղܾͰ͖ΔͩΖ͏͔? 10

Slide 11

Slide 11 text

• ՝୊1: Ϧιʔε؅ཧͰؒҧ͍΍͍͢ • → Ϧιʔε؅ཧΛϑΝΠϧಡΈࠐΈ Actor ʹہॴԽͰ͖Δɻͦͷ Actor ֎ Ͱ͸ؒҧ͑ͳ͍ɻ • ՝୊2: εςʔτϑϧͳͷͰؒҧ͑΍͍͢ • → Actor Λ࡞Δࡍͷ akka.actor.Props ͸ immutable ͳ஋ɻ࠶ར༻Մೳɻ • ՝୊3: εϨου͕ block ͞ΕΔ • → ϝοηʔδۦಈͳͷͰඇಉظ API Λࣗવʹѻ͑Δ ׬શղܾʂ 11

Slide 12

Slide 12 text

৽ͨͳ՝୊ 12

Slide 13

Slide 13 text

ࠓճ঺հ͢Δ akka-stream Ͱ͸ Actor ͱൺ΂ͯ • ߏ੒ཁૉ͕ग़ྗ/ೖྗ͢Δཁૉʹܕ͕͍͍ͭͯΔ • όοϑΝᷓΕ͠ͳ͍ Iterator ͱൺ΂ͯ • Ϧιʔε؅ཧ͕ہॴԽ͞Εؒҧ͑ʹ͍͘ • ߏ੒ཁૉ͕ immutable ͳ஋ • ඇಉظ API ͱ૊Έ߹ΘͤΒΕΔ 13

Slide 14

Slide 14 text

akka-stream 14

Slide 15

Slide 15 text

ґଘੑ௥Ճ Play 2.5 ͔Β Play ͷ࣮૷ʹ akka-stream ͕࢖ΘΕ͍ͯΔ → ৽ͨʹґଘੑΛ௥Ճ͢Δඞཁͳ͠ Play ؀ڥͰͳ͍৔߹͸ґଘੑΛ௥Ճ libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.4.11" 15

Slide 16

Slide 16 text

୯७ͳετϦʔϜ 16

Slide 17

Slide 17 text

ྫ: 1~5 Λྲྀͯ͠ * 2 ͯ͠ println() import akka.actor.ActorSystem import akka.stream.ActorMaterializer import akka.stream.scaladsl._ val src = Source(1 to 5) val flow = Flow[Int].map(_ * 2) val sink = Sink.foreach[Int](println) val stream = src.via(flow).to(sink) implicit val system = ActorSystem() implicit val materializer = ActorMaterializer() stream.run() 17

Slide 18

Slide 18 text

ετϦʔϜͷهड़ val src = Source(1 to 5) val flow = Flow[Int].map(_ * 2) val sink = Sink.foreach[Int](println) val stream = src.via(flow).to(sink) • هड़ ͱ࣮ߦ͕෼཭͞Ε͍ͯΔ • هड़͸ઃܭॻΛॻ͘࡞ۀ • ෭࡞༻ͳ͠ • src, ﬂow, sink, stream ͷ࠶ར༻OK! 18

Slide 19

Slide 19 text

Source, Flow, Sink 19

Slide 20

Slide 20 text

Source, Flow, Sink ͷ߹੒ 20

Slide 21

Slide 21 text

RunnableGraph RunnableGraph ͸ run() ՄೳͳετϦ ʔϜͷهड़ɻ run() ͢Δʹ͸ೖग़ྗ͕શͯ઀ଓ͞Ε ͍ͯΔඞཁ͕͋Δɻ Source ΍ Flowɺ Sink ͸ run() Ͱ͖ͳ ͍ɻ 21

Slide 22

Slide 22 text

ετϦʔϜͷ࣮ߦ • ࣮ߦ։࢝͢Δ͜ͱΛ materialize (͋Δ͍͸ run)ͱݺͿ • ࣮ߦʹ͸ Materializer ͕ඞཁ • Ұ౓هड़ͨ͠ετϦʔϜ͸Կ౓Ͱ΋࣮ߦͰ͖Δ implicit val system = ActorSystem() implicit val materializer = ActorMaterializer()(system) stream.run()(materializer) 22

Slide 23

Slide 23 text

Materialized value 23

Slide 24

Slide 24 text

run() ͭ·Γ materialize ͷ໭Γ஋ɻ஋ ͱͯ͠ಘΒΕΔͷ͸ Materialized value ͷΈɻ ༻్ • ੍ޚϙΠϯτ • ֎෦͔ΒετϦʔϜΛ੍ޚ͢Δ • ؂ࢹϙΠϯτ • ࣮ߦঢ়ଶ΍ॲཧ݁Ռ 24

Slide 25

Slide 25 text

val stream: RunnableGraph[Future[IOResult]] = FileIO.fromPath(Paths.get("src_file")) .to(FileIO.toPath(Paths.get("dest_file"))) val result: Future[IOResult] = stream.run() // ↑ Materialized value result.onSuccess { case ioResult => println(s"${ioResult.count} byteॲཧ͠·ͨ͠") } 25

Slide 26

Slide 26 text

RunnableGraph ͷ(ຊ౰ͷ)ܕ RunnableGraph[+Mat] Mat ͕ materialized value ͷܕ run() ͢Δͱ Mat ܕͷ materialized value ͕໭Γ஋ͱͯ͠ฦ͞ ΕΔɻ 26

Slide 27

Slide 27 text

Source, Flow, Sink ͷ(ຊ౰ͷ)ܕ Source[+Out,+Mat] Flow[-In,+Out,+Mat] Sink[-In,+Mat] ߏ੒ཁૉ (Source, Flow, Sink) ͕ͦΕͧΕ Mat ܕͷ materialized value Λ࣋ͭɻ ͜ΕΒΛ૊Έ߹ͤͯ RunnableGraph Λ࡞ͬͯ run() ͢Δɻ run() ͷ໭Γ஋ͱͯ͠ಘΒΕΔͷ͸ RunnableGraph ͷ Mat ͚ͩɻ 27

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

߹੒࣌ͷ materialized value • via(), to() ͸ࠨଆͷ materialzed value Λ߹੒ޙͷ materialized value ʹ࠾༻ • viaMat(), toMat() Ͱࣗ༝ʹͰ͖Δ import akka.stream.scaladsl.Keep source.viaMat(flow)(Keep.left) // ࠨͷ Mat (σϑΥϧτ) source.viaMat(flow)(Keep.right) // ӈͷ Mat source.viaMat(flow)(Keep.both) // ྆ Mat ͷλϓϧ source.viaMat(flow)(Keep.none) // NotUsed 30

Slide 31

Slide 31 text

Source, Flow, Sink 31

Slide 32

Slide 32 text

Source, Flow ڞ௨ͷجຊతͳૢ࡞ map(), filter(), filterNot(), take(), takeWhile(), drop(), dropWhile(), fold(), reduce() ඪ४ϥΠϒϥϦͷίϨΫγϣϯAPIʹࣅͨϝιου܈ Source ΍ Flow ͷग़ྗཁૉ͕ίϨΫγϣϯͷཁૉʹରԠ͢Δͱ ଊ͑Ε͹௚ײతʹૢ࡞Մೳ flatMap() ͸ͳ͍ɻmapConcat() ͋Δ͍͸ flatMapConcat() ͕૬౰ɻ 32

Slide 33

Slide 33 text

ετϦʔϜͳΒͰ͸ͷૢ࡞ • takeWithin(d: FiniteDuration) • dropWithin(d: FiniteDuration) • ࢦఆ࣌ؒ಺ʹ౸ணͨ͠΋ͷ͚ͩऔΓग़͢/ࣺͯΔ • groupedWithin(n: Int, d: FiniteDuration) • ࢦఆͨ࣌ؒ͠ʹ౸ணͨ͠ཁૉʢ࠷େn݅ʣΛ·ͱΊͯ Seq Ͱग़ྗ 33

Slide 34

Slide 34 text

Source ͷ࡞Γํ (1) • Source.apply[T](iterable: Iterable[T]): Source[T, NotUsed] Source.fromIterator[T](f: () 㱺 Iterator[T]): Source[T, NotUsed] • Iterable ΍ Iterator ͔Β Source Λ࡞Δ • Source.repeat[T](element: T): Source[T, NotUsed] • elementΛӬԕʹग़͢ 34

Slide 35

Slide 35 text

Source ͷ࡞Γํ (2) • Source.tick[T](initialDelay: FiniteDuration, interval: FiniteDuration, tick: T): Source[T, Cancellable] • Ұఆ࣌ؒຖʹ tick Λग़͢ • Source.actorRef[T](bufferSize: Int, overflowStrategy: OverflowStrategy): Source[T, ActorRef] • materialized value ͷ ActorRef ʹϝοηʔδΛ౤͛Δͱ Source ͔ Βग़ྗ͞ΕΔ ੍ޚϙΠϯτͱͯ͠ materialized value ͕࢖ΘΕΔྫ 35

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Sink ͷ࡞Γํ • Sink.foreach[T](f: T 㱺 Unit): Sink[T, Future[Done]] • TܕͷཁૉΛडऔΔͨͼʹfΛ࣮ߦ͢Δ • Sink.fold[U, T](zero: U)(f: (U, T) 㱺 U): Sink[T, Future[U]] • Sink.reduce[T](f: (T, T) 㱺 T): Sink[T, Future[T]] • ৞ΈࠐΈ 37

Slide 38

Slide 38 text

Flow ͷ࡞Γํ • Flow.apply[T]: Flow[T,T,NotUsed] • ্ྲྀ͔Βड৴ͨ͠ཁૉΛͦͷ··Լྲྀ΁ૹ৴͢Δ • Flow[Int].map(_.toString) ͷΑ͏ʹม׵ͷελʔτϙΠϯτ ͱͯ͠࢖͏ • Flow.fromSinkAndSource[I,O](sink: Graph[SinkShape[I],_], source: Graph[SourceShape[O],_): Flow[I,O,NotUsed] • Sink ͱ Source Λ·ͱΊͯ Flow ʹ͢Δ 38

Slide 39

Slide 39 text

Async Boundary ඇಉظڥք 39

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

ඇಉظڥքΛೖΕΔ΂͖͔? ڥքΛೖΕΕ͹ෳ਺ͷCPUͰฒྻ࣮ߦ (pipeline࣮ߦ) Ͱ͖Δ v.s. ڥքΛ·͙ͨ΍ΓऔΓͷίετ .async ΛೖΕΔ͚ͩͰ pipeline ͕ߏஙͰ͖Δ 42

Slide 43

Slide 43 text

Materialize ࣌ʹɺ1 actor Ͱ࣮ߦ͢ΔΑ͏ʹࣗಈతʹ༥߹(fuse) ͞ΕΔɻ༥߹ʹ͸Φʔόϔου͕͋Δɻ͋Β͔͡Ί༥߹͢Δ͜ ͱͰ materialize ࣌ͷΦʔόʔϔουΛআڈͰ͖Δɻ val fusedStream = akka.stream.Fusing.aggressive(myStream) application.conf Ͱࣗಈతͳ༥߹ΛແޮԽʢσόοά༻ʣ akka.stream.materializer.auto-fusing=off 43

Slide 44

Slide 44 text

ͱ͜ΖͰ .async ͷ࣮૷͸ override def async: Repr[Out] = addAttributes(Attributes.asyncBoundary) ଐੑ (attribute) ͰετϦʔϜߏ੒ཁૉΛϚʔΫɻ ࣮ߦ࣌ʹ Materializer ͕ଐੑΛݟͯ༥߹͢Δ͔൑அɻ ଞʹ͸ͲΜͳଐੑ͕ʁ 44

Slide 45

Slide 45 text

import akka.event.Logging.WarningLevel import akka.stream.{ActorAttributes, Attributes} // ߏ੒ཁૉʹ໊લΛ෇͚Δ src.named("My source") // ཁૉΛϩάग़ྗ͢ΔࡍͷϩάϨϕϧ src.log("foo").withAttributes(Attributes.logLevels(WarningLevel)) // ActorMaterializer Ͱ࣮ߦ͢Δͱ͖ͷ dispatcher Λࢦఆ src.addAttributes(ActorAttributes.dispatcher("my-dispatcher")) // Τϥʔ࣌ͷ supervision strategy src.addAttributes(ActorAttributes.supervisionStrategy(decider)) 45

Slide 46

Slide 46 text

Back pressure എѹ੍ޚ 46

Slide 47

Slide 47 text

ϦΞΫςΟϒγεςϜ(ϦΞΫςΟϒϚχϑΣετ1ΑΓ) ϝοηʔδۦಈ (Message Driven): ϦΞΫςΟϒγεςϜ͸ ඇಉ ظͳϝοηʔδύογϯά ʹґͬͯίϯϙʔωϯτؒͷڥքΛ ཱ֬͢Δɻ… ·ͨɺγεςϜ಺ʹ ϝοηʔδΩϡʔ Λ࡞੒ͯ͠ ؂ࢹ͠ɺඞཁͳΒ όοΫϓϨογϟʔ Λద༻͢Δ͜ͱͰϑϩʔ ੍ޚ͕ՄೳʹͳΔɻ… 1 http://www.reactivemanifesto.org/ja 47

Slide 48

Slide 48 text

ඇಉظϝοηʔδΛҰํతʹ౤͛ΔͱϝοηʔδΩϡʔ͕ᷓΕΔ 48

Slide 49

Slide 49 text

back pressure 49

Slide 50

Slide 50 text

ඇಉظڥքΛ·͙ͨϝοηʔδͷड৴όοϑΝ akka.stream.materializer.max-input-buffer-size = 16 val materializer = ActorMaterializer( ActorMaterializerSettings(system) .withInputBuffer( initialSize = 64, maxSize = 64)) ੑೳνϡʔχϯά༻ 50

Slide 51

Slide 51 text

Slide 52

Slide 52 text

໌ࣔతͳྲྀྔௐ੔ Source/Flow ݻఆαΠζͷόοϑΝΛઃ͚Δɻ def buffer(size: Int, overflowStrategy: OverflowStrategy): Repr[Out] Ұఆͷ଎౓ʹ཈͑Δ def throttle(elements: Int, per: FiniteDuration, maximumBurst: Int, mode: ThrottleMode): Repr[Out] 52

Slide 53

Slide 53 text

֎෦࿈ܞ 53

Slide 54

Slide 54 text

mapAsync(), mapAsyncUnordered() def send(email: Email): Future[X] = ??? val src: Source[Email] = ??? // 4 ฒྻͰϝʔϧΛૹ৴ (໭Γ஋͸ Source[X]) src.mapAsync(4)(email) // ޙ͔Βདྷͨ΋ͷͰ΋ऴΘͬͨॱʹԼྲྀ΁ྲྀ͢ src.mapAsyncUnordered(4)(email) 54

Slide 55

Slide 55 text

Reactive streams Reactive stream ͷ PublisherɺSubscriber ͔Β Source, Sink Λ࡞Δ Source.fromPublisher(myPublisher) Sink.fromSubscriber(mySubscriber) 55

Slide 56

Slide 56 text

ઐ༻ͷ stage • Reactive kafka • Akka stream contrib • MQTT, AMQP ͳͲ ࣗ෼Ͱ GraphStage Λ࣮૷͢Δ (akka blog) Writing Akka Streams Connectors for existing APIs 56

Slide 57

Slide 57 text

akka-http akka-stream Λ࡞ͬͯ࡞ΒΕͨ HTTP toolkitɻ Server ΋ client ΋ॻ͚Δɻ 57

Slide 58

Slide 58 text

Graph ͱͯ͠ͷετϦʔϜ 58

Slide 59

Slide 59 text

γϯϓϧͳ෼ذɺ߹ྲྀ (merge, alsoTo) val mergedSrc = src1 .merge(src2) .merge(src3) val branchedSrc = src1 .alsoTo(sink1) .alsoTo(sink2) 59

Slide 60

Slide 60 text

ෳࡶͳ෼ذɺ߹ྲྀ͸ Graph DSL RunnableGraph.fromGraph(GraphDSL.create() { implicit builder => import GraphDSL.Implicits._ val in = Source(1 to 10) val out = Sink.ignore val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge ClosedShape }) 60

Slide 61

Slide 61 text

in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge ݟͨ໨͕ͦͬ͘Γʂ௚ײతʂ 61

Slide 62

Slide 62 text

GraphDSL Ͱ͙͢࢖͑ Δ෼ذͱ߹ྲྀ • ߹ྲྀ (fan-in) • Merge, MergePreferred • Zip, ZipWith • ෼ذ (fan-out) • Broadcast • Balance • Unzip, UnzipWith 62

Slide 63

Slide 63 text

·ͱΊ • ετϦʔϜॲཧΛೃછΈͷ͋Δ API Ͱߏ੒Ͱ͖Δ • ϓϩάϥϜΛؒҧ͑ʹ͍͘Α͏ʹ޻෉͞Ε͍ͯΔ • ඇಉظ࣮ߦͱback pressure ʹΑΔϑϩʔ੍ޚ͕σϑΥϧτ Ͱ͍ͭͯ͘Δ • ෳࡶͳ෼ذɺ߹ྲྀ΋ graph DSL Ͱ௚ײతʹॻ͚Δ 63

Slide 64

Slide 64 text

Tips: akka.stream.javadsl Λิ׬͠ͳ͍ IntelliJ IDEA ͷิ׬͸ύοέʔδ͝ͱʹແޮԽͰ͖Δ 64

Slide 65

Slide 65 text

News: GearpumpMaterializer Apache Gearpump ͷ࣍ϦϦʔε(0.8.2)͔Β Gearpump ্Ͱ stream Λ materialize ͢Δ GearpumpMaterializer ؚ͕·ΕΔɻ ෼ࢄίϯϐϡʔςΟϯά + akka-stream ͷ໷໌͚ The Gearpump Materializer, Kam Kasravi 65

Slide 66

Slide 66 text

Thank you! Any questions? 66