Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed System and Functional Programming

Distributed System and Functional Programming

「分散処理と
関数型プログラミング」 by @okapies

2016/03/21 第2回 分散システム勉強会 / 第三十回 P2P SIP 勉強会

Ef45a84b10aa3cb5acd2ea0c9846bd3b?s=128

Yuta Okamoto

March 21, 2016
Tweet

Transcript

  1. ୈ2ճ ෼ࢄγεςϜษڧձ / ୈࡾेճ P2P SIP ษڧձ ෼ࢄॲཧͱ
 ؔ਺ܕϓϩάϥϛϯά Ԭຊ

    ༤ଠ (@okapies) 2016/03/21
  2. ࣗݾ঺հ • Ԭຊ ༤ଠ(@okapies) • ੡଄ۀͰಇ͘ιϑτ԰͞Μ • Scala ͱ Scala

    OSSs Ѫ޷Ո • ࠷ۙͷ࢓ࣄ͸ΠϯϑϥΤϯδχΞͬΆ͍ײ͡ • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ
  3. http://www.slideshare.net/okapies/scalamatsuri-58141520

  4. http://www.slideshare.net/okapies/reactive-architecture-20160218-58403521

  5. ͳͥ෼ࢄγεςϜ͔ʁ • ੑೳͷεέʔϥϏϦςΟ • ଱ো֐ੑ • ։ൃ૊৫ͷεέʔϥϏϦςΟ (NEW!) ΧδϡΞϧʹ෼ࢄϓϩάϥϛϯά Λ΍Δधཁ͕ग़͖ͯͨ

  6. http://martinfowler.com/microservices/

  7. ϚΠΫϩαʔϏεΞʔΩςΫνϟ • ҰͭͷΞϓϦέʔγϣϯΛখ͞ͳαʔϏε܈ ͷ૊Έ߹Θͤͱͯ͠ߏங͢Δख๏ http://martinfowler.com/articles/microservices.html

  8. MSA Λ࠾༻͍ͯ͠Δاۀ • Amazon • Twitter • Netflix • PayPal

    • LinkedIn • Tumblr • SoundCloud • LINE • υϫϯΰ • etc…
  9. http://anond.hatelabo.jp/20111018190933 Amazon ͷ࠷ॳظ͔Βͷࣄྫ

  10. େن໛ͳ MSA ͷྫ http://www.slideshare.net/adriancockcroft/goto-berlin/76

  11. ϚΠΫϩαʔϏεͷಛੑ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 5.

    ඇதԝूݖతͳΨόφϯε 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ 9. ਐԽతઃܭ http://martinfowler.com/microservices/
  12. ϚΠΫϩαʔϏεͷಛੑ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 5. ඇதԝूݖతͳΨόφϯε 9. ਐԽతઃܭ 1.

    αʔϏεʹΑΔίϯϙʔωϯτԽ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ ૊৫ͷಛੑ (໨త) ٕज़ͷಛੑ (खஈ) http://martinfowler.com/microservices/
  13. ϚΠΫϩαʔϏεͷ໨త • ։ൃ૊৫ͷεέʔϥϏϦςΟͷ֬อ • Ϗδωε਱ߦೳྗΛ֩ͱͨ͠ඇதԝूݖతͳ ػೳԣஅܕνʔϜͱͯ͠૊৫Խ͢Δ
 (c.f. Conway’s Law) http://martinfowler.com/articles/microservices.html

  14. ϚΠΫϩαʔϏεͷํ๏࿦ • αʔϏεؒʹϏδωε਱ߦೳྗ΍υϝΠϯͷ ϥΠϑαΠΫϧʹैͬͯϞδϡʔϧڥքΛ࣋ͭ • αʔϏεؒ͸ REST ΍୯७ͳ MQ Ͱܨ͙ɻτ

    ϥϯβΫγϣϯ͸ͳΔ΂͘࢖Θͳ͍ • αʔϏεಉ࢜ͷґଘؔ܎ΛݮΒͯ͠ಠཱʹ഑ උͰ͖ΔΑ͏ʹ͢Δ
  15. ϚΠΫϩαʔϏεͷར఺ • ڧ͍ϞδϡʔϧڥքɿϚΠΫϩαʔϏε͸Ϟδϡʔ ϧߏ଄ΛڧԽ͢Δɻ͜ͷ͜ͱ͸ɺνʔϜͷن໛͕େ ͖͘ͳΔʹͭΕͯಛʹॏཁʹͳΔ • ಠཱͨ͠഑උɿ୯७ͳαʔϏε͸഑උ͕༰қͰ͋ Γɺ·ͨࣗ཯తͳͷͰɺݸʑʹෆ۩߹͕ى͖ͯ΋γ εςϜো֐ΛҾ͖ى͜͠ʹ͍͘ •

    ٕज़తଟ༷ੑɿෳ਺ͷݴޠɺ։ൃϑϨʔϜϫʔΫɺ σʔλετϨʔδٕज़Λ૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ http://martinfowler.com/articles/microservice-trade-offs.html
  16. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ʁʁʁʁʁʁʁʁʁʁ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html

  17. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ΦϒδΣΫτΛ෼ࢄͤ͞Δͳ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html ෼ࢄԽʹ͸σϝϦοτ
 ͕ͨ͘͞Μ͋Δ

  18. ʮ෼ࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ 1. ωοτϫʔΫ͸৴པͰ͖Δɻ 2. ϨΠςϯγ͸θϩͰ͋Δɻ 3. ଳҬ෯͸ແݶͰ͋Δɻ 4. ωοτϫʔΫ͸ηΩϡΞͰ͋Δɻ 5.

    ωοτϫʔΫߏ੒͸มԽͤͣҰఆͰ͋Δɻ 6. ؅ཧऀ͸̍ਓͰ͋Δɻ 7. τϥϯεϙʔτίετ͸θϩͰ͋Δɻ 8. ωοτϫʔΫ͸ۉ࣭Ͱ͋Δɻ
  19. ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ

    αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html
  20. ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ

    αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html ࠓ೔͸෼ࢄϓϩάϥϛϯάͷ࿩Λ͠·͢
  21. ෼ࢄϓϩάϥϛϯάͷͭΒΈ • RPC ͷஞ࣍ॲཧ΍ฒྻॲཧ • ωοτϫʔΫͷো֐ɾ஗Ԇɾ෼அ • ϦϞʔτϗετͷো֐ • εϩοτϦϯάʢ࿈࠯ো֐ͷ๷ࢭʣ

    • αʔϏεσΟεΧόϦ • ϞχλϦϯάɺϦΫΤετͷ෼ࢄτϨʔε
  22. ໋ྩతͳಉظݺͼग़͠ val users = UserService(…) ! // block until the

    request is completed result = users.findByUserId(“okapies”)
  23. ωοτϫʔΫϨΠςϯγ Latency Comparison Numbers (Originally by Peter Norvig) -------------------------- L1

    cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us Disk seek 10,000,000 ns 10,000 us 10 ms Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms https://gist.github.com/jboner/2841832
  24. ωοτϫʔΫϨΠςϯγ http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

  25. ໋ྩɾಉظܕϓϩάϥϛϯάͷܽ఺ • ಉظݺͼग़͠͸ޮ཰͕ѱ͍ • ී௨͸εϨου΍ϓϩηεΛฒ΂ͯରॲ͢Δ • εϨουʗϓϩηεΛ૿΍͢ͱϦιʔεΛେ ͖͘ফඅ͢Δ (Java thread

    = 1 MB) • ʢgreen thread ͕࢖͑Δ؀ڥ͸ݶΒΕΔʣ
  26. ίʔϧόοΫ def findByUserId(f: User => Unit): Unit ! val users

    = UserService(…) 
 // register a callback as an argument users.findByUserId(“okapies”) { user => println(s”$user”) } ίʔϧόοΫ
  27. ίʔϧόοΫ஍ࠈ • ίʔυͷϞδϡʔϧԽ͕ࠔ೉ • ঢ়ଶʢ෭࡞༻ʣ΍σʔλͷґଘؔ܎ͷ؅ཧ͕ ࠔ೉ • ࣮ߦॱংͷ੍ޚ͕ࠔ೉ʢ֎෦Πϕϯτ࣍ୈͰ มԽ͢Δʣ

  28. ഁ໓ͷϐϥϛου var g = ... ! step1 { a =>

    step2 { b => step3 { c => step4 { d => // do something with a, b, c, d and g } } } } ґଘ͢Δඇಉظεςοϓ͕
 ϐϥϛουͷΑ͏ʹੵΈ্͕Δ ֎ଆͷείʔϓͷঢ়ଶΛ҉໧ʹ
 ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕௿͍
  29. ෼ࢄͱؔ਺ܕϓϩάϥϛϯά • ʮ͋ΕͰ͠ΐʁϜʔΞͷ๏ଇ͕ഁΕͯϑϦʔ ϥϯν͸ऴΘͬͨɺ͜Ε͔Β͸ϚϧνίΞͰ ฒྻॲཧͩؔ਺ܕͩͬͯ΍ͭʯ • ෼ࢄॲཧʹ͓͚Δؔ਺ܕͷຊ࣭͸ɺϏδωε ϩδοΫΛඇಉظॲཧ΍෼ࢄ؀ڥͰͷԣஅత ؔ৺ࣄ͔Β෼཭ͯ͠ɺ༰қʹϞδϡʔϧԽͰ ͖ΔΑ͏ʹ͢Δ͜ͱ

  30. ؔ਺ܕϓϩάϥϛϯά • ࣮༻తʹ͸ϞδϡʔϧੑΛߴΊΔͨΊͷख๏ https://twitter.com/mfeathers/status/29581296216

  31. ʢ਺ֶతʣؔ਺ • y = f (x) f x f (x)

    × ೖྗ͕ܾ·Δͱ
 ग़ྗ͕Ұҙʹܾ·Δ
  32. ؔ਺ͷ߹੒ • z = g ◦ f (x) = g

    ( f (x) ) f x f (x) g ◦ f (x) g f ͷ஋Ҭͱ g ͷఆٛҬ͕ Ұக͢Ε͹߹੒Ͱ͖Δ
  33. ७ਮؔ਺ͱ෭࡞༻ • ϓϩάϥϜΛঢ়ଶΛ࣋ͨͳ͍७ਮ (pure) ͳؔ਺ Ͱهड़͠ɺ෭࡞༻Λ֎քͱͷڥքʹू໿͢Δ • Ϟδϡʔϧੑ΍ςετ༰қੑ͕޲্ • ؔ਺͕ϞδϡʔϧʹͳΔ

    • composability ͕伴
  34. ७ਮؔ਺ͱ෭࡞༻ http://www.iij.ad.jp/company/development/external/lecture/pdf/yamamoto.pdf

  35. ͳͥؔ਺ϓϩάϥϛϯά͸ॏཁ͔ • QuickCheck ͷ։ൃ΍ QuviQ ͷ૑ۀऀͱͯ͠ ஌ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳ࿦จ • ॳ൛͸ 1984

    ೥ʢ30 ೥લʂʣ • ؔ਺ܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍ͭͯ࿦͍ͯ͡ Δ http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html
  36. ؔ਺ܕͷʪކʫ • ؔ਺ܕʹ͓͚Δೋͭͷॏཁͳʪކʫ • ߴ֊ؔ਺ʢίϯϏωʔλʣ • ஗ԆධՁ ʮݩͷ໰୊Λ෼ׂ͢Δํ๏͸ɺղͱղΛషΓ߹ ΘͤΔํ๏ʹ௚઀ʹґଘ͢Δɻʯ

  37. ߴ֊ؔ਺ • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔ਺ͱϢʔεέʔ εʹಛԽͨؔ͠਺ʹ෼͚ͯϞδϡʔϧԽ ! ! • ϏδωεϩδοΫͱɺͦΕ͕৐͔ͬΔσʔλ ܕͷจ຺Λ෼཭Ͱ͖Δ set.

    map(_ + 1) // Set[A] map. map(_ + 1) // Map[A, B] list.map(_ + 1) // List[A] ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ࢖͍ճͤΔ
  38. ஗ԆධՁ class Cons[A](hd: A, tl: => List[A]) extends List[A] !

    def nats(n: Int): List[Int] = new Cons(n, nats(n+1)) def fizzbuzz(n: Int) = n match { case _ if n % 15 == 0 => "FizzBuzz" case _ if n % 3 == 0 => "Fizz" case _ if n % 5 == 0 => "Buzz" case _ => n.toString } nats.map(fizzbuzz).take(100).foreach(println) ඞཁݺͼ ʢϓϧܕʣ ίʔυΛੜ੒ثͱબ୒ثͷ ૊Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ ແݶϦετ
  39. What ͱ How ͷ෼཭ Input Input Output (2) ϥϯλΠϜ (1)

    ϓϩάϥϛϯάϞσϧ (DSL) (how Λ࣮ߦ͢Δ) (what Λهड़͢Δ)
  40. ؔ਺ܕͰ෭࡞༻Λѻ͏ʹ͸ʁ • ७ਮؔ਺͸ʢఆ্ٛʣ෭࡞༻Λѻ͑ͳ͍ • Ͳ͏͢Δ͔ʁ • ෭࡞༻΍จ຺Λѻ͏ίϯςφʢശʣΛ࡞ͬ ͯɺ஋ΛͦͷதʹೖΕΔʢϞφυͱݺͿʣ a

  41. ശͷதͷ஋Λม׵͢Δ • y = x.map(f) (= map f x) •

    ७ਮؔ਺ͱ෭࡞༻Λ෼཭ͨ͠··ܭࢉՄೳ • ෭࡞༻͸ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ a b f map(f) औΓग़͢ ໭͢
  42. ʮശʯͷछྨ • ৭ʑͳʮശʯ͕͋Δ • Identity • List • State •

    Reader ! • Writer • Continuation • IO • Future/Promise
  43. • y = x.map(f).map(g) Future/Promise f x f (x) g

    ◦ f (x) g ඇಉظॲཧͷจ຺Λ ѻ͏ʮശʯ
  44. • y = x.map(f).map(g) Future/Promise f x f (x) g

    ◦ f (x) g ࣦഊͷจ຺͸ࣗಈ తʹҾ͖ܧ͕ΕΔ ੒ޭ ࣦഊ
  45. Future/Promise // ശͷதͷ஋Λม׵͢Δ def map[B](f: A => B): Future[B] !

    // จ຺Λ੾Γସ͑ΔೳྗΛ࣋ͬͨ map def flatMap[B](f: A => Future[B]): Future[B] ! // ෳ਺ͷ Future Λू໿ͯ͠଴ͪ߹Θͤ͢Δ def sequence[A](fs: List[Future[A]]): Future[List[A]]
  46. ෳ਺ϦΫΤετͷू໿ val userAndTweets = Future.join( userService.findByUserId(userId), tweetService.findByUserId(userId) ) find find

    userId userAndTweets User
 Service Tweet
 Service http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16 join ଞͷϚΠΫϩαʔϏε΁ΫΤϦ Λ౤͛ɺશͯͷԠ౴͕ἧͬͨΒ ඇಉظʹू໿ͯ͠λϓϧʹ͢Δ
  47. Future/Promise ͷϝϦοτ • ඇಉظॲཧಉ࢜ͷґଘؔ܎ʢσʔλϑϩʔʣΛ એݴతʹهड़͢Δ͜ͱͰϓϩάϥϜΛॻ͘ • ϓϩάϥϚ͸ DSL Λॻ͚͹Α͘ɺඇಉظॲ ཧͷৄࡉ͸ॲཧܥ͕Ӆṭ͢Δ

    • ਖ਼ৗܥͱɺͦͷଞͷؔ৺ࣄʢো֐ɺ஗ԆͳͲʣ Λ௚ަతʹѻ͏͜ͱ͕Ͱ͖ɺ෦඼Խ͕༰қʹ
  48. ϚΠΫϩαʔϏε޲͚ϑϨʔϜϫʔΫ • େن໛ͳϚΠΫϩαʔϏεΛӡ༻͍ͯ͠Δا ۀͷ͏ͪɺ͍͔ͭ͘͸ࣗࣾͷج൫ιϑτ΢Σ ΞΛ OSS Խ͍ͯ͠Δ • Netflix OSS

    • Twitter Finagle, Zipkin, etc…
  49. Finagle Twitter ͷ Finagle ͸ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͠೉͍ཁ ҼͩɻFinagle ͸ɺ෼ࢄτϨʔεɺαʔϏεσΟεΧόϦ΍αʔ

    Ϗεొ࿥ͱ͍ͬͨ෼ࢄͷ՝୊ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ • Finagle ͸ɺTumblr ΍ SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ Ϗε΁ͷ Scala ಋೖʹ౿Έ੾Δେ͖ͳཁҼʹͳ͍ͬͯΔ • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏ ͭͷཧ༝ʯ Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter
  50. http://monkey.org/~marius/funsrv.pdf Your Server as a Function

  51. Your Server as a Function Most of our systems are

    phrased as big future transformers. — Marius Eriksen • Future ܕͱɺͦΕΛม׵͢Δ Service, Filter ͱ ͍͏ؔ਺ͷ૊Έ߹ΘͤͰαʔόΛهड़͢Δ https://monkey.org/~marius/scala2015.pdf
  52. Future/Service/Filter trait Service[Req, Rep] extends (Req => Future[Rep]) ! !

    ! ! ! ! abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut]) ϦΫΤετΛड͚औͬͯϨεϙϯεΛ แΜͩ Future Λฦؔ͢਺ ϦΫΤετͱ Service Λड͚औͬͯ
 ϨεϙϯεΛแΜͩ Future Λฦؔ͢਺
  53. ԣஅతؔ৺ࣄͷ߹੒ recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest

    andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer FilterΛService ʹ߹੒͢Δͩ ͚Ͱɺαʔόʹ༷ʑͳԣஅత ؔ৺ࣄΛ௥ՃͰ͖Δ http://monkey.org/~marius/funsrv.pdf
  54. ӡ༻ج൫ͱͷ࿈ܞ • ྫ͑͹ɺFinagle Ͱॻ͍ͨαʔό͔ΒɺZipkin ͳͲͷ෼ࢄτϨʔεγεςϜʹτϨʔε৘ใΛ ૹ৴ͯ͠ՄࢹԽͰ͖Δ

  55. σʔλϑϩʔ DSL • σʔλϑϩʔ DSL ͱϥϯλΠϜͷ૊Έ߹Θͤ͸ɺۙ ೥ɺ༷ʑͳ෼໺Ͱద༻͞Ε͍ͯΔ • Akka Streams,

    ReactiveX, … • Պֶٕज़ܭࢉ: TensorFlow, Halide • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow, Asakusa Framework, Gearpump
  56. FRP (Reactive Extensions) ϚϧνΫϦοΫετϦʔϜ https://gist.github.com/staltz/868e7e9bc2a7b8c1f754 250 ϛϦඵຖʹ
 ΫϦοΫΛूੵ ϦετΛ௕͞
 ʹϚοϐϯά

    RxJS ʹΑΔඇಉظΫϦοΫετϦʔϜ
  57. Akka Streams implicit val system = ActorSystem() implicit val mat

    = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) A B C +1 —1 ×2 +
  58. Akka Streams implicit val system = ActorSystem() implicit val mat

    = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) ೖྗʹద༻͢Δؔ਺Λ
 ߴ֊ؔ਺ map Ͱܨ͗߹ΘͤΔ ؔ਺ ೖྗ A B C +1 —1 ×2 +
  59. FlowGraph DSL: val c = RunnableGraph.fromGraph(FlowGraph.create(out) { implicit b =>

    out => import FlowGraph.Implicits._ ! val f1 = Flow[Int].map(_ + 1) ... ! a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out b ~> f2 ~> f3 ~> zip.in1 ClosedShape }) c.run() A B C +1 —1 ×2 +
  60. TensorFlow http://download.tensorflow.org/paper/whitepaper2015.pdf

  61. DAG on Spark https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png ࣮ߦதͷ Spark δϣϒΛ DAG ͱͯ͠ՄࢹԽͨ͠ྫ

  62. Google Cloud Dataflow https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru

  63. Apache Beam (New!) http://googlecloudplatform.blogspot.co.uk/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html Google ओಋʹΑΔσʔλϑϩʔهड़ͷඪ४ԽͷऔΓ૊Έ

  64. STORAGE & RETRIEVAL LOGIC PRESENTATION ROUTING Redis Memcache Flock T-Bird

    MySQL Tweet User Timeline Social Graph DMs API Web Monorail TFE HTTP Thrift “Stuff” http://monkey.org/~marius/scala2015.pdf Twitter ͷ
 ϚΠΫϩαʔϏεߏ଄
  65. γεςϜϨϕϧͷσʔλϑϩʔ • MSA ͸ϚΠΫϩαʔϏεͱɺͦͷϏδωε্ ͷґଘؔ܎Ͱߏ੒͞ΕΔ • ͭ·ΓγεςϜϨϕϧͷσʔλϑϩʔ • ϚΠΫϩαʔϏεશମΛίʔυͱͯ͠هड़Ͱ ͖ͳ͍ͩΖ͏͔ʁ

    A B C
  66. https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru ϥϯλΠϜͱͯ͠ͷ Google Cloud Optimize Schedule Flow of pipeline User

    code & SDK Monitoring UI σʔλϑϩʔఆٛ σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥ΢υ͕ σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏
  67. ϚΠΫϩαʔϏεσʔλϑϩʔʁ • Ϋϥ΢υϨϕϧͷϥϯλΠϜ͸: • ࢦఆ͞ΕͨσʔλϑϩʔΛ࠷దԽ͢ΔͱڞʹɺϦΞΫ ςΟϒͳίϯϙʔωϯτΛεέδϡʔϦϯά͢Δ • YARN ΍ɹɹɹɹɹɹͷΑ͏ͳ෼ࢄϦιʔεϚωʔδϟ ͔ΒϦιʔεΛ֫ಘ͢Δ

    • ֬อͨ͠ϦιʔεʹίϯϙʔωϯτΛ഑උͯ͠γες Ϝͱ࣮ͯ͠ߦ͢Δ