Distributed System and Functional Programming

「分散処理と
関数型プログラミング」 by @okapies

2016/03/21 第2回 分散システム勉強会 / 第三十回 P2P SIP 勉強会

Transcript

1. ୈ2ճ ෼ࢄγεςϜษڧձ / ୈࡾेճ P2P SIP ษڧձ ෼ࢄॲཧͱ
 ؔ਺ܕϓϩάϥϛϯά Ԭຊ ༤ଠ (@okapies) 2016/03/21
2. ࣗݾ঺հ • Ԭຊ ༤ଠ(@okapies) • ੡଄ۀͰಇ͘ιϑτ԰͞Μ • Scala ͱ Scala OSSs Ѫ޷Ո • ࠷ۙͷ࢓ࣄ͸ΠϯϑϥΤϯδχΞͬΆ͍ײ͡ • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ
3. http://www.slideshare.net/okapies/scalamatsuri-58141520
4. http://www.slideshare.net/okapies/reactive-architecture-20160218-58403521
5. ͳͥ෼ࢄγεςϜ͔ʁ • ੑೳͷεέʔϥϏϦςΟ • ଱ো֐ੑ • ։ൃ૊৫ͷεέʔϥϏϦςΟ (NEW!) ΧδϡΞϧʹ෼ࢄϓϩάϥϛϯά Λ΍Δधཁ͕ग़͖ͯͨ
6. http://martinfowler.com/microservices/
7. ϚΠΫϩαʔϏεΞʔΩςΫνϟ • ҰͭͷΞϓϦέʔγϣϯΛখ͞ͳαʔϏε܈ ͷ૊Έ߹Θͤͱͯ͠ߏங͢Δख๏ http://martinfowler.com/articles/microservices.html
8. MSA Λ࠾༻͍ͯ͠Δاۀ • Amazon • Twitter • Netflix • PayPal • LinkedIn • Tumblr • SoundCloud • LINE • υϫϯΰ • etc…
9. http://anond.hatelabo.jp/20111018190933 Amazon ͷ࠷ॳظ͔Βͷࣄྫ
10. େن໛ͳ MSA ͷྫ http://www.slideshare.net/adriancockcroft/goto-berlin/76
11. ϚΠΫϩαʔϏεͷಛੑ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 5. ඇதԝूݖతͳΨόφϯε 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ 9. ਐԽతઃܭ http://martinfowler.com/microservices/
12. ϚΠΫϩαʔϏεͷಛੑ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 5. ඇதԝूݖతͳΨόφϯε 9. ਐԽతઃܭ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ ૊৫ͷಛੑ (໨త) ٕज़ͷಛੑ (खஈ) http://martinfowler.com/microservices/
13. ϚΠΫϩαʔϏεͷ໨త • ։ൃ૊৫ͷεέʔϥϏϦςΟͷ֬อ • Ϗδωε਱ߦೳྗΛ֩ͱͨ͠ඇதԝूݖతͳ ػೳԣஅܕνʔϜͱͯ͠૊৫Խ͢Δ
 (c.f. Conway’s Law) http://martinfowler.com/articles/microservices.html
14. ϚΠΫϩαʔϏεͷํ๏࿦ • αʔϏεؒʹϏδωε਱ߦೳྗ΍υϝΠϯͷ ϥΠϑαΠΫϧʹैͬͯϞδϡʔϧڥքΛ࣋ͭ • αʔϏεؒ͸ REST ΍୯७ͳ MQ Ͱܨ͙ɻτ ϥϯβΫγϣϯ͸ͳΔ΂͘࢖Θͳ͍ • αʔϏεಉ࢜ͷґଘؔ܎ΛݮΒͯ͠ಠཱʹ഑ උͰ͖ΔΑ͏ʹ͢Δ
15. ϚΠΫϩαʔϏεͷར఺ • ڧ͍ϞδϡʔϧڥքɿϚΠΫϩαʔϏε͸Ϟδϡʔ ϧߏ଄ΛڧԽ͢Δɻ͜ͷ͜ͱ͸ɺνʔϜͷن໛͕େ ͖͘ͳΔʹͭΕͯಛʹॏཁʹͳΔ • ಠཱͨ͠഑උɿ୯७ͳαʔϏε͸഑උ͕༰қͰ͋ Γɺ·ͨࣗ཯తͳͷͰɺݸʑʹෆ۩߹͕ى͖ͯ΋γ εςϜো֐ΛҾ͖ى͜͠ʹ͍͘ • ٕज़తଟ༷ੑɿෳ਺ͷݴޠɺ։ൃϑϨʔϜϫʔΫɺ σʔλετϨʔδٕज़Λ૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ http://martinfowler.com/articles/microservice-trade-offs.html
16. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ʁʁʁʁʁʁʁʁʁʁ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html
17. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ΦϒδΣΫτΛ෼ࢄͤ͞Δͳ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html ෼ࢄԽʹ͸σϝϦοτ
 ͕ͨ͘͞Μ͋Δ
18. ʮ෼ࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ 1. ωοτϫʔΫ͸৴པͰ͖Δɻ 2. ϨΠςϯγ͸θϩͰ͋Δɻ 3. ଳҬ෯͸ແݶͰ͋Δɻ 4. ωοτϫʔΫ͸ηΩϡΞͰ͋Δɻ 5. ωοτϫʔΫߏ੒͸มԽͤͣҰఆͰ͋Δɻ 6. ؅ཧऀ͸̍ਓͰ͋Δɻ 7. τϥϯεϙʔτίετ͸θϩͰ͋Δɻ 8. ωοτϫʔΫ͸ۉ࣭Ͱ͋Δɻ
19. ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html
20. ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html ࠓ೔͸෼ࢄϓϩάϥϛϯάͷ࿩Λ͠·͢
21. ෼ࢄϓϩάϥϛϯάͷͭΒΈ • RPC ͷஞ࣍ॲཧ΍ฒྻॲཧ • ωοτϫʔΫͷো֐ɾ஗Ԇɾ෼அ • ϦϞʔτϗετͷো֐ • εϩοτϦϯάʢ࿈࠯ো֐ͷ๷ࢭʣ • αʔϏεσΟεΧόϦ • ϞχλϦϯάɺϦΫΤετͷ෼ࢄτϨʔε
22. ໋ྩతͳಉظݺͼग़͠ val users = UserService(…) ! // block until the request is completed result = users.findByUserId(“okapies”)
23. ωοτϫʔΫϨΠςϯγ Latency Comparison Numbers (Originally by Peter Norvig) -------------------------- L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us Disk seek 10,000,000 ns 10,000 us 10 ms Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms https://gist.github.com/jboner/2841832
24. ωοτϫʔΫϨΠςϯγ http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html
25. ໋ྩɾಉظܕϓϩάϥϛϯάͷܽ఺ • ಉظݺͼग़͠͸ޮ཰͕ѱ͍ • ී௨͸εϨου΍ϓϩηεΛฒ΂ͯରॲ͢Δ • εϨουʗϓϩηεΛ૿΍͢ͱϦιʔεΛେ ͖͘ফඅ͢Δ (Java thread = 1 MB) • ʢgreen thread ͕࢖͑Δ؀ڥ͸ݶΒΕΔʣ
26. ίʔϧόοΫ def findByUserId(f: User => Unit): Unit ! val users = UserService(…) 
 // register a callback as an argument users.findByUserId(“okapies”) { user => println(s”$user”) } ίʔϧόοΫ
27. ίʔϧόοΫ஍ࠈ • ίʔυͷϞδϡʔϧԽ͕ࠔ೉ • ঢ়ଶʢ෭࡞༻ʣ΍σʔλͷґଘؔ܎ͷ؅ཧ͕ ࠔ೉ • ࣮ߦॱংͷ੍ޚ͕ࠔ೉ʢ֎෦Πϕϯτ࣍ୈͰ มԽ͢Δʣ
28. ഁ໓ͷϐϥϛου var g = ... ! step1 { a => step2 { b => step3 { c => step4 { d => // do something with a, b, c, d and g } } } } ґଘ͢Δඇಉظεςοϓ͕
 ϐϥϛουͷΑ͏ʹੵΈ্͕Δ ֎ଆͷείʔϓͷঢ়ଶΛ҉໧ʹ
 ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕௿͍
29. ෼ࢄͱؔ਺ܕϓϩάϥϛϯά • ʮ͋ΕͰ͠ΐʁϜʔΞͷ๏ଇ͕ഁΕͯϑϦʔ ϥϯν͸ऴΘͬͨɺ͜Ε͔Β͸ϚϧνίΞͰ ฒྻॲཧͩؔ਺ܕͩͬͯ΍ͭʯ • ෼ࢄॲཧʹ͓͚Δؔ਺ܕͷຊ࣭͸ɺϏδωε ϩδοΫΛඇಉظॲཧ΍෼ࢄ؀ڥͰͷԣஅత ؔ৺ࣄ͔Β෼཭ͯ͠ɺ༰қʹϞδϡʔϧԽͰ ͖ΔΑ͏ʹ͢Δ͜ͱ
30. ؔ਺ܕϓϩάϥϛϯά • ࣮༻తʹ͸ϞδϡʔϧੑΛߴΊΔͨΊͷख๏ https://twitter.com/mfeathers/status/29581296216
31. ʢ਺ֶతʣؔ਺ • y = f (x) f x f (x) × ೖྗ͕ܾ·Δͱ
 ग़ྗ͕Ұҙʹܾ·Δ
32. ؔ਺ͷ߹੒ • z = g ◦ f (x) = g ( f (x) ) f x f (x) g ◦ f (x) g f ͷ஋Ҭͱ g ͷఆٛҬ͕ Ұக͢Ε͹߹੒Ͱ͖Δ
33. ७ਮؔ਺ͱ෭࡞༻ • ϓϩάϥϜΛঢ়ଶΛ࣋ͨͳ͍७ਮ (pure) ͳؔ਺ Ͱهड़͠ɺ෭࡞༻Λ֎քͱͷڥքʹू໿͢Δ • Ϟδϡʔϧੑ΍ςετ༰қੑ͕޲্ • ؔ਺͕ϞδϡʔϧʹͳΔ • composability ͕伴
34. ७ਮؔ਺ͱ෭࡞༻ http://www.iij.ad.jp/company/development/external/lecture/pdf/yamamoto.pdf
35. ͳͥؔ਺ϓϩάϥϛϯά͸ॏཁ͔ • QuickCheck ͷ։ൃ΍ QuviQ ͷ૑ۀऀͱͯ͠ ஌ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳ࿦จ • ॳ൛͸ 1984 ೥ʢ30 ೥લʂʣ • ؔ਺ܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍ͭͯ࿦͍ͯ͡ Δ http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html
36. ؔ਺ܕͷʪކʫ • ؔ਺ܕʹ͓͚Δೋͭͷॏཁͳʪކʫ • ߴ֊ؔ਺ʢίϯϏωʔλʣ • ஗ԆධՁ ʮݩͷ໰୊Λ෼ׂ͢Δํ๏͸ɺղͱղΛషΓ߹ ΘͤΔํ๏ʹ௚઀ʹґଘ͢Δɻʯ
37. ߴ֊ؔ਺ • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔ਺ͱϢʔεέʔ εʹಛԽͨؔ͠਺ʹ෼͚ͯϞδϡʔϧԽ ! ! • ϏδωεϩδοΫͱɺͦΕ͕৐͔ͬΔσʔλ ܕͷจ຺Λ෼཭Ͱ͖Δ set. map(_ + 1) // Set[A] map. map(_ + 1) // Map[A, B] list.map(_ + 1) // List[A] ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ࢖͍ճͤΔ
38. ஗ԆධՁ class Cons[A](hd: A, tl: => List[A]) extends List[A] ! def nats(n: Int): List[Int] = new Cons(n, nats(n+1)) def fizzbuzz(n: Int) = n match { case _ if n % 15 == 0 => "FizzBuzz" case _ if n % 3 == 0 => "Fizz" case _ if n % 5 == 0 => "Buzz" case _ => n.toString } nats.map(fizzbuzz).take(100).foreach(println) ඞཁݺͼ ʢϓϧܕʣ ίʔυΛੜ੒ثͱબ୒ثͷ ૊Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ ແݶϦετ
39. What ͱ How ͷ෼཭ Input Input Output (2) ϥϯλΠϜ (1) ϓϩάϥϛϯάϞσϧ (DSL) (how Λ࣮ߦ͢Δ) (what Λهड़͢Δ)
40. ؔ਺ܕͰ෭࡞༻Λѻ͏ʹ͸ʁ • ७ਮؔ਺͸ʢఆ্ٛʣ෭࡞༻Λѻ͑ͳ͍ • Ͳ͏͢Δ͔ʁ • ෭࡞༻΍จ຺Λѻ͏ίϯςφʢശʣΛ࡞ͬ ͯɺ஋ΛͦͷதʹೖΕΔʢϞφυͱݺͿʣ a
41. ശͷதͷ஋Λม׵͢Δ • y = x.map(f) (= map f x) • ७ਮؔ਺ͱ෭࡞༻Λ෼཭ͨ͠··ܭࢉՄೳ • ෭࡞༻͸ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ a b f map(f) औΓग़͢ ໭͢
42. ʮശʯͷछྨ • ৭ʑͳʮശʯ͕͋Δ • Identity • List • State • Reader ! • Writer • Continuation • IO • Future/Promise
43. • y = x.map(f).map(g) Future/Promise f x f (x) g ◦ f (x) g ඇಉظॲཧͷจ຺Λ ѻ͏ʮശʯ
44. • y = x.map(f).map(g) Future/Promise f x f (x) g ◦ f (x) g ࣦഊͷจ຺͸ࣗಈ తʹҾ͖ܧ͕ΕΔ ੒ޭ ࣦഊ
45. Future/Promise // ശͷதͷ஋Λม׵͢Δ def map[B](f: A => B): Future[B] ! // จ຺Λ੾Γସ͑ΔೳྗΛ࣋ͬͨ map def flatMap[B](f: A => Future[B]): Future[B] ! // ෳ਺ͷ Future Λू໿ͯ͠଴ͪ߹Θͤ͢Δ def sequence[A](fs: List[Future[A]]): Future[List[A]]
46. ෳ਺ϦΫΤετͷू໿ val userAndTweets = Future.join( userService.findByUserId(userId), tweetService.findByUserId(userId) ) find find userId userAndTweets User
 Service Tweet
 Service http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16 join ଞͷϚΠΫϩαʔϏε΁ΫΤϦ Λ౤͛ɺશͯͷԠ౴͕ἧͬͨΒ ඇಉظʹू໿ͯ͠λϓϧʹ͢Δ
47. Future/Promise ͷϝϦοτ • ඇಉظॲཧಉ࢜ͷґଘؔ܎ʢσʔλϑϩʔʣΛ એݴతʹهड़͢Δ͜ͱͰϓϩάϥϜΛॻ͘ • ϓϩάϥϚ͸ DSL Λॻ͚͹Α͘ɺඇಉظॲ ཧͷৄࡉ͸ॲཧܥ͕Ӆṭ͢Δ • ਖ਼ৗܥͱɺͦͷଞͷؔ৺ࣄʢো֐ɺ஗ԆͳͲʣ Λ௚ަతʹѻ͏͜ͱ͕Ͱ͖ɺ෦඼Խ͕༰қʹ
48. ϚΠΫϩαʔϏε޲͚ϑϨʔϜϫʔΫ • େن໛ͳϚΠΫϩαʔϏεΛӡ༻͍ͯ͠Δا ۀͷ͏ͪɺ͍͔ͭ͘͸ࣗࣾͷج൫ιϑτ΢Σ ΞΛ OSS Խ͍ͯ͠Δ • Netflix OSS • Twitter Finagle, Zipkin, etc…
49. Finagle Twitter ͷ Finagle ͸ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͠೉͍ཁ ҼͩɻFinagle ͸ɺ෼ࢄτϨʔεɺαʔϏεσΟεΧόϦ΍αʔ Ϗεొ࿥ͱ͍ͬͨ෼ࢄͷ՝୊ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ • Finagle ͸ɺTumblr ΍ SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ Ϗε΁ͷ Scala ಋೖʹ౿Έ੾Δେ͖ͳཁҼʹͳ͍ͬͯΔ • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏ ͭͷཧ༝ʯ Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter
50. http://monkey.org/~marius/funsrv.pdf Your Server as a Function
51. Your Server as a Function Most of our systems are phrased as big future transformers. — Marius Eriksen • Future ܕͱɺͦΕΛม׵͢Δ Service, Filter ͱ ͍͏ؔ਺ͷ૊Έ߹ΘͤͰαʔόΛهड़͢Δ https://monkey.org/~marius/scala2015.pdf
52. Future/Service/Filter trait Service[Req, Rep] extends (Req => Future[Rep]) ! ! ! ! ! ! abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut]) ϦΫΤετΛड͚औͬͯϨεϙϯεΛ แΜͩ Future Λฦؔ͢਺ ϦΫΤετͱ Service Λड͚औͬͯ
 ϨεϙϯεΛแΜͩ Future Λฦؔ͢਺
53. ԣஅతؔ৺ࣄͷ߹੒ recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer FilterΛService ʹ߹੒͢Δͩ ͚Ͱɺαʔόʹ༷ʑͳԣஅత ؔ৺ࣄΛ௥ՃͰ͖Δ http://monkey.org/~marius/funsrv.pdf
54. ӡ༻ج൫ͱͷ࿈ܞ • ྫ͑͹ɺFinagle Ͱॻ͍ͨαʔό͔ΒɺZipkin ͳͲͷ෼ࢄτϨʔεγεςϜʹτϨʔε৘ใΛ ૹ৴ͯ͠ՄࢹԽͰ͖Δ
55. σʔλϑϩʔ DSL • σʔλϑϩʔ DSL ͱϥϯλΠϜͷ૊Έ߹Θͤ͸ɺۙ ೥ɺ༷ʑͳ෼໺Ͱద༻͞Ε͍ͯΔ • Akka Streams, ReactiveX, … • Պֶٕज़ܭࢉ: TensorFlow, Halide • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow, Asakusa Framework, Gearpump
56. FRP (Reactive Extensions) ϚϧνΫϦοΫετϦʔϜ https://gist.github.com/staltz/868e7e9bc2a7b8c1f754 250 ϛϦඵຖʹ
 ΫϦοΫΛूੵ ϦετΛ௕͞
 ʹϚοϐϯά RxJS ʹΑΔඇಉظΫϦοΫετϦʔϜ
57. Akka Streams implicit val system = ActorSystem() implicit val mat = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) A B C +1 —1 ×2 +
58. Akka Streams implicit val system = ActorSystem() implicit val mat = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) ೖྗʹద༻͢Δؔ਺Λ
 ߴ֊ؔ਺ map Ͱܨ͗߹ΘͤΔ ؔ਺ ೖྗ A B C +1 —1 ×2 +
59. FlowGraph DSL: val c = RunnableGraph.fromGraph(FlowGraph.create(out) { implicit b => out => import FlowGraph.Implicits._ ! val f1 = Flow[Int].map(_ + 1) ... ! a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out b ~> f2 ~> f3 ~> zip.in1 ClosedShape }) c.run() A B C +1 —1 ×2 +
60. TensorFlow http://download.tensorflow.org/paper/whitepaper2015.pdf
61. DAG on Spark https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png ࣮ߦதͷ Spark δϣϒΛ DAG ͱͯ͠ՄࢹԽͨ͠ྫ
62. Google Cloud Dataflow https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru
63. Apache Beam (New!) http://googlecloudplatform.blogspot.co.uk/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html Google ओಋʹΑΔσʔλϑϩʔهड़ͷඪ४ԽͷऔΓ૊Έ
64. STORAGE & RETRIEVAL LOGIC PRESENTATION ROUTING Redis Memcache Flock T-Bird MySQL Tweet User Timeline Social Graph DMs API Web Monorail TFE HTTP Thrift “Stuff” http://monkey.org/~marius/scala2015.pdf Twitter ͷ
 ϚΠΫϩαʔϏεߏ଄
65. γεςϜϨϕϧͷσʔλϑϩʔ • MSA ͸ϚΠΫϩαʔϏεͱɺͦͷϏδωε্ ͷґଘؔ܎Ͱߏ੒͞ΕΔ • ͭ·ΓγεςϜϨϕϧͷσʔλϑϩʔ • ϚΠΫϩαʔϏεશମΛίʔυͱͯ͠هड़Ͱ ͖ͳ͍ͩΖ͏͔ʁ A B C
66. https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru ϥϯλΠϜͱͯ͠ͷ Google Cloud Optimize Schedule Flow of pipeline User code & SDK Monitoring UI σʔλϑϩʔఆٛ σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥ΢υ͕ σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏
67. ϚΠΫϩαʔϏεσʔλϑϩʔʁ • Ϋϥ΢υϨϕϧͷϥϯλΠϜ͸: • ࢦఆ͞ΕͨσʔλϑϩʔΛ࠷దԽ͢ΔͱڞʹɺϦΞΫ ςΟϒͳίϯϙʔωϯτΛεέδϡʔϦϯά͢Δ • YARN ΍ɹɹɹɹɹɹͷΑ͏ͳ෼ࢄϦιʔεϚωʔδϟ ͔ΒϦιʔεΛ֫ಘ͢Δ • ֬อͨ͠ϦιʔεʹίϯϙʔωϯτΛ഑උͯ͠γες Ϝͱ࣮ͯ͠ߦ͢Δ