Slide 1

Slide 1 text

ୈ2ճ ෼ࢄγεςϜษڧձ / ୈࡾेճ P2P SIP ษڧձ ෼ࢄॲཧͱ
 ؔ਺ܕϓϩάϥϛϯά Ԭຊ ༤ଠ (@okapies) 2016/03/21

Slide 2

Slide 2 text

ࣗݾ঺հ • Ԭຊ ༤ଠ(@okapies) • ੡଄ۀͰಇ͘ιϑτ԰͞Μ • Scala ͱ Scala OSSs Ѫ޷Ո • ࠷ۙͷ࢓ࣄ͸ΠϯϑϥΤϯδχΞͬΆ͍ײ͡ • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ

Slide 3

Slide 3 text

http://www.slideshare.net/okapies/scalamatsuri-58141520

Slide 4

Slide 4 text

http://www.slideshare.net/okapies/reactive-architecture-20160218-58403521

Slide 5

Slide 5 text

ͳͥ෼ࢄγεςϜ͔ʁ • ੑೳͷεέʔϥϏϦςΟ • ଱ো֐ੑ • ։ൃ૊৫ͷεέʔϥϏϦςΟ (NEW!) ΧδϡΞϧʹ෼ࢄϓϩάϥϛϯά Λ΍Δधཁ͕ग़͖ͯͨ

Slide 6

Slide 6 text

http://martinfowler.com/microservices/

Slide 7

Slide 7 text

ϚΠΫϩαʔϏεΞʔΩςΫνϟ • ҰͭͷΞϓϦέʔγϣϯΛখ͞ͳαʔϏε܈ ͷ૊Έ߹Θͤͱͯ͠ߏங͢Δख๏ http://martinfowler.com/articles/microservices.html

Slide 8

Slide 8 text

MSA Λ࠾༻͍ͯ͠Δاۀ • Amazon • Twitter • Netflix • PayPal • LinkedIn • Tumblr • SoundCloud • LINE • υϫϯΰ • etc…

Slide 9

Slide 9 text

http://anond.hatelabo.jp/20111018190933 Amazon ͷ࠷ॳظ͔Βͷࣄྫ

Slide 10

Slide 10 text

େن໛ͳ MSA ͷྫ http://www.slideshare.net/adriancockcroft/goto-berlin/76

Slide 11

Slide 11 text

ϚΠΫϩαʔϏεͷಛੑ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 5. ඇதԝूݖతͳΨόφϯε 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ 9. ਐԽతઃܭ http://martinfowler.com/microservices/

Slide 12

Slide 12 text

ϚΠΫϩαʔϏεͷಛੑ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 5. ඇதԝूݖతͳΨόφϯε 9. ਐԽతઃܭ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ ૊৫ͷಛੑ (໨త) ٕज़ͷಛੑ (खஈ) http://martinfowler.com/microservices/

Slide 13

Slide 13 text

ϚΠΫϩαʔϏεͷ໨త • ։ൃ૊৫ͷεέʔϥϏϦςΟͷ֬อ • Ϗδωε਱ߦೳྗΛ֩ͱͨ͠ඇதԝूݖతͳ ػೳԣஅܕνʔϜͱͯ͠૊৫Խ͢Δ
 (c.f. Conway’s Law) http://martinfowler.com/articles/microservices.html

Slide 14

Slide 14 text

ϚΠΫϩαʔϏεͷํ๏࿦ • αʔϏεؒʹϏδωε਱ߦೳྗ΍υϝΠϯͷ ϥΠϑαΠΫϧʹैͬͯϞδϡʔϧڥքΛ࣋ͭ • αʔϏεؒ͸ REST ΍୯७ͳ MQ Ͱܨ͙ɻτ ϥϯβΫγϣϯ͸ͳΔ΂͘࢖Θͳ͍ • αʔϏεಉ࢜ͷґଘؔ܎ΛݮΒͯ͠ಠཱʹ഑ උͰ͖ΔΑ͏ʹ͢Δ

Slide 15

Slide 15 text

ϚΠΫϩαʔϏεͷར఺ • ڧ͍ϞδϡʔϧڥքɿϚΠΫϩαʔϏε͸Ϟδϡʔ ϧߏ଄ΛڧԽ͢Δɻ͜ͷ͜ͱ͸ɺνʔϜͷن໛͕େ ͖͘ͳΔʹͭΕͯಛʹॏཁʹͳΔ • ಠཱͨ͠഑උɿ୯७ͳαʔϏε͸഑උ͕༰қͰ͋ Γɺ·ͨࣗ཯తͳͷͰɺݸʑʹෆ۩߹͕ى͖ͯ΋γ εςϜো֐ΛҾ͖ى͜͠ʹ͍͘ • ٕज़తଟ༷ੑɿෳ਺ͷݴޠɺ։ൃϑϨʔϜϫʔΫɺ σʔλετϨʔδٕज़Λ૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ http://martinfowler.com/articles/microservice-trade-offs.html

Slide 16

Slide 16 text

෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ʁʁʁʁʁʁʁʁʁʁ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html

Slide 17

Slide 17 text

෼ࢄΦϒδΣΫτઃܭͷ๏ଇ • ୈҰ๏ଇ: ΦϒδΣΫτΛ෼ࢄͤ͞Δͳ by Martin Fowler http://martinfowler.com/bliki/FirstLaw.html ෼ࢄԽʹ͸σϝϦοτ
 ͕ͨ͘͞Μ͋Δ

Slide 18

Slide 18 text

ʮ෼ࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ 1. ωοτϫʔΫ͸৴པͰ͖Δɻ 2. ϨΠςϯγ͸θϩͰ͋Δɻ 3. ଳҬ෯͸ແݶͰ͋Δɻ 4. ωοτϫʔΫ͸ηΩϡΞͰ͋Δɻ 5. ωοτϫʔΫߏ੒͸มԽͤͣҰఆͰ͋Δɻ 6. ؅ཧऀ͸̍ਓͰ͋Δɻ 7. τϥϯεϙʔτίετ͸θϩͰ͋Δɻ 8. ωοτϫʔΫ͸ۉ࣭Ͱ͋Δɻ

Slide 19

Slide 19 text

ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html

Slide 20

Slide 20 text

ϚΠΫϩαʔϏεͷίετ • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ ͠ͳ͚Ε͹ͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html ࠓ೔͸෼ࢄϓϩάϥϛϯάͷ࿩Λ͠·͢

Slide 21

Slide 21 text

෼ࢄϓϩάϥϛϯάͷͭΒΈ • RPC ͷஞ࣍ॲཧ΍ฒྻॲཧ • ωοτϫʔΫͷো֐ɾ஗Ԇɾ෼அ • ϦϞʔτϗετͷো֐ • εϩοτϦϯάʢ࿈࠯ো֐ͷ๷ࢭʣ • αʔϏεσΟεΧόϦ • ϞχλϦϯάɺϦΫΤετͷ෼ࢄτϨʔε

Slide 22

Slide 22 text

໋ྩతͳಉظݺͼग़͠ val users = UserService(…) ! // block until the request is completed result = users.findByUserId(“okapies”)

Slide 23

Slide 23 text

ωοτϫʔΫϨΠςϯγ Latency Comparison Numbers (Originally by Peter Norvig) -------------------------- L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us Disk seek 10,000,000 ns 10,000 us 10 ms Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms https://gist.github.com/jboner/2841832

Slide 24

Slide 24 text

ωοτϫʔΫϨΠςϯγ http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

Slide 25

Slide 25 text

໋ྩɾಉظܕϓϩάϥϛϯάͷܽ఺ • ಉظݺͼग़͠͸ޮ཰͕ѱ͍ • ී௨͸εϨου΍ϓϩηεΛฒ΂ͯରॲ͢Δ • εϨουʗϓϩηεΛ૿΍͢ͱϦιʔεΛେ ͖͘ফඅ͢Δ (Java thread = 1 MB) • ʢgreen thread ͕࢖͑Δ؀ڥ͸ݶΒΕΔʣ

Slide 26

Slide 26 text

ίʔϧόοΫ def findByUserId(f: User => Unit): Unit ! val users = UserService(…) 
 // register a callback as an argument users.findByUserId(“okapies”) { user => println(s”$user”) } ίʔϧόοΫ

Slide 27

Slide 27 text

ίʔϧόοΫ஍ࠈ • ίʔυͷϞδϡʔϧԽ͕ࠔ೉ • ঢ়ଶʢ෭࡞༻ʣ΍σʔλͷґଘؔ܎ͷ؅ཧ͕ ࠔ೉ • ࣮ߦॱংͷ੍ޚ͕ࠔ೉ʢ֎෦Πϕϯτ࣍ୈͰ มԽ͢Δʣ

Slide 28

Slide 28 text

ഁ໓ͷϐϥϛου var g = ... ! step1 { a => step2 { b => step3 { c => step4 { d => // do something with a, b, c, d and g } } } } ґଘ͢Δඇಉظεςοϓ͕
 ϐϥϛουͷΑ͏ʹੵΈ্͕Δ ֎ଆͷείʔϓͷঢ়ଶΛ҉໧ʹ
 ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕௿͍

Slide 29

Slide 29 text

෼ࢄͱؔ਺ܕϓϩάϥϛϯά • ʮ͋ΕͰ͠ΐʁϜʔΞͷ๏ଇ͕ഁΕͯϑϦʔ ϥϯν͸ऴΘͬͨɺ͜Ε͔Β͸ϚϧνίΞͰ ฒྻॲཧͩؔ਺ܕͩͬͯ΍ͭʯ • ෼ࢄॲཧʹ͓͚Δؔ਺ܕͷຊ࣭͸ɺϏδωε ϩδοΫΛඇಉظॲཧ΍෼ࢄ؀ڥͰͷԣஅత ؔ৺ࣄ͔Β෼཭ͯ͠ɺ༰қʹϞδϡʔϧԽͰ ͖ΔΑ͏ʹ͢Δ͜ͱ

Slide 30

Slide 30 text

ؔ਺ܕϓϩάϥϛϯά • ࣮༻తʹ͸ϞδϡʔϧੑΛߴΊΔͨΊͷख๏ https://twitter.com/mfeathers/status/29581296216

Slide 31

Slide 31 text

ʢ਺ֶతʣؔ਺ • y = f (x) f x f (x) × ೖྗ͕ܾ·Δͱ
 ग़ྗ͕Ұҙʹܾ·Δ

Slide 32

Slide 32 text

ؔ਺ͷ߹੒ • z = g ◦ f (x) = g ( f (x) ) f x f (x) g ◦ f (x) g f ͷ஋Ҭͱ g ͷఆٛҬ͕ Ұக͢Ε͹߹੒Ͱ͖Δ

Slide 33

Slide 33 text

७ਮؔ਺ͱ෭࡞༻ • ϓϩάϥϜΛঢ়ଶΛ࣋ͨͳ͍७ਮ (pure) ͳؔ਺ Ͱهड़͠ɺ෭࡞༻Λ֎քͱͷڥքʹू໿͢Δ • Ϟδϡʔϧੑ΍ςετ༰қੑ͕޲্ • ؔ਺͕ϞδϡʔϧʹͳΔ • composability ͕伴

Slide 34

Slide 34 text

७ਮؔ਺ͱ෭࡞༻ http://www.iij.ad.jp/company/development/external/lecture/pdf/yamamoto.pdf

Slide 35

Slide 35 text

ͳͥؔ਺ϓϩάϥϛϯά͸ॏཁ͔ • QuickCheck ͷ։ൃ΍ QuviQ ͷ૑ۀऀͱͯ͠ ஌ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳ࿦จ • ॳ൛͸ 1984 ೥ʢ30 ೥લʂʣ • ؔ਺ܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍ͭͯ࿦͍ͯ͡ Δ http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html

Slide 36

Slide 36 text

ؔ਺ܕͷʪކʫ • ؔ਺ܕʹ͓͚Δೋͭͷॏཁͳʪކʫ • ߴ֊ؔ਺ʢίϯϏωʔλʣ • ஗ԆධՁ ʮݩͷ໰୊Λ෼ׂ͢Δํ๏͸ɺղͱղΛషΓ߹ ΘͤΔํ๏ʹ௚઀ʹґଘ͢Δɻʯ

Slide 37

Slide 37 text

ߴ֊ؔ਺ • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔ਺ͱϢʔεέʔ εʹಛԽͨؔ͠਺ʹ෼͚ͯϞδϡʔϧԽ ! ! • ϏδωεϩδοΫͱɺͦΕ͕৐͔ͬΔσʔλ ܕͷจ຺Λ෼཭Ͱ͖Δ set. map(_ + 1) // Set[A] map. map(_ + 1) // Map[A, B] list.map(_ + 1) // List[A] ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ࢖͍ճͤΔ

Slide 38

Slide 38 text

஗ԆධՁ class Cons[A](hd: A, tl: => List[A]) extends List[A] ! def nats(n: Int): List[Int] = new Cons(n, nats(n+1)) def fizzbuzz(n: Int) = n match { case _ if n % 15 == 0 => "FizzBuzz" case _ if n % 3 == 0 => "Fizz" case _ if n % 5 == 0 => "Buzz" case _ => n.toString } nats.map(fizzbuzz).take(100).foreach(println) ඞཁݺͼ ʢϓϧܕʣ ίʔυΛੜ੒ثͱબ୒ثͷ ૊Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ ແݶϦετ

Slide 39

Slide 39 text

What ͱ How ͷ෼཭ Input Input Output (2) ϥϯλΠϜ (1) ϓϩάϥϛϯάϞσϧ (DSL) (how Λ࣮ߦ͢Δ) (what Λهड़͢Δ)

Slide 40

Slide 40 text

ؔ਺ܕͰ෭࡞༻Λѻ͏ʹ͸ʁ • ७ਮؔ਺͸ʢఆ্ٛʣ෭࡞༻Λѻ͑ͳ͍ • Ͳ͏͢Δ͔ʁ • ෭࡞༻΍จ຺Λѻ͏ίϯςφʢശʣΛ࡞ͬ ͯɺ஋ΛͦͷதʹೖΕΔʢϞφυͱݺͿʣ a

Slide 41

Slide 41 text

ശͷதͷ஋Λม׵͢Δ • y = x.map(f) (= map f x) • ७ਮؔ਺ͱ෭࡞༻Λ෼཭ͨ͠··ܭࢉՄೳ • ෭࡞༻͸ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ a b f map(f) औΓग़͢ ໭͢

Slide 42

Slide 42 text

ʮശʯͷछྨ • ৭ʑͳʮശʯ͕͋Δ • Identity • List • State • Reader ! • Writer • Continuation • IO • Future/Promise

Slide 43

Slide 43 text

• y = x.map(f).map(g) Future/Promise f x f (x) g ◦ f (x) g ඇಉظॲཧͷจ຺Λ ѻ͏ʮശʯ

Slide 44

Slide 44 text

• y = x.map(f).map(g) Future/Promise f x f (x) g ◦ f (x) g ࣦഊͷจ຺͸ࣗಈ తʹҾ͖ܧ͕ΕΔ ੒ޭ ࣦഊ

Slide 45

Slide 45 text

Future/Promise // ശͷதͷ஋Λม׵͢Δ def map[B](f: A => B): Future[B] ! // จ຺Λ੾Γସ͑ΔೳྗΛ࣋ͬͨ map def flatMap[B](f: A => Future[B]): Future[B] ! // ෳ਺ͷ Future Λू໿ͯ͠଴ͪ߹Θͤ͢Δ def sequence[A](fs: List[Future[A]]): Future[List[A]]

Slide 46

Slide 46 text

ෳ਺ϦΫΤετͷू໿ val userAndTweets = Future.join( userService.findByUserId(userId), tweetService.findByUserId(userId) ) find find userId userAndTweets User
 Service Tweet
 Service http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16 join ଞͷϚΠΫϩαʔϏε΁ΫΤϦ Λ౤͛ɺશͯͷԠ౴͕ἧͬͨΒ ඇಉظʹू໿ͯ͠λϓϧʹ͢Δ

Slide 47

Slide 47 text

Future/Promise ͷϝϦοτ • ඇಉظॲཧಉ࢜ͷґଘؔ܎ʢσʔλϑϩʔʣΛ એݴతʹهड़͢Δ͜ͱͰϓϩάϥϜΛॻ͘ • ϓϩάϥϚ͸ DSL Λॻ͚͹Α͘ɺඇಉظॲ ཧͷৄࡉ͸ॲཧܥ͕Ӆṭ͢Δ • ਖ਼ৗܥͱɺͦͷଞͷؔ৺ࣄʢো֐ɺ஗ԆͳͲʣ Λ௚ަతʹѻ͏͜ͱ͕Ͱ͖ɺ෦඼Խ͕༰қʹ

Slide 48

Slide 48 text

ϚΠΫϩαʔϏε޲͚ϑϨʔϜϫʔΫ • େن໛ͳϚΠΫϩαʔϏεΛӡ༻͍ͯ͠Δا ۀͷ͏ͪɺ͍͔ͭ͘͸ࣗࣾͷج൫ιϑτ΢Σ ΞΛ OSS Խ͍ͯ͠Δ • Netflix OSS • Twitter Finagle, Zipkin, etc…

Slide 49

Slide 49 text

Finagle Twitter ͷ Finagle ͸ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͠೉͍ཁ ҼͩɻFinagle ͸ɺ෼ࢄτϨʔεɺαʔϏεσΟεΧόϦ΍αʔ Ϗεొ࿥ͱ͍ͬͨ෼ࢄͷ՝୊ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ • Finagle ͸ɺTumblr ΍ SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ Ϗε΁ͷ Scala ಋೖʹ౿Έ੾Δେ͖ͳཁҼʹͳ͍ͬͯΔ • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏ ͭͷཧ༝ʯ Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter

Slide 50

Slide 50 text

http://monkey.org/~marius/funsrv.pdf Your Server as a Function

Slide 51

Slide 51 text

Your Server as a Function Most of our systems are phrased as big future transformers. — Marius Eriksen • Future ܕͱɺͦΕΛม׵͢Δ Service, Filter ͱ ͍͏ؔ਺ͷ૊Έ߹ΘͤͰαʔόΛهड़͢Δ https://monkey.org/~marius/scala2015.pdf

Slide 52

Slide 52 text

Future/Service/Filter trait Service[Req, Rep] extends (Req => Future[Rep]) ! ! ! ! ! ! abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut]) ϦΫΤετΛड͚औͬͯϨεϙϯεΛ แΜͩ Future Λฦؔ͢਺ ϦΫΤετͱ Service Λड͚औͬͯ
 ϨεϙϯεΛแΜͩ Future Λฦؔ͢਺

Slide 53

Slide 53 text

ԣஅతؔ৺ࣄͷ߹੒ recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer FilterΛService ʹ߹੒͢Δͩ ͚Ͱɺαʔόʹ༷ʑͳԣஅత ؔ৺ࣄΛ௥ՃͰ͖Δ http://monkey.org/~marius/funsrv.pdf

Slide 54

Slide 54 text

ӡ༻ج൫ͱͷ࿈ܞ • ྫ͑͹ɺFinagle Ͱॻ͍ͨαʔό͔ΒɺZipkin ͳͲͷ෼ࢄτϨʔεγεςϜʹτϨʔε৘ใΛ ૹ৴ͯ͠ՄࢹԽͰ͖Δ

Slide 55

Slide 55 text

σʔλϑϩʔ DSL • σʔλϑϩʔ DSL ͱϥϯλΠϜͷ૊Έ߹Θͤ͸ɺۙ ೥ɺ༷ʑͳ෼໺Ͱద༻͞Ε͍ͯΔ • Akka Streams, ReactiveX, … • Պֶٕज़ܭࢉ: TensorFlow, Halide • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow, Asakusa Framework, Gearpump

Slide 56

Slide 56 text

FRP (Reactive Extensions) ϚϧνΫϦοΫετϦʔϜ https://gist.github.com/staltz/868e7e9bc2a7b8c1f754 250 ϛϦඵຖʹ
 ΫϦοΫΛूੵ ϦετΛ௕͞
 ʹϚοϐϯά RxJS ʹΑΔඇಉظΫϦοΫετϦʔϜ

Slide 57

Slide 57 text

Akka Streams implicit val system = ActorSystem() implicit val mat = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) A B C +1 —1 ×2 +

Slide 58

Slide 58 text

Akka Streams implicit val system = ActorSystem() implicit val mat = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) ೖྗʹద༻͢Δؔ਺Λ
 ߴ֊ؔ਺ map Ͱܨ͗߹ΘͤΔ ؔ਺ ೖྗ A B C +1 —1 ×2 +

Slide 59

Slide 59 text

FlowGraph DSL: val c = RunnableGraph.fromGraph(FlowGraph.create(out) { implicit b => out => import FlowGraph.Implicits._ ! val f1 = Flow[Int].map(_ + 1) ... ! a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out b ~> f2 ~> f3 ~> zip.in1 ClosedShape }) c.run() A B C +1 —1 ×2 +

Slide 60

Slide 60 text

TensorFlow http://download.tensorflow.org/paper/whitepaper2015.pdf

Slide 61

Slide 61 text

DAG on Spark https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png ࣮ߦதͷ Spark δϣϒΛ DAG ͱͯ͠ՄࢹԽͨ͠ྫ

Slide 62

Slide 62 text

Google Cloud Dataflow https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru

Slide 63

Slide 63 text

Apache Beam (New!) http://googlecloudplatform.blogspot.co.uk/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html Google ओಋʹΑΔσʔλϑϩʔهड़ͷඪ४ԽͷऔΓ૊Έ

Slide 64

Slide 64 text

STORAGE & RETRIEVAL LOGIC PRESENTATION ROUTING Redis Memcache Flock T-Bird MySQL Tweet User Timeline Social Graph DMs API Web Monorail TFE HTTP Thrift “Stuff” http://monkey.org/~marius/scala2015.pdf Twitter ͷ
 ϚΠΫϩαʔϏεߏ଄

Slide 65

Slide 65 text

γεςϜϨϕϧͷσʔλϑϩʔ • MSA ͸ϚΠΫϩαʔϏεͱɺͦͷϏδωε্ ͷґଘؔ܎Ͱߏ੒͞ΕΔ • ͭ·ΓγεςϜϨϕϧͷσʔλϑϩʔ • ϚΠΫϩαʔϏεશମΛίʔυͱͯ͠هड़Ͱ ͖ͳ͍ͩΖ͏͔ʁ A B C

Slide 66

Slide 66 text

https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru ϥϯλΠϜͱͯ͠ͷ Google Cloud Optimize Schedule Flow of pipeline User code & SDK Monitoring UI σʔλϑϩʔఆٛ σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥ΢υ͕ σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏

Slide 67

Slide 67 text

ϚΠΫϩαʔϏεσʔλϑϩʔʁ • Ϋϥ΢υϨϕϧͷϥϯλΠϜ͸: • ࢦఆ͞ΕͨσʔλϑϩʔΛ࠷దԽ͢ΔͱڞʹɺϦΞΫ ςΟϒͳίϯϙʔωϯτΛεέδϡʔϦϯά͢Δ • YARN ΍ɹɹɹɹɹɹͷΑ͏ͳ෼ࢄϦιʔεϚωʔδϟ ͔ΒϦιʔεΛ֫ಘ͢Δ • ֬อͨ͠ϦιʔεʹίϯϙʔωϯτΛ഑උͯ͠γες Ϝͱ࣮ͯ͠ߦ͢Δ