Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed System and Functional Programming

Distributed System and Functional Programming

「分散処理と
関数型プログラミング」 by @okapies

2016/03/21 第2回 分散システム勉強会 / 第三十回 P2P SIP 勉強会

Yuta Okamoto

March 21, 2016
Tweet

More Decks by Yuta Okamoto

Other Decks in Programming

Transcript

  1. ୈ2ճ ෼ࢄγεςϜษڧձ / ୈࡾेճ P2P SIP ษڧձ
    ෼ࢄॲཧͱ

    ؔ਺ܕϓϩάϥϛϯά
    Ԭຊ ༤ଠ (@okapies)
    2016/03/21

    View full-size slide

  2. ࣗݾ঺հ
    • Ԭຊ ༤ଠ(@okapies)
    • ੡଄ۀͰಇ͘ιϑτ԰͞Μ
    • Scala ͱ Scala OSSs Ѫ޷Ո
    • ࠷ۙͷ࢓ࣄ͸ΠϯϑϥΤϯδχΞͬΆ͍ײ͡
    • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ

    View full-size slide

  3. http://www.slideshare.net/okapies/scalamatsuri-58141520

    View full-size slide

  4. http://www.slideshare.net/okapies/reactive-architecture-20160218-58403521

    View full-size slide

  5. ͳͥ෼ࢄγεςϜ͔ʁ
    • ੑೳͷεέʔϥϏϦςΟ
    • ଱ো֐ੑ
    • ։ൃ૊৫ͷεέʔϥϏϦςΟ (NEW!)
    ΧδϡΞϧʹ෼ࢄϓϩάϥϛϯά
    Λ΍Δधཁ͕ग़͖ͯͨ

    View full-size slide

  6. http://martinfowler.com/microservices/

    View full-size slide

  7. ϚΠΫϩαʔϏεΞʔΩςΫνϟ
    • ҰͭͷΞϓϦέʔγϣϯΛখ͞ͳαʔϏε܈
    ͷ૊Έ߹Θͤͱͯ͠ߏங͢Δख๏
    http://martinfowler.com/articles/microservices.html

    View full-size slide

  8. MSA Λ࠾༻͍ͯ͠Δاۀ
    • Amazon
    • Twitter
    • Netflix
    • PayPal
    • LinkedIn
    • Tumblr
    • SoundCloud
    • LINE
    • υϫϯΰ
    • etc…

    View full-size slide

  9. http://anond.hatelabo.jp/20111018190933
    Amazon ͷ࠷ॳظ͔Βͷࣄྫ

    View full-size slide

  10. େن໛ͳ MSA ͷྫ
    http://www.slideshare.net/adriancockcroft/goto-berlin/76

    View full-size slide

  11. ϚΠΫϩαʔϏεͷಛੑ
    1. αʔϏεʹΑΔίϯϙʔωϯτԽ
    2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ
    3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ
    4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ
    5. ඇதԝूݖతͳΨόφϯε
    6. ඇதԝूݖతͳσʔλ؅ཧ
    7. ΠϯϑϥͷࣗಈԽ
    8. ো֐Λߟྀͨ͠ઃܭ
    9. ਐԽతઃܭ
    http://martinfowler.com/microservices/

    View full-size slide

  12. ϚΠΫϩαʔϏεͷಛੑ
    2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ
    3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ
    5. ඇதԝूݖతͳΨόφϯε
    9. ਐԽతઃܭ
    1. αʔϏεʹΑΔίϯϙʔωϯτԽ
    4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ
    6. ඇதԝूݖతͳσʔλ؅ཧ
    7. ΠϯϑϥͷࣗಈԽ
    8. ো֐Λߟྀͨ͠ઃܭ
    ૊৫ͷಛੑ (໨త)
    ٕज़ͷಛੑ (खஈ)
    http://martinfowler.com/microservices/

    View full-size slide

  13. ϚΠΫϩαʔϏεͷ໨త
    • ։ൃ૊৫ͷεέʔϥϏϦςΟͷ֬อ
    • Ϗδωε਱ߦೳྗΛ֩ͱͨ͠ඇதԝूݖతͳ
    ػೳԣஅܕνʔϜͱͯ͠૊৫Խ͢Δ

    (c.f. Conway’s Law)
    http://martinfowler.com/articles/microservices.html

    View full-size slide

  14. ϚΠΫϩαʔϏεͷํ๏࿦
    • αʔϏεؒʹϏδωε਱ߦೳྗ΍υϝΠϯͷ
    ϥΠϑαΠΫϧʹैͬͯϞδϡʔϧڥքΛ࣋ͭ
    • αʔϏεؒ͸ REST ΍୯७ͳ MQ Ͱܨ͙ɻτ
    ϥϯβΫγϣϯ͸ͳΔ΂͘࢖Θͳ͍
    • αʔϏεಉ࢜ͷґଘؔ܎ΛݮΒͯ͠ಠཱʹ഑
    උͰ͖ΔΑ͏ʹ͢Δ

    View full-size slide

  15. ϚΠΫϩαʔϏεͷར఺
    • ڧ͍ϞδϡʔϧڥքɿϚΠΫϩαʔϏε͸Ϟδϡʔ
    ϧߏ଄ΛڧԽ͢Δɻ͜ͷ͜ͱ͸ɺνʔϜͷن໛͕େ
    ͖͘ͳΔʹͭΕͯಛʹॏཁʹͳΔ
    • ಠཱͨ͠഑උɿ୯७ͳαʔϏε͸഑උ͕༰қͰ͋
    Γɺ·ͨࣗ཯తͳͷͰɺݸʑʹෆ۩߹͕ى͖ͯ΋γ
    εςϜো֐ΛҾ͖ى͜͠ʹ͍͘
    • ٕज़తଟ༷ੑɿෳ਺ͷݴޠɺ։ൃϑϨʔϜϫʔΫɺ
    σʔλετϨʔδٕज़Λ૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ
    http://martinfowler.com/articles/microservice-trade-offs.html

    View full-size slide

  16. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ
    • ୈҰ๏ଇ: ʁʁʁʁʁʁʁʁʁʁ
    by Martin Fowler
    http://martinfowler.com/bliki/FirstLaw.html

    View full-size slide

  17. ෼ࢄΦϒδΣΫτઃܭͷ๏ଇ
    • ୈҰ๏ଇ: ΦϒδΣΫτΛ෼ࢄͤ͞Δͳ
    by Martin Fowler
    http://martinfowler.com/bliki/FirstLaw.html
    ෼ࢄԽʹ͸σϝϦοτ

    ͕ͨ͘͞Μ͋Δ

    View full-size slide

  18. ʮ෼ࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ
    1. ωοτϫʔΫ͸৴པͰ͖Δɻ
    2. ϨΠςϯγ͸θϩͰ͋Δɻ
    3. ଳҬ෯͸ແݶͰ͋Δɻ
    4. ωοτϫʔΫ͸ηΩϡΞͰ͋Δɻ
    5. ωοτϫʔΫߏ੒͸มԽͤͣҰఆͰ͋Δɻ
    6. ؅ཧऀ͸̍ਓͰ͋Δɻ
    7. τϥϯεϙʔτίετ͸θϩͰ͋Δɻ
    8. ωοτϫʔΫ͸ۉ࣭Ͱ͋Δɻ

    View full-size slide

  19. ϚΠΫϩαʔϏεͷίετ
    • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ
    Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ
    • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋
    ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ
    ͠ͳ͚Ε͹ͳΒͳ͍
    • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ
    αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ
    http://martinfowler.com/articles/microservice-trade-offs.html

    View full-size slide

  20. ϚΠΫϩαʔϏεͷίετ
    • ෼ࢄɿ෼ࢄγεςϜ͸ϓϩάϥϜ͕೉͍͠ɻϦ
    Ϟʔτݺग़͸஗͍͠ɺࣦഊͷϦεΫ͕͋Δ
    • ݁Ռ੔߹ੑɿ෼ࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋
    ͢Δͷ͸ඇৗʹࠔ೉ͳͷͰɺ݁Ռ੔߹ੑΛ؅ཧ
    ͠ͳ͚Ε͹ͳΒͳ͍
    • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶഑උ͢Δେྔͷ
    αʔϏεΛ؅ཧ͢Δ੒ख़ͨ͠ӡ༻νʔϜ͕ඞཁ
    http://martinfowler.com/articles/microservice-trade-offs.html
    ࠓ೔͸෼ࢄϓϩάϥϛϯάͷ࿩Λ͠·͢

    View full-size slide

  21. ෼ࢄϓϩάϥϛϯάͷͭΒΈ
    • RPC ͷஞ࣍ॲཧ΍ฒྻॲཧ
    • ωοτϫʔΫͷো֐ɾ஗Ԇɾ෼அ
    • ϦϞʔτϗετͷো֐
    • εϩοτϦϯάʢ࿈࠯ো֐ͷ๷ࢭʣ
    • αʔϏεσΟεΧόϦ
    • ϞχλϦϯάɺϦΫΤετͷ෼ࢄτϨʔε

    View full-size slide

  22. ໋ྩతͳಉظݺͼग़͠
    val users = UserService(…)
    !
    // block until the request is completed
    result = users.findByUserId(“okapies”)

    View full-size slide

  23. ωοτϫʔΫϨΠςϯγ
    Latency Comparison Numbers (Originally by Peter Norvig)
    --------------------------
    L1 cache reference 0.5 ns
    Branch mispredict 5 ns
    L2 cache reference 7 ns
    Mutex lock/unlock 25 ns
    Main memory reference 100 ns
    Compress 1K bytes with Zippy 3,000 ns 3 us
    Send 1K bytes over 1 Gbps network 10,000 ns 10 us
    Read 4K randomly from SSD* 150,000 ns 150 us
    Read 1 MB sequentially from memory 250,000 ns 250 us
    Round trip within same datacenter 500,000 ns 500 us
    Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us
    Disk seek 10,000,000 ns 10,000 us 10 ms
    Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms
    Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms
    https://gist.github.com/jboner/2841832

    View full-size slide

  24. ωοτϫʔΫϨΠςϯγ
    http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

    View full-size slide

  25. ໋ྩɾಉظܕϓϩάϥϛϯάͷܽ఺
    • ಉظݺͼग़͠͸ޮ཰͕ѱ͍
    • ී௨͸εϨου΍ϓϩηεΛฒ΂ͯରॲ͢Δ
    • εϨουʗϓϩηεΛ૿΍͢ͱϦιʔεΛେ
    ͖͘ফඅ͢Δ (Java thread = 1 MB)
    • ʢgreen thread ͕࢖͑Δ؀ڥ͸ݶΒΕΔʣ

    View full-size slide

  26. ίʔϧόοΫ
    def findByUserId(f: User => Unit): Unit
    !
    val users = UserService(…)

    // register a callback as an argument
    users.findByUserId(“okapies”) { user =>
    println(s”$user”)
    }
    ίʔϧόοΫ

    View full-size slide

  27. ίʔϧόοΫ஍ࠈ
    • ίʔυͷϞδϡʔϧԽ͕ࠔ೉
    • ঢ়ଶʢ෭࡞༻ʣ΍σʔλͷґଘؔ܎ͷ؅ཧ͕
    ࠔ೉
    • ࣮ߦॱংͷ੍ޚ͕ࠔ೉ʢ֎෦Πϕϯτ࣍ୈͰ
    มԽ͢Δʣ

    View full-size slide

  28. ഁ໓ͷϐϥϛου
    var g = ...
    !
    step1 { a =>
    step2 { b =>
    step3 { c =>
    step4 { d =>
    // do something with a, b, c, d and g
    }
    }
    }
    }
    ґଘ͢Δඇಉظεςοϓ͕

    ϐϥϛουͷΑ͏ʹੵΈ্͕Δ
    ֎ଆͷείʔϓͷঢ়ଶΛ҉໧ʹ

    ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕௿͍

    View full-size slide

  29. ෼ࢄͱؔ਺ܕϓϩάϥϛϯά
    • ʮ͋ΕͰ͠ΐʁϜʔΞͷ๏ଇ͕ഁΕͯϑϦʔ
    ϥϯν͸ऴΘͬͨɺ͜Ε͔Β͸ϚϧνίΞͰ
    ฒྻॲཧͩؔ਺ܕͩͬͯ΍ͭʯ
    • ෼ࢄॲཧʹ͓͚Δؔ਺ܕͷຊ࣭͸ɺϏδωε
    ϩδοΫΛඇಉظॲཧ΍෼ࢄ؀ڥͰͷԣஅత
    ؔ৺ࣄ͔Β෼཭ͯ͠ɺ༰қʹϞδϡʔϧԽͰ
    ͖ΔΑ͏ʹ͢Δ͜ͱ

    View full-size slide

  30. ؔ਺ܕϓϩάϥϛϯά
    • ࣮༻తʹ͸ϞδϡʔϧੑΛߴΊΔͨΊͷख๏
    https://twitter.com/mfeathers/status/29581296216

    View full-size slide

  31. ʢ਺ֶతʣؔ਺
    • y = f (x)
    f
    x f (x)
    ×
    ೖྗ͕ܾ·Δͱ

    ग़ྗ͕Ұҙʹܾ·Δ

    View full-size slide

  32. ؔ਺ͷ߹੒
    • z = g ◦ f (x) = g ( f (x) )
    f
    x f (x) g ◦ f (x)
    g
    f ͷ஋Ҭͱ g ͷఆٛҬ͕
    Ұக͢Ε͹߹੒Ͱ͖Δ

    View full-size slide

  33. ७ਮؔ਺ͱ෭࡞༻
    • ϓϩάϥϜΛঢ়ଶΛ࣋ͨͳ͍७ਮ (pure) ͳؔ਺
    Ͱهड़͠ɺ෭࡞༻Λ֎քͱͷڥքʹू໿͢Δ
    • Ϟδϡʔϧੑ΍ςετ༰қੑ͕޲্
    • ؔ਺͕ϞδϡʔϧʹͳΔ
    • composability ͕伴

    View full-size slide

  34. ७ਮؔ਺ͱ෭࡞༻
    http://www.iij.ad.jp/company/development/external/lecture/pdf/yamamoto.pdf

    View full-size slide

  35. ͳͥؔ਺ϓϩάϥϛϯά͸ॏཁ͔
    • QuickCheck ͷ։ൃ΍ QuviQ ͷ૑ۀऀͱͯ͠
    ஌ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳ࿦จ
    • ॳ൛͸ 1984 ೥ʢ30 ೥લʂʣ
    • ؔ਺ܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ
    ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍ͭͯ࿦͍ͯ͡
    Δ
    http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html

    View full-size slide

  36. ؔ਺ܕͷʪކʫ
    • ؔ਺ܕʹ͓͚Δೋͭͷॏཁͳʪކʫ
    • ߴ֊ؔ਺ʢίϯϏωʔλʣ
    • ஗ԆධՁ
    ʮݩͷ໰୊Λ෼ׂ͢Δํ๏͸ɺղͱղΛషΓ߹
    ΘͤΔํ๏ʹ௚઀ʹґଘ͢Δɻʯ

    View full-size slide

  37. ߴ֊ؔ਺
    • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔ਺ͱϢʔεέʔ
    εʹಛԽͨؔ͠਺ʹ෼͚ͯϞδϡʔϧԽ
    !
    !
    • ϏδωεϩδοΫͱɺͦΕ͕৐͔ͬΔσʔλ
    ܕͷจ຺Λ෼཭Ͱ͖Δ
    set. map(_ + 1) // Set[A]
    map. map(_ + 1) // Map[A, B]
    list.map(_ + 1) // List[A]
    ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ࢖͍ճͤΔ

    View full-size slide

  38. ஗ԆධՁ
    class Cons[A](hd: A, tl: => List[A]) extends List[A]
    !
    def nats(n: Int): List[Int] = new Cons(n, nats(n+1))
    def fizzbuzz(n: Int) = n match {
    case _ if n % 15 == 0 => "FizzBuzz"
    case _ if n % 3 == 0 => "Fizz"
    case _ if n % 5 == 0 => "Buzz"
    case _ => n.toString
    }
    nats.map(fizzbuzz).take(100).foreach(println)
    ඞཁݺͼ
    ʢϓϧܕʣ
    ίʔυΛੜ੒ثͱબ୒ثͷ
    ૊Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ
    ແݶϦετ

    View full-size slide

  39. What ͱ How ͷ෼཭
    Input
    Input
    Output
    (2) ϥϯλΠϜ
    (1) ϓϩάϥϛϯάϞσϧ (DSL)
    (how Λ࣮ߦ͢Δ)
    (what Λهड़͢Δ)

    View full-size slide

  40. ؔ਺ܕͰ෭࡞༻Λѻ͏ʹ͸ʁ
    • ७ਮؔ਺͸ʢఆ্ٛʣ෭࡞༻Λѻ͑ͳ͍
    • Ͳ͏͢Δ͔ʁ
    • ෭࡞༻΍จ຺Λѻ͏ίϯςφʢശʣΛ࡞ͬ
    ͯɺ஋ΛͦͷதʹೖΕΔʢϞφυͱݺͿʣ
    a

    View full-size slide

  41. ശͷதͷ஋Λม׵͢Δ
    • y = x.map(f) (= map f x)
    • ७ਮؔ਺ͱ෭࡞༻Λ෼཭ͨ͠··ܭࢉՄೳ
    • ෭࡞༻͸ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ
    a b
    f
    map(f)
    औΓग़͢ ໭͢

    View full-size slide

  42. ʮശʯͷछྨ
    • ৭ʑͳʮശʯ͕͋Δ
    • Identity
    • List
    • State
    • Reader
    !
    • Writer
    • Continuation
    • IO
    • Future/Promise

    View full-size slide

  43. • y = x.map(f).map(g)
    Future/Promise
    f
    x f (x) g ◦ f (x)
    g
    ඇಉظॲཧͷจ຺Λ
    ѻ͏ʮശʯ

    View full-size slide

  44. • y = x.map(f).map(g)
    Future/Promise
    f
    x f (x) g ◦ f (x)
    g
    ࣦഊͷจ຺͸ࣗಈ
    తʹҾ͖ܧ͕ΕΔ
    ੒ޭ
    ࣦഊ

    View full-size slide

  45. Future/Promise
    // ശͷதͷ஋Λม׵͢Δ
    def map[B](f: A => B): Future[B]
    !
    // จ຺Λ੾Γସ͑ΔೳྗΛ࣋ͬͨ map
    def flatMap[B](f: A => Future[B]): Future[B]
    !
    // ෳ਺ͷ Future Λू໿ͯ͠଴ͪ߹Θͤ͢Δ
    def sequence[A](fs: List[Future[A]]):
    Future[List[A]]

    View full-size slide

  46. ෳ਺ϦΫΤετͷू໿
    val userAndTweets = Future.join(
    userService.findByUserId(userId),
    tweetService.findByUserId(userId)
    )
    find
    find
    userId userAndTweets
    User

    Service
    Tweet

    Service
    http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16
    join
    ଞͷϚΠΫϩαʔϏε΁ΫΤϦ
    Λ౤͛ɺશͯͷԠ౴͕ἧͬͨΒ
    ඇಉظʹू໿ͯ͠λϓϧʹ͢Δ

    View full-size slide

  47. Future/Promise ͷϝϦοτ
    • ඇಉظॲཧಉ࢜ͷґଘؔ܎ʢσʔλϑϩʔʣΛ
    એݴతʹهड़͢Δ͜ͱͰϓϩάϥϜΛॻ͘
    • ϓϩάϥϚ͸ DSL Λॻ͚͹Α͘ɺඇಉظॲ
    ཧͷৄࡉ͸ॲཧܥ͕Ӆṭ͢Δ
    • ਖ਼ৗܥͱɺͦͷଞͷؔ৺ࣄʢো֐ɺ஗ԆͳͲʣ
    Λ௚ަతʹѻ͏͜ͱ͕Ͱ͖ɺ෦඼Խ͕༰қʹ

    View full-size slide

  48. ϚΠΫϩαʔϏε޲͚ϑϨʔϜϫʔΫ
    • େن໛ͳϚΠΫϩαʔϏεΛӡ༻͍ͯ͠Δا
    ۀͷ͏ͪɺ͍͔ͭ͘͸ࣗࣾͷج൫ιϑτ΢Σ
    ΞΛ OSS Խ͍ͯ͠Δ
    • Netflix OSS
    • Twitter Finagle, Zipkin, etc…

    View full-size slide

  49. Finagle
    Twitter ͷ Finagle ͸ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͠೉͍ཁ
    ҼͩɻFinagle ͸ɺ෼ࢄτϨʔεɺαʔϏεσΟεΧόϦ΍αʔ
    Ϗεొ࿥ͱ͍ͬͨ෼ࢄͷ՝୊ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ
    • Finagle ͸ɺTumblr ΍ SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ
    Ϗε΁ͷ Scala ಋೖʹ౿Έ੾Δେ͖ͳཁҼʹͳ͍ͬͯΔ
    • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏
    ͭͷཧ༝ʯ
    Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter

    View full-size slide

  50. http://monkey.org/~marius/funsrv.pdf
    Your Server as a Function

    View full-size slide

  51. Your Server as a Function
    Most of our systems are phrased as big future
    transformers. — Marius Eriksen
    • Future ܕͱɺͦΕΛม׵͢Δ Service, Filter ͱ
    ͍͏ؔ਺ͷ૊Έ߹ΘͤͰαʔόΛهड़͢Δ
    https://monkey.org/~marius/scala2015.pdf

    View full-size slide

  52. Future/Service/Filter
    trait Service[Req, Rep] extends (Req => Future[Rep])
    !
    !
    !
    !
    !
    !
    abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn]
    extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])
    ϦΫΤετΛड͚औͬͯϨεϙϯεΛ
    แΜͩ Future Λฦؔ͢਺
    ϦΫΤετͱ Service Λड͚औͬͯ

    ϨεϙϯεΛแΜͩ Future Λฦؔ͢਺

    View full-size slide

  53. ԣஅతؔ৺ࣄͷ߹੒
    recordHandletime andThen
    traceRequest andThen
    collectJvmStats andThen
    parseRequest andThen
    logRequest andThen
    recordClientStats andThen
    sanitize andThen
    respondToHealthCheck andThen
    applyTrafficControl andThen
    virtualHostServer
    FilterΛService ʹ߹੒͢Δͩ
    ͚Ͱɺαʔόʹ༷ʑͳԣஅత
    ؔ৺ࣄΛ௥ՃͰ͖Δ
    http://monkey.org/~marius/funsrv.pdf

    View full-size slide

  54. ӡ༻ج൫ͱͷ࿈ܞ
    • ྫ͑͹ɺFinagle Ͱॻ͍ͨαʔό͔ΒɺZipkin
    ͳͲͷ෼ࢄτϨʔεγεςϜʹτϨʔε৘ใΛ
    ૹ৴ͯ͠ՄࢹԽͰ͖Δ

    View full-size slide

  55. σʔλϑϩʔ DSL
    • σʔλϑϩʔ DSL ͱϥϯλΠϜͷ૊Έ߹Θͤ͸ɺۙ
    ೥ɺ༷ʑͳ෼໺Ͱద༻͞Ε͍ͯΔ
    • Akka Streams, ReactiveX, …
    • Պֶٕज़ܭࢉ: TensorFlow, Halide
    • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow,
    Asakusa Framework, Gearpump

    View full-size slide

  56. FRP (Reactive Extensions)
    ϚϧνΫϦοΫετϦʔϜ
    https://gist.github.com/staltz/868e7e9bc2a7b8c1f754
    250 ϛϦඵຖʹ

    ΫϦοΫΛूੵ
    ϦετΛ௕͞

    ʹϚοϐϯά
    RxJS ʹΑΔඇಉظΫϦοΫετϦʔϜ

    View full-size slide

  57. Akka Streams
    implicit val system = ActorSystem()
    implicit val mat = ActorMaterializer()
    !
    val a = Source(...)
    val b = Source(...)
    !
    val a1 = a.map(_ + 1)
    val b1 = b.map(_ - 1).map(_ * 2)
    !
    val c = (a1 zip b1).map{case (a, b) => a + b}
    !
    c.runWith(Sink.foreach(println))(mat)
    A
    B C
    +1
    —1
    ×2
    +

    View full-size slide

  58. Akka Streams
    implicit val system = ActorSystem()
    implicit val mat = ActorMaterializer()
    !
    val a = Source(...)
    val b = Source(...)
    !
    val a1 = a.map(_ + 1)
    val b1 = b.map(_ - 1).map(_ * 2)
    !
    val c = (a1 zip b1).map{case (a, b) => a + b}
    !
    c.runWith(Sink.foreach(println))(mat)
    ೖྗʹద༻͢Δؔ਺Λ

    ߴ֊ؔ਺ map Ͱܨ͗߹ΘͤΔ
    ؔ਺
    ೖྗ
    A
    B C
    +1
    —1
    ×2
    +

    View full-size slide

  59. FlowGraph DSL:
    val c = RunnableGraph.fromGraph(FlowGraph.create(out) {
    implicit b => out =>
    import FlowGraph.Implicits._
    !
    val f1 = Flow[Int].map(_ + 1)
    ...
    !
    a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out
    b ~> f2 ~> f3 ~> zip.in1
    ClosedShape
    })
    c.run()
    A
    B C
    +1
    —1
    ×2
    +

    View full-size slide

  60. TensorFlow
    http://download.tensorflow.org/paper/whitepaper2015.pdf

    View full-size slide

  61. DAG on Spark
    https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png
    ࣮ߦதͷ Spark δϣϒΛ DAG ͱͯ͠ՄࢹԽͨ͠ྫ

    View full-size slide

  62. Google Cloud Dataflow
    https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru

    View full-size slide

  63. Apache Beam (New!)
    http://googlecloudplatform.blogspot.co.uk/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html
    Google ओಋʹΑΔσʔλϑϩʔهड़ͷඪ४ԽͷऔΓ૊Έ

    View full-size slide

  64. STORAGE &
    RETRIEVAL
    LOGIC
    PRESENTATION
    ROUTING
    Redis
    Memcache
    Flock
    T-Bird
    MySQL
    Tweet
    User
    Timeline
    Social
    Graph
    DMs
    API
    Web
    Monorail
    TFE
    HTTP Thrift “Stuff”
    http://monkey.org/~marius/scala2015.pdf
    Twitter ͷ

    ϚΠΫϩαʔϏεߏ଄

    View full-size slide

  65. γεςϜϨϕϧͷσʔλϑϩʔ
    • MSA ͸ϚΠΫϩαʔϏεͱɺͦͷϏδωε্
    ͷґଘؔ܎Ͱߏ੒͞ΕΔ
    • ͭ·ΓγεςϜϨϕϧͷσʔλϑϩʔ
    • ϚΠΫϩαʔϏεશମΛίʔυͱͯ͠هड़Ͱ
    ͖ͳ͍ͩΖ͏͔ʁ
    A
    B C

    View full-size slide

  66. https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru
    ϥϯλΠϜͱͯ͠ͷ
    Google Cloud
    Optimize
    Schedule
    Flow of pipeline User code & SDK Monitoring UI
    σʔλϑϩʔఆٛ
    σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥ΢υ͕
    σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏

    View full-size slide

  67. ϚΠΫϩαʔϏεσʔλϑϩʔʁ
    • Ϋϥ΢υϨϕϧͷϥϯλΠϜ͸:
    • ࢦఆ͞ΕͨσʔλϑϩʔΛ࠷దԽ͢ΔͱڞʹɺϦΞΫ
    ςΟϒͳίϯϙʔωϯτΛεέδϡʔϦϯά͢Δ
    • YARN ΍ɹɹɹɹɹɹͷΑ͏ͳ෼ࢄϦιʔεϚωʔδϟ
    ͔ΒϦιʔεΛ֫ಘ͢Δ
    • ֬อͨ͠ϦιʔεʹίϯϙʔωϯτΛ഑උͯ͠γες
    Ϝͱ࣮ͯ͠ߦ͢Δ

    View full-size slide