Distributed System and Functional Programming

Distributed System and Functional Programming

「分散処理と
関数型プログラミング」 by @okapies

2016/03/21 第2回 分散システム勉強会 / 第三十回 P2P SIP 勉強会

Ef45a84b10aa3cb5acd2ea0c9846bd3b?s=128

Yuta Okamoto

March 21, 2016
Tweet

Transcript

  1. 2.

    ࣗݾ঺հ • Ԭຊ ༤ଠ(@okapies) • ੡଄ۀͰಇ͘ιϑτ԰͞Μ • Scala ͱ Scala

    OSSs Ѫ޷Ո • ࠷ۙͷ࢓ࣄ͸ΠϯϑϥΤϯδχΞͬΆ͍ײ͡ • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ
  2. 8.

    MSA Λ࠾༻͍ͯ͠Δاۀ • Amazon • Twitter • Netflix • PayPal

    • LinkedIn • Tumblr • SoundCloud • LINE • υϫϯΰ • etc…
  3. 11.

    ϚΠΫϩαʔϏεͷಛੑ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 5.

    ඇதԝूݖతͳΨόφϯε 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ 9. ਐԽతઃܭ http://martinfowler.com/microservices/
  4. 12.

    ϚΠΫϩαʔϏεͷಛੑ 2. Ϗδωε਱ߦೳྗʹجͮ͘૊৫Խ 3. ϓϩδΣΫτͰ͸ͳ͘ϓϩμΫτ 5. ඇதԝूݖతͳΨόφϯε 9. ਐԽతઃܭ 1.

    αʔϏεʹΑΔίϯϙʔωϯτԽ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 6. ඇதԝूݖతͳσʔλ؅ཧ 7. ΠϯϑϥͷࣗಈԽ 8. ো֐Λߟྀͨ͠ઃܭ ૊৫ͷಛੑ (໨త) ٕज़ͷಛੑ (खஈ) http://martinfowler.com/microservices/
  5. 18.

    ʮ෼ࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ 1. ωοτϫʔΫ͸৴པͰ͖Δɻ 2. ϨΠςϯγ͸θϩͰ͋Δɻ 3. ଳҬ෯͸ແݶͰ͋Δɻ 4. ωοτϫʔΫ͸ηΩϡΞͰ͋Δɻ 5.

    ωοτϫʔΫߏ੒͸มԽͤͣҰఆͰ͋Δɻ 6. ؅ཧऀ͸̍ਓͰ͋Δɻ 7. τϥϯεϙʔτίετ͸θϩͰ͋Δɻ 8. ωοτϫʔΫ͸ۉ࣭Ͱ͋Δɻ
  6. 22.

    ໋ྩతͳಉظݺͼग़͠ val users = UserService(…) ! // block until the

    request is completed result = users.findByUserId(“okapies”)
  7. 23.

    ωοτϫʔΫϨΠςϯγ Latency Comparison Numbers (Originally by Peter Norvig) -------------------------- L1

    cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us Disk seek 10,000,000 ns 10,000 us 10 ms Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms https://gist.github.com/jboner/2841832
  8. 26.

    ίʔϧόοΫ def findByUserId(f: User => Unit): Unit ! val users

    = UserService(…) 
 // register a callback as an argument users.findByUserId(“okapies”) { user => println(s”$user”) } ίʔϧόοΫ
  9. 28.

    ഁ໓ͷϐϥϛου var g = ... ! step1 { a =>

    step2 { b => step3 { c => step4 { d => // do something with a, b, c, d and g } } } } ґଘ͢Δඇಉظεςοϓ͕
 ϐϥϛουͷΑ͏ʹੵΈ্͕Δ ֎ଆͷείʔϓͷঢ়ଶΛ҉໧ʹ
 ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕௿͍
  10. 31.

    ʢ਺ֶతʣؔ਺ • y = f (x) f x f (x)

    × ೖྗ͕ܾ·Δͱ
 ग़ྗ͕Ұҙʹܾ·Δ
  11. 32.

    ؔ਺ͷ߹੒ • z = g ◦ f (x) = g

    ( f (x) ) f x f (x) g ◦ f (x) g f ͷ஋Ҭͱ g ͷఆٛҬ͕ Ұக͢Ε͹߹੒Ͱ͖Δ
  12. 35.

    ͳͥؔ਺ϓϩάϥϛϯά͸ॏཁ͔ • QuickCheck ͷ։ൃ΍ QuviQ ͷ૑ۀऀͱͯ͠ ஌ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳ࿦จ • ॳ൛͸ 1984

    ೥ʢ30 ೥લʂʣ • ؔ਺ܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍ͭͯ࿦͍ͯ͡ Δ http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html
  13. 37.

    ߴ֊ؔ਺ • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔ਺ͱϢʔεέʔ εʹಛԽͨؔ͠਺ʹ෼͚ͯϞδϡʔϧԽ ! ! • ϏδωεϩδοΫͱɺͦΕ͕৐͔ͬΔσʔλ ܕͷจ຺Λ෼཭Ͱ͖Δ set.

    map(_ + 1) // Set[A] map. map(_ + 1) // Map[A, B] list.map(_ + 1) // List[A] ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ࢖͍ճͤΔ
  14. 38.

    ஗ԆධՁ class Cons[A](hd: A, tl: => List[A]) extends List[A] !

    def nats(n: Int): List[Int] = new Cons(n, nats(n+1)) def fizzbuzz(n: Int) = n match { case _ if n % 15 == 0 => "FizzBuzz" case _ if n % 3 == 0 => "Fizz" case _ if n % 5 == 0 => "Buzz" case _ => n.toString } nats.map(fizzbuzz).take(100).foreach(println) ඞཁݺͼ ʢϓϧܕʣ ίʔυΛੜ੒ثͱબ୒ثͷ ૊Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ ແݶϦετ
  15. 39.

    What ͱ How ͷ෼཭ Input Input Output (2) ϥϯλΠϜ (1)

    ϓϩάϥϛϯάϞσϧ (DSL) (how Λ࣮ߦ͢Δ) (what Λهड़͢Δ)
  16. 41.

    ശͷதͷ஋Λม׵͢Δ • y = x.map(f) (= map f x) •

    ७ਮؔ਺ͱ෭࡞༻Λ෼཭ͨ͠··ܭࢉՄೳ • ෭࡞༻͸ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ a b f map(f) औΓग़͢ ໭͢
  17. 42.

    ʮശʯͷछྨ • ৭ʑͳʮശʯ͕͋Δ • Identity • List • State •

    Reader ! • Writer • Continuation • IO • Future/Promise
  18. 43.

    • y = x.map(f).map(g) Future/Promise f x f (x) g

    ◦ f (x) g ඇಉظॲཧͷจ຺Λ ѻ͏ʮശʯ
  19. 44.

    • y = x.map(f).map(g) Future/Promise f x f (x) g

    ◦ f (x) g ࣦഊͷจ຺͸ࣗಈ తʹҾ͖ܧ͕ΕΔ ੒ޭ ࣦഊ
  20. 45.

    Future/Promise // ശͷதͷ஋Λม׵͢Δ def map[B](f: A => B): Future[B] !

    // จ຺Λ੾Γସ͑ΔೳྗΛ࣋ͬͨ map def flatMap[B](f: A => Future[B]): Future[B] ! // ෳ਺ͷ Future Λू໿ͯ͠଴ͪ߹Θͤ͢Δ def sequence[A](fs: List[Future[A]]): Future[List[A]]
  21. 46.

    ෳ਺ϦΫΤετͷू໿ val userAndTweets = Future.join( userService.findByUserId(userId), tweetService.findByUserId(userId) ) find find

    userId userAndTweets User
 Service Tweet
 Service http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16 join ଞͷϚΠΫϩαʔϏε΁ΫΤϦ Λ౤͛ɺશͯͷԠ౴͕ἧͬͨΒ ඇಉظʹू໿ͯ͠λϓϧʹ͢Δ
  22. 49.

    Finagle Twitter ͷ Finagle ͸ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͠೉͍ཁ ҼͩɻFinagle ͸ɺ෼ࢄτϨʔεɺαʔϏεσΟεΧόϦ΍αʔ

    Ϗεొ࿥ͱ͍ͬͨ෼ࢄͷ՝୊ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ • Finagle ͸ɺTumblr ΍ SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ Ϗε΁ͷ Scala ಋೖʹ౿Έ੾Δେ͖ͳཁҼʹͳ͍ͬͯΔ • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏ ͭͷཧ༝ʯ Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter
  23. 51.

    Your Server as a Function Most of our systems are

    phrased as big future transformers. — Marius Eriksen • Future ܕͱɺͦΕΛม׵͢Δ Service, Filter ͱ ͍͏ؔ਺ͷ૊Έ߹ΘͤͰαʔόΛهड़͢Δ https://monkey.org/~marius/scala2015.pdf
  24. 52.

    Future/Service/Filter trait Service[Req, Rep] extends (Req => Future[Rep]) ! !

    ! ! ! ! abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut]) ϦΫΤετΛड͚औͬͯϨεϙϯεΛ แΜͩ Future Λฦؔ͢਺ ϦΫΤετͱ Service Λड͚औͬͯ
 ϨεϙϯεΛแΜͩ Future Λฦؔ͢਺
  25. 53.

    ԣஅతؔ৺ࣄͷ߹੒ recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest

    andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer FilterΛService ʹ߹੒͢Δͩ ͚Ͱɺαʔόʹ༷ʑͳԣஅత ؔ৺ࣄΛ௥ՃͰ͖Δ http://monkey.org/~marius/funsrv.pdf
  26. 55.

    σʔλϑϩʔ DSL • σʔλϑϩʔ DSL ͱϥϯλΠϜͷ૊Έ߹Θͤ͸ɺۙ ೥ɺ༷ʑͳ෼໺Ͱద༻͞Ε͍ͯΔ • Akka Streams,

    ReactiveX, … • Պֶٕज़ܭࢉ: TensorFlow, Halide • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow, Asakusa Framework, Gearpump
  27. 57.

    Akka Streams implicit val system = ActorSystem() implicit val mat

    = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) A B C +1 —1 ×2 +
  28. 58.

    Akka Streams implicit val system = ActorSystem() implicit val mat

    = ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) ೖྗʹద༻͢Δؔ਺Λ
 ߴ֊ؔ਺ map Ͱܨ͗߹ΘͤΔ ؔ਺ ೖྗ A B C +1 —1 ×2 +
  29. 59.

    FlowGraph DSL: val c = RunnableGraph.fromGraph(FlowGraph.create(out) { implicit b =>

    out => import FlowGraph.Implicits._ ! val f1 = Flow[Int].map(_ + 1) ... ! a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out b ~> f2 ~> f3 ~> zip.in1 ClosedShape }) c.run() A B C +1 —1 ×2 +
  30. 64.

    STORAGE & RETRIEVAL LOGIC PRESENTATION ROUTING Redis Memcache Flock T-Bird

    MySQL Tweet User Timeline Social Graph DMs API Web Monorail TFE HTTP Thrift “Stuff” http://monkey.org/~marius/scala2015.pdf Twitter ͷ
 ϚΠΫϩαʔϏεߏ଄
  31. 66.

    https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru ϥϯλΠϜͱͯ͠ͷ Google Cloud Optimize Schedule Flow of pipeline User

    code & SDK Monitoring UI σʔλϑϩʔఆٛ σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥ΢υ͕ σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏