Learning functional programming the hard way with Spark

Transcript

1. MAD · NOV 22-23 · 2019 David Muñoz - @voiser Learning FP to the donkey the hard way with
2. MAD · NOV 22-23 · 2019
3. MAD · NOV 22-23 · 2019 You probably have heard of the FP Way: Pure functions Immutability HOFs map, flatmap, foldLeft…
4. MAD · NOV 22-23 · 2019 MapReduce’s hello world: word count
5. MAD · NOV 22-23 · 2019 val counts = mutable.Map[String, Int]() foreach(word in words) { val currentCount = counts.getOrElse(word, 0) counts.put(word, currentCount + 1) } pseudocode
6. MAD · NOV 22-23 · 2019 What if we have trillions of words? Let’s use Spark!
7. MAD · NOV 22-23 · 2019 RDD[T]
8. MAD · NOV 22-23 · 2019 Driver Worker Worker Worker Worker RDD i really like scala i want to learn to code in scala as soon as I can
9. MAD · NOV 22-23 · 2019 RDD[T]. map(f: (T) !=> U): RDD[U] RDD[T]. !flatMap(f: (T) !=> TraversableOnce[U]): RDD[U] RDD[T]. reduce(f: (T, T) !=> T): T RDD[T]. fold(z: T)(f: (T, T) !=> T): T
10. MAD · NOV 22-23 · 2019 Lazy evaluation Retrials
11. MAD · NOV 22-23 · 2019 val counts = mutable.Map[String, Int]() rddWords.foreach { word => val currentCount = counts.getOrElse(word, 0) counts.put(word, currentCount + 1) } pseudocode
12. MAD · NOV 22-23 · 2019 Driver Worker Worker Worker Worker RDD i really like scala i want to learn to code in scala as soon as I can Map[String, Int]() Map[String, Int]() Map[String, Int]() Map[String, Int]() Map[String, Int]()
13. MAD · NOV 22-23 · 2019 Count words avoiding side effects
14. MAD · NOV 22-23 · 2019 Let’s represent a word-count like an object: word count hello 87
15. MAD · NOV 22-23 · 2019 Example: “hello world” word count hello 1 word count world 1
16. MAD · NOV 22-23 · 2019 Example: “hello hello world” word count world 1 word count hello 1 word count hello 1
17. MAD · NOV 22-23 · 2019 word count hello A+B word count hello B word count hello A Let’s define a combination operation: * =
18. MAD · NOV 22-23 · 2019 word count hello A Let’s define a combination operation: word count world B word count hello A world B * =
19. MAD · NOV 22-23 · 2019 Example: “hello hello world” word count world 1 word count hello 1 word count hello 1
20. MAD · NOV 22-23 · 2019 word count hello 1 word count world 1 word count hello 1 *
21. MAD · NOV 22-23 · 2019 word count world 1 word count hello 2
22. MAD · NOV 22-23 · 2019 word count world 1 word count hello 2 *
23. MAD · NOV 22-23 · 2019 word count hello 2 world 1
24. MAD · NOV 22-23 · 2019 Semigroup Design Pattern:
25. MAD · NOV 22-23 · 2019 Semigroups give you Parallelism
26. MAD · NOV 22-23 · 2019 Semigroup If a type A can form a Semigroup it has an associative binary operation. trait Semigroup[A] { def combine(x: A, y: A): A } Associativity means the following equality must hold for any choice of x, y, and z. combine(x, combine(y, z)) = combine(combine(x, y), z) https://typelevel.org/cats/typeclasses/semigroup.html
27. MAD · NOV 22-23 · 2019 val result = rdd .map(x => (x, 1)) .reduceByKey((a, b) => a + b)
28. MAD · NOV 22-23 · 2019 val result = rdd .map(x => (x, 1)) .reduceByKey((a, b) => a + b) word count hello 1 * = word count hello A world C word count hello B scala D word count hello A+B world C scala D
29. MAD · NOV 22-23 · 2019 Why associative?
30. MAD · NOV 22-23 · 2019 Worker Worker word count world 1 word count hello 1 word count hello 1
31. MAD · NOV 22-23 · 2019 Worker Worker word count world 1 word count hello 1 word count hello 1 word count hello 2 word count hello 2 world 1
32. MAD · NOV 22-23 · 2019 Worker Worker word count world 1 word count hello 1 word count hello 1
33. MAD · NOV 22-23 · 2019 Worker Worker word count world 1 word count hello 1 word count hello 1 word count hello 2 world 1 word count hello 1 world 1
34. MAD · NOV 22-23 · 2019 Spark asks for commutative + associative operations
35. MAD · NOV 22-23 · 2019 WordCount as a semigroup class WordCount … { !// represents a set of (word, count) } def combine(a: WordCount, b: WordCount): WordCount = !// combines to WordCount into one def combineMany(xs: List[WordCount]): WordCount = ???
36. MAD · NOV 22-23 · 2019 word count
37. MAD · NOV 22-23 · 2019 Monoid Design Pattern:
38. MAD · NOV 22-23 · 2019 = Monoid Semigroup + empty
39. MAD · NOV 22-23 · 2019 Living without side effects is a dream.
40. MAD · NOV 22-23 · 2019 Living without side effects is a dream. Seriously.
41. MAD · NOV 22-23 · 2019 Living without side effects is a dream. Seriously. It’s not real.
42. MAD · NOV 22-23 · 2019 Living without side effects is a dream. Seriously. It’s not real. Face it, vegans functional programmers!!
43. MAD · NOV 22-23 · 2019 RDD i want to learn scala f f f f f made up numbers, doesn’t matter RDD 5 8 ???? 4 19
44. MAD · NOV 22-23 · 2019 val result = words .map { w !=> w.length } .map { l !=> 1. / (l - 1) } .foreach(println)
45. MAD · NOV 22-23 · 2019 val result = words .map { w !=> w.length } .map { l !=> if (l !!= 1) 1 / (l - 1) else ??? } .foreach(println) val result = words .map { w !=> w.length } .map { l !=> try { 1 / (l - 1) } catch { case e:Exception !=> ??? } } .foreach(println)
46. MAD · NOV 22-23 · 2019 Spark is not very functional about this. Let’s help him. Describing the problem is the key to control it.
47. MAD · NOV 22-23 · 2019 Try[T]
48. MAD · NOV 22-23 · 2019 trait Try[T] { def map(f: T !=> U): Try[U] } class Success[T](t: T) extends Try[T] {…} class Failure[T] extends Try[T] {…} Sum type
49. MAD · NOV 22-23 · 2019 “hello” “hello” 5 map(_.length) Try(value) 6 map(_ + 1) 6.0 map(_.to!Float) ERROR map(_ / 0) map(_.to!Float) map(_ + 1)
50. MAD · NOV 22-23 · 2019 Functor Design Pattern:
51. MAD · NOV 22-23 · 2019 Functors give you computational context
52. MAD · NOV 22-23 · 2019 Functor trait Functor[F[_]] { def map[A, B](fa: F[A])(f: A !=> B): F[B] } A Functor instance must obey two laws: •Composition: ◦fa.map(f).map(g) = fa.map(f.andThen(g)) •Identity: ◦fa.map(x !=> x) = fa https://typelevel.org/cats/typeclasses/functor.html
53. MAD · NOV 22-23 · 2019 The problem with Functors def combine(ta: Try[Int], tb: Try[Int] = ta.map(a !=> tb.map(b !=> a + b))
54. MAD · NOV 22-23 · 2019 The problem with Functors def combine(ta: Try[Int], tb: Try[Int]: Try[Try[Int]] = ta.map(a !=> tb.map(b !=> a + b))
55. MAD · NOV 22-23 · 2019 Monad Design Pattern:
56. MAD · NOV 22-23 · 2019 Monads give you Composability
57. MAD · NOV 22-23 · 2019 Monad trait Monad[M[_]] { def !flatMap[A, B](ma: M[A])(f: A !=> M[B]): M[B] def unit[A](x: A): M[A] } A Monad instance must obey three laws: •Associativity: ◦m.flatMap(f).flatMap(g) == m.flatMap(x => f(x).flatMap(g)) •Left Identity: ◦unit(x).!flatMap(f) !== f(x) •Right Identity: ◦m.!flatMap(unit) !== m
58. MAD · NOV 22-23 · 2019 Monads compose def combine(ta: Try[Int], tb: Try[Int]: Try[Int] = ta.!flatMap(a !=> tb.map(b !=> a + b))
59. MAD · NOV 22-23 · 2019 Monads compose def combine(ta: Try[Int], tb: Try[Int]: Try[Int] = for { a !<- ta b !<- tb } yield a + b
60. MAD · NOV 22-23 · 2019 Try[T] Option[T] Future[T] List[T] Writer[T]
61. MAD · NOV 22-23 · 2019
62. MAD · NOV 22-23 · 2019 RDD[T]
63. MAD · NOV 22-23 · 2019 RDD[T]. map(f: (T) !=> U): RDD[U] RDD[T]. !flatMap(f: (T) !=> TraversableOnce[U]): RDD[U] RDD[T]. reduce(f: (T, T) !=> T): T RDD[T]. fold(z: T)(f: (T, T) !=> T): T Functor Monad??? Semigroup Monoid
64. MAD · NOV 22-23 · 2019 QUESTION
65. MAD · NOV 22-23 · 2019 def foldLeft [B](z: B)(op: (B, A) ⇒ B): B def foldRight[B](z: B)(op: (A, B) ⇒ B): B
66. MAD · NOV 22-23 · 2019 NOT QUITE A MONOID
67. MAD · NOV 22-23 · 2019 wiki.haskell.org/Typeclassopedia typelevel.org/cats/typeclasses.html
68. MAD · NOV 22-23 · 2019 Sorry for the brasa New York 3537 36th St +1 (646) 403-3104 Madrid Alameda, 22 +34 910 754 276 www.meaningcloud.com