The Rise and Rise of Dataflow in the Javaverse

Slides from my DevoxxUK 2016 session 2016-06-08. It's about how functional style, dataflow thinking, and parallelism are the way forward.

Transcript

1. Copyright © 2016 Russel Winder 1 The Rise and Rise of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk
2. Copyright © 2016 Russel Winder 2 The Plan for the Session • Stuff. • More stuff. • Even more stuff – possibly. • Summary and conclusions – maybe. • Q & A Notice the flow going on here.
3. Copyright © 2016 Russel Winder 3 Some Definitions • Dataflow — moving data from one transformation to another. • Javaverse — – the Java Platform: the JVM the Java compiler, and the standard library; – other languages such as Kotlin, Ceylon, Groovy, Scala, Clojure, etc; – other libraries from Bintray, Maven Central, etc.
4. Copyright © 2016 Russel Winder 4 More on the Term Dataflow • Wikipedia: https://en.wikipedia.org/wiki/Dataflow – Dataflow can also be called stream processing or reactive programming. – Dataflow programming is about pipelines. – …data flow programming promotes high-level functional style… Streams, pipelines imply one-dimensional.
5. Copyright © 2016 Russel Winder 5 Some Libraries of Interest • GPars — http://www.gpars.org/ • Akka — http://akka.io/ • Quasar — http://docs.paralleluniverse.co/quasar/ • RxJava — https://github.com/ReactiveX/RxJava • Apache Spark — http://spark.apache.org/
6. Copyright © 2016 Russel Winder 6 What is a Program? n m
7. Copyright © 2016 Russel Winder 7 Example
8. Copyright © 2016 Russel Winder 8 Let’s be a bit abstract for the moment.
9. Copyright © 2016 Russel Winder 9 Let Us Assume Java and… public static Integer f1(final Integer i) { return i * 2; } public static Integer f2(final Integer i) { return i * 3; } public static Integer f3(final Integer i) { return i * 4; } All this in a class.
10. Copyright © 2016 Russel Winder 10 Compute a Value Given an Input public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented.
11. Copyright © 2016 Russel Winder 11 Diagrammatically
12. Copyright © 2016 Russel Winder 12 Compute a Value Given an Input public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented.
13. Copyright © 2016 Russel Winder 13 Compute a Value Given an Input public static Integer statementSequence(final Integer i) { Integer x = f1(i); x = f2(x); x = f3(x); return x; } Very traditional Imperative style.
14. Copyright © 2016 Russel Winder 14 State change vs. Data flow
15. Copyright © 2016 Russel Winder 15 Compute a Value Given an Input A more functional and declarative approach. public static Integer functionApplication(final Integer i) { return f3(f2(f1(i))); }
16. Copyright © 2016 Russel Winder 16 Diagrammatically
17. Copyright © 2016 Russel Winder 17 Compilers deal in “dataflow analysis”, let us not forget this view of a program.
18. Copyright © 2016 Russel Winder 18 That is all very Java, how about doing it with Frege?
19. Copyright © 2016 Russel Winder 19 The Frege Transforms f1 = (* 2) f2 = (* 3) f3 = (* 4) Pointfree definition of functions using partial evaluation. Frege has top-level functions.
20. Copyright © 2016 Russel Winder 20 Compute a Value Given an Input bindingSequenceScalar i = f3 i3 where i2 = f1 i i3 = f2 i2 An Imperative style functional approach.
21. Copyright © 2016 Russel Winder 21 Diagrammatically
22. Copyright © 2016 Russel Winder 22 Compute a Value Given an Input functionApplicationScalarExplicit i = f3 (f2 (f1 i)) A more functional and declarative approach.
23. Copyright © 2016 Russel Winder 23 Compute a Value Given an Input A more functional and declarative approach. functionApplicationScalar i = f3 $ f2 $ f1 $ i
24. Copyright © 2016 Russel Winder 24 Compute a Value Given an Input A more functional and declarative approach. functionCompositionScalar i = f3 . f2 . f1 $ i
25. Copyright © 2016 Russel Winder 25 Diagrammatically
26. Copyright © 2016 Russel Winder 26 Let’s do the maths…
27. Copyright © 2016 Russel Winder 27 Function Composition f 3 (f 2 (f 1 (x))) = (f 3 ∘f 2 ∘f 1 )(x) It’s only a bit of maths, do not be afraid.
28. Copyright © 2016 Russel Winder 28 Compute a Value Given an Input A more functional and declarative approach. functionSavedCompositionScalar i = f i where f = f3 . f2 . f1
29. Copyright © 2016 Russel Winder 29 This example is seriously unrealistic.
30. Copyright © 2016 Russel Winder 30 Make it a wee bit more realistic by switching to a potentially infinite dataset…
31. Copyright © 2016 Russel Winder 31 …let’s stage this by doing a finite sequence first…
32. Copyright © 2016 Russel Winder 32 …let’s go (statically compiled) Groovy… Do not have to have classes, top-level functions are allowed.
33. Copyright © 2016 Russel Winder 33 The fs but Groovy Integer f1(final Integer i) { i * 2 } Integer f2(final Integer i) { i * 3 } Integer f3(final Integer i) { i * 4 }
34. Copyright © 2016 Russel Winder 34 Compute a Value Given an Input List<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { final i1 = f1(i) final i2 = f2(i1) final i3 = f3(i2) result.add(i3) } result }
35. Copyright © 2016 Russel Winder 35 Compute a Value Given an Input List<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { def x = f1(i) x = f2(x) x = f3(x) result.add(x) } result }
36. Copyright © 2016 Russel Winder 36 Compute a Value Given an Input List<Integer> functionApplication(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i : l) { result.add(f3(f2(f1(i)))) } result }
37. Copyright © 2016 Russel Winder 37 But this is all about state, no real dataflow. So…
38. Copyright © 2016 Russel Winder 38 Compute a Value Given an Input List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList()) } This is using Streams from the Java Platform library.
39. Copyright © 2016 Russel Winder 39 Diagrammatically
40. Copyright © 2016 Russel Winder 40 Diagrammatically
41. Copyright © 2016 Russel Winder 41 Compute a Value Given an Input List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList()) } This is using Streams from the Java Platform library. Intermediate Terminal
42. Copyright © 2016 Russel Winder 42 Do this with explicit composition?
43. Copyright © 2016 Russel Winder 43 Frege functionCompositionSequence l = map (f3 . f2 . f1) l
44. Copyright © 2016 Russel Winder 44 Let’s introduce a new language: Kotlin Do not have to have classes, top-level functions are allowed.
45. Copyright © 2016 Russel Winder 45 The Three Functions fun f1(i:Int):Int = i * 2 fun f2(i:Int):Int = i * 3 fun f3(i:Int):Int = i * 4 We can already tell that Kotlin will be lots of fun.
46. Copyright © 2016 Russel Winder 46 Using Streams fun usingStream(l:List<Int>):List<Int> = l.stream() .map(::f1) .map(::f2) .map(::f3) .collect(Collectors.toList<Int>())
47. Copyright © 2016 Russel Winder 47 Kotlin version fun usingMap(l:List<Int>):List<Int> = l.map(::f1).map(::f2).map(::f3)
48. Copyright © 2016 Russel Winder 48 Using composition fun usingComposedMap(l:List<Int>):List<Int > = l.map(::f1 compose ::f2 compose ::f3)
49. Copyright © 2016 Russel Winder 49 Kotlin Compose infix fun<V, T, R> Function1<T, R>.compose(before: (V) -> T): (V) -> R { return { v: V -> this(before(v)) } } t ∘b(i) = t(b(i))
50. Copyright © 2016 Russel Winder 50 But this is still finite, what about potentially infinite?
51. Copyright © 2016 Russel Winder 51 Cannot do collect. Map is another name for collect in most circumstances.
52. Copyright © 2016 Russel Winder 52 Infinite Data • With an infinite data sequence you can: – Do some form of reduction Or windowing. – Perform a side-effect, e.g. output.
53. Copyright © 2016 Russel Winder 53 New (more realistic) problem…
54. Copyright © 2016 Russel Winder 54 Cumulative mean and Standard deviation.
55. Copyright © 2016 Russel Winder 55 Equation warning: please do not be afraid.
56. Copyright © 2016 Russel Winder 56 ¯ x= 1 n ∑ i=0 n x i s= √ 1 n−1 ∑ i=0 n (x i −¯ x)2
57. Copyright © 2016 Russel Winder 57 ¯ x= 1 n ∑ i=0 n x i s= √ 1 n−1 ((∑ i=0 n x i 2)−n¯ x2)
58. Copyright © 2016 Russel Winder 58
59. Copyright © 2016 Russel Winder 59 I only do equations after being fed…
60. Copyright © 2016 Russel Winder 60 Code?
61. Copyright © 2016 Russel Winder 61 A little architecture first.
62. Copyright © 2016 Russel Winder 62
63. Copyright © 2016 Russel Winder 63 Code.
64. Copyright © 2016 Russel Winder 64 What’s the Message? • Small, single threaded, communicating processes are easy to program. (Communicating Sequential Processes, CSP) • Threadpools and processpools make parallelism easy to realize, without manual locks. • Most calculations and dataset are now very big, hence “Big Data”.
65. Copyright © 2016 Russel Winder 65 Parallelism is mandatory For “Big Data”.
66. Copyright © 2016 Russel Winder 66 Dataflow not state is required for parallelism.
67. Copyright © 2016 Russel Winder 67 The Rise and Rise of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk
68. Copyright © 2016 Russel Winder 68 But before I go…
69. Copyright © 2016 Russel Winder 69
70. Copyright © 2016 Russel Winder 70 Q & A
71. Copyright © 2016 Russel Winder 71 The Rise and Rise of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk