Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Rise and Rise of Dataflow in the Javaverse

The Rise and Rise of Dataflow in the Javaverse

Slides from my DevoxxUK 2016 session 2016-06-08. It's about how functional style, dataflow thinking, and parallelism are the way forward.

Ca04455902d1b207348b6d406432718f?s=128

Russel Winder

June 10, 2016
Tweet

More Decks by Russel Winder

Other Decks in Technology

Transcript

  1. Copyright © 2016 Russel Winder 1 The Rise and Rise

    of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk
  2. Copyright © 2016 Russel Winder 2 The Plan for the

    Session • Stuff. • More stuff. • Even more stuff – possibly. • Summary and conclusions – maybe. • Q & A Notice the flow going on here.
  3. Copyright © 2016 Russel Winder 3 Some Definitions • Dataflow

    — moving data from one transformation to another. • Javaverse — – the Java Platform: the JVM the Java compiler, and the standard library; – other languages such as Kotlin, Ceylon, Groovy, Scala, Clojure, etc; – other libraries from Bintray, Maven Central, etc.
  4. Copyright © 2016 Russel Winder 4 More on the Term

    Dataflow • Wikipedia: https://en.wikipedia.org/wiki/Dataflow – Dataflow can also be called stream processing or reactive programming. – Dataflow programming is about pipelines. – …data flow programming promotes high-level functional style… Streams, pipelines imply one-dimensional.
  5. Copyright © 2016 Russel Winder 5 Some Libraries of Interest

    • GPars — http://www.gpars.org/ • Akka — http://akka.io/ • Quasar — http://docs.paralleluniverse.co/quasar/ • RxJava — https://github.com/ReactiveX/RxJava • Apache Spark — http://spark.apache.org/
  6. Copyright © 2016 Russel Winder 6 What is a Program?

    n m
  7. Copyright © 2016 Russel Winder 7 Example

  8. Copyright © 2016 Russel Winder 8 Let’s be a bit

    abstract for the moment.
  9. Copyright © 2016 Russel Winder 9 Let Us Assume Java

    and… public static Integer f1(final Integer i) { return i * 2; } public static Integer f2(final Integer i) { return i * 3; } public static Integer f3(final Integer i) { return i * 4; } All this in a class.
  10. Copyright © 2016 Russel Winder 10 Compute a Value Given

    an Input public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented.
  11. Copyright © 2016 Russel Winder 11 Diagrammatically

  12. Copyright © 2016 Russel Winder 12 Compute a Value Given

    an Input public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented.
  13. Copyright © 2016 Russel Winder 13 Compute a Value Given

    an Input public static Integer statementSequence(final Integer i) { Integer x = f1(i); x = f2(x); x = f3(x); return x; } Very traditional Imperative style.
  14. Copyright © 2016 Russel Winder 14 State change vs. Data

    flow
  15. Copyright © 2016 Russel Winder 15 Compute a Value Given

    an Input A more functional and declarative approach. public static Integer functionApplication(final Integer i) { return f3(f2(f1(i))); }
  16. Copyright © 2016 Russel Winder 16 Diagrammatically

  17. Copyright © 2016 Russel Winder 17 Compilers deal in “dataflow

    analysis”, let us not forget this view of a program.
  18. Copyright © 2016 Russel Winder 18 That is all very

    Java, how about doing it with Frege?
  19. Copyright © 2016 Russel Winder 19 The Frege Transforms f1

    = (* 2) f2 = (* 3) f3 = (* 4) Pointfree definition of functions using partial evaluation. Frege has top-level functions.
  20. Copyright © 2016 Russel Winder 20 Compute a Value Given

    an Input bindingSequenceScalar i = f3 i3 where i2 = f1 i i3 = f2 i2 An Imperative style functional approach.
  21. Copyright © 2016 Russel Winder 21 Diagrammatically

  22. Copyright © 2016 Russel Winder 22 Compute a Value Given

    an Input functionApplicationScalarExplicit i = f3 (f2 (f1 i)) A more functional and declarative approach.
  23. Copyright © 2016 Russel Winder 23 Compute a Value Given

    an Input A more functional and declarative approach. functionApplicationScalar i = f3 $ f2 $ f1 $ i
  24. Copyright © 2016 Russel Winder 24 Compute a Value Given

    an Input A more functional and declarative approach. functionCompositionScalar i = f3 . f2 . f1 $ i
  25. Copyright © 2016 Russel Winder 25 Diagrammatically

  26. Copyright © 2016 Russel Winder 26 Let’s do the maths…

  27. Copyright © 2016 Russel Winder 27 Function Composition f 3

    (f 2 (f 1 (x))) = (f 3 ∘f 2 ∘f 1 )(x) It’s only a bit of maths, do not be afraid.
  28. Copyright © 2016 Russel Winder 28 Compute a Value Given

    an Input A more functional and declarative approach. functionSavedCompositionScalar i = f i where f = f3 . f2 . f1
  29. Copyright © 2016 Russel Winder 29 This example is seriously

    unrealistic.
  30. Copyright © 2016 Russel Winder 30 Make it a wee

    bit more realistic by switching to a potentially infinite dataset…
  31. Copyright © 2016 Russel Winder 31 …let’s stage this by

    doing a finite sequence first…
  32. Copyright © 2016 Russel Winder 32 …let’s go (statically compiled)

    Groovy… Do not have to have classes, top-level functions are allowed.
  33. Copyright © 2016 Russel Winder 33 The fs but Groovy

    Integer f1(final Integer i) { i * 2 } Integer f2(final Integer i) { i * 3 } Integer f3(final Integer i) { i * 4 }
  34. Copyright © 2016 Russel Winder 34 Compute a Value Given

    an Input List<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { final i1 = f1(i) final i2 = f2(i1) final i3 = f3(i2) result.add(i3) } result }
  35. Copyright © 2016 Russel Winder 35 Compute a Value Given

    an Input List<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { def x = f1(i) x = f2(x) x = f3(x) result.add(x) } result }
  36. Copyright © 2016 Russel Winder 36 Compute a Value Given

    an Input List<Integer> functionApplication(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i : l) { result.add(f3(f2(f1(i)))) } result }
  37. Copyright © 2016 Russel Winder 37 But this is all

    about state, no real dataflow. So…
  38. Copyright © 2016 Russel Winder 38 Compute a Value Given

    an Input List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList()) } This is using Streams from the Java Platform library.
  39. Copyright © 2016 Russel Winder 39 Diagrammatically

  40. Copyright © 2016 Russel Winder 40 Diagrammatically

  41. Copyright © 2016 Russel Winder 41 Compute a Value Given

    an Input List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList()) } This is using Streams from the Java Platform library. Intermediate Terminal
  42. Copyright © 2016 Russel Winder 42 Do this with explicit

    composition?
  43. Copyright © 2016 Russel Winder 43 Frege functionCompositionSequence l =

    map (f3 . f2 . f1) l
  44. Copyright © 2016 Russel Winder 44 Let’s introduce a new

    language: Kotlin Do not have to have classes, top-level functions are allowed.
  45. Copyright © 2016 Russel Winder 45 The Three Functions fun

    f1(i:Int):Int = i * 2 fun f2(i:Int):Int = i * 3 fun f3(i:Int):Int = i * 4 We can already tell that Kotlin will be lots of fun.
  46. Copyright © 2016 Russel Winder 46 Using Streams fun usingStream(l:List<Int>):List<Int>

    = l.stream() .map(::f1) .map(::f2) .map(::f3) .collect(Collectors.toList<Int>())
  47. Copyright © 2016 Russel Winder 47 Kotlin version fun usingMap(l:List<Int>):List<Int>

    = l.map(::f1).map(::f2).map(::f3)
  48. Copyright © 2016 Russel Winder 48 Using composition fun usingComposedMap(l:List<Int>):List<Int

    > = l.map(::f1 compose ::f2 compose ::f3)
  49. Copyright © 2016 Russel Winder 49 Kotlin Compose infix fun<V,

    T, R> Function1<T, R>.compose(before: (V) -> T): (V) -> R { return { v: V -> this(before(v)) } } t ∘b(i) = t(b(i))
  50. Copyright © 2016 Russel Winder 50 But this is still

    finite, what about potentially infinite?
  51. Copyright © 2016 Russel Winder 51 Cannot do collect. Map

    is another name for collect in most circumstances.
  52. Copyright © 2016 Russel Winder 52 Infinite Data • With

    an infinite data sequence you can: – Do some form of reduction Or windowing. – Perform a side-effect, e.g. output.
  53. Copyright © 2016 Russel Winder 53 New (more realistic) problem…

  54. Copyright © 2016 Russel Winder 54 Cumulative mean and Standard

    deviation.
  55. Copyright © 2016 Russel Winder 55 Equation warning: please do

    not be afraid.
  56. Copyright © 2016 Russel Winder 56 ¯ x= 1 n

    ∑ i=0 n x i s= √ 1 n−1 ∑ i=0 n (x i −¯ x)2
  57. Copyright © 2016 Russel Winder 57 ¯ x= 1 n

    ∑ i=0 n x i s= √ 1 n−1 ((∑ i=0 n x i 2)−n¯ x2)
  58. Copyright © 2016 Russel Winder 58

  59. Copyright © 2016 Russel Winder 59 I only do equations

    after being fed…
  60. Copyright © 2016 Russel Winder 60 Code?

  61. Copyright © 2016 Russel Winder 61 A little architecture first.

  62. Copyright © 2016 Russel Winder 62

  63. Copyright © 2016 Russel Winder 63 Code.

  64. Copyright © 2016 Russel Winder 64 What’s the Message? •

    Small, single threaded, communicating processes are easy to program. (Communicating Sequential Processes, CSP) • Threadpools and processpools make parallelism easy to realize, without manual locks. • Most calculations and dataset are now very big, hence “Big Data”.
  65. Copyright © 2016 Russel Winder 65 Parallelism is mandatory For

    “Big Data”.
  66. Copyright © 2016 Russel Winder 66 Dataflow not state is

    required for parallelism.
  67. Copyright © 2016 Russel Winder 67 The Rise and Rise

    of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk
  68. Copyright © 2016 Russel Winder 68 But before I go…

  69. Copyright © 2016 Russel Winder 69

  70. Copyright © 2016 Russel Winder 70 Q & A

  71. Copyright © 2016 Russel Winder 71 The Rise and Rise

    of Dataflow in the JavaVerse Russel Winder @russel_winder russel@winder.org.uk https://www.russel.org.uk