Streams a la carte

This is the slide deck of our talk at ECOOP15 about extensible pipelines with Object Algebras.

Paper: http://drops.dagstuhl.de/opus/frontdoor.php?source_opus=5239
Code: http://biboudis.github.io/streamalg/
Video: https://www.youtube.com/watch?v=rHzgkMW2AZ0
Conference: http://2015.ecoop.org/

Transcript

1. Streams à la carte Extensible Pipelines with Object Algebras 29th European Conference on Object-Oriented Programming (ECOOP 2015) Aggelos Biboudis¹, Nick Palladinos², George Fourtounis¹, Yannis Smaragdakis¹ @biboudis, @NickPalladinos, @gf0ur University of Athens¹ Nessos Information Technologies² 10 July 2015
2. Stream libraries: what are they? • Functional-inspired pipelines • Lazy 10/7/2015 University of Athens 2 int sum = IntStream.of(v) .filter(x -> x % 2 == 0) .map(x -> x * x) .sum();
3. Stream libraries: everywhere! • Great implementation variety, Java surprising! • C# (LINQ), F#(Seq), Scala(Views) implement Pull-streams • Java 8 implements Push-streams 10/7/2015 University of Athens 3 int sum = IntStream.of(v) .filter(x -> x % 2 == 0) .map(x -> x * x) .sum();
4. What's missing from current libs? • Current libraries: fixed behavior and operators • We want pluggable semantics (e.g., push vs. pull) • And the ability to retrofit new operators 10/7/2015 University of Athens 4 int sum = IntStream.of(v) .filter(x -> x % 2 == 0) .map(x -> x * x) .sum();
5. Why pluggable semantics? • Operators are naturally push or pull ⇒ variable performance • Extra functionality mixed in, e.g.: oLog with push oFuse with pull oBlocking or not with push or pull University of Athens 5 10/7/2015
6. Background: Push vs. Pull Streams (both are lazy!!!) Push (consumer side: no loop) Pull (consumer side: loop) 10/7/2015 University of Athens 6 Pull<Integer> s = source(v).map(i->i+1); while(s.hasNext()) { v = s.next(); /*use v*/ } Push<Integer> s = source(v).map(i->i+1); s(v -> /*use v*/);
7. Background: Push vs. Pull Streams (both are lazy!!!) Push (producer side: loop) Pull (producer side: no loop) 10/7/2015 University of Athens 7 Pull<T> source(T[] arr) { return new Pull<T>() { boolean hasNext() { … } T next() { … } }; } Push<T> source(T[] arr) { return k -> { for (int i = 0; i<arr.length; i++) k(array[i]); }; }
8. Difference in semantics not just academic (actual Java 8 snippet) Iterator<Long> iterator = Stream .of(v) .flatMap(x -> Stream.iterate(0L, i -> i + 2) .map(y -> x * y)) .iterator(); iterator.hasNext(); // Out-of-memory :-( University of Athens 8 10/7/2015
9. Expression problem: adding behaviors and operators http://homepages.inf.ed.ac.uk/wadler/papers/expression/expression.txt University of Athens 9 10/7/2015
10. Object Algebras: A design pattern to the rescue https://www.cs.utexas.edu/~wcook/Drafts/2012/ecoop2012.pdf University of Athens 10 10/7/2015
11. A factory interface ExpFactory { Exp lit(int x); Exp add(Exp e1, Exp e2); } University of Athens 11 10/7/2015
12. A generic factory interface ExpFactory<Exp> { Exp lit(int x); Exp add(Exp e1, Exp e2); } University of Athens 12 10/7/2015
13. 1 + 2 + 3 <Exp> Exp mkAnExp(ExpFactory<Exp> f) { return f.add(f.lit(1), f.add(f.lit(2), f.lit(3))); } University of Athens 13 10/7/2015
14. In the Object Algebras realm • interfaces are named algebras • implementations are named factories • new cases (by extending the algebra) • new functions (by implementing the algebra) 10/7/2015 University of Athens 14
15. We propose • A library design • Inspired by Object Algebras • Provide extensible streams with: • Pluggable operators • Pluggable behaviors • Mixedin behaviors • Affect performance (in a good way) 10/7/2015 University of Athens 15
16. What is the object algebra of streams? (for intermediate operators) interface StreamAlg<C<_>> { <T> C<T> source(T[] array); <T,R> C<R> map(Function<T,R> f, C<T> s); <T> C<T> filter(Predicate<T> f, C<T> s); } University of Athens 16 10/7/2015
17. What is the object algebra of streams? (for terminal operators) interface ExecStreamAlg<E<_>, C<_>> extends StreamAlg<C> { <T> E<Long> count (C<T> s); <T> E<T> reduce(T identity, BinaryOperator<T> acc, C<T> s); } University of Athens 17 10/7/2015
18. How do we extend streams? Add new operators (by extending the algebra) Add new behavior (by implementing the algebra) interface TakeStreamAlg<C<_>> extends StreamAlg<C> { <T> C<T> take(int n, C<T> s); } class PushFactory implements StreamAlg<Push> University of Athens 18 10/7/2015
19. Let’s use a stream PushFactory alg = new PushFactory(); int sum = alg.sum( alg.map(x -> x * x, alg.filter(x -> x % 2 == 0, alg.source(v)))); University of Athens 19 10/7/2015
20. Semantics as a run-time parameter • Declaring streams: reducing a Cartesian product 10/7/2015 University of Athens 20 <E,C> E<Long> cart(ExecStreamAlg<E,C> alg) { return alg.reduce(0L, Long::sum, alg.flatMap(x -> alg.map(y -> x*y, alg.source(v2)), alg.source(v1))); }
21. Stream semantics à la carte • Using streams with various factories 10/7/2015 University of Athens 21 cart(new ExecPushFactory()); cart(new ExecPullFactory()); cart(new ExecFusedPullFactory()); cart(new LogFactory<>(new ExecPushFactory())); cart(new LogFactory<>(new ExecPullFactory())); cart(new ExecFutureFactory<>( new ExecPushFactory())).get(); cart(new ExecFutureFactory<>( new ExecPullFactory())).get(); …
22. Object Algebras are for construction University of Athens 22 10/7/2015
23. An algebra that fuses maps & filters University of Athens 23 10/7/2015
24. Sometimes we need fully fledged pull University of Athens 24 10/7/2015
25. Our pathogenic case from earlier with large nested stream University of Athens 25 10/7/2015
26. How did we encode higher-kinded types? • Clever technique, already used in Java and C# libraries • Gronau: HighJ • Magi (http://stackoverflow.com/a/1466226) • Also recently presented in an OCaml publication 10/7/2015 University of Athens 26 https://ocamllabs.github.io/higher/lightweight-higher-kinded-polymorphism.pdf
27. How did we encode higher-kinded types? interface StreamAlg<C<_>> { <T> C<T> source(T[] array); <T, R> C<R> map(Function<T, R> f, C<T> s); <T> C<T> filter(Predicate<T> f, C<T> s); } University of Athens 27 10/7/2015
28. How did we encode higher-kinded types? interface StreamAlg<C> { <T> App<C,T> source(T[] array); <T, R> App<C,R> map(Function<T,R> f, App<C,T> s); <T> App<C,T> filter(Predicate<T> f, App<C,T> s); } University of Athens 28 10/7/2015
29. To sum up • A library implementation • Inspired by Object Algebras • Extensible operators • Pluggable behaviors • Mixedin behaviors • Performance is still there 10/7/2015 University of Athens 29
30. Artifact 10/7/2015 University of Athens 30 http://biboudis.github.io/streamalg/
31. Thank you, Questions? • This deck: speakerdeck.com/biboudis/streams-a-la-carte University of Athens 31 10/7/2015