Stream Processing and Functional Programming

Everyone's doing Stream Processing these days. During last 5 years there's been more CEP engines released than Web Frameworks. This is a great pattern, but without Functional Programming concepts you're just touching a surface. How to make your streams immutable, composable and reusable, and why many out-of-the-box distributions are limiting your basically endless opportunities.
Stateful streams is a current state-of-art. Although much too often state makes the Stream less composable and performant. People are using stores such as Redis to store the intermediate state, and never think about better ways to make state available for every worker.

With currently existing single-box RX Frameworks and Message Queues it's possible to build Streams in ways that entirely change the perspective and thinking about the stream, taking complexity out and letting developers think about each entity in a stream as an isolated event, allowing better composition and testability.

Transcript

1. @ifesdjeen Stream Processing and Functional Programming
2. http://instana.com/
3. Introduction
4. Being Ukrainian
5. Being Ukrainian
6. Compuater Scientist
7. Evaluating Ideas
8. Talking to the People
9. Data Science
10. Going sub-second
11. An ultimate problem-solving algorithm
12. Identify the problem Think Real Hard Write down the solution
13. Functional Programming 101
14. Partial Application Currying Closures
15. Currying ((a → b) → c) → (a → (b → c)) = λf. λx. λy. f (x, y)
16. Partial Application (a → b → c) = a → (b → c)
17. Closures BiFunction<Integer, Integer, Integer> fn = (a, b) -> a + b; Function<Integer, Function<Integer, Integer>> curried = a -> { return (b) -> a + b; };
18. Stream Processing
19. Producer Consumer Pipe
20. Producer Yields the values Allows Consumers to subscribe
21. Consumer Awaits for the values Subscribes to Producer
22. Pipe Connects consumers and producers
23. Pipe data Step a i = Yield a i | Skip i data Pipe a = Pipe (i -> Step a i) i - - Monads in Stream type omitted for brevity
24. Stream Combination of all “Container” Burrito
25. Stream is a Functor fmap id = id fmap (g . h) = (fmap g) . (fmap h) class Functor f where fmap :: (a -> b) -> f a -> f b Functor Laws morphism, mapping between categories
26. Stream is an Applicative pure id <*> v = v pure f <*> pure x = pure (f x) u <*> pure y = pure ($ y) <*> u u <*> (v <*> w) = pure (.) <*> u <*> v <*> w class Functor f => Applicative f where pure :: a -> f a (<*>) :: f (a -> b) -> f a -> f b Applicative Laws
27. Basic Stream Operations map (Stream s) :: (a -> b) -> s a -> s b filter (Stream s) :: (a -> Boolean) -> s a -> s a slide (Stream s) :: ([a] -> [a]) -> s a -> s [a] partition (Stream s) :: ([a] -> Boolean) -> s a -> s [a] consume (Stream a) :: (a -> IO ()) -> s a -> s a
28. Building Blocks
29. <V> Firehose on(final K key, Consumer<V> consumer) <V> Firehose notify(final K key, final V ev) Implementation Idea
30. Firehose<Key> firehose = new Firehose<>(); firehose.on(Key.wrap("myKey"), (v) -> { /* ... */} ); firehose.notify(Key.wrap("key1"), 1); Example
31. Logical Consequence Named Pipe
32. SRC source, DST destination, Function<V, V1> mapper Named Pipe
33. key1 key2 key3 Intuition
34. (SRC source, DST destination, Function<V, V1> mapper) -> { firehose.on(source, (SRC key, V value) -> { firehose.notify(destination, mapper.apply(value)); } }); return new NamedPipe<>(firehose); } Implementation Idea
35. NamedPipe<Integer> intPipe = new NamedPipe<>(); intPipe.map("key1", "key2", (i) -> i + 1); intPipe.map("key2", "key3", (i) -> i * 2); intPipe.consume("key2", System.out::println); Example
36. Reducing Boilerplate Anonymous Pipe
37. Function<V, V1> mapper SRC and DST are implicit Anonymous Pipe
38. key key’ key’’ Intuition
39. private final Key upstream; private final NamedPipe pipe; <V1> AnonymousPipe<V1> map(Function<V, V1> mapper) { Key downstream = upstream.derive(); pipe.map(upstream, downstream, mapper); return new AnonymousPipe<>(downstream, pipe); } Implementation Idea
40. Example NamedPipe<Integer> pipe = new NamedPipe<>(); pipe.anonymous("key") .map((i) -> i + 1) .map(i -> i * 2) .consume(System.out::println); pipe.notify("key", 1);
41. Adding Flexibility Matched Pipes
42. <V> Firehose on(K key, Consumer<V> consumer) <V> Firehose on(Predicate<K> key, Consumer<V> consumer) vs
43. Intuition ^(key)(\d+)$ ^(key)(\d+)$’ ^(key)(\d+)$’’ key1 key1’ key1’’ key2 key2’ key2’’
44. Implementation Idea <V1> MatchedPipe<V1> map(Function<V, V1> mapper) { this.suppliers.add((Key src, Key dst, NamedPipe<V1> pipe) -> { return (key, value) -> { pipe.notify(dst, mapper.apply(value)); }; }); return new MatchedPipe<>(suppliers); }
45. Example NamedPipe<Integer> pipe = new NamedPipe<>(); pipe.matched(key -> key.startsWith("key")) .map(i -> i + 1) .map(i -> i * 2) .consume(System.out::println); pipe.notify("key1", 1); pipe.notify("key2", 2); pipe.notify("key3", 3);
46. Composition (+) <$> stream1 <*> stream2 (+) <$> stream1 <*> stream2 <*> stream3
47. (+) <$> (Stream 1) <*> (Stream 2) (Stream +1) <*> (Stream 2)
48. Composition Pipe<Integer> -> Pipe<Consumer<Integer>>
49. Composition Simple with lists, not so with streams Order, frequency, amounts Needs explicit tagging
50. Common pitfalls and Algebras to overcome them
51. Reordering a • b = b • a ∀ a,b ∈ S (a • b) • c = a • (b • c) ∀ a,b, c ∈ S Commutativity Associativity
52. Exactly-once delivery (lack of thereof) a • a = a ∀ a,b ∈ S Idempotence
53. Use Cases Decoupling independent actions Data pipelining, aggregation Building blocks for any async actions Database Drivers Request/Response servers
54. Decomposition Separate downstreams Make downstream location transparent
55. Questions?
56. @ifesdjeen