Tweet
Tweet

Slide 1

Slide 1 text

Pipeline Oriented Programming @ScottWlaschin fsharpforfunandprofit.com/pipeline Code examples will be in C# and F#

Slide 2

Slide 2 text

What is “Pipeline-oriented Programming"? Part I

Slide 3

Slide 3 text

Object-oriented Arrows go in all directions

Slide 4

Slide 4 text

Pipeline-oriented All arrows go left to right

Slide 5

Slide 5 text

A pipeline-oriented web backend One directional flow, even with branching

Slide 6

Slide 6 text

What are the benefits of a pipeline-oriented approach?

Slide 7

Slide 7 text

Benefits • Pipelines encourage composability • Pipelines follow good design principles • Pipelines are easier to maintain • Pipelines make testing easier • Pipelines fit well with modern architectures – E.g. Onion/Clean/Hexagonal, etc

Slide 8

Slide 8 text

Benefit 1: Pipelines encourage composability Composable == “Like Lego”

Slide 9

Slide 9 text

Connect two pieces together and get another "piece" that can still be connected You don't need to create a special adapter to make connections.

Slide 10

Slide 10 text

Component Component Component Pipelines composition Pipelines encourage you to design so that: • Any pieces can be connected • You don’t need special adapters If you do this, you get nice composable components

Slide 11

Slide 11 text

Bigger Component Sub- Component Sub- Component Sub- Component Sub-components composed into a new bigger unit

Slide 12

Slide 12 text

Benefit 2: Pipelines follow good design principles Many design patterns work naturally with a pipeline-oriented approach

Slide 13

Slide 13 text

Single responsibility principle Component Can't get more single responsibility than this!

Slide 14

Slide 14 text

Open/Closed principle You could be able to add new functionality (“open for extension”) without changing existing code (“closed for modification”) New Behavior Component Component Component Extension Not changed

Slide 15

Slide 15 text

Strategy Pattern Parameterized Component Component Component Component

Slide 16

Slide 16 text

Decorator Pattern Decorator Decorator Original Component

Slide 17

Slide 17 text

Decorated Component Decorator Pattern Decorator Decorator "decorated" component works just like the original one

Slide 18

Slide 18 text

Benefit 3: Pipelines are easier to maintain

Slide 19

Slide 19 text

This is easier to maintain This is harder to maintain

Slide 20

Slide 20 text

Object-oriented: Real-world dependency graph Multiple circular dependencies  Arrows go in all directions

Slide 21

Slide 21 text

Pipeline-oriented: Real-world dependency graph All arrows go left to right  Same functionality as the OO example

Slide 22

Slide 22 text

More benefits: • Pipelines make testing easier • Pipelines fit well with modern architectures

Slide 23

Slide 23 text

Pipeline-oriented Programming in practice Part 2

Slide 24

Slide 24 text

var count = 0; foreach (var i in list) { var j = i + 2; if (j > 3) { count++; } } return count; Here is a traditional for- loop in C#

Slide 25

Slide 25 text

return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Here is the same code using LINQ in C#

Slide 26

Slide 26 text

return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Composability The LINQ components have been designed to fit together in many different ways You could write your code this way too!

Slide 27

Slide 27 text

return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Single responsibility Each LINQ component does one thing only. Easier to understand and test

Slide 28

Slide 28 text

return list .Select(x => x + 2) .Where(x => x > 3) .Where(x => x < 10) .Count(); Benefit: open for extension It is easy to add new steps in the pipeline without touching anything else

Slide 29

Slide 29 text

list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#

Slide 30

Slide 30 text

list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#

Slide 31

Slide 31 text

Immutability and pipelines If you have immutable data you will probably need a pipeline

Slide 32

Slide 32 text

// Person is immutable var p = new Person("Scott","[email protected]",21); var p2 = p.WithName("Tom"); var p3 = p2.WithEmail("[email protected]"); var p4 = p3.WithAge(42); When each call returns a new object, it gets repetitive

Slide 33

Slide 33 text

// Person is immutable var p = new Person("Scott","[email protected]",21); p .WithName("Tom") .WithEmail("[email protected]") .WithAge(42); Pipelines make it look nicer

Slide 34

Slide 34 text

Pipes vs. Extension methods Pipes are more general

Slide 35

Slide 35 text

int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; }

Slide 36

Slide 36 text

int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; } int NestedCalls() { return Double(Square(Add1(5))); } How can I make this look nicer? How about a pipeline?

Slide 37

Slide 37 text

return 5 .Add1 .Square .Double; But this doesn't work, because these are not extension methods.

Slide 38

Slide 38 text

public static TOut Pipe (this TIn input, Func fn) { return fn(input); } Introducing "Pipe"

Slide 39

Slide 39 text

public static TOut Pipe (this TIn input, Func fn) { return fn(input); }

Slide 40

Slide 40 text

public static TOut Pipe (this TIn input, Func fn) { return fn(input); }

Slide 41

Slide 41 text

return 5 .Pipe(Add1) .Pipe(Square) .Pipe(Double); And now we can create a pipeline out of any* existing functions

Slide 42

Slide 42 text

int Add(int i, int j) { return i + j; } int Mult(int i, int j) { return i * j; } A two parameter function

Slide 43

Slide 43 text

public static TOut Pipe (this TIn input, Func fn, TParam p1) { return fn(input, p1); }

Slide 44

Slide 44 text

int Add(int i, int j) { return i + j; } int Mult(int i, int j) { return i * j; } public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Mult, 2); } We can now create a pipeline out of any existing functions with parameters

Slide 45

Slide 45 text

public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Mult, 2); } Why bother? Because now we get all the benefits of a pipeline

Slide 46

Slide 46 text

public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Mult, 2) .Pipe(Add, 42) .Pipe(Square); } Why bother? Because now we get all the benefits of a pipeline, such as adding things to the pipeline easily (diffs look nicer too!)

Slide 47

Slide 47 text

let add x y = x + y let mult x y = x * y let square x = x * x 5 |> add 1 |> mult 2 |> add 42 |> square And here's what the same code looks like in F# F# uses pipelines everywhere! Most functions in F# are "pipeline-compatible" out of the box

Slide 48

Slide 48 text

Pipes vs. Extension methods, Part 2 Extension methods cannot be parameters

Slide 49

Slide 49 text

int CodeWithStrategy(... list, ... strategyFn) { return list .Select(x => x + 2) .strategyFn .Where(x => x > 3) .Count(); }  My "strategy" We cant use a function parameter as an extension method

Slide 50

Slide 50 text

int CodeWithStrategy(... list, ... strategyFn) { return list .Select(x => x + 2) .Pipe(strategyFn) .Where(x => x > 3) .Count(); } But we can use a function parameter in a pipeline! 

Slide 51

Slide 51 text

let codeWithStrategy list strategyFn = list |> List.map (fun x -> x + 2) |> strategyFn |> List.filter (fun x -> x > 3) |> List.length Again, F# functions are pipeline-compatible. No special helper needed. same code in F#

Slide 52

Slide 52 text

Pipelines in practice Part 3:

Slide 53

Slide 53 text

Three demonstrations • Pipeline-oriented Roman Numerals • Pipeline-oriented FizzBuzz • Pipeline-oriented web API

Slide 54

Slide 54 text

Roman Numerals

Slide 55

Slide 55 text

To Roman Numerals Task: convert an integer to roman numerals • 5 => "V" • 12 => "XII" • 107 => "CVII"

Slide 56

Slide 56 text

To Roman Numerals

Slide 57

Slide 57 text

To Roman Numerals • Use the "tally" approach – Start with N copies of "I" – Replace five "I"s with a "V" – Replace two "V"s with a "X" – Replace five "X"s with a "L" – Replace two "L"s with a "C" – etc

Slide 58

Slide 58 text

To Roman Numerals number etc Replicate "I" Replace_IIIII_V Replace_VV_X Replace_XXXXX_L

Slide 59

Slide 59 text

string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example

Slide 60

Slide 60 text

string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example .Replace("LL", "C") .Replace("VIIII", "IX") .Replace("IIII", "IV") .Replace("LXXXX", "XC") .Replace("XXXX", "XL"); } We can extend functionality without touching existing code!

Slide 61

Slide 61 text

let toRomanNumerals n = String.replicate n "I" |> replace "IIIII" "V" |> replace "VV" "X" |> replace "XXXXX" "L" |> replace "LL" "C" // special cases |> replace "VIIII" "IX" |> replace "IIII" "IV" |> replace "LXXXX" "XC" |> replace "XXXX" "XL" F# example

Slide 62

Slide 62 text

Demo: Roman Numerals in C# and F#

Slide 63

Slide 63 text

FizzBuzz

Slide 64

Slide 64 text

FizzBuzz definition • Write a program that prints the numbers from 1 to N • But: – For multiples of three print "Fizz" instead – For multiples of five print "Buzz" instead – For multiples of both three and five print "FizzBuzz" instead.

Slide 65

Slide 65 text

for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example

Slide 66

Slide 66 text

for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example

Slide 67

Slide 67 text

Pipeline implementation number Handle 15 case

Slide 68

Slide 68 text

Pipeline implementation Handle 3 case number Handle 15 case

Slide 69

Slide 69 text

Pipeline implementation Handle 3 case Handle 5 case number Handle 15 case

Slide 70

Slide 70 text

Pipeline implementation Handle 3 case Handle 5 case number Answer Handle 15 case Final step

Slide 71

Slide 71 text

number Handle case Processed (e.g. "Fizz", "Buzz") Unprocessed (e.g. 2, 7, 13) record FizzBuzzData(string Output, int Number);

Slide 72

Slide 72 text

record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); }

Slide 73

Slide 73 text

record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); } Extension method

Slide 74

Slide 74 text

static string FizzBuzzPipeline(int i) { return new FizzBuzzData("", i) .Handle(15, "FizzBuzz") .Handle(3, "Fizz") .Handle(5, "Buzz") .FinalStep(); } static void FizzBuzz() { var words = Enumerable.Range(1, 30) .Select(FizzBuzzPipeline); Console.WriteLine(string.Join(",", words)); }

Slide 75

Slide 75 text

Demo: FizzBuzz Bonus: Parallelization

Slide 76

Slide 76 text

A pipeline-oriented web API

Slide 77

Slide 77 text

HttpContext Web Component Not Handled Handled

Slide 78

Slide 78 text

(async) HttpContext null HttpContext

Slide 79

Slide 79 text

GET Define a component matches verb doesn't match verb HttpContext

Slide 80

Slide 80 text

route "/hello" Define a component matches route doesn't match route HttpContext

Slide 81

Slide 81 text

setStatusCode 200 Define a component Always succeeds HttpContext

Slide 82

Slide 82 text

GET route "/hello" >=> A special pipe for web components setStatusCode 200 >=>

Slide 83

Slide 83 text

Are you wondering how do we compose these?

Slide 84

Slide 84 text

Easy!

Slide 85

Slide 85 text

choose [ ] Picks first component that succeeds Define a new component

Slide 86

Slide 86 text

choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" ] Pick first path that succeeds

Slide 87

Slide 87 text

choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/bad" >=> BAD_REQUEST ] All the benefits of pipeline-oriented programming

Slide 88

Slide 88 text

choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming

Slide 89

Slide 89 text

choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming

Slide 90

Slide 90 text

Demo: A pipeline oriented web app For more on the web framework I'm using, search the internet for "F# Giraffe"

Slide 91

Slide 91 text

Testing and architecture Part 4

Slide 92

Slide 92 text

Benefit 4: Pipelines encourage good architecture

Slide 93

Slide 93 text

The "onion" architecture Core domain

Slide 94

Slide 94 text

Domain Logic I/O I/O The "onion" architecture  Core domain is pure, and all I/O is at the edges

Slide 95

Slide 95 text

In a well-designed pipeline, all I/O is at the edges. Easy to enforce this with a pipeline-oriented approach

Slide 96

Slide 96 text

Database Database Load data Process it Save it

Slide 97

Slide 97 text

I/O in the middle of a pipeline Keep them separate

Slide 98

Slide 98 text

Benefit 5: Easier to test

Slide 99

Slide 99 text

Deterministic Non-deterministic Non-deterministic

Slide 100

Slide 100 text

This makes testing easy!

Slide 101

Slide 101 text

In Conclusion

Slide 102

Slide 102 text

Why bother? • Reusable components • Understandable – data flows in one direction • Extendable – add new parts without touching old code • Testable – parts can be tested in isolation • A different way of thinking – it's good for your brain to learn new things!

Slide 103

Slide 103 text

Databases Object-oriented programmng Pipeline-oriented programming Domain-driven design Does pipeline-oriented programming work for every situation? No! But it should be part of your toolkit

Slide 104

Slide 104 text

Pipeline Oriented Programming – Slides and video will be posted at • fsharpforfunandprofit.com/pipeline Related talks – "The Power of Composition" • fsharpforfunandprofit.com/composition – “Railway Oriented Programming" • fsharpforfunandprofit.com/rop Thanks! Twitter: @ScottWlaschin