Workers and the Rise of Reactive Infrastructure

By Paul Burt!

In this presentation, I argue that Functional Programming is great for infrastructure. But, the Worker pattern is more familiar to OO folks, while offering many of the same advantages.

SLIDE NOTES:
2)
I’m a Developer Evangelist at iron.io.
We make easy to use, high throughput, infrastructure solutions.
4)
Functional Programming and Reactive Infrastructure are trendy.
For good reason, they require fewer mental gymnastics than traditional approaches.
7)
https://medium.com/javascript-scene/the-two-pillars-of-javascript-pt-2-functional-programming-a63aa53a41a4
8)
http://www.reactivemanifesto.org/
9)
Image CC BY 2.0
tracy the astonishing https://www.flickr.com/photos/tracy_the_astonishing/3398886840
12)
Image CC BY 2.0 https://upload.wikimedia.org/wikipedia/commons/e/e0/The_Grumpy_Old_Punks.jpg
14)
Quote from https://en.wikipedia.org/wiki/Idempotence#Examples
15)
The response is returning a webpage. This may change between requests. The “outcome” is the resource returns a page.
17)
Fish is treats other fish just like any other piece of food.
Functions should be treated just like any other variable.
CC BY 2.0 source https://www.flickr.com/photos/38446022@N00/2874171158
18)
One thing to note, this code is odd to write. But, it’s easy to read Hungry(bear). It gets even easier with piping syntax.
20)
One function does work.
The output is passed to the next one.
CC BY 2.0 from Alden Jewell https://www.flickr.com/photos/autohistorian/16617915558
22)
Image CC BY 2.0 https://upload.wikimedia.org/wikipedia/commons/4/4a/Todd_Huffman_-_Lattice_(by).jpg
25)
Like this banana slicer that does one thing, and does it well...
26)
Write code purpose built for one task. Chain tasks together. Yay composability!
28)
CC BY 2.0 Beate Meier https://www.flickr.com/photos/beate_meier/8337014543
31)
CC BY 2.0 khrawlings https://www.flickr.com/photos/khrawlings/3622143862/
32)
http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/
33) This Greek structure was shoddily repurposed, just like many try to repurpose yesterday's infrastructure.
Image CC BY 2.0 https://upload.wikimedia.org/wikipedia/commons/1/11/Vilnius_Cathedral_Facade.jpg
34)
Image from http://www.reactivemanifesto.org/
39)
FP offers a declarative approach.
This simplifies reading, reasoning, and debugging code.
Writing is harder, which makes it hard to get a big team to adopt.
41)
They’re fast,
fail gracefully, message driven,
and are easy to scale
42)
Devs already embrace OOP.
Just need to make code more modular. Embrace the Unix philosophy.
Workers are a natural way to get many benefits of FP, without the learning curve.

Transcript

1. Workers and the Rise of
   Reactive Infrastructure
   “Hey bruh, how many nines are in your SLA?” [sic]
   

   View Slide

2. Who is this guy?
   

   View Slide

3. Workers?
   Reactive whats?
   

   View Slide

4. FP + Reactive Infra
   

   View Slide

5. View Slide

6. FP + Reactive
   Workers
   

   View Slide

7. View Slide

8. View Slide

9. Functional Programming
   “The only way we’ll be
   able to keep up with the
   exponentially increasing
   complexity will be to
   decrease the complexity
   of understanding
   programs.”
   

   View Slide

10. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    

    View Slide

11. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    

    View Slide

12. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists
    We should
    Change.
    No.
    

    View Slide

13. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    Function add(a, b) {

    return a + b

    }
    add(2, 3) // 5

    add(2, 3) // 5
    Same input produces 

    the same output
    State is

    wholly inside

    the function.

    (no side effects)
    

    View Slide

14. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists
    Function add(a, b) {

    return a + b

    }
    add(2, 3) // 5

    add(2, 3) // 5
    Same input produces 

    the same output
    Idempotence 

    

    the ability of a system to 

    produce the same outcome, 

    even if an event or message is
    received more than once.
    

    View Slide

15. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists
    Function add(a, b) {

    return a + b

    }
    add(2, 3) // 5

    add(2, 3) // 5
    Same input produces 

    the same output
    Idempotence 

    

    Caveat!

    The response might be different,
    even if the “outcome” is the same.
    eg

    HTTP GET requests
    

    View Slide

16. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    

    View Slide

17. First Class
    Functions?
    A fish, is a fish, is a fish.
    

    View Slide

18. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    var bear = function(err) {

    if (err) throw err

    console.log(“Was hungry.”)

    }
    function hungry(func) {

    console.log(

    “Has now eaten.”

    )

    func(null)

    }

    

    hungry(bear)
    Function stored 

    like a var

    (first class)
    Pass functions

    as an arg

    (higher order)
    

    View Slide

19. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    

    View Slide

20. Streams
    Kind of like an 

    assembly line.
    

    View Slide

21. Functional Programming
    • Immutable & Idempotent
    • First Class Functions
    • Lists => Streams
    // Finally, a concrete example
    var nums = [1, 4, 9, 16]

    

    nums.map(Math.sqrt) // [1, 2, 3, 4]

    .filter(function(n) { 

    return n > 2 

    }) // [3, 4]

    .reduce(function(pre, cur) { 

    return pre + cur 

    }) // 7

    Functions

    operating

    on the stream
    

    View Slide

22. Workers
    Unix philosophy

    + at scale.

    Do one thing, 

    do it well

    + do a lot of it.
    

    View Slide

23. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    

    View Slide

24. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    

    View Slide

25. View Slide

26. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    > find ./ironio/**/
    > curl iron.io
    > grep -E ‘meta' ironio.html
    # composability is key
    > curl iron.io | grep 'html'
    Workers are chainable

    in the same way.
    

    View Slide

27. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    

    View Slide

28. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    Scaling?
    Use 

    message queues
    

    View Slide

29. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    # use CLI to queue jobs
    > iron worker queue 

    --payload-file
    hello.payload.json 

    --wait projectName
    # need speed?

    # just add workers to 

    # consume faster
    Payload files:

    may be serialized data,

    or a URL to a video

    (anything to be processed)
    

    View Slide

30. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    

    View Slide

31. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    

    View Slide

32. Workers
    • Do one thing, Do it well
    • Do it at Scale
    • Great designers steal
    Steal idempotence from FP.
    Why?
    “Within the context of a
    distributed system, you cannot
    have exactly-once message
    delivery.“
    

    View Slide

33. Reactive Manifesto
    Today's demands are
    simply not met by
    yesterday’s software
    architectures.
    

    View Slide

34. Reactive Manifesto
    

    View Slide

35. Reactive Manifesto
    Fast!

    Responds to actions quickly.
    

    View Slide

36. Reactive Manifesto
    Fails gracefully

    (isolation + replication)
    

    View Slide

37. Reactive Manifesto
    + Non-blocking

    + Real time

    + Loosely coupled
    (NOT event emmitters)

    Messages are:

    

    View Slide

38. Reactive Manifesto
    Scales simply

    & horizontally
    

    View Slide

39. FP + Reactive

    = a neat trend
    

    View Slide

40. FP + Reactive

    = a neat trend
    Big learning

    curve
    sad face :(
    

    View Slide

41. Workers
    What about Workers?
    

    View Slide

42. Workers
    

    View Slide

43. OOP FP + Reactive

    = a neat trend
    Use workers & . . .
    Yay! :D
    Build

    modular programs!
    Embrace the

    Unix

    Philosophy
    

    View Slide

44. Let’s take a look at a Worker

    iron.io/worker
    

    View Slide