Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pipeline Architectures

Pipeline Architectures

Pipeline Architectures at PHPNW15

Michael Heap

October 03, 2015
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. Pipeline Architectures

    View Slide

  2. @mheap

    View Slide

  3. Pipelines

    View Slide

  4. Tea/Coffee

    View Slide

  5. Waterfall

    View Slide

  6. Manufacturing

    View Slide

  7. View Slide

  8. Engine: 20 mins
    Frame: 40 mins
    Windows: 10 mins

    View Slide

  9. 1x Engine: 20 mins
    1x Frame: 40 mins
    1x Windows: 10 mins

    View Slide

  10. 2x Engine: 20 mins
    4x Frame: 40 mins
    1x Windows: 10 mins

    View Slide

  11. Code Pipelines

    View Slide

  12. This is a unix system
    I know this

    View Slide

  13. cat attendees.txt

    View Slide

  14. cat attendees.txt | grep
    Michael

    View Slide

  15. cat attendees.txt | grep
    Michael > michaels.txt

    View Slide

  16. cat attendees.txt | grep
    Michael | sort >
    michaels.txt

    View Slide

  17. Write programs that do one thing
    and do it well.
    Write programs to work together.
    Write programs to handle text
    streams, because that is a
    universal interface.

    View Slide

  18. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)

    View Slide

  19. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)

    View Slide

  20. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)

    View Slide

  21. producer.go
    channel <- sum
    foo.go
    val <- channel

    View Slide

  22. app.php
    $bus->handle(
    new UserRegisteredEvent
    )

    View Slide

  23. Input
    Processing
    Output

    View Slide

  24. Controller
    Model
    View

    View Slide

  25. Action
    Domain
    Responder

    View Slide

  26. POST /foo
    Domain
    echo $twig->render()

    View Slide

  27. $ php foo.php ‘bar’
    Domain
    echo cli_output($output);

    View Slide

  28. zmqpp

    zmqpp

    View Slide

  29. zmqpp

    redis-pipe

    View Slide

  30. nsq

    nsq

    View Slide

  31. Human Pipelines

    View Slide

  32. + =

    View Slide

  33. View Slide

  34. Data
    ???
    Profit!

    View Slide

  35. Data
    Profit!

    View Slide

  36. Data
    Profit!

    View Slide

  37. Data
    Profit!

    View Slide

  38. Data
    Profit!
    JSON
    JSON
    JSON
    JSON

    View Slide

  39. Data
    Profit!
    JSON
    JSON
    JSON
    JSON

    View Slide

  40. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    ?
    ?

    View Slide

  41. Data
    Profit!
    PHP
    PHP
    C++

    View Slide

  42. Data
    Profit!
    PHP
    Golang
    C++

    View Slide

  43. Data
    Profit!
    PHP
    Golang
    Michael

    View Slide

  44. Building Pipelines

    View Slide

  45. Data-Driven

    View Slide

  46. Demand-Driven

    View Slide

  47. Data-Driven

    View Slide

  48. push/pull

    View Slide

  49. Back Pressure

    View Slide

  50. pub/sub

    View Slide

  51. Sequence Numbers

    View Slide

  52. Demand-Driven

    View Slide

  53. Unique Jobs

    View Slide

  54. Transport

    View Slide

  55. Serialization

    View Slide

  56. Error Handling

    View Slide

  57. Testing Pipelines

    View Slide

  58. Pros/Cons

    View Slide

  59. Examples

    View Slide

  60. https://www.flickr.com/photos/neotsn/2246759100/
    https://www.flickr.com/photos/theenmoy/15255488518
    https://www.flickr.com/photos/stuckincustoms/5109788502
    https://www.flickr.com/photos/danielfoster/4725849931
    https://www.flickr.com/photos/mzmo/8666989650
    https://www.flickr.com/photos/usacehq/5529362681
    https://www.flickr.com/photos/alpha_auer/5068482201/
    https://www.flickr.com/photos/[email protected]/14863759614/
    https://www.flickr.com/photos/[email protected]/9007117038
    https://www.flickr.com/photos/swamibu/2868288357/
    https://www.flickr.com/photos/sepehrehsani/5766453552/
    https://www.flickr.com/photos/scubasteveo/296747958/
    https://www.flickr.com/photos/[email protected]/15371262455/
    https://www.flickr.com/photos/jvk/4417106
    https://www.flickr.com/photos/[email protected]/14448815581
    https://www.flickr.com/photos/jamesshade/473206000
    https://www.flickr.com/photos/albertogp123/5843577306/
    https://www.flickr.com/photos/dpstyles/4835354126/

    View Slide

  61. Questions?
    @mheap joind.in/15428

    View Slide