Pipeline Architectures

Pipeline Architectures at PHPNW15

Transcript

1. Pipeline Architectures
   

   View Slide

2. @mheap
   

   View Slide

3. Pipelines
   

   View Slide

4. Tea/Coffee
   

   View Slide

5. Waterfall
   

   View Slide

6. Manufacturing
   

   View Slide

7. View Slide

8. Engine: 20 mins
   Frame: 40 mins
   Windows: 10 mins
   

   View Slide

9. 1x Engine: 20 mins
   1x Frame: 40 mins
   1x Windows: 10 mins
   

   View Slide

10. 2x Engine: 20 mins
    4x Frame: 40 mins
    1x Windows: 10 mins
    

    View Slide

11. Code Pipelines
    

    View Slide

12. This is a unix system
    I know this
    

    View Slide

13. cat attendees.txt
    

    View Slide

14. cat attendees.txt | grep
    Michael
    

    View Slide

15. cat attendees.txt | grep
    Michael > michaels.txt
    

    View Slide

16. cat attendees.txt | grep
    Michael | sort >
    michaels.txt
    

    View Slide

17. Write programs that do one thing
    and do it well.
    Write programs to work together.
    Write programs to handle text
    streams, because that is a
    universal interface.
    

    View Slide

18. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

19. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

20. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

21. producer.go
    channel <- sum
    foo.go
    val <- channel
    

    View Slide

22. app.php
    $bus->handle(
    new UserRegisteredEvent
    )
    

    View Slide

23. Input
    Processing
    Output
    

    View Slide

24. Controller
    Model
    View
    

    View Slide

25. Action
    Domain
    Responder
    

    View Slide

26. POST /foo
    Domain
    echo $twig->render()
    

    View Slide

27. $ php foo.php ‘bar’
    Domain
    echo cli_output($output);
    

    View Slide

28. zmqpp
    
    zmqpp
    

    View Slide

29. zmqpp
    
    redis-pipe
    

    View Slide

30. nsq
    
    nsq
    

    View Slide

31. Human Pipelines
    

    View Slide

32. + =
    

    View Slide

33. View Slide

34. Data
    ???
    Profit!
    

    View Slide

35. Data
    Profit!
    

    View Slide

36. Data
    Profit!
    

    View Slide

37. Data
    Profit!
    

    View Slide

38. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    

    View Slide

39. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    

    View Slide

40. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    ?
    ?
    

    View Slide

41. Data
    Profit!
    PHP
    PHP
    C++
    

    View Slide

42. Data
    Profit!
    PHP
    Golang
    C++
    

    View Slide

43. Data
    Profit!
    PHP
    Golang
    Michael
    

    View Slide

44. Building Pipelines
    

    View Slide

45. Data-Driven
    

    View Slide

46. Demand-Driven
    

    View Slide

47. Data-Driven
    

    View Slide

48. push/pull
    

    View Slide

49. Back Pressure
    

    View Slide

50. pub/sub
    

    View Slide

51. Sequence Numbers
    

    View Slide

52. Demand-Driven
    

    View Slide

53. Unique Jobs
    

    View Slide

54. Transport
    

    View Slide

55. Serialization
    

    View Slide

56. Error Handling
    

    View Slide

57. Testing Pipelines
    

    View Slide

58. Pros/Cons
    

    View Slide

59. Examples
    

    View Slide

60. https://www.flickr.com/photos/neotsn/2246759100/
    https://www.flickr.com/photos/theenmoy/15255488518
    https://www.flickr.com/photos/stuckincustoms/5109788502
    https://www.flickr.com/photos/danielfoster/4725849931
    https://www.flickr.com/photos/mzmo/8666989650
    https://www.flickr.com/photos/usacehq/5529362681
    https://www.flickr.com/photos/alpha_auer/5068482201/
    https://www.flickr.com/photos/109144586@N08/14863759614/
    https://www.flickr.com/photos/94418464@N08/9007117038
    https://www.flickr.com/photos/swamibu/2868288357/
    https://www.flickr.com/photos/sepehrehsani/5766453552/
    https://www.flickr.com/photos/scubasteveo/296747958/
    https://www.flickr.com/photos/56218409@N03/15371262455/
    https://www.flickr.com/photos/jvk/4417106
    https://www.flickr.com/photos/13523064@N03/14448815581
    https://www.flickr.com/photos/jamesshade/473206000
    https://www.flickr.com/photos/albertogp123/5843577306/
    https://www.flickr.com/photos/dpstyles/4835354126/
    

    View Slide

61. Questions?
    @mheap joind.in/15428
    

    View Slide