Pipeline Architectures

Pipelines help split your architecture into distinct components. They're also a fantastic way to design software. By designing your architecture as a pipeline, you decouple all of your components and make sure that each part has a very specific responsibility.

This talk will look at how to start putting together a pipeline architecture and how to test them, as well as looking at a few case studies of how existing systems can be refactored to behave as a pipeline.

Transcript

1. pipeline:architectures
   

   View Slide

2. @mheap
   

   View Slide

3. Pipelines
   

   View Slide

4. Tea/Coffee
   

   View Slide

5. Waterfall
   

   View Slide

6. Manufacturing
   

   View Slide

7. View Slide

8. Engine: 20 mins
   Frame: 40 mins
   Windows: 10 mins
   

   View Slide

9. 2x Engine: 20 mins
   4x Frame: 40 mins
   1x Windows: 10 mins
   

   View Slide

10. Code Pipelines
    

    View Slide

11. This is a unix system
    I know this
    

    View Slide

12. cat attendees.txt
    

    View Slide

13. cat attendees.txt | grep
    Michael
    

    View Slide

14. cat attendees.txt | grep
    Michael > michaels.txt
    

    View Slide

15. cat attendees.txt | grep
    Michael | sort >
    michaels.txt
    

    View Slide

16. Write programs that do one thing
    and do it well.
    Write programs to work together.
    Write programs to handle text
    streams, because that is a
    universal interface.
    

    View Slide

17. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

18. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

19. read(file.txt).
    map(u => return u.name).
    filter(n => return n ==
    “michael”)
    

    View Slide

20. producer.go
    channel <- sum
    foo.go
    val <- channel
    

    View Slide

21. app.php
    $bus->handle(
    new UserRegisteredEvent
    )
    

    View Slide

22. Input
    Processing
    Output
    

    View Slide

23. View Slide

24. Data
    ???
    Profit!
    

    View Slide

25. Data
    Profit!
    

    View Slide

26. Data
    Profit!
    

    View Slide

27. Data
    Profit!
    

    View Slide

28. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    

    View Slide

29. Data
    Profit!
    JSON
    JSON
    JSON
    JSON
    

    View Slide

30. Data
    Profit!
    PHP
    PHP
    C++
    

    View Slide

31. Data
    Profit!
    PHP
    Golang
    C++
    

    View Slide

32. Data
    Profit!
    PHP
    Golang
    Michael
    

    View Slide

33. Building Pipelines
    

    View Slide

34. Data-Driven
    

    View Slide

35. Demand-Driven
    

    View Slide

36. Data-Driven
    

    View Slide

37. push/pull
    

    View Slide

38. Back Pressure
    

    View Slide

39. pub/sub
    

    View Slide

40. Sequence Numbers
    

    View Slide

41. Demand-Driven
    

    View Slide

42. Unique Jobs
    

    View Slide

43. Transport
    

    View Slide

44. Serialization
    

    View Slide

45. Error Handling
    

    View Slide

46. Pros/Cons
    

    View Slide

47. Case Study
    

    View Slide

48. https://www.flickr.com/photos/neotsn/2246759100/
    https://www.flickr.com/photos/theenmoy/15255488518
    https://www.flickr.com/photos/stuckincustoms/5109788502
    https://www.flickr.com/photos/danielfoster/4725849931
    https://www.flickr.com/photos/mzmo/8666989650
    https://www.flickr.com/photos/usacehq/5529362681
    https://www.flickr.com/photos/alpha_auer/5068482201/
    https://www.flickr.com/photos/109144586@N08/14863759614/
    https://www.flickr.com/photos/94418464@N08/9007117038
    https://www.flickr.com/photos/swamibu/2868288357/
    https://www.flickr.com/photos/sepehrehsani/5766453552/
    https://www.flickr.com/photos/scubasteveo/296747958/
    https://www.flickr.com/photos/56218409@N03/15371262455/
    

    View Slide

49. @mheap
    

    View Slide

50. Questions?
    

    View Slide