Pipeline Architectures

Pipelines help split your architecture into distinct components. They're also a fantastic way to design software. By designing your architecture as a pipeline, you decouple all of your components and make sure that each part has a very specific responsibility.

This talk will look at how to start putting together a pipeline architecture and how to test them, as well as looking at a few case studies of how existing systems can be refactored to behave as a pipeline.

Transcript

1. pipeline:architectures
2. @mheap
3. Pipelines
4. Tea/Coffee
5. Waterfall
6. Manufacturing
8. Engine: 20 mins Frame: 40 mins Windows: 10 mins
9. 2x Engine: 20 mins 4x Frame: 40 mins 1x Windows: 10 mins
10. Code Pipelines
11. This is a unix system I know this
12. cat attendees.txt
13. cat attendees.txt | grep Michael
14. cat attendees.txt | grep Michael > michaels.txt
15. cat attendees.txt | grep Michael | sort > michaels.txt
16. Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
17. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
18. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
19. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
20. producer.go channel <- sum foo.go val <- channel
21. app.php $bus->handle( new UserRegisteredEvent )
22. Input Processing Output
24. Data ??? Profit!
25. Data Profit!
26. Data Profit!
27. Data Profit!
28. Data Profit! JSON JSON JSON JSON
29. Data Profit! JSON JSON JSON JSON
30. Data Profit! PHP PHP C++
31. Data Profit! PHP Golang C++
32. Data Profit! PHP Golang Michael
33. Building Pipelines
34. Data-Driven
35. Demand-Driven
36. Data-Driven
37. push/pull
38. Back Pressure
39. pub/sub
40. Sequence Numbers
41. Demand-Driven
42. Unique Jobs
43. Transport
44. Serialization
45. Error Handling
46. Pros/Cons
47. Case Study
48. https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/ https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/
49. @mheap
50. Questions?