Pipeline Architectures

Pipeline Architectures at PHPNW15

Transcript

1. Pipeline Architectures
2. @mheap
3. Pipelines
4. Tea/Coffee
5. Waterfall
6. Manufacturing
8. Engine: 20 mins Frame: 40 mins Windows: 10 mins
9. 1x Engine: 20 mins 1x Frame: 40 mins 1x Windows: 10 mins
10. 2x Engine: 20 mins 4x Frame: 40 mins 1x Windows: 10 mins
11. Code Pipelines
12. This is a unix system I know this
13. cat attendees.txt
14. cat attendees.txt | grep Michael
15. cat attendees.txt | grep Michael > michaels.txt
16. cat attendees.txt | grep Michael | sort > michaels.txt
17. Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
18. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
19. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
20. read(file.txt). map(u => return u.name). filter(n => return n == “michael”)
21. producer.go channel <- sum foo.go val <- channel
22. app.php $bus->handle( new UserRegisteredEvent )
23. Input Processing Output
24. Controller Model View
25. Action Domain Responder
26. POST /foo Domain echo $twig->render()
27. $ php foo.php ‘bar’ Domain echo cli_output($output);
28. zmqpp <your application> zmqpp
29. zmqpp <your application> redis-pipe
30. nsq <your application> nsq
31. Human Pipelines
32. + =
34. Data ??? Profit!
35. Data Profit!
36. Data Profit!
37. Data Profit!
38. Data Profit! JSON JSON JSON JSON
39. Data Profit! JSON JSON JSON JSON
40. Data Profit! JSON JSON JSON JSON ? ?
41. Data Profit! PHP PHP C++
42. Data Profit! PHP Golang C++
43. Data Profit! PHP Golang Michael
44. Building Pipelines
45. Data-Driven
46. Demand-Driven
47. Data-Driven
48. push/pull
49. Back Pressure
50. pub/sub
51. Sequence Numbers
52. Demand-Driven
53. Unique Jobs
54. Transport
55. Serialization
56. Error Handling
57. Testing Pipelines
58. Pros/Cons
59. Examples
60. https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/ https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
61. Questions? @mheap joind.in/15428