Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
630
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
340
But why does the business care?
mheap
0
200
Advanced GitHub Actions
mheap
0
450
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
390
API Standards 2.0
mheap
1
1k
API Standards 2.0
mheap
0
520
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
570
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
950
Behat for characterization tests
mheap
0
420
Other Decks in Technology
See All in Technology
LangChain Interrupt & LangChain Ambassadors meetingレポート
os1ma
2
320
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
230
american aa airlines®️ USA Contact Numbers: Complete 2025 Support Guide
aaguide
0
180
Glacierだからってコストあきらめてない? / JAWS Meet Glacier Cost
taishin
1
160
面倒な作業はAIにおまかせ。Flutter開発をスマートに効率化
ruideengineer
0
260
Lufthansa ®️ USA Contact Numbers: Complete 2025 Support Guide
lufthanahelpsupport
0
200
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
150
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
200
LLM時代の検索
shibuiwilliam
2
170
Model Mondays S2E04: AI Developer Experiences
nitya
0
140
React開発にStorybookとCopilotを導入して、爆速でUIを編集・確認する方法
yu_kod
1
280
2025-07-06 QGIS初級ハンズオン「はじめてのQGIS」
kou_kita
0
170
Featured
See All Featured
RailsConf 2023
tenderlove
30
1.1k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Facilitating Awesome Meetings
lara
54
6.4k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Designing Experiences People Love
moore
142
24k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428