Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
630
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
340
But why does the business care?
mheap
0
200
Advanced GitHub Actions
mheap
0
440
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
380
API Standards 2.0
mheap
1
1k
API Standards 2.0
mheap
0
510
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
560
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
930
Behat for characterization tests
mheap
0
410
Other Decks in Technology
See All in Technology
変化する開発、進化する体系時代に適応するソフトウェアエンジニアの知識と考え方(JaSST'25 Kansai)
mizunori
1
210
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
100
OpenHands🤲にContributeしてみた
kotauchisunsun
1
430
Snowflake Summit 2025 データエンジニアリング関連新機能紹介 / Snowflake Summit 2025 What's New about Data Engineering
tiltmax3
0
310
Witchcraft for Memory
pocke
1
310
第9回情シス転職ミートアップ_テックタッチ株式会社
forester3003
0
230
Agentic Workflowという選択肢を考える
tkikuchi1002
1
500
Amazon Bedrockで実現する 新たな学習体験
kzkmaeda
2
540
フィンテック養成勉強会#54
finengine
0
180
PostgreSQL 18 cancel request key長の変更とRailsへの関連
yahonda
0
120
Navigation3でViewModelにデータを渡す方法
mikanichinose
0
220
AWS テクニカルサポートとエンドカスタマーの中間地点から見えるより良いサポートの活用方法
kazzpapa3
2
540
Featured
See All Featured
Become a Pro
speakerdeck
PRO
28
5.4k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
210
4 Signs Your Business is Dying
shpigford
184
22k
Typedesign – Prime Four
hannesfritz
42
2.7k
How GitHub (no longer) Works
holman
314
140k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
VelocityConf: Rendering Performance Case Studies
addyosmani
330
24k
GitHub's CSS Performance
jonrohan
1031
460k
Code Reviewing Like a Champion
maltzj
524
40k
Speed Design
sergeychernyshev
32
1k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428