Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
640
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
350
But why does the business care?
mheap
0
220
Advanced GitHub Actions
mheap
0
460
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
410
API Standards 2.0
mheap
1
1.1k
API Standards 2.0
mheap
0
540
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
600
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
980
Behat for characterization tests
mheap
0
430
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructure:2025年7月度サービス・アップデート
oracle4engineer
PRO
1
180
人に寄り添うAIエージェントとアーキテクチャ #BetAIDay
layerx
PRO
9
2.2k
AI時代の経営、Bet AI Vision #BetAIDay
layerx
PRO
1
2k
Rubyの国のPerlMonger
anatofuz
3
730
大規模イベントに向けた ABEMA アーキテクチャの遍歴 ~ Platform Strategy 詳細解説 ~
nagapad
0
220
データモデリング通り #2オンライン勉強会 ~方法論の話をしよう~
datayokocho
0
150
LTに影響を受けてテンプレリポジトリを作った話
hol1kgmg
0
360
Claude Codeは仕様駆動の夢を見ない
gotalab555
23
6.4k
Vision Language Modelと自動運転AIの最前線_20250730
yuyamaguchi
4
1.3k
Amazon Bedrock AgentCoreのフロントエンドを探す旅 (Next.js編)
kmiya84377
1
140
S3 Glacier のデータを Athena からクエリしようとしたらどうなるのか/try-to-query-s3-glacier-from-athena
emiki
0
220
形式手法特論:位相空間としての並行プログラミング #kernelvm / Kernel VM Study Tokyo 18th
ytaka23
3
1.3k
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
96
6.2k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
For a Future-Friendly Web
brad_frost
179
9.9k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
183
54k
Why Our Code Smells
bkeepers
PRO
337
57k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
332
22k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
800
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Done Done
chrislema
185
16k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428