Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
660
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
380
But why does the business care?
mheap
0
240
Advanced GitHub Actions
mheap
0
500
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
460
API Standards 2.0
mheap
1
1.1k
API Standards 2.0
mheap
0
560
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
630
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
1k
Behat for characterization tests
mheap
0
450
Other Decks in Technology
See All in Technology
AWS Control Tower に学ぶ! IAM Identity Center 権限設計の第一歩 / IAM Identity Center with Control Tower
y___u
0
170
ガバメントクラウド(AWS)へのデータ移行戦略の立て方【虎の巻】 / 20251011 Mitsutosi Matsuo
shift_evolve
PRO
2
200
リセラー企業のテクサポ担当が考える、生成 AI 時代のトラブルシュート 2025
kazzpapa3
1
350
大規模サーバーレスAPIの堅牢性・信頼性設計 〜AWSのベストプラクティスから始まる現実的制約との向き合い方〜
maimyyym
10
4.8k
新規事業におけるGORM+SQLx併用アーキテクチャ
hacomono
PRO
0
300
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
ComposeではないコードをCompose化する case ビズリーチ / DroidKaigi 2025 koyasai
visional_engineering_and_design
0
110
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
「れきちず」のこれまでとこれから - 誰にでもわかりやすい歴史地図を目指して / FOSS4G 2025 Japan
hjmkth
1
310
RDS の負荷が高い場合に AWS で取りうる具体策 N 連発/a-series-of-specific-countermeasures-available-on-aws-when-rds-is-under-high-load
emiki
2
1.4k
プロダクトのコードから見るGoによるデザインパターンの実践 #go_night_talk
bengo4com
1
2.5k
Geospatialの世界最前線を探る [2025年版]
dayjournal
1
220
Featured
See All Featured
How GitHub (no longer) Works
holman
315
140k
The Language of Interfaces
destraynor
162
25k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Making the Leap to Tech Lead
cromwellryan
135
9.6k
Producing Creativity
orderedlist
PRO
347
40k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
37
2.6k
Speed Design
sergeychernyshev
32
1.2k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428