Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
650
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
360
But why does the business care?
mheap
0
230
Advanced GitHub Actions
mheap
0
480
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
430
API Standards 2.0
mheap
1
1.1k
API Standards 2.0
mheap
0
540
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
610
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
1k
Behat for characterization tests
mheap
0
440
Other Decks in Technology
See All in Technology
Product Management Conference -AI時代に進化するPdM-
kojima111
0
270
実践アプリケーション設計 ②トランザクションスクリプトへの対応
recruitengineers
PRO
4
1.1k
新規案件の立ち上げ専門チームから見たAI駆動開発の始め方
shuyakinjo
0
620
ソフトウェア エンジニアとしての 姿勢と心構え
recruitengineers
PRO
25
12k
生成AI時代のデータ基盤
shibuiwilliam
1
1.1k
「魔法少女まどか☆マギカ Magia Exedra」での負荷試験の実践と学び
gree_tech
PRO
0
390
人と組織に偏重したEMへのアンチテーゼ──なぜ、EMに設計力が必要なのか/An antithesis to the overemphasis of people and organizations in EM
dskst
7
810
AWSで推進するデータマネジメント
kawanago
0
610
『FailNet~やらかし共有SNS~』エレベーターピッチ
yokomachi
1
190
JuniorからSeniorまで: DevOpsエンジニアの成長ロードマップ
yuriemori
2
330
TypeScript入門
recruitengineers
PRO
33
11k
AWS環境のリソース調査を Claude Code で効率化 / aws investigate with cc devio2025
masahirokawahara
2
900
Featured
See All Featured
Done Done
chrislema
185
16k
Gamification - CAS2011
davidbonilla
81
5.4k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Into the Great Unknown - MozCon
thekraken
40
2k
Navigating Team Friction
lara
189
15k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Visualization
eitanlees
147
16k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
570
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
51
5.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
A Tale of Four Properties
chriscoyier
160
23k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428