Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
550
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
190
But why does the business care?
mheap
0
97
Advanced GitHub Actions
mheap
0
280
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
170
API Standards 2.0
mheap
1
820
API Standards 2.0
mheap
0
370
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
380
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
640
Behat for characterization tests
mheap
0
300
Other Decks in Technology
See All in Technology
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
120
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
240
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
5
490
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
120
20240418_Google ColabにLLMが搭載されたようなのでPython x データ分析の勉強方法を考えてみる
doradora09
0
130
プロトタイピングによる不確実性の低減 / Reducing Uncertainty through Prototyping
ohbarye
5
380
ChatworkのSRE部って実は 半分くらいPlatform Engineering部かもしれない
saramune
0
160
開発パフォーマンスを最大化するための開発体制
ham0215
2
330
VS CodeでAWSを操作しよう
smt7174
7
1.6k
長期間TiDBを使ってきた話 @ 私たちはなぜNewSQLを使うのかTiDB選定5社が語る選定理由と活用LT / Experiences with TiDB Over Time
chibiegg
2
890
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
910
Reducing Cross-Zone Egress at Spotify with Custom gRPC Load Balancing Recap
koh_naga
0
200
Featured
See All Featured
Atom: Resistance is Futile
akmur
259
25k
The Illustrated Children's Guide to Kubernetes
chrisshort
31
46k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Designing for humans not robots
tammielis
248
25k
Practical Orchestrator
shlominoach
182
9.7k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Why Our Code Smells
bkeepers
PRO
331
56k
Testing 201, or: Great Expectations
jmmastey
28
6.3k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.6k
Bash Introduction
62gerente
604
210k
We Have a Design System, Now What?
morganepeng
43
6.7k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
244
20k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428