Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
570
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
220
But why does the business care?
mheap
0
110
Advanced GitHub Actions
mheap
0
300
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
210
API Standards 2.0
mheap
1
850
API Standards 2.0
mheap
0
400
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
400
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
680
Behat for characterization tests
mheap
0
320
Other Decks in Technology
See All in Technology
RAGのサービスをリリースして1年3ヶ月が経ちました
segavvy
4
900
開発生産性をむしろ向上させる セキュリティパートナーの作り方 / Dev Productivity Con 2024
flatt_security
0
360
エンジニア向け会社紹介資料
caddi_eng
14
220k
Classmethod流のPlatform Engineering / classmethod-platform-engineering-devio2024
tomoki10
0
470
データベース研修 分析向けSQL入門【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
110
DevIO2024_レガシー運用からの脱却 -クラウド活用の実践事例とベストプラクティス-
jun2882
0
210
開発と事業を繋ぐ!SREのオブザーバビリティ戦略 ~ Developers Summit 2024 Summer ~
leveragestech
0
620
データ分析を支える技術 生成AI再入門
ishikawa_satoru
0
380
セキュリティ研修 Day1【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
150
OSSコミットしてZennの課題を解決した話
dyoshikawa1993
0
150
JBUG岡山 #6 WordCamp男木島の チームビルディング
takeshifurusato
0
150
頼られるのが大好きな 皆さんへ - 支援相手との期待の合わせ方、突き放し方 -/For_people_who_like_to_be_relied_on
naitosatoshi
1
290
Featured
See All Featured
What's in a price? How to price your products and services
michaelherold
239
11k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.9k
How to Ace a Technical Interview
jacobian
274
23k
Leading Effective Engineering Teams 2024
addyosmani
3
300
Ruby is Unlike a Banana
tanoku
96
10k
A designer walks into a library…
pauljervisheath
201
24k
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
Embracing the Ebb and Flow
colly
81
4.3k
Optimizing for Happiness
mojombo
373
69k
Agile that works and the tools we love
rasmusluckow
325
20k
The Cult of Friendly URLs
andyhume
75
5.9k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428