Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
580
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
260
But why does the business care?
mheap
0
130
Advanced GitHub Actions
mheap
0
340
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
250
API Standards 2.0
mheap
1
880
API Standards 2.0
mheap
0
440
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
440
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
750
Behat for characterization tests
mheap
0
350
Other Decks in Technology
See All in Technology
生成AIの強みと弱みを理解して、生成AIがもたらすパワーをプロダクトの価値へ繋げるために実践したこと / advance-ai-generating
cyberagentdevelopers
PRO
1
190
Apple/Google/Amazonの決済システムの違いを踏まえた定期購読課金システムの構築 / abema-billing-system
cyberagentdevelopers
PRO
1
220
Fargateを使った研修の話
takesection
0
140
オーティファイ会社紹介資料 / Autify Company Deck
autifyhq
9
120k
国土交通省 データコンペ参加者向け勉強会
takehikohashimoto
0
180
【技術書典17】OpenFOAM(自宅で極める流体解析)2次元円柱まわりの流れ
kamakiri1225
0
230
GitHub Universe: Evaluating RAG apps in GitHub Actions
pamelafox
0
180
一休.comレストランにおけるRustの活用
kymmt90
3
610
【若手エンジニア応援LT会】AWSで繋がり、共に成長! ~コミュニティ活動と新人教育への挑戦~
kazushi_ohata
0
230
小規模に始めるデータメッシュとデータガバナンスの実践
kimujun
4
610
Jr. Championsになって、強く連携しながらAWSをもっと使いたい!~AWSに対する期待と行動~
amixedcolor
0
240
新卒1年目が挑む!生成AI × マルチエージェントで実現する次世代オンボーディング / operation-ai-onboarding
cyberagentdevelopers
PRO
1
180
Featured
See All Featured
It's Worth the Effort
3n
183
27k
Code Reviewing Like a Champion
maltzj
519
39k
Code Review Best Practice
trishagee
64
17k
Building an army of robots
kneath
302
42k
How To Stay Up To Date on Web Technology
chriscoyier
788
250k
We Have a Design System, Now What?
morganepeng
50
7.2k
The Language of Interfaces
destraynor
154
24k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
How STYLIGHT went responsive
nonsquared
95
5.2k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.6k
Fashionably flexible responsive web design (full day workshop)
malarkey
404
65k
For a Future-Friendly Web
brad_frost
175
9.4k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428