Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
660
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
370
But why does the business care?
mheap
0
230
Advanced GitHub Actions
mheap
0
500
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
460
API Standards 2.0
mheap
1
1.1k
API Standards 2.0
mheap
0
550
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
620
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
1k
Behat for characterization tests
mheap
0
440
Other Decks in Technology
See All in Technology
AI ReadyなData PlatformとしてのAutonomous Databaseアップデート
oracle4engineer
PRO
0
180
成長自己責任時代のあるきかた/How to navigate the era of personal responsibility for growth
kwappa
3
270
ZOZOのAI活用実践〜社内基盤からサービス応用まで〜
zozotech
PRO
0
170
多様な事業ドメインのクリエイターへ 価値を届けるための営みについて
massyuu
0
110
神回のメカニズムと再現方法/Mechanisms and Playbook for Kamikai scrumat2025
moriyuya
4
530
Why React!?? Next.jsそしてReactを改めてイチから選ぶ
ypresto
10
4.5k
GA technologiesでのAI-Readyの取り組み@DataOps Night
yuto16
0
270
Modern_Data_Stack最新動向クイズ_買収_AI_激動の2025年_.pdf
sagara
0
200
AWSにおけるTrend Vision Oneの効果について
shimak
0
130
社内お問い合わせBotの仕組みと学び
nish01
0
330
AIAgentの限界を超え、 現場を動かすWorkflowAgentの設計と実践
miyatakoji
0
130
10年の共創が示す、これからの開発者と企業の関係 ~ Crossroad
soracom
PRO
1
190
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
Scaling GitHub
holman
463
140k
Gamification - CAS2011
davidbonilla
81
5.5k
Done Done
chrislema
185
16k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.2k
How to train your dragon (web standard)
notwaldorf
96
6.3k
Site-Speed That Sticks
csswizardry
11
880
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
2.6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
Building Applications with DynamoDB
mza
96
6.6k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
19
1.2k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428