Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Pipeline Architectures
Search
Michael Heap
October 03, 2015
Technology
0
640
Pipeline Architectures
Pipeline Architectures at PHPNW15
Michael Heap
October 03, 2015
Tweet
Share
More Decks by Michael Heap
See All by Michael Heap
WTF are OKRs?
mheap
0
340
But why does the business care?
mheap
0
210
Advanced GitHub Actions
mheap
0
450
Controlling Your Kong Gateway With decK and CI/CD
mheap
0
390
API Standards 2.0
mheap
1
1k
API Standards 2.0
mheap
0
520
Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)
mheap
0
580
Dr Sheldon Cooper Presents: Fun with Flags
mheap
0
950
Behat for characterization tests
mheap
0
420
Other Decks in Technology
See All in Technology
United™️ Airlines®️ Customer®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedguide
0
800
今だから言えるセキュリティLT_Wordpress5.7.2未満を一斉アップデートせよ
cuebic9bic
2
170
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
2
940
Bill One 開発エンジニア 紹介資料
sansan33
PRO
4
13k
モニタリング統一への道のり - 分散モニタリングツール統合のためのオブザーバビリティプロジェクト
niftycorp
PRO
1
520
Maintainer Meetupで「生の声」を聞く ~講演だけじゃないKubeCon
logica0419
0
110
スタックチャン家庭用アシスタントへの道
kanekoh
0
120
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing
tomzoh
2
120
無理しない AI 活用サービス / #jazug
koudaiii
0
100
サービスを止めるな! DDoS攻撃へのスマートな備えと最前線の事例
coconala_engineer
1
180
AWS CDK 入門ガイド これだけは知っておきたいヒント集
anank
5
750
大量配信システムにおけるSLOの実践:「見えない」信頼性をSLOで可視化
plaidtech
PRO
0
390
Featured
See All Featured
Navigating Team Friction
lara
187
15k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Building Applications with DynamoDB
mza
95
6.5k
Thoughts on Productivity
jonyablonski
69
4.7k
RailsConf 2023
tenderlove
30
1.1k
Testing 201, or: Great Expectations
jmmastey
43
7.6k
Bash Introduction
62gerente
613
210k
The Pragmatic Product Professional
lauravandoore
35
6.7k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Transcript
Pipeline Architectures
@mheap
Pipelines
Tea/Coffee
Waterfall
Manufacturing
None
Engine: 20 mins Frame: 40 mins Windows: 10 mins
1x Engine: 20 mins 1x Frame: 40 mins 1x Windows:
10 mins
2x Engine: 20 mins 4x Frame: 40 mins 1x Windows:
10 mins
Code Pipelines
This is a unix system I know this
cat attendees.txt
cat attendees.txt | grep Michael
cat attendees.txt | grep Michael > michaels.txt
cat attendees.txt | grep Michael | sort > michaels.txt
Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams, because that is a universal interface.
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
read(file.txt). map(u => return u.name). filter(n => return n ==
“michael”)
producer.go channel <- sum foo.go val <- channel
app.php $bus->handle( new UserRegisteredEvent )
Input Processing Output
Controller Model View
Action Domain Responder
POST /foo Domain echo $twig->render()
$ php foo.php ‘bar’ Domain echo cli_output($output);
zmqpp <your application> zmqpp
zmqpp <your application> redis-pipe
nsq <your application> nsq
Human Pipelines
+ =
None
Data ??? Profit!
Data Profit!
Data Profit!
Data Profit!
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON
Data Profit! JSON JSON JSON JSON ? ?
Data Profit! PHP PHP C++
Data Profit! PHP Golang C++
Data Profit! PHP Golang Michael
Building Pipelines
Data-Driven
Demand-Driven
Data-Driven
push/pull
Back Pressure
pub/sub
Sequence Numbers
Demand-Driven
Unique Jobs
Transport
Serialization
Error Handling
Testing Pipelines
Pros/Cons
Examples
https://www.flickr.com/photos/neotsn/2246759100/ https://www.flickr.com/photos/theenmoy/15255488518 https://www.flickr.com/photos/stuckincustoms/5109788502 https://www.flickr.com/photos/danielfoster/4725849931 https://www.flickr.com/photos/mzmo/8666989650 https://www.flickr.com/photos/usacehq/5529362681 https://www.flickr.com/photos/alpha_auer/5068482201/ https://www.flickr.com/photos/109144586@N08/14863759614/ https://www.flickr.com/photos/94418464@N08/9007117038 https://www.flickr.com/photos/swamibu/2868288357/
https://www.flickr.com/photos/sepehrehsani/5766453552/ https://www.flickr.com/photos/scubasteveo/296747958/ https://www.flickr.com/photos/56218409@N03/15371262455/ https://www.flickr.com/photos/jvk/4417106 https://www.flickr.com/photos/13523064@N03/14448815581 https://www.flickr.com/photos/jamesshade/473206000 https://www.flickr.com/photos/albertogp123/5843577306/ https://www.flickr.com/photos/dpstyles/4835354126/
Questions? @mheap joind.in/15428