Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Lessons from 6 Months of using Luigi
Search
peteowlett
May 07, 2016
Technology
1k
4
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Lessons from 6 Months of using Luigi
AKA Why it's better to be woken up by you cat than by the server alarm
peteowlett
May 07, 2016
Transcript
105 slides
Lessons from 6 months of using Luigi in production @peterowlett
@deliveroo
Hello! I’m Pete
None
I work for these folks
WE DO THIS
Why it’s better to be woken up by your cat
than by the server alarm A BETTER TITLE
This is Kitty
NO RESPECT FOR PERSONAL SPACE
This is PagerDuty
Even less respect for personal space
Let’s Compare! - Goes off at any time, day or
night - Loud ring tone, text messages, answer phone messages and flashing - Resolution can take hours - Goes off only once at precisely 6am - Cute batting motion to wake - Resolved in time it takes to open cat food packet
I think we can all agree with my premise Kitty
>> PagerDuty
Lets get to it
Chapter 1 “The Model”
Let’s build a model
I’m ready, where’s the data?
“Just pg_dump the prod db”
OH PLS PLS NO DON’T DO THAT
Lets spin up a read slave and ETL the data
to a warehouse …
… then train our models from that
How do we ensure tasks run in Order?
I want them to run one after the other 2
1 3 4 5 6 7 8
Directed Acyclic Graph
None
Enter Stage Left …
Simple Task
Postgres Loader Task
We string these together to make DAGs CHECK MAX ROW
ID LOAD DATA MOD DATA MAKE MODEL CHECK MAX ROW ID LOAD DATA TABLE1 TABLE2
DAGs solve the dependency problem
Bung it all on EC2
Define an entry point
Run the scheduler
Kick it all off with CRON
With luigi we were up and running in a few
hours
Chapter 2 “The Nuclear Option”
A few weeks later, something happened …
None
Schema can change anytime without warning HAS THE SCHEMA CHANGED?
RELOAD JUST NEW ROWS DROP AND CREATE WHOLE SCHEMA RELOAD ALL TABLES NO! YES!
None
Handle schema changes robustly
Let’s test our pipeline before we deploy it. But how?
Two new operating modes TEST MODE Run the whole pipeline
but only write to a test schema UNIT MODE Run the current task, ignoring its dependencies
Configure these modes in the pipeline using luigi.Parameter
Now nothing will ever go wrong, ever again …
Make your testing comprehensive
Make your testing fast
Adding in external API services
Build Loaders for each API
Loading Schedules
Plumbing them in
Keep def rows as short as possible
Be consistent in loader design pattern
Expect external API services to misbehave
Expect external API services to misbehave X
Trust external API services as if they actively want to
hurt you
Hey cool, all our data is in one place, we
might as well use it for BI Reporting
Chapter 3 “The Management Report”
This happened
And then your ad hoc database is now supporting global
business critical apps
None
Stuff that was happening • Irrelevant upstream failures • Low
priority upstream failures • Flakey Data (but it worked!)
Our DAG looked like this: START LOAD2 LOAD1 LOAD3 LOAD
DONE MAKE1 MAKE2 MAKE3 END
And this was happening START LOAD2 LOAD1 LOAD3 LOAD DONE
MAKE1 MAKE2 MAKE3 END
This one doesn’t need LOAD3 START LOAD2 LOAD1 LOAD3 LOAD
DONE MAKE1 MAKE2 MAKE3 END
So we changed it to this START LOAD2 LOAD1 LOAD3
LOAD ALL MAKE1 MAKE2 MAKE3 END
Now when 3 fails: START LOAD2 LOAD1 LOAD3 LOAD ALL
MAKE1 MAKE2 MAKE3 END
Decide about what can be allowed to fail, and what
can’t
Isolate the path to critical ops jobs
Loading tables more reliably 5AM SATURDAY
The currency table failed to update
Loading tables more reliably DROP TABLE CREATE TABLE LOAD DATA
Task 1 Task 2 Task 3
Loading tables more reliably DROP TABLE CREATE TABLE THIS CAN
GO WRONG LOAD DATA Task 1 Task 2 Task 3 THIS CAN GO WRONG THIS CAN GO WRONG
Expect Failure, Rollback Transaction CREATE TEMP TABLE Task 1 (There
is no task 2) LOAD DATA RENAME OLD TABLE RENAME NEW TABLE ROLLBACK
None
Encapsulate logic in bigger chunks
None
Anticipating problems early
Going beyond system monitoring
Defined Monitoring Tests
Measuring outcomes directly
And get gentler alerts in slack
Monitor (and alert on) outcomes as well as system metrics
Try / Except / Slack Alert low priority tasks
Chapter 4 “Hi, this is Australia calling …”
In the beginning there was the UK YAY! DOWNTIME!! Midnight
Midnight Midday UK Ops
Then Europe Ok cool still loads of downtime Midnight Midnight
Midday
Then Some Other Places No such thing as downtime anymore
Midnight Midnight Midday
Table Loading - Take 2 HASH THE TABLE SCHEMA COMPARE
TO LAST HASH SAME! CHANGED! DROP AND REBUILD JUST LOAD ROWS
Get rid of the nuclear option
SORRY RIPLEY
Chapter 5 “Moving to Scale”
When it comes to BI, Old School Rules Still Apply
Configuration Management (Docker + ECS)
Distributed workers make some pain go away
Protobuf3 on Message Bus
Final Thoughts
I regret nothing!
Everything is defined in code
Two people, tiny budget
Time spent speeding up the build process is time well
spent
Think carefully about what dependencies *mean*
To finish …
None
We’re hiring! Grab me after :) https://roo.it/peteo Also £5 off
your first order!
Sleep Well! @peterowlett @deliveroo Sleep Well!
More Decks by peteowlett
See All by peteowlett
Takeaway Tales
peteowlett
1
230
Send More Riders
peteowlett
4
970
Other Decks in Technology
See All in Technology
Kubernetes, Agones, WebRTC, GPU で作る クラウドゲーミング基盤を本番運用に乗せるまで
vivion
0
270
New Relicのアラート対応をClaude Codeで自動修正するワークフローを作った話
manabuseki
2
560
[NW-JAWS DAY]「うちのAWS、インターネット出られないんで」― AWS外の自社サービスをPrivateLinkでシュッと公開する方法
ykimi
3
920
現場に渡し、構造で守る AI エージェントを活用したセルフサーブのデータ基盤の運用
legalontechnologies
PRO
0
100
みん強Lightning! データ基盤ひとりでできるかな
endonanana
0
190
Amazon Bedrock AgentCore の基礎とセキュリティ・ガバナンス
ren8k
0
270
AIエージェントのアーキテクチャ設計
nsatoh
0
500
HTTP通信に使おうとすると実はめんどくさいfile_get_contents
maguroalternative
0
260
AWS Blocksで AIエージェント作れるの知ってます?
kmiya84377
1
170
【Oracle AI SPOTLIGHT ウェビナー】AI時代、狙われるデータと止められない業務をどう守るか?
oracle4engineer
PRO
1
140
認知負荷の減らし方 ~開発にも日常業務にも活かせるノウハウ~
minorun365
PRO
16
10k
AWS DevOps Agentの構造から読み解く運用 AI エージェントの設計原則/aws-devops-agent-design-principles
tomoki10
0
270
Featured
See All Featured
Navigating the moral maze — ethical principles for Al-driven product design
skipperchong
2
700
State of Search Keynote: SEO is Dead Long Live SEO
ryanjones
0
360
Agile that works and the tools we love
rasmusluckow
331
22k
Making the Leap to Tech Lead
cromwellryan
135
10k
Highjacked: Video Game Concept Design
rkendrick25
PRO
2
590
How to Align SEO within the Product Triangle To Get Buy-In & Support - #RIMC
aleyda
2
1.9k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
18k
Building Adaptive Systems
keathley
44
3.3k
Git: the NoSQL Database
bkeepers
PRO
433
67k
The AI Search Optimization Roadmap by Aleyda Solis
aleyda
2
6.5k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
33
5.9k
Automating Front-end Workflow
addyosmani
1370
210k