Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Rapid Release at Massive Scale
Search
Proko Chou
November 06, 2017
Technology
0
79
Rapid Release at Massive Scale
The study of Facebook's release architect: Rapid Release at Massive Scale
Proko Chou
November 06, 2017
Tweet
Share
More Decks by Proko Chou
See All by Proko Chou
Software Testing
prokochou
0
72
Introduce ISTQB Test
prokochou
0
68
Context Driven Testing
prokochou
0
74
Mobile App Testing
prokochou
0
57
Mining Patterns from Lyrics of Chinese Popular Music
prokochou
0
150
Other Decks in Technology
See All in Technology
AI時代にも変わらぬ価値を発揮したい: インフラ・クラウドを切り口にユーザー価値と非機能要件に向き合ってエンジニアとしての地力を培う
netmarkjp
0
220
PHPでResult型やってみよう
higaki_program
0
180
AI Ready API ─ AI時代に求められるAPI設計とは?/ AI-Ready API - Designing MCP and APIs in the AI Era
yokawasa
20
5.7k
Data Engineering Study#30 LT資料
tetsuroito
1
550
ゼロから始めるSREの事業貢献 - 生成AI時代のSRE成長戦略と実践 / Starting SRE from Day One
shinyorke
PRO
0
230
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
18k
Bliki (ja), and the Cathedral, and the Bazaar
koic
8
1.3k
Recoil脱却の現状と挑戦
kirik
2
230
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
6
520
Railsの限界を超えろ!「家族アルバム みてね」の画像・動画の大規模アップロードを支えるアーキテクチャの変遷
ojima_h
3
370
AI エンジニアの立場からみた、AI コーディング時代の開発の品質向上の取り組みと妄想
soh9834
5
130
Maintainer Meetupで「生の声」を聞く ~講演だけじゃないKubeCon
logica0419
1
160
Featured
See All Featured
The Language of Interfaces
destraynor
158
25k
A designer walks into a library…
pauljervisheath
207
24k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
Unsuck your backbone
ammeep
671
58k
Become a Pro
speakerdeck
PRO
29
5.4k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
KATA
mclloyd
30
14k
Building Adaptive Systems
keathley
43
2.7k
Typedesign – Prime Four
hannesfritz
42
2.7k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
BBQ
matthewcrist
89
9.7k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Transcript
Rapid Release at Massive Scale 20171106 Proko
About Author
Agenda • Canary Deployment • The reasons why adopt rapid
release at massive scale • Solutions • Limitations
Canary Deployment
Canary Deployment (cont.) • Canaries were once regularly used in
coal mining as an early warning system • Distributed to a small number of users • As an early warning indicator with less impact on downtime
Canary Deployment • Deploy to one or more canary servers
• Test, or wait until satisfied • Deploy to the remaining servers
Why
Previous deployment process Three times a day Cherry-picks
Weekly Web Branch
Time Zone Issue Release duration: 14 hours every single week
London New York Menlo Park
How
Quasi-continuous “push from master” system
Tool and Process
Improve tools and process
Continuous Integration
Gatekeeper System
Test Tool Deployment (cont.)
Test Tool Deployment (cont.)
Test Tool Deployment (cont.)
Test Tool Deployment (cont.)
Test Tool Deployment
Continuous Delivery at Scale
Continuous Delivery at Scale (cont.)
Continuous Delivery at Scale: C1 (internal)
Canary Deployment Continuous Delivery at Scale: C2 (production)
Continuous Delivery at Scale: C3
Result
System
No More Hotfix
Deploy anytime, anywhere All engineers everywhere in the world can
develop and deliver their code when it makes sense for them.
Forcing Function
Make the user experience better, faster
Limitation
None
None
Android OS Version
Solution
What’s more • Use Puffin for Facebook (For Facebook) •
Add more checking point which can be executed automatically • The result can be checked by developers if it is failed
Reference 1. Article: https://code.facebook.com/posts/270314900139291/rapid-release-at-mas sive-scale/ 2. Video: https://atscaleconference.com/videos/rapid-release-at-massive-scale/ 3. Canary
Deployment: https://blog.ccjeng.com/2015/06/canary-deployment.html https://octopus.com/docs/patterns/canary-deployments
Q&A