Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Continuous Delivery at Netflix: Uberconf
Search
Rob Spieldenner
June 27, 2014
Programming
0
210
Continuous Delivery at Netflix: Uberconf
Uberconf presentation of Continuous Delivery at Netflix: Speed at Scale
Rob Spieldenner
June 27, 2014
Tweet
Share
More Decks by Rob Spieldenner
See All by Rob Spieldenner
Migrating Nebula Plugins to the New Model and TestKit
rspieldenner
1
140
Nebula: Netflix's OSS Gradle Plugins (Bay Area Gradle Users Meetup)
rspieldenner
0
110
Nebula: Netflix's OSS Gradle Plugins
rspieldenner
0
770
Continuous Delivery at Netflix
rspieldenner
3
250
Other Decks in Programming
See All in Programming
Go言語の特性を活かした公式MCP SDKの設計
hond0413
1
210
Web フロントエンドエンジニアに開かれる AI Agent プロダクト開発 - Vercel AI SDK を観察して AI Agent と仲良くなろう! #FEC余熱NIGHT
izumin5210
3
490
Goで実践するドメイン駆動開発 AIと歩み始めた新規プロダクト開発の現在地
imkaoru
4
790
Introducing ReActionView: A new ActionView-Compatible ERB Engine @ Kaigi on Rails 2025, Tokyo, Japan
marcoroth
3
980
階層構造を表現するデータ構造とリファクタリング 〜1年で10倍成長したプロダクトの変化と課題〜
yuhisatoxxx
3
970
CSC305 Lecture 03
javiergs
PRO
0
240
理論と実務のギャップを超える
eycjur
0
120
Go Conference 2025: Goで体感するMultipath TCP ― Go 1.24 時代の MPTCP Listener を理解する
takehaya
8
1.6k
Web Components で実現する Hotwire とフロントエンドフレームワークの橋渡し / Bridging with Web Components
da1chi
3
2k
SpecKitでどこまでできる? コストはどれくらい?
leveragestech
0
660
Catch Up: Go Style Guide Update
andpad
0
210
uniqueパッケージの内部実装を支えるweak pointerの話
magavel
0
960
Featured
See All Featured
The Pragmatic Product Professional
lauravandoore
36
6.9k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
45
2.5k
Six Lessons from altMBA
skipperchong
28
4k
Building Adaptive Systems
keathley
43
2.8k
Rails Girls Zürich Keynote
gr2m
95
14k
Building an army of robots
kneath
306
46k
Statistics for Hackers
jakevdp
799
220k
Into the Great Unknown - MozCon
thekraken
40
2.1k
RailsConf 2023
tenderlove
30
1.2k
A Modern Web Designer's Workflow
chriscoyier
697
190k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Transcript
Continuous Delivery at Netflix Speed at Scale
Rob Spieldenner Senior Software Engineer on Engineering Tools team at
Netflix @robspieldenner -> Twitter
Speed breaks software Scale breaks hardware Speed at
Scale breaks everything
Netflix Culture
Culture Deck http://jobs.netflix.com/
Freedom & Responsibility
Context, not Control
Highly Aligned, Loosely Coupled
Act in Netflix’s Best Interest
Self Service
Developers on call for their services
Netflix OSS http://netflix.github.io/
OODA Loop - Observe, Orient, Decide, Act
Commit Unit Test Build Bake Deploy To Test Integration &
Acceptance Tests Red/Black Push Continuous Delivery Pipeline
Commit and Code Review
Pull Requests
Build simplify build file with plugins add metadata calculate version
output jar/war, sources, javadoc, rpm/deb lock dynamic dependencies
Ant build.xml <project name="example"> <property environment="env"/> <import file="${env.CBF_HOME}/library.xml"/> </project> ivy.xml
list dependencies list publications
Gradle build.gradle apply plugin: ‘nebula’ apply plugin: ‘java’ dependencies
{ testCompile ‘junit:junit:4.+’ }
Nebula Plugins http://nebula-plugins.github.io/
apply plugin: ‘nebula.nebula-project’ Builds Javadoc, Sources and a Test jar
Record information about the build and stores it in the .jar, via gradle-info-plugin Easy specification of people involved in a project via gradle-contacts-plugin
What Goes in the Manifest Module-Source=/mpex-common Module-Origin=ssh://
[email protected]
flix.com:7999/nexmpl/multiprojectexample.git Change=767e5d7 Manifest-Version=1.0 Implementation-Title=netflix#mpex-common;3.0.0
Implementation-Version=3.0.0 Built-Status=snapshot Built-By=rspieldenner Build-Date=2014-06-12_11:24:00 Gradle-Version=1.12-20140604072223+0000 Build-Host=localhost Build-Job=LOCAL Build-Number=LOCAL Build-Id=LOCAL Created-By=1.7.0_51-b13 (Oracle Corporation) Build-Java-Version=1.7.0_51 Nebula-Version=1.48 Implementation-Vendor=Netflix, Inc. X-Compile-Target-JDK=1.7 X-Compile-Source-JDK=1.7 Module-Owner=person@netflix.com Module-Email=team@netflix.com
apply plugin: ‘nebula.gradle- dependency-lock’ dependencies { compile ‘com.google.guava:guava:14.+’ testCompile ‘junit:junit:4.+’
}
dependencies.lock { "com.google.guava:guava": { "locked": "14.0.1", "requested": "14.+" },
”junit:junit": { "locked": "4.11", "requested": "4.+" } }
Jenkins
Job-DSL https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL +Plugin
Job DSL Example example def project = 'quidryan/aws-sdk-test' def branchApi
= new URL(“https://api.github.com/repos/${project}/branches”) def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader()) branches.each { def branchName = it.name job { name “${project}-${branchName}”.replaceAll(‘/’, ‘-’) scm { git(“git://github.com/${project}.git”, branchName) } steps { gradle(‘test’) } } }
Bake
aminator https://github.com/Netflix/aminator
chef for base image
Apply deb/rpm, snapshot
Unit of deployment
Deploy
Asgard https://github.com/Netflix/asgard
None
UI or REST API
Canary Deploys
Red / Black Push
Testing Webdriver Geb Betamax and manual
Monitoring How many people are watching? Can the new version
handle load? Errors and logging suro, jmx
Alert on issues
Maybe automatic rollback
Resiliency / Anti-Fragility
Simian Army https://github.com/Netflix/SimianArmy Chaos Monkey Chaos Gorilla Chaos Kong Latency
Monkey Janitor Monkey Conformity Monkey
Chaos Monkey randomly kill running instances
Chaos Gorilla take down an availability zone
Chaos Kong take down a region
Latency Monkey inject delays, timeout, latency
Janitor Monkey clean up unused instances
Conformity Monkey make sure instances follow rules
Assume services will break, disappear, have latency
Continuous Delivery Pipeline Commit Unit Test Build Bake Deploy To
Test Integration & Acceptance Tests Red/Black Push
Future Work Mesos based workflow engine Docker containers Build Insight/Metrics
Books
Adrian Cockcroft Flowcon Keynote - Velocity and Volume (or Speed
Wins) - http://www.youtube.com/ watch?v=wyWI3gLpB8o Roy Rapoport - Cloud Operations at Netflix - http:// www.youtube.com/watch?v=7779Wrun5fo AWS re:Invent - http://reinvent.awsevents.com/ recap2013.html Other Presentations
Netflix is Hiring http://jobs.netflix.com/