Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Continuous Delivery at Netflix: Uberconf
Search
Rob Spieldenner
June 27, 2014
Programming
0
210
Continuous Delivery at Netflix: Uberconf
Uberconf presentation of Continuous Delivery at Netflix: Speed at Scale
Rob Spieldenner
June 27, 2014
Tweet
Share
More Decks by Rob Spieldenner
See All by Rob Spieldenner
Migrating Nebula Plugins to the New Model and TestKit
rspieldenner
1
140
Nebula: Netflix's OSS Gradle Plugins (Bay Area Gradle Users Meetup)
rspieldenner
0
110
Nebula: Netflix's OSS Gradle Plugins
rspieldenner
0
770
Continuous Delivery at Netflix
rspieldenner
3
250
Other Decks in Programming
See All in Programming
私はどうやって技術力を上げたのか
yusukebe
43
19k
Devvox Belgium - Agentic AI Patterns
kdubois
1
120
作って理解するGOCACHEPROG / Go Conference 2025(Workshop)
mazrean
0
100
Le côté obscur des IA génératives
pascallemerrer
0
150
ALL CODE BASE ARE BELONG TO STUDY
uzulla
25
6k
Go Conference 2025: Goで体感するMultipath TCP ― Go 1.24 時代の MPTCP Listener を理解する
takehaya
9
1.7k
私達はmodernize packageに夢を見るか feat. go/analysis, go/ast / Go Conference 2025
kaorumuta
2
570
Foundation Modelsを実装日本語学習アプリを作ってみた!
hypebeans
0
110
その面倒な作業、「Dart」にやらせませんか? Flutter開発者のための業務効率化
yordgenome03
1
130
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
410
組込みだけじゃない!TinyGo で始める無料クラウド開発入門
otakakot
0
280
Things You Thought You Didn’t Need To Care About That Have a Big Impact On Your Job
hollycummins
0
230
Featured
See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
189
55k
Optimising Largest Contentful Paint
csswizardry
37
3.5k
How STYLIGHT went responsive
nonsquared
100
5.8k
Making Projects Easy
brettharned
120
6.4k
Side Projects
sachag
455
43k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
For a Future-Friendly Web
brad_frost
180
10k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
35
6.1k
Documentation Writing (for coders)
carmenintech
75
5.1k
Balancing Empowerment & Direction
lara
4
690
KATA
mclloyd
32
15k
Designing for humans not robots
tammielis
254
26k
Transcript
Continuous Delivery at Netflix Speed at Scale
Rob Spieldenner Senior Software Engineer on Engineering Tools team at
Netflix @robspieldenner -> Twitter
Speed breaks software Scale breaks hardware Speed at
Scale breaks everything
Netflix Culture
Culture Deck http://jobs.netflix.com/
Freedom & Responsibility
Context, not Control
Highly Aligned, Loosely Coupled
Act in Netflix’s Best Interest
Self Service
Developers on call for their services
Netflix OSS http://netflix.github.io/
OODA Loop - Observe, Orient, Decide, Act
Commit Unit Test Build Bake Deploy To Test Integration &
Acceptance Tests Red/Black Push Continuous Delivery Pipeline
Commit and Code Review
Pull Requests
Build simplify build file with plugins add metadata calculate version
output jar/war, sources, javadoc, rpm/deb lock dynamic dependencies
Ant build.xml <project name="example"> <property environment="env"/> <import file="${env.CBF_HOME}/library.xml"/> </project> ivy.xml
list dependencies list publications
Gradle build.gradle apply plugin: ‘nebula’ apply plugin: ‘java’ dependencies
{ testCompile ‘junit:junit:4.+’ }
Nebula Plugins http://nebula-plugins.github.io/
apply plugin: ‘nebula.nebula-project’ Builds Javadoc, Sources and a Test jar
Record information about the build and stores it in the .jar, via gradle-info-plugin Easy specification of people involved in a project via gradle-contacts-plugin
What Goes in the Manifest Module-Source=/mpex-common Module-Origin=ssh://
[email protected]
flix.com:7999/nexmpl/multiprojectexample.git Change=767e5d7 Manifest-Version=1.0 Implementation-Title=netflix#mpex-common;3.0.0
Implementation-Version=3.0.0 Built-Status=snapshot Built-By=rspieldenner Build-Date=2014-06-12_11:24:00 Gradle-Version=1.12-20140604072223+0000 Build-Host=localhost Build-Job=LOCAL Build-Number=LOCAL Build-Id=LOCAL Created-By=1.7.0_51-b13 (Oracle Corporation) Build-Java-Version=1.7.0_51 Nebula-Version=1.48 Implementation-Vendor=Netflix, Inc. X-Compile-Target-JDK=1.7 X-Compile-Source-JDK=1.7 Module-Owner=person@netflix.com Module-Email=team@netflix.com
apply plugin: ‘nebula.gradle- dependency-lock’ dependencies { compile ‘com.google.guava:guava:14.+’ testCompile ‘junit:junit:4.+’
}
dependencies.lock { "com.google.guava:guava": { "locked": "14.0.1", "requested": "14.+" },
”junit:junit": { "locked": "4.11", "requested": "4.+" } }
Jenkins
Job-DSL https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL +Plugin
Job DSL Example example def project = 'quidryan/aws-sdk-test' def branchApi
= new URL(“https://api.github.com/repos/${project}/branches”) def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader()) branches.each { def branchName = it.name job { name “${project}-${branchName}”.replaceAll(‘/’, ‘-’) scm { git(“git://github.com/${project}.git”, branchName) } steps { gradle(‘test’) } } }
Bake
aminator https://github.com/Netflix/aminator
chef for base image
Apply deb/rpm, snapshot
Unit of deployment
Deploy
Asgard https://github.com/Netflix/asgard
None
UI or REST API
Canary Deploys
Red / Black Push
Testing Webdriver Geb Betamax and manual
Monitoring How many people are watching? Can the new version
handle load? Errors and logging suro, jmx
Alert on issues
Maybe automatic rollback
Resiliency / Anti-Fragility
Simian Army https://github.com/Netflix/SimianArmy Chaos Monkey Chaos Gorilla Chaos Kong Latency
Monkey Janitor Monkey Conformity Monkey
Chaos Monkey randomly kill running instances
Chaos Gorilla take down an availability zone
Chaos Kong take down a region
Latency Monkey inject delays, timeout, latency
Janitor Monkey clean up unused instances
Conformity Monkey make sure instances follow rules
Assume services will break, disappear, have latency
Continuous Delivery Pipeline Commit Unit Test Build Bake Deploy To
Test Integration & Acceptance Tests Red/Black Push
Future Work Mesos based workflow engine Docker containers Build Insight/Metrics
Books
Adrian Cockcroft Flowcon Keynote - Velocity and Volume (or Speed
Wins) - http://www.youtube.com/ watch?v=wyWI3gLpB8o Roy Rapoport - Cloud Operations at Netflix - http:// www.youtube.com/watch?v=7779Wrun5fo AWS re:Invent - http://reinvent.awsevents.com/ recap2013.html Other Presentations
Netflix is Hiring http://jobs.netflix.com/