Continuous Delivery at Netflix: Uberconf

Uberconf presentation of Continuous Delivery at Netflix: Speed at Scale

Transcript

1. Continuous Delivery at Netflix Speed at Scale
2. Rob Spieldenner Senior Software Engineer on Engineering Tools team at Netflix @robspieldenner -> Twitter
3. Speed breaks software
 
 Scale breaks hardware
 
 Speed at Scale breaks everything
4. Netflix Culture
5. Culture Deck
   http://jobs.netflix.com/
6. Freedom & Responsibility
7. Context, not Control
8. Highly Aligned, Loosely Coupled
9. Act in Netflix’s Best Interest
10. Self Service
11. Developers on call for their services
12. Netflix OSS
    http://netflix.github.io/
13. OODA Loop - Observe, Orient, Decide, Act
14. Commit Unit Test Build Bake Deploy To Test Integration & Acceptance Tests Red/Black Push Continuous Delivery Pipeline
15. Commit and Code Review
16. Pull Requests
17. Build simplify build file with plugins add metadata calculate version output jar/war, sources, javadoc, rpm/deb lock dynamic dependencies
18. Ant build.xml
 <project name="example">
 <property environment="env"/>
 <import file="${env.CBF_HOME}/library.xml"/>
 </project> ivy.xml list dependencies list publications

19. Gradle build.gradle
 apply plugin: ‘nebula’
 apply plugin: ‘java’
 
 dependencies {
 testCompile ‘junit:junit:4.+’
 }
20. Nebula Plugins
 http://nebula-plugins.github.io/
21. apply plugin: ‘nebula.nebula-project’ Builds Javadoc, Sources and a Test jar Record information about the build and stores it in the .jar, via gradle-info-plugin Easy specification of people involved in a project via gradle-contacts-plugin
22. What Goes in the Manifest Module-Source=/mpex-common Module-Origin=ssh://git@stash.corp.netflix.com:7999/nexmpl/multiprojectexample.git Change=767e5d7 Manifest-Version=1.0 Implementation-Title=netflix#mpex-common;3.0.0 Implementation-Version=3.0.0 Built-Status=snapshot Built-By=rspieldenner Build-Date=2014-06-12_11:24:00 Gradle-Version=1.12-20140604072223+0000 Build-Host=localhost Build-Job=LOCAL Build-Number=LOCAL Build-Id=LOCAL Created-By=1.7.0_51-b13 (Oracle Corporation) Build-Java-Version=1.7.0_51 Nebula-Version=1.48 Implementation-Vendor=Netflix, Inc. X-Compile-Target-JDK=1.7 X-Compile-Source-JDK=1.7 Module-Owner=person@netflix.com Module-Email=team@netflix.com
23. apply plugin: ‘nebula.gradle- dependency-lock’ dependencies {
 compile ‘com.google.guava:guava:14.+’
 testCompile ‘junit:junit:4.+’
 }
24. dependencies.lock {
 "com.google.guava:guava": { "locked": "14.0.1", 
 "requested": "14.+" },
 ”junit:junit": { "locked": "4.11", "requested": "4.+" }
 }

25. Jenkins
26. Job-DSL
    https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL +Plugin
27. Job DSL Example example
 def project = 'quidryan/aws-sdk-test'
 def branchApi = new URL(“https://api.github.com/repos/${project}/branches”)
 def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
 branches.each {
 def branchName = it.name
 job {
 name “${project}-${branchName}”.replaceAll(‘/’, ‘-’)
 scm {
 git(“git://github.com/${project}.git”, branchName)
 }
 steps {
 gradle(‘test’)
 }
 }
 }
28. Bake
29. aminator
    https://github.com/Netflix/aminator
30. chef for base image
31. Apply deb/rpm, snapshot
32. Unit of deployment
33. Deploy
34. Asgard
    https://github.com/Netflix/asgard
36. UI or REST API
37. Canary Deploys
38. Red / Black Push
39. Testing Webdriver Geb Betamax and manual
40. Monitoring How many people are watching? Can the new version handle load? Errors and logging suro, jmx
41. Alert on issues
42. Maybe automatic rollback
43. Resiliency / Anti-Fragility
44. Simian Army
    https://github.com/Netflix/SimianArmy Chaos Monkey Chaos Gorilla Chaos Kong Latency Monkey Janitor Monkey Conformity Monkey
45. Chaos Monkey
 randomly kill running instances
46. Chaos Gorilla take down an availability zone
47. Chaos Kong take down a region
48. Latency Monkey inject delays, timeout, latency
49. Janitor Monkey clean up unused instances
50. Conformity Monkey make sure instances follow rules
51. Assume services will break, disappear, have latency
52. Continuous Delivery Pipeline Commit Unit Test Build Bake Deploy To Test Integration & Acceptance Tests Red/Black Push
53. Future Work Mesos based workflow engine Docker containers Build Insight/Metrics
54. Books
55. Adrian Cockcroft Flowcon Keynote - Velocity and Volume (or Speed Wins) - http://www.youtube.com/ watch?v=wyWI3gLpB8o Roy Rapoport - Cloud Operations at Netflix - http:// www.youtube.com/watch?v=7779Wrun5fo AWS re:Invent - http://reinvent.awsevents.com/ recap2013.html Other Presentations
56. Netflix is Hiring
    http://jobs.netflix.com/