Continuous Delivery at Netflix: Uberconf

Transcript

1. 1.

   Continuous Delivery at Netflix Speed at Scale

2. 2.

   Rob Spieldenner Senior Software Engineer on Engineering Tools team at

   Netflix @robspieldenner -> Twitter
3. 3.

   Speed breaks software
 
 Scale breaks hardware
 
 Speed at

   Scale breaks everything
4. 4.

   Netflix Culture

5. 5.

   Culture Deck
   http://jobs.netflix.com/

6. 6.

   Freedom & Responsibility

7. 7.

   Context, not Control

8. 8.

   Highly Aligned, Loosely Coupled

9. 9.

   Act in Netflix’s Best Interest

10. 10.

    Self Service

11. 11.

    Developers on call for their services

12. 12.

    Netflix OSS
    http://netflix.github.io/

13. 13.

    OODA Loop - Observe, Orient, Decide, Act

14. 14.

    Commit Unit Test Build Bake Deploy To Test Integration &

    Acceptance Tests Red/Black Push Continuous Delivery Pipeline
15. 15.

    Commit and Code Review

16. 16.

    Pull Requests

17. 17.

    Build simplify build file with plugins add metadata calculate version

    output jar/war, sources, javadoc, rpm/deb lock dynamic dependencies
18. 18.

    Ant build.xml
 <project name="example">
 <property environment="env"/>
 <import file="${env.CBF_HOME}/library.xml"/>
 </project> ivy.xml

    list dependencies list publications

19. 19.

    Gradle build.gradle
 apply plugin: ‘nebula’
 apply plugin: ‘java’
 
 dependencies

    {
 testCompile ‘junit:junit:4.+’
 }
20. 20.

    Nebula Plugins
 http://nebula-plugins.github.io/

21. 21.

    apply plugin: ‘nebula.nebula-project’ Builds Javadoc, Sources and a Test jar

    Record information about the build and stores it in the .jar, via gradle-info-plugin Easy specification of people involved in a project via gradle-contacts-plugin
22. 22.

    What Goes in the Manifest Module-Source=/mpex-common Module-Origin=ssh://git@stash.corp.netflix.com:7999/nexmpl/multiprojectexample.git Change=767e5d7 Manifest-Version=1.0 Implementation-Title=netflix#mpex-common;3.0.0

    Implementation-Version=3.0.0 Built-Status=snapshot Built-By=rspieldenner Build-Date=2014-06-12_11:24:00 Gradle-Version=1.12-20140604072223+0000 Build-Host=localhost Build-Job=LOCAL Build-Number=LOCAL Build-Id=LOCAL Created-By=1.7.0_51-b13 (Oracle Corporation) Build-Java-Version=1.7.0_51 Nebula-Version=1.48 Implementation-Vendor=Netflix, Inc. X-Compile-Target-JDK=1.7 X-Compile-Source-JDK=1.7 Module-Owner=person@netflix.com Module-Email=team@netflix.com
23. 23.

    apply plugin: ‘nebula.gradle- dependency-lock’ dependencies {
 compile ‘com.google.guava:guava:14.+’
 testCompile ‘junit:junit:4.+’


    }
24. 24.

    dependencies.lock {
 "com.google.guava:guava": { "locked": "14.0.1", 
 "requested": "14.+" },


    ”junit:junit": { "locked": "4.11", "requested": "4.+" }
 }

25. 25.

    Jenkins

26. 26.

    Job-DSL
    https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL +Plugin

27. 27.

    Job DSL Example example
 def project = 'quidryan/aws-sdk-test'
 def branchApi

    = new URL(“https://api.github.com/repos/${project}/branches”)
 def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
 branches.each {
 def branchName = it.name
 job {
 name “${project}-${branchName}”.replaceAll(‘/’, ‘-’)
 scm {
 git(“git://github.com/${project}.git”, branchName)
 }
 steps {
 gradle(‘test’)
 }
 }
 }
28. 28.

    Bake

29. 29.

    aminator
    https://github.com/Netflix/aminator

30. 30.

    chef for base image

31. 31.

    Apply deb/rpm, snapshot

32. 32.

    Unit of deployment

33. 33.

    Deploy

34. 34.

    Asgard
    https://github.com/Netflix/asgard

35. 35.

    None
36. 36.

    UI or REST API

37. 37.

    Canary Deploys

38. 38.

    Red / Black Push

39. 39.

    Testing Webdriver Geb Betamax and manual

40. 40.

    Monitoring How many people are watching? Can the new version

    handle load? Errors and logging suro, jmx
41. 41.

    Alert on issues

42. 42.

    Maybe automatic rollback

43. 43.

    Resiliency / Anti-Fragility

44. 44.

    Simian Army
    https://github.com/Netflix/SimianArmy Chaos Monkey Chaos Gorilla Chaos Kong Latency

    Monkey Janitor Monkey Conformity Monkey
45. 45.

    Chaos Monkey
 randomly kill running instances

46. 46.

    Chaos Gorilla take down an availability zone

47. 47.

    Chaos Kong take down a region

48. 48.

    Latency Monkey inject delays, timeout, latency

49. 49.

    Janitor Monkey clean up unused instances

50. 50.

    Conformity Monkey make sure instances follow rules

51. 51.

    Assume services will break, disappear, have latency

52. 52.

    Continuous Delivery Pipeline Commit Unit Test Build Bake Deploy To

    Test Integration & Acceptance Tests Red/Black Push
53. 53.

    Future Work Mesos based workflow engine Docker containers Build Insight/Metrics

54. 54.

    Books

55. 55.

    Adrian Cockcroft Flowcon Keynote - Velocity and Volume (or Speed

    Wins) - http://www.youtube.com/ watch?v=wyWI3gLpB8o Roy Rapoport - Cloud Operations at Netflix - http:// www.youtube.com/watch?v=7779Wrun5fo AWS re:Invent - http://reinvent.awsevents.com/ recap2013.html Other Presentations
56. 56.

    Netflix is Hiring
    http://jobs.netflix.com/