In Praise of Slow Continuous Delivery

Talk from the Pipeline Conference in London in March 2017. Covers the differences between the common perception of Continuous Delivery and the reality of shipping packaged software. Discusses patterns like dark launches, feature flags and analytics and how they are applied to this different domain.

Transcript

1. (without introducing more risk)
   In Praise of Slow
   Continuous Delivery
   Puppet
   Gareth Rushgrove
   On-premise software and continuous delivery
   

   View Slide

2. (without introducing more risk)
   @garethr
   

   View Slide

3. (without introducing more risk)
   Gareth Rushgrove
   

   View Slide

4. (without introducing more risk)
   What we’ll cover
   This talk
   

   View Slide

5. - What do we mean when we say continuous
   - The world of Enterprise Software vendors
   - Why is this different?
   - Examples of evolving stable software
   Gareth Rushgrove
   

   View Slide

6. Note
   Most of the examples are infrastructure tools,
   because I’m an infrastructure tools geek. The
   patterns and practices are more general
   Gareth Rushgrove
   

   View Slide

7. (without introducing more risk)
   What people think of when you
   
   say continuous delivery
   What does good
   look like?
   

   View Slide

8. Gareth Rushgrove
   

   View Slide

9. Gareth Rushgrove
   “Continuous Delivery is the ability to get
   changes of all types—including new features,
   configuration changes, bug fixes and
   experiments—into production, or into the
   hands of users, safely and quickly in a
   sustainable way.
   Jez Humble, Continuous Delivery
   

   View Slide

10. Gareth Rushgrove
    

    View Slide

11. Gareth Rushgrove
    “10 deploys a day at Flickr
    John Allspaw and Paul Hammond, VelocityConf, 2009
    

    View Slide

12. Gareth Rushgrove
    Jon Jenkins, VelocityConf, 2011
    “Amazon deploys every 11.6 seconds
    

    View Slide

13. How fast are you deploying your
    software today?
    Gareth Rushgrove
    

    View Slide

14. (without introducing more risk)
    The land the internet forgot
    Enterprise Software
    

    View Slide

15. Gareth Rushgrove
    

    View Slide

16. But everyone wants to be an enterprise
    software company
    Gareth Rushgrove
    

    View Slide

17. Gareth Rushgrove
    

    View Slide

18. No really. Everyone wants to be an
    enterprise software company
    Gareth Rushgrove
    

    View Slide

19. Gareth Rushgrove
    0
    50
    100
    150
    200
    ORCL IBM SAP VMware
    Market cap in billions
    

    View Slide

20. - Puppet Enterprise ships every 3 months
    - Docker EE ships every 3 months
    - vSphere ships about every 6 months
    - Oracle ship a new major every 3 years
    Gareth Rushgrove
    

    View Slide

21. Can this be continuous delivery?
    Gareth Rushgrove
    

    View Slide

22. Gareth Rushgrove
    

    View Slide

23. Gareth Rushgrove
    HP LaserJet Firmware
    Builds
    Commits per day
    Regression tests
    1 per week
    1
    6 weeks
    10 per day
    100
    24 hours
    2008 2011
    

    View Slide

24. Can this be continuous delivery?
    Gareth Rushgrove
    

    View Slide

25. (without introducing more risk)
    Non-SaaS delivery models
    Why is this different?
    

    View Slide

26. 1. Consuming software via pull vs push
    Gareth Rushgrove
    

    View Slide

27. Gareth Rushgrove
    A typical web app is pushed
    Deployment Running
    application
    Users
    

    View Slide

28. Gareth Rushgrove
    New versions replace old
    Deployment
    New version
    Users
    

    View Slide

29. But deployment of on-premise software is
    disconnected from it’s consumption by users
    Gareth Rushgrove
    

    View Slide

30. Gareth Rushgrove
    Deployment is disconnected
    Deployment
    Package available
    to download
    

    View Slide

31. Gareth Rushgrove
    Deployment is disconnected
    Deployment
    New package
    available
    Old packages
    still available
    

    View Slide

32. Gareth Rushgrove
    Gareth Rushgrove
    Consumers choose when to pull
    Download
    package
    Users
    Local copy
    per user
    

    View Slide

33. Problem
    Users can choose not to update
    Gareth Rushgrove
    

    View Slide

34. Problem
    Other people may choose to package your
    software if it’s open source
    Gareth Rushgrove
    

    View Slide

35. 2. The environment permutation explosion
    Gareth Rushgrove
    

    View Slide

36. How many platforms does your web
    applications need to run on?
    Gareth Rushgrove
    

    View Slide

37. Gareth Rushgrove
    15 supported platforms
    

    View Slide

38. i386 and x86_64. IBM z Systems and Power.
    Solaris. RHEL 4 (released 2005). 3 versions
    of macOS, 4 versions of AIX, 11 versions of
    Windows across server and desktop.
    Gareth Rushgrove
    80 more platforms!
    

    View Slide

39. We run about 500 pipelines a day. That’s
    about 35,000 individual jobs a week. For a
    development team of less than 100 people
    Gareth Rushgrove
    

    View Slide

40. Problem
    Your users control the environment not you.
    Limiting the environments you support costs
    you users
    Gareth Rushgrove
    

    View Slide

41. 3. Say semantic versioning one more time
    Gareth Rushgrove
    

    View Slide

42. What version of GitHub did I just use?
    Gareth Rushgrove
    

    View Slide

43. Gareth Rushgrove
    What version of GitHub Enterprise did I just use?
    

    View Slide

44. Gareth Rushgrove
    So Windows 8 is really Windows 6.2?
    

    View Slide

45. Versions numbers are marketing too
    Gareth Rushgrove
    

    View Slide

46. Docker 17.03, Ubuntu 16.04, PE 2017.1,
    Office 2016, Office 365, OpenStack Mitaka,
    Newton, Ocata, etc.
    Gareth Rushgrove
    

    View Slide

47. (without introducing more risk)
    CalVer
    

    View Slide

48. Gareth Rushgrove
    Constantly incrementing values
    

    View Slide

49. (without introducing more risk)
    Recommended reading
    

    View Slide

50. Problem
    Versioning packaged software is
    a wicked problem
    Gareth Rushgrove
    A wicked problem is a problem that is difficult or impossible to solve because of
    incomplete, contradictory, and changing requirements that are often difficult to recognise.
    

    View Slide

51. 4. How many versions of your software can
    you support at once?
    Gareth Rushgrove
    

    View Slide

52. The ideal of continuous delivery is one
    Gareth Rushgrove
    

    View Slide

53. How long do you support that one version?
    Until you ship the next one (maybe later the
    same day?)
    Gareth Rushgrove
    

    View Slide

54. (without introducing more risk)
    RHEL support life-cycle
    Gareth Rushgrove
    

    View Slide

55. (without introducing more risk)
    RHEL Support dates
    Gareth Rushgrove
    

    View Slide

56. RHEL 4 end of life is this month. It’s been
    supported for 12 years
    Gareth Rushgrove
    

    View Slide

57. Gareth Rushgrove
    12 years
    

    View Slide

58. (without introducing more risk)
    Docker support scheme
    

    View Slide

59. Release quarterly, each release supported
    for a year means 4 supported major versions
    Gareth Rushgrove
    

    View Slide

60. Gareth Rushgrove
    “
    Docker not supporting a version for longer
    than a year is a no-go. This is not
    enterprise ready.
    Hacker News comment
    

    View Slide

61. Gareth Rushgrove
    “In large enterprise it will take 6 months to
    get a version certified to work with tooling
    so need 3 year at the minimum to put any
    effort on it.
    Hacker News comment
    

    View Slide

62. Gareth Rushgrove
    “Docker initial release was just 4 years ago,
    your asking for a support timeframe that's
    around the same length as the age of the
    project itself.
    Hacker News comment
    

    View Slide

63. Problem
    User expectations vary wildly across
    organisations, sectors and types of software
    Gareth Rushgrove
    

    View Slide

64. 5. Licenses matter
    Gareth Rushgrove
    

    View Slide

65. Who has every had to do a license audit?
    Gareth Rushgrove
    

    View Slide

66. (without introducing more risk)
    Puppet Enterprise licenses
    

    View Slide

67. Puppet Server (just one component of Puppet
    Enterprise) has 170 different open source
    components, each with individual license terms
    Gareth Rushgrove
    

    View Slide

68. In total we ship third-party software with 19
    different licenses, from MIT, Apache2 and
    BSD to EDL 1.0, EPL 1.0, Artistic 1.0 and 2.0,
    Ruby, Boost Software License and more
    Gareth Rushgrove
    

    View Slide

69. Problem
    Does this commit break the law? Is this
    release legal? Who has legal checks in
    their continuous delivery pipeline?
    Gareth Rushgrove
    

    View Slide

70. (without introducing more risk)
    Adopting continuous delivery patterns
    Evolving stable
    software
    

    View Slide

71. 1. Feature flags and conducting experiments
    Gareth Rushgrove
    

    View Slide

72. Allow users to opt-in to some functionality
    Gareth Rushgrove
    

    View Slide

73. (without introducing more risk)
    Docker experimental flag
    

    View Slide

74. (without introducing more risk)
    Puppet future parser
    

    View Slide

75. Dark launch features and test
    in the background
    Gareth Rushgrove
    

    View Slide

76. (without introducing more risk)
    Terraform experiments
    Gareth Rushgrove
    

    View Slide

77. 2. Releasing every commit
    Gareth Rushgrove
    

    View Slide

78. (without introducing more risk)
    Puppet nightly builds
    

    View Slide

79. Gareth Rushgrove
    Puppet docs
    “Our automated systems create new
    “nightly” repositories for builds that pass
    our acceptance testing on the most
    popular platforms.
    

    View Slide

80. (without introducing more risk)
    gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt
    

    View Slide

81. The difference between being able to ship
    every day vs actually shipping every day
    Gareth Rushgrove
    

    View Slide

82. 3. The importance of product analytics
    Gareth Rushgrove
    

    View Slide

83. Part of Continuous Delivery is Continuous
    Improvement. And without data about usage
    how do you know what to improve?
    Gareth Rushgrove
    

    View Slide

84. Do you expect a web application adopting
    continuous delivery to use things like:
    - Google Analytics
    - Kissmetrics
    - New Relic
    - AppDynamics
    - Dynatrace
    Gareth Rushgrove
    

    View Slide

85. Do you expect an open source CLI based
    tool to do the same?
    Gareth Rushgrove
    

    View Slide

86. (without introducing more risk)
    HashiCorp Checkpoint
    

    View Slide

87. Gareth Rushgrove
    “Consul makes use of a HashiCorp service
    called Checkpoint which is used to check
    for updates and critical security bulletins.
    Only anonymous information, which cannot
    be used to identify the user or host, is sent
    to Checkpoint
    Consul FAQ
    

    View Slide

88. Gareth Rushgrove
    Docker for Mac statistics
    

    View Slide

89. (without introducing more risk)
    Puppet Enterprise data
    

    View Slide

90. Gareth Rushgrove
    “Data like this helps us understand how you
    use the product, which in turn helps us
    improve the product to meet your needs
    How does sharing this data benefit PE users?
    

    View Slide

91. Problem
    Not every one is connected to the internet
    Gareth Rushgrove
    

    View Slide

92. (without introducing more risk)
    Offline data gathering
    

    View Slide

93. Questions around packaged software,
    product analytics, network usage, firewalls
    and user consent can be tricky
    Gareth Rushgrove
    

    View Slide

94. (without introducing more risk)
    If all you remember is
    Conclusions
    

    View Slide

95. Enterprise user expectations clash with
    some of the expected implementations of
    the practices of continuous delivery
    Gareth Rushgrove
    

    View Slide

96. But the practices of continuous delivery
    are still relevant to enterprise software
    Gareth Rushgrove
    

    View Slide

97. Continuous Delivery is about speed
    Gareth Rushgrove
    

    View Slide

98. But speed is a relative measure that
    depends on a frame of reference
    Gareth Rushgrove
    My Physics degree coming in handy
    

    View Slide

99. Always remember to put your users
    and your context ahead of any
    absolute measures
    Gareth Rushgrove
    Especially those you read on the internet
    

    View Slide

100. (without introducing more risk)
     Questions?
     And thanks for listening
     

     View Slide