Upgrade to Pro — share decks privately, control downloads, hide ads and more …

In Praise of Slow Continuous Delivery

In Praise of Slow Continuous Delivery

Talk from the Pipeline Conference in London in March 2017. Covers the differences between the common perception of Continuous Delivery and the reality of shipping packaged software. Discusses patterns like dark launches, feature flags and analytics and how they are applied to this different domain.

Gareth Rushgrove

March 21, 2017

More Decks by Gareth Rushgrove

Other Decks in Technology


  1. (without introducing more risk) In Praise of Slow Continuous Delivery

    Puppet Gareth Rushgrove On-premise software and continuous delivery
  2. (without introducing more risk) @garethr

  3. (without introducing more risk) Gareth Rushgrove

  4. (without introducing more risk) What we’ll cover This talk

  5. - What do we mean when we say continuous -

    The world of Enterprise Software vendors - Why is this different? - Examples of evolving stable software Gareth Rushgrove
  6. Note Most of the examples are infrastructure tools, because I’m

    an infrastructure tools geek. The patterns and practices are more general Gareth Rushgrove
  7. (without introducing more risk) What people think of when you

    say continuous delivery What does good look like?
  8. Gareth Rushgrove

  9. Gareth Rushgrove “Continuous Delivery is the ability to get changes

    of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. Jez Humble, Continuous Delivery
  10. Gareth Rushgrove

  11. Gareth Rushgrove “10 deploys a day at Flickr John Allspaw

    and Paul Hammond, VelocityConf, 2009
  12. Gareth Rushgrove Jon Jenkins, VelocityConf, 2011 “Amazon deploys every 11.6

  13. How fast are you deploying your software today? Gareth Rushgrove

  14. (without introducing more risk) The land the internet forgot Enterprise

  15. Gareth Rushgrove

  16. But everyone wants to be an enterprise software company Gareth

  17. Gareth Rushgrove

  18. No really. Everyone wants to be an enterprise software company

    Gareth Rushgrove
  19. Gareth Rushgrove 0 50 100 150 200 ORCL IBM SAP

    VMware Market cap in billions
  20. - Puppet Enterprise ships every 3 months - Docker EE

    ships every 3 months - vSphere ships about every 6 months - Oracle ship a new major every 3 years Gareth Rushgrove
  21. Can this be continuous delivery? Gareth Rushgrove

  22. Gareth Rushgrove

  23. Gareth Rushgrove HP LaserJet Firmware Builds Commits per day Regression

    tests 1 per week 1 6 weeks 10 per day 100 24 hours 2008 2011
  24. Can this be continuous delivery? Gareth Rushgrove

  25. (without introducing more risk) Non-SaaS delivery models Why is this

  26. 1. Consuming software via pull vs push Gareth Rushgrove

  27. Gareth Rushgrove A typical web app is pushed Deployment Running

    application Users
  28. Gareth Rushgrove New versions replace old Deployment New version Users

  29. But deployment of on-premise software is disconnected from it’s consumption

    by users Gareth Rushgrove
  30. Gareth Rushgrove Deployment is disconnected Deployment Package available to download

  31. Gareth Rushgrove Deployment is disconnected Deployment New package available Old

    packages still available
  32. Gareth Rushgrove Gareth Rushgrove Consumers choose when to pull Download

    package Users Local copy per user
  33. Problem Users can choose not to update Gareth Rushgrove

  34. Problem Other people may choose to package your software if

    it’s open source Gareth Rushgrove
  35. 2. The environment permutation explosion Gareth Rushgrove

  36. How many platforms does your web applications need to run

    on? Gareth Rushgrove
  37. Gareth Rushgrove 15 supported platforms

  38. i386 and x86_64. IBM z Systems and Power. Solaris. RHEL

    4 (released 2005). 3 versions of macOS, 4 versions of AIX, 11 versions of Windows across server and desktop. Gareth Rushgrove 80 more platforms!
  39. We run about 500 pipelines a day. That’s about 35,000

    individual jobs a week. For a development team of less than 100 people Gareth Rushgrove
  40. Problem Your users control the environment not you. Limiting the

    environments you support costs you users Gareth Rushgrove
  41. 3. Say semantic versioning one more time Gareth Rushgrove

  42. What version of GitHub did I just use? Gareth Rushgrove

  43. Gareth Rushgrove What version of GitHub Enterprise did I just

  44. Gareth Rushgrove So Windows 8 is really Windows 6.2?

  45. Versions numbers are marketing too Gareth Rushgrove

  46. Docker 17.03, Ubuntu 16.04, PE 2017.1, Office 2016, Office 365,

    OpenStack Mitaka, Newton, Ocata, etc. Gareth Rushgrove
  47. (without introducing more risk) CalVer

  48. Gareth Rushgrove Constantly incrementing values

  49. (without introducing more risk) Recommended reading

  50. Problem Versioning packaged software is a wicked problem Gareth Rushgrove

    A wicked problem is a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognise.
  51. 4. How many versions of your software can you support

    at once? Gareth Rushgrove
  52. The ideal of continuous delivery is one Gareth Rushgrove

  53. How long do you support that one version? Until you

    ship the next one (maybe later the same day?) Gareth Rushgrove
  54. (without introducing more risk) RHEL support life-cycle Gareth Rushgrove

  55. (without introducing more risk) RHEL Support dates Gareth Rushgrove

  56. RHEL 4 end of life is this month. It’s been

    supported for 12 years Gareth Rushgrove
  57. Gareth Rushgrove 12 years

  58. (without introducing more risk) Docker support scheme

  59. Release quarterly, each release supported for a year means 4

    supported major versions Gareth Rushgrove
  60. Gareth Rushgrove “ Docker not supporting a version for longer

    than a year is a no-go. This is not enterprise ready. Hacker News comment
  61. Gareth Rushgrove “In large enterprise it will take 6 months

    to get a version certified to work with tooling so need 3 year at the minimum to put any effort on it. Hacker News comment
  62. Gareth Rushgrove “Docker initial release was just 4 years ago,

    your asking for a support timeframe that's around the same length as the age of the project itself. Hacker News comment
  63. Problem User expectations vary wildly across organisations, sectors and types

    of software Gareth Rushgrove
  64. 5. Licenses matter Gareth Rushgrove

  65. Who has every had to do a license audit? Gareth

  66. (without introducing more risk) Puppet Enterprise licenses

  67. Puppet Server (just one component of Puppet Enterprise) has 170

    different open source components, each with individual license terms Gareth Rushgrove
  68. In total we ship third-party software with 19 different licenses,

    from MIT, Apache2 and BSD to EDL 1.0, EPL 1.0, Artistic 1.0 and 2.0, Ruby, Boost Software License and more Gareth Rushgrove
  69. Problem Does this commit break the law? Is this release

    legal? Who has legal checks in their continuous delivery pipeline? Gareth Rushgrove
  70. (without introducing more risk) Adopting continuous delivery patterns Evolving stable

  71. 1. Feature flags and conducting experiments Gareth Rushgrove

  72. Allow users to opt-in to some functionality Gareth Rushgrove

  73. (without introducing more risk) Docker experimental flag

  74. (without introducing more risk) Puppet future parser

  75. Dark launch features and test in the background Gareth Rushgrove

  76. (without introducing more risk) Terraform experiments Gareth Rushgrove

  77. 2. Releasing every commit Gareth Rushgrove

  78. (without introducing more risk) Puppet nightly builds

  79. Gareth Rushgrove Puppet docs “Our automated systems create new “nightly”

    repositories for builds that pass our acceptance testing on the most popular platforms.
  80. (without introducing more risk) gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt

  81. The difference between being able to ship every day vs

    actually shipping every day Gareth Rushgrove
  82. 3. The importance of product analytics Gareth Rushgrove

  83. Part of Continuous Delivery is Continuous Improvement. And without data

    about usage how do you know what to improve? Gareth Rushgrove
  84. Do you expect a web application adopting continuous delivery to

    use things like: - Google Analytics - Kissmetrics - New Relic - AppDynamics - Dynatrace Gareth Rushgrove
  85. Do you expect an open source CLI based tool to

    do the same? Gareth Rushgrove
  86. (without introducing more risk) HashiCorp Checkpoint

  87. Gareth Rushgrove “Consul makes use of a HashiCorp service called

    Checkpoint which is used to check for updates and critical security bulletins. Only anonymous information, which cannot be used to identify the user or host, is sent to Checkpoint Consul FAQ
  88. Gareth Rushgrove Docker for Mac statistics

  89. (without introducing more risk) Puppet Enterprise data

  90. Gareth Rushgrove “Data like this helps us understand how you

    use the product, which in turn helps us improve the product to meet your needs How does sharing this data benefit PE users?
  91. Problem Not every one is connected to the internet Gareth

  92. (without introducing more risk) Offline data gathering

  93. Questions around packaged software, product analytics, network usage, firewalls and

    user consent can be tricky Gareth Rushgrove
  94. (without introducing more risk) If all you remember is Conclusions

  95. Enterprise user expectations clash with some of the expected implementations

    of the practices of continuous delivery Gareth Rushgrove
  96. But the practices of continuous delivery are still relevant to

    enterprise software Gareth Rushgrove
  97. Continuous Delivery is about speed Gareth Rushgrove

  98. But speed is a relative measure that depends on a

    frame of reference Gareth Rushgrove My Physics degree coming in handy
  99. Always remember to put your users and your context ahead

    of any absolute measures Gareth Rushgrove Especially those you read on the internet
  100. (without introducing more risk) Questions? And thanks for listening