In Praise of Slow Continuous Delivery

Talk from the Pipeline Conference in London in March 2017. Covers the differences between the common perception of Continuous Delivery and the reality of shipping packaged software. Discusses patterns like dark launches, feature flags and analytics and how they are applied to this different domain.

Transcript

1. (without introducing more risk) In Praise of Slow Continuous Delivery Puppet Gareth Rushgrove On-premise software and continuous delivery
2. (without introducing more risk) @garethr
3. (without introducing more risk) Gareth Rushgrove
4. (without introducing more risk) What we’ll cover This talk
5. - What do we mean when we say continuous - The world of Enterprise Software vendors - Why is this different? - Examples of evolving stable software Gareth Rushgrove
6. Note Most of the examples are infrastructure tools, because I’m an infrastructure tools geek. The patterns and practices are more general Gareth Rushgrove
7. (without introducing more risk) What people think of when you say continuous delivery What does good look like?
8. Gareth Rushgrove
9. Gareth Rushgrove “Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. Jez Humble, Continuous Delivery
10. Gareth Rushgrove
11. Gareth Rushgrove “10 deploys a day at Flickr John Allspaw and Paul Hammond, VelocityConf, 2009
12. Gareth Rushgrove Jon Jenkins, VelocityConf, 2011 “Amazon deploys every 11.6 seconds
13. How fast are you deploying your software today? Gareth Rushgrove
14. (without introducing more risk) The land the internet forgot Enterprise Software
15. Gareth Rushgrove
16. But everyone wants to be an enterprise software company Gareth Rushgrove
17. Gareth Rushgrove
18. No really. Everyone wants to be an enterprise software company Gareth Rushgrove
19. Gareth Rushgrove 0 50 100 150 200 ORCL IBM SAP VMware Market cap in billions
20. - Puppet Enterprise ships every 3 months - Docker EE ships every 3 months - vSphere ships about every 6 months - Oracle ship a new major every 3 years Gareth Rushgrove
21. Can this be continuous delivery? Gareth Rushgrove
22. Gareth Rushgrove
23. Gareth Rushgrove HP LaserJet Firmware Builds Commits per day Regression tests 1 per week 1 6 weeks 10 per day 100 24 hours 2008 2011
24. Can this be continuous delivery? Gareth Rushgrove
25. (without introducing more risk) Non-SaaS delivery models Why is this different?
26. 1. Consuming software via pull vs push Gareth Rushgrove
27. Gareth Rushgrove A typical web app is pushed Deployment Running application Users
28. Gareth Rushgrove New versions replace old Deployment New version Users
29. But deployment of on-premise software is disconnected from it’s consumption by users Gareth Rushgrove
30. Gareth Rushgrove Deployment is disconnected Deployment Package available to download
31. Gareth Rushgrove Deployment is disconnected Deployment New package available Old packages still available
32. Gareth Rushgrove Gareth Rushgrove Consumers choose when to pull Download package Users Local copy per user
33. Problem Users can choose not to update Gareth Rushgrove
34. Problem Other people may choose to package your software if it’s open source Gareth Rushgrove
35. 2. The environment permutation explosion Gareth Rushgrove
36. How many platforms does your web applications need to run on? Gareth Rushgrove
37. Gareth Rushgrove 15 supported platforms
38. i386 and x86_64. IBM z Systems and Power. Solaris. RHEL 4 (released 2005). 3 versions of macOS, 4 versions of AIX, 11 versions of Windows across server and desktop. Gareth Rushgrove 80 more platforms!
39. We run about 500 pipelines a day. That’s about 35,000 individual jobs a week. For a development team of less than 100 people Gareth Rushgrove
40. Problem Your users control the environment not you. Limiting the environments you support costs you users Gareth Rushgrove
41. 3. Say semantic versioning one more time Gareth Rushgrove
42. What version of GitHub did I just use? Gareth Rushgrove
43. Gareth Rushgrove What version of GitHub Enterprise did I just use?
44. Gareth Rushgrove So Windows 8 is really Windows 6.2?
45. Versions numbers are marketing too Gareth Rushgrove
46. Docker 17.03, Ubuntu 16.04, PE 2017.1, Office 2016, Office 365, OpenStack Mitaka, Newton, Ocata, etc. Gareth Rushgrove
47. (without introducing more risk) CalVer
48. Gareth Rushgrove Constantly incrementing values
49. (without introducing more risk) Recommended reading
50. Problem Versioning packaged software is a wicked problem Gareth Rushgrove A wicked problem is a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognise.
51. 4. How many versions of your software can you support at once? Gareth Rushgrove
52. The ideal of continuous delivery is one Gareth Rushgrove
53. How long do you support that one version? Until you ship the next one (maybe later the same day?) Gareth Rushgrove
54. (without introducing more risk) RHEL support life-cycle Gareth Rushgrove
55. (without introducing more risk) RHEL Support dates Gareth Rushgrove
56. RHEL 4 end of life is this month. It’s been supported for 12 years Gareth Rushgrove
57. Gareth Rushgrove 12 years
58. (without introducing more risk) Docker support scheme
59. Release quarterly, each release supported for a year means 4 supported major versions Gareth Rushgrove
60. Gareth Rushgrove “ Docker not supporting a version for longer than a year is a no-go. This is not enterprise ready. Hacker News comment
61. Gareth Rushgrove “In large enterprise it will take 6 months to get a version certified to work with tooling so need 3 year at the minimum to put any effort on it. Hacker News comment
62. Gareth Rushgrove “Docker initial release was just 4 years ago, your asking for a support timeframe that's around the same length as the age of the project itself. Hacker News comment
63. Problem User expectations vary wildly across organisations, sectors and types of software Gareth Rushgrove
64. 5. Licenses matter Gareth Rushgrove
65. Who has every had to do a license audit? Gareth Rushgrove
66. (without introducing more risk) Puppet Enterprise licenses
67. Puppet Server (just one component of Puppet Enterprise) has 170 different open source components, each with individual license terms Gareth Rushgrove
68. In total we ship third-party software with 19 different licenses, from MIT, Apache2 and BSD to EDL 1.0, EPL 1.0, Artistic 1.0 and 2.0, Ruby, Boost Software License and more Gareth Rushgrove
69. Problem Does this commit break the law? Is this release legal? Who has legal checks in their continuous delivery pipeline? Gareth Rushgrove
70. (without introducing more risk) Adopting continuous delivery patterns Evolving stable software
71. 1. Feature flags and conducting experiments Gareth Rushgrove
72. Allow users to opt-in to some functionality Gareth Rushgrove
73. (without introducing more risk) Docker experimental flag
74. (without introducing more risk) Puppet future parser
75. Dark launch features and test in the background Gareth Rushgrove
76. (without introducing more risk) Terraform experiments Gareth Rushgrove
77. 2. Releasing every commit Gareth Rushgrove
78. (without introducing more risk) Puppet nightly builds
79. Gareth Rushgrove Puppet docs “Our automated systems create new “nightly” repositories for builds that pass our acceptance testing on the most popular platforms.
80. (without introducing more risk) gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt
81. The difference between being able to ship every day vs actually shipping every day Gareth Rushgrove
82. 3. The importance of product analytics Gareth Rushgrove
83. Part of Continuous Delivery is Continuous Improvement. And without data about usage how do you know what to improve? Gareth Rushgrove
84. Do you expect a web application adopting continuous delivery to use things like: - Google Analytics - Kissmetrics - New Relic - AppDynamics - Dynatrace Gareth Rushgrove
85. Do you expect an open source CLI based tool to do the same? Gareth Rushgrove
86. (without introducing more risk) HashiCorp Checkpoint
87. Gareth Rushgrove “Consul makes use of a HashiCorp service called Checkpoint which is used to check for updates and critical security bulletins. Only anonymous information, which cannot be used to identify the user or host, is sent to Checkpoint Consul FAQ
88. Gareth Rushgrove Docker for Mac statistics
89. (without introducing more risk) Puppet Enterprise data
90. Gareth Rushgrove “Data like this helps us understand how you use the product, which in turn helps us improve the product to meet your needs How does sharing this data benefit PE users?
91. Problem Not every one is connected to the internet Gareth Rushgrove
92. (without introducing more risk) Offline data gathering
93. Questions around packaged software, product analytics, network usage, firewalls and user consent can be tricky Gareth Rushgrove
94. (without introducing more risk) If all you remember is Conclusions
95. Enterprise user expectations clash with some of the expected implementations of the practices of continuous delivery Gareth Rushgrove
96. But the practices of continuous delivery are still relevant to enterprise software Gareth Rushgrove
97. Continuous Delivery is about speed Gareth Rushgrove
98. But speed is a relative measure that depends on a frame of reference Gareth Rushgrove My Physics degree coming in handy
99. Always remember to put your users and your context ahead of any absolute measures Gareth Rushgrove Especially those you read on the internet
100. (without introducing more risk) Questions? And thanks for listening