(without introducing more risk)
In Praise of Slow
Continuous Delivery
Puppet
Gareth Rushgrove
On-premise software and continuous delivery
Slide 2
Slide 2 text
(without introducing more risk)
@garethr
Slide 3
Slide 3 text
(without introducing more risk)
Gareth Rushgrove
Slide 4
Slide 4 text
(without introducing more risk)
What we’ll cover
This talk
Slide 5
Slide 5 text
- What do we mean when we say continuous
- The world of Enterprise Software vendors
- Why is this different?
- Examples of evolving stable software
Gareth Rushgrove
Slide 6
Slide 6 text
Note
Most of the examples are infrastructure tools,
because I’m an infrastructure tools geek. The
patterns and practices are more general
Gareth Rushgrove
Slide 7
Slide 7 text
(without introducing more risk)
What people think of when you
say continuous delivery
What does good
look like?
Slide 8
Slide 8 text
Gareth Rushgrove
Slide 9
Slide 9 text
Gareth Rushgrove
“Continuous Delivery is the ability to get
changes of all types—including new features,
configuration changes, bug fixes and
experiments—into production, or into the
hands of users, safely and quickly in a
sustainable way.
Jez Humble, Continuous Delivery
Slide 10
Slide 10 text
Gareth Rushgrove
Slide 11
Slide 11 text
Gareth Rushgrove
“10 deploys a day at Flickr
John Allspaw and Paul Hammond, VelocityConf, 2009
Slide 12
Slide 12 text
Gareth Rushgrove
Jon Jenkins, VelocityConf, 2011
“Amazon deploys every 11.6 seconds
Slide 13
Slide 13 text
How fast are you deploying your
software today?
Gareth Rushgrove
Slide 14
Slide 14 text
(without introducing more risk)
The land the internet forgot
Enterprise Software
Slide 15
Slide 15 text
Gareth Rushgrove
Slide 16
Slide 16 text
But everyone wants to be an enterprise
software company
Gareth Rushgrove
Slide 17
Slide 17 text
Gareth Rushgrove
Slide 18
Slide 18 text
No really. Everyone wants to be an
enterprise software company
Gareth Rushgrove
Slide 19
Slide 19 text
Gareth Rushgrove
0
50
100
150
200
ORCL IBM SAP VMware
Market cap in billions
Slide 20
Slide 20 text
- Puppet Enterprise ships every 3 months
- Docker EE ships every 3 months
- vSphere ships about every 6 months
- Oracle ship a new major every 3 years
Gareth Rushgrove
Slide 21
Slide 21 text
Can this be continuous delivery?
Gareth Rushgrove
Slide 22
Slide 22 text
Gareth Rushgrove
Slide 23
Slide 23 text
Gareth Rushgrove
HP LaserJet Firmware
Builds
Commits per day
Regression tests
1 per week
1
6 weeks
10 per day
100
24 hours
2008 2011
Slide 24
Slide 24 text
Can this be continuous delivery?
Gareth Rushgrove
Slide 25
Slide 25 text
(without introducing more risk)
Non-SaaS delivery models
Why is this different?
Slide 26
Slide 26 text
1. Consuming software via pull vs push
Gareth Rushgrove
Slide 27
Slide 27 text
Gareth Rushgrove
A typical web app is pushed
Deployment Running
application
Users
Slide 28
Slide 28 text
Gareth Rushgrove
New versions replace old
Deployment
New version
Users
Slide 29
Slide 29 text
But deployment of on-premise software is
disconnected from it’s consumption by users
Gareth Rushgrove
Slide 30
Slide 30 text
Gareth Rushgrove
Deployment is disconnected
Deployment
Package available
to download
Slide 31
Slide 31 text
Gareth Rushgrove
Deployment is disconnected
Deployment
New package
available
Old packages
still available
Slide 32
Slide 32 text
Gareth Rushgrove
Gareth Rushgrove
Consumers choose when to pull
Download
package
Users
Local copy
per user
Slide 33
Slide 33 text
Problem
Users can choose not to update
Gareth Rushgrove
Slide 34
Slide 34 text
Problem
Other people may choose to package your
software if it’s open source
Gareth Rushgrove
Slide 35
Slide 35 text
2. The environment permutation explosion
Gareth Rushgrove
Slide 36
Slide 36 text
How many platforms does your web
applications need to run on?
Gareth Rushgrove
Slide 37
Slide 37 text
Gareth Rushgrove
15 supported platforms
Slide 38
Slide 38 text
i386 and x86_64. IBM z Systems and Power.
Solaris. RHEL 4 (released 2005). 3 versions
of macOS, 4 versions of AIX, 11 versions of
Windows across server and desktop.
Gareth Rushgrove
80 more platforms!
Slide 39
Slide 39 text
We run about 500 pipelines a day. That’s
about 35,000 individual jobs a week. For a
development team of less than 100 people
Gareth Rushgrove
Slide 40
Slide 40 text
Problem
Your users control the environment not you.
Limiting the environments you support costs
you users
Gareth Rushgrove
Slide 41
Slide 41 text
3. Say semantic versioning one more time
Gareth Rushgrove
Slide 42
Slide 42 text
What version of GitHub did I just use?
Gareth Rushgrove
Slide 43
Slide 43 text
Gareth Rushgrove
What version of GitHub Enterprise did I just use?
Slide 44
Slide 44 text
Gareth Rushgrove
So Windows 8 is really Windows 6.2?
Slide 45
Slide 45 text
Versions numbers are marketing too
Gareth Rushgrove
Slide 46
Slide 46 text
Docker 17.03, Ubuntu 16.04, PE 2017.1,
Office 2016, Office 365, OpenStack Mitaka,
Newton, Ocata, etc.
Gareth Rushgrove
Slide 47
Slide 47 text
(without introducing more risk)
CalVer
Slide 48
Slide 48 text
Gareth Rushgrove
Constantly incrementing values
Slide 49
Slide 49 text
(without introducing more risk)
Recommended reading
Slide 50
Slide 50 text
Problem
Versioning packaged software is
a wicked problem
Gareth Rushgrove
A wicked problem is a problem that is difficult or impossible to solve because of
incomplete, contradictory, and changing requirements that are often difficult to recognise.
Slide 51
Slide 51 text
4. How many versions of your software can
you support at once?
Gareth Rushgrove
Slide 52
Slide 52 text
The ideal of continuous delivery is one
Gareth Rushgrove
Slide 53
Slide 53 text
How long do you support that one version?
Until you ship the next one (maybe later the
same day?)
Gareth Rushgrove
Slide 54
Slide 54 text
(without introducing more risk)
RHEL support life-cycle
Gareth Rushgrove
Slide 55
Slide 55 text
(without introducing more risk)
RHEL Support dates
Gareth Rushgrove
Slide 56
Slide 56 text
RHEL 4 end of life is this month. It’s been
supported for 12 years
Gareth Rushgrove
Slide 57
Slide 57 text
Gareth Rushgrove
12 years
Slide 58
Slide 58 text
(without introducing more risk)
Docker support scheme
Slide 59
Slide 59 text
Release quarterly, each release supported
for a year means 4 supported major versions
Gareth Rushgrove
Slide 60
Slide 60 text
Gareth Rushgrove
“
Docker not supporting a version for longer
than a year is a no-go. This is not
enterprise ready.
Hacker News comment
Slide 61
Slide 61 text
Gareth Rushgrove
“In large enterprise it will take 6 months to
get a version certified to work with tooling
so need 3 year at the minimum to put any
effort on it.
Hacker News comment
Slide 62
Slide 62 text
Gareth Rushgrove
“Docker initial release was just 4 years ago,
your asking for a support timeframe that's
around the same length as the age of the
project itself.
Hacker News comment
Slide 63
Slide 63 text
Problem
User expectations vary wildly across
organisations, sectors and types of software
Gareth Rushgrove
Slide 64
Slide 64 text
5. Licenses matter
Gareth Rushgrove
Slide 65
Slide 65 text
Who has every had to do a license audit?
Gareth Rushgrove
Slide 66
Slide 66 text
(without introducing more risk)
Puppet Enterprise licenses
Slide 67
Slide 67 text
Puppet Server (just one component of Puppet
Enterprise) has 170 different open source
components, each with individual license terms
Gareth Rushgrove
Slide 68
Slide 68 text
In total we ship third-party software with 19
different licenses, from MIT, Apache2 and
BSD to EDL 1.0, EPL 1.0, Artistic 1.0 and 2.0,
Ruby, Boost Software License and more
Gareth Rushgrove
Slide 69
Slide 69 text
Problem
Does this commit break the law? Is this
release legal? Who has legal checks in
their continuous delivery pipeline?
Gareth Rushgrove
1. Feature flags and conducting experiments
Gareth Rushgrove
Slide 72
Slide 72 text
Allow users to opt-in to some functionality
Gareth Rushgrove
Slide 73
Slide 73 text
(without introducing more risk)
Docker experimental flag
Slide 74
Slide 74 text
(without introducing more risk)
Puppet future parser
Slide 75
Slide 75 text
Dark launch features and test
in the background
Gareth Rushgrove
Slide 76
Slide 76 text
(without introducing more risk)
Terraform experiments
Gareth Rushgrove
Slide 77
Slide 77 text
2. Releasing every commit
Gareth Rushgrove
Slide 78
Slide 78 text
(without introducing more risk)
Puppet nightly builds
Slide 79
Slide 79 text
Gareth Rushgrove
Puppet docs
“Our automated systems create new
“nightly” repositories for builds that pass
our acceptance testing on the most
popular platforms.
Slide 80
Slide 80 text
(without introducing more risk)
gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt
Slide 81
Slide 81 text
The difference between being able to ship
every day vs actually shipping every day
Gareth Rushgrove
Slide 82
Slide 82 text
3. The importance of product analytics
Gareth Rushgrove
Slide 83
Slide 83 text
Part of Continuous Delivery is Continuous
Improvement. And without data about usage
how do you know what to improve?
Gareth Rushgrove
Slide 84
Slide 84 text
Do you expect a web application adopting
continuous delivery to use things like:
- Google Analytics
- Kissmetrics
- New Relic
- AppDynamics
- Dynatrace
Gareth Rushgrove
Slide 85
Slide 85 text
Do you expect an open source CLI based
tool to do the same?
Gareth Rushgrove
Slide 86
Slide 86 text
(without introducing more risk)
HashiCorp Checkpoint
Slide 87
Slide 87 text
Gareth Rushgrove
“Consul makes use of a HashiCorp service
called Checkpoint which is used to check
for updates and critical security bulletins.
Only anonymous information, which cannot
be used to identify the user or host, is sent
to Checkpoint
Consul FAQ
Slide 88
Slide 88 text
Gareth Rushgrove
Docker for Mac statistics
Slide 89
Slide 89 text
(without introducing more risk)
Puppet Enterprise data
Slide 90
Slide 90 text
Gareth Rushgrove
“Data like this helps us understand how you
use the product, which in turn helps us
improve the product to meet your needs
How does sharing this data benefit PE users?
Slide 91
Slide 91 text
Problem
Not every one is connected to the internet
Gareth Rushgrove
Slide 92
Slide 92 text
(without introducing more risk)
Offline data gathering
Slide 93
Slide 93 text
Questions around packaged software,
product analytics, network usage, firewalls
and user consent can be tricky
Gareth Rushgrove
Slide 94
Slide 94 text
(without introducing more risk)
If all you remember is
Conclusions
Slide 95
Slide 95 text
Enterprise user expectations clash with
some of the expected implementations of
the practices of continuous delivery
Gareth Rushgrove
Slide 96
Slide 96 text
But the practices of continuous delivery
are still relevant to enterprise software
Gareth Rushgrove
Slide 97
Slide 97 text
Continuous Delivery is about speed
Gareth Rushgrove
Slide 98
Slide 98 text
But speed is a relative measure that
depends on a frame of reference
Gareth Rushgrove
My Physics degree coming in handy
Slide 99
Slide 99 text
Always remember to put your users
and your context ahead of any
absolute measures
Gareth Rushgrove
Especially those you read on the internet
Slide 100
Slide 100 text
(without introducing more risk)
Questions?
And thanks for listening