$30 off During Our Annual Pro Sale. View Details »

In Praise of Slow Continuous Delivery

In Praise of Slow Continuous Delivery

Talk from the Pipeline Conference in London in March 2017. Covers the differences between the common perception of Continuous Delivery and the reality of shipping packaged software. Discusses patterns like dark launches, feature flags and analytics and how they are applied to this different domain.

Gareth Rushgrove

March 21, 2017
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. (without introducing more risk)
    In Praise of Slow
    Continuous Delivery
    Puppet
    Gareth Rushgrove
    On-premise software and continuous delivery

    View Slide

  2. (without introducing more risk)
    @garethr

    View Slide

  3. (without introducing more risk)
    Gareth Rushgrove

    View Slide

  4. (without introducing more risk)
    What we’ll cover
    This talk

    View Slide

  5. - What do we mean when we say continuous
    - The world of Enterprise Software vendors
    - Why is this different?
    - Examples of evolving stable software
    Gareth Rushgrove

    View Slide

  6. Note
    Most of the examples are infrastructure tools,
    because I’m an infrastructure tools geek. The
    patterns and practices are more general
    Gareth Rushgrove

    View Slide

  7. (without introducing more risk)
    What people think of when you

    say continuous delivery
    What does good
    look like?

    View Slide

  8. Gareth Rushgrove

    View Slide

  9. Gareth Rushgrove
    “Continuous Delivery is the ability to get
    changes of all types—including new features,
    configuration changes, bug fixes and
    experiments—into production, or into the
    hands of users, safely and quickly in a
    sustainable way.
    Jez Humble, Continuous Delivery

    View Slide

  10. Gareth Rushgrove

    View Slide

  11. Gareth Rushgrove
    “10 deploys a day at Flickr
    John Allspaw and Paul Hammond, VelocityConf, 2009

    View Slide

  12. Gareth Rushgrove
    Jon Jenkins, VelocityConf, 2011
    “Amazon deploys every 11.6 seconds

    View Slide

  13. How fast are you deploying your
    software today?
    Gareth Rushgrove

    View Slide

  14. (without introducing more risk)
    The land the internet forgot
    Enterprise Software

    View Slide

  15. Gareth Rushgrove

    View Slide

  16. But everyone wants to be an enterprise
    software company
    Gareth Rushgrove

    View Slide

  17. Gareth Rushgrove

    View Slide

  18. No really. Everyone wants to be an
    enterprise software company
    Gareth Rushgrove

    View Slide

  19. Gareth Rushgrove
    0
    50
    100
    150
    200
    ORCL IBM SAP VMware
    Market cap in billions

    View Slide

  20. - Puppet Enterprise ships every 3 months
    - Docker EE ships every 3 months
    - vSphere ships about every 6 months
    - Oracle ship a new major every 3 years
    Gareth Rushgrove

    View Slide

  21. Can this be continuous delivery?
    Gareth Rushgrove

    View Slide

  22. Gareth Rushgrove

    View Slide

  23. Gareth Rushgrove
    HP LaserJet Firmware
    Builds
    Commits per day
    Regression tests
    1 per week
    1
    6 weeks
    10 per day
    100
    24 hours
    2008 2011

    View Slide

  24. Can this be continuous delivery?
    Gareth Rushgrove

    View Slide

  25. (without introducing more risk)
    Non-SaaS delivery models
    Why is this different?

    View Slide

  26. 1. Consuming software via pull vs push
    Gareth Rushgrove

    View Slide

  27. Gareth Rushgrove
    A typical web app is pushed
    Deployment Running
    application
    Users

    View Slide

  28. Gareth Rushgrove
    New versions replace old
    Deployment
    New version
    Users

    View Slide

  29. But deployment of on-premise software is
    disconnected from it’s consumption by users
    Gareth Rushgrove

    View Slide

  30. Gareth Rushgrove
    Deployment is disconnected
    Deployment
    Package available
    to download

    View Slide

  31. Gareth Rushgrove
    Deployment is disconnected
    Deployment
    New package
    available
    Old packages
    still available

    View Slide

  32. Gareth Rushgrove
    Gareth Rushgrove
    Consumers choose when to pull
    Download
    package
    Users
    Local copy
    per user

    View Slide

  33. Problem
    Users can choose not to update
    Gareth Rushgrove

    View Slide

  34. Problem
    Other people may choose to package your
    software if it’s open source
    Gareth Rushgrove

    View Slide

  35. 2. The environment permutation explosion
    Gareth Rushgrove

    View Slide

  36. How many platforms does your web
    applications need to run on?
    Gareth Rushgrove

    View Slide

  37. Gareth Rushgrove
    15 supported platforms

    View Slide

  38. i386 and x86_64. IBM z Systems and Power.
    Solaris. RHEL 4 (released 2005). 3 versions
    of macOS, 4 versions of AIX, 11 versions of
    Windows across server and desktop.
    Gareth Rushgrove
    80 more platforms!

    View Slide

  39. We run about 500 pipelines a day. That’s
    about 35,000 individual jobs a week. For a
    development team of less than 100 people
    Gareth Rushgrove

    View Slide

  40. Problem
    Your users control the environment not you.
    Limiting the environments you support costs
    you users
    Gareth Rushgrove

    View Slide

  41. 3. Say semantic versioning one more time
    Gareth Rushgrove

    View Slide

  42. What version of GitHub did I just use?
    Gareth Rushgrove

    View Slide

  43. Gareth Rushgrove
    What version of GitHub Enterprise did I just use?

    View Slide

  44. Gareth Rushgrove
    So Windows 8 is really Windows 6.2?

    View Slide

  45. Versions numbers are marketing too
    Gareth Rushgrove

    View Slide

  46. Docker 17.03, Ubuntu 16.04, PE 2017.1,
    Office 2016, Office 365, OpenStack Mitaka,
    Newton, Ocata, etc.
    Gareth Rushgrove

    View Slide

  47. (without introducing more risk)
    CalVer

    View Slide

  48. Gareth Rushgrove
    Constantly incrementing values

    View Slide

  49. (without introducing more risk)
    Recommended reading

    View Slide

  50. Problem
    Versioning packaged software is
    a wicked problem
    Gareth Rushgrove
    A wicked problem is a problem that is difficult or impossible to solve because of
    incomplete, contradictory, and changing requirements that are often difficult to recognise.

    View Slide

  51. 4. How many versions of your software can
    you support at once?
    Gareth Rushgrove

    View Slide

  52. The ideal of continuous delivery is one
    Gareth Rushgrove

    View Slide

  53. How long do you support that one version?
    Until you ship the next one (maybe later the
    same day?)
    Gareth Rushgrove

    View Slide

  54. (without introducing more risk)
    RHEL support life-cycle
    Gareth Rushgrove

    View Slide

  55. (without introducing more risk)
    RHEL Support dates
    Gareth Rushgrove

    View Slide

  56. RHEL 4 end of life is this month. It’s been
    supported for 12 years
    Gareth Rushgrove

    View Slide

  57. Gareth Rushgrove
    12 years

    View Slide

  58. (without introducing more risk)
    Docker support scheme

    View Slide

  59. Release quarterly, each release supported
    for a year means 4 supported major versions
    Gareth Rushgrove

    View Slide

  60. Gareth Rushgrove

    Docker not supporting a version for longer
    than a year is a no-go. This is not
    enterprise ready.
    Hacker News comment

    View Slide

  61. Gareth Rushgrove
    “In large enterprise it will take 6 months to
    get a version certified to work with tooling
    so need 3 year at the minimum to put any
    effort on it.
    Hacker News comment

    View Slide

  62. Gareth Rushgrove
    “Docker initial release was just 4 years ago,
    your asking for a support timeframe that's
    around the same length as the age of the
    project itself.
    Hacker News comment

    View Slide

  63. Problem
    User expectations vary wildly across
    organisations, sectors and types of software
    Gareth Rushgrove

    View Slide

  64. 5. Licenses matter
    Gareth Rushgrove

    View Slide

  65. Who has every had to do a license audit?
    Gareth Rushgrove

    View Slide

  66. (without introducing more risk)
    Puppet Enterprise licenses

    View Slide

  67. Puppet Server (just one component of Puppet
    Enterprise) has 170 different open source
    components, each with individual license terms
    Gareth Rushgrove

    View Slide

  68. In total we ship third-party software with 19
    different licenses, from MIT, Apache2 and
    BSD to EDL 1.0, EPL 1.0, Artistic 1.0 and 2.0,
    Ruby, Boost Software License and more
    Gareth Rushgrove

    View Slide

  69. Problem
    Does this commit break the law? Is this
    release legal? Who has legal checks in
    their continuous delivery pipeline?
    Gareth Rushgrove

    View Slide

  70. (without introducing more risk)
    Adopting continuous delivery patterns
    Evolving stable
    software

    View Slide

  71. 1. Feature flags and conducting experiments
    Gareth Rushgrove

    View Slide

  72. Allow users to opt-in to some functionality
    Gareth Rushgrove

    View Slide

  73. (without introducing more risk)
    Docker experimental flag

    View Slide

  74. (without introducing more risk)
    Puppet future parser

    View Slide

  75. Dark launch features and test
    in the background
    Gareth Rushgrove

    View Slide

  76. (without introducing more risk)
    Terraform experiments
    Gareth Rushgrove

    View Slide

  77. 2. Releasing every commit
    Gareth Rushgrove

    View Slide

  78. (without introducing more risk)
    Puppet nightly builds

    View Slide

  79. Gareth Rushgrove
    Puppet docs
    “Our automated systems create new
    “nightly” repositories for builds that pass
    our acceptance testing on the most
    popular platforms.

    View Slide

  80. (without introducing more risk)
    gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt

    View Slide

  81. The difference between being able to ship
    every day vs actually shipping every day
    Gareth Rushgrove

    View Slide

  82. 3. The importance of product analytics
    Gareth Rushgrove

    View Slide

  83. Part of Continuous Delivery is Continuous
    Improvement. And without data about usage
    how do you know what to improve?
    Gareth Rushgrove

    View Slide

  84. Do you expect a web application adopting
    continuous delivery to use things like:
    - Google Analytics
    - Kissmetrics
    - New Relic
    - AppDynamics
    - Dynatrace
    Gareth Rushgrove

    View Slide

  85. Do you expect an open source CLI based
    tool to do the same?
    Gareth Rushgrove

    View Slide

  86. (without introducing more risk)
    HashiCorp Checkpoint

    View Slide

  87. Gareth Rushgrove
    “Consul makes use of a HashiCorp service
    called Checkpoint which is used to check
    for updates and critical security bulletins.
    Only anonymous information, which cannot
    be used to identify the user or host, is sent
    to Checkpoint
    Consul FAQ

    View Slide

  88. Gareth Rushgrove
    Docker for Mac statistics

    View Slide

  89. (without introducing more risk)
    Puppet Enterprise data

    View Slide

  90. Gareth Rushgrove
    “Data like this helps us understand how you
    use the product, which in turn helps us
    improve the product to meet your needs
    How does sharing this data benefit PE users?

    View Slide

  91. Problem
    Not every one is connected to the internet
    Gareth Rushgrove

    View Slide

  92. (without introducing more risk)
    Offline data gathering

    View Slide

  93. Questions around packaged software,
    product analytics, network usage, firewalls
    and user consent can be tricky
    Gareth Rushgrove

    View Slide

  94. (without introducing more risk)
    If all you remember is
    Conclusions

    View Slide

  95. Enterprise user expectations clash with
    some of the expected implementations of
    the practices of continuous delivery
    Gareth Rushgrove

    View Slide

  96. But the practices of continuous delivery
    are still relevant to enterprise software
    Gareth Rushgrove

    View Slide

  97. Continuous Delivery is about speed
    Gareth Rushgrove

    View Slide

  98. But speed is a relative measure that
    depends on a frame of reference
    Gareth Rushgrove
    My Physics degree coming in handy

    View Slide

  99. Always remember to put your users
    and your context ahead of any
    absolute measures
    Gareth Rushgrove
    Especially those you read on the internet

    View Slide

  100. (without introducing more risk)
    Questions?
    And thanks for listening

    View Slide