Faster, safer, smarter releases with data-driven feedback @davekarow Title: A Layered Approach to Progressive Delivery: Build Your Way to Faster, Safer, Smarter Releases Abstract: Progressive Delivery is the practice of decoupling deploy from release, allowing changes to be safely pushed all the way to production and verified there *before* releasing to users. Selectively dialing up and down the exposure of code in production without a new deploy, rollback, or hotfix is the foundation of Progressive Delivery, but the higher-level benefits of safety and fast feedback come from layering practices on top of that. Key Takeaways: Whether you are new to Progressive Delivery or are already practicing some aspect of it, you'll learn/refresh on the basics and then come away with a powerful model for layering higher-value benefits on top of that foundation: -> Decouple Deploy from Release o Incremental Feature Development o Testing In Production o Kill Switch (big red button instead of hotfixes and rollbacks) -> Automate Guardrails/Do No Harm Metrics o Alert on Exception / Performance (early in rollout)

o “Limit The Blast Radius” (without manual heroics) -> Measure Release Impact o Boost Team Pride o Avoid the Feature Factory Trap -> Test To Learn (A/B Testing) o Taking Bigger Risks Safely o Learning Faster With Less Waste (painted door experiments, dynamic config) Bio: Dave has three decades of experience in developer tools, developer communities, and evangelizing sustainable software delivery practices. He has held programming, product management, and product marketing roles at Sun Microsystems, Gupta Technologies, Remedy Software, Marimba, Keynote Systems (Dynatrace), SOASTA, and BlazeMeter. Dave’s current passion is demystifying progressive delivery, especially the ways it enables better outcomes by removing constraints and building-in feedback loops.

BarCampManchesterUk - 40 empty pizza boxes (David Graham) https://www.flickr.com/photos/davidistesting/ https://creativecommons.org/licenses/by-nd/2.0/ Do release nights still make sense? I remember release WEEKS, building box + docs software 25 years ago. Bunching up changes and releasing them all at once is likely to lead to problems, which is why there’s a late at night pizza-fueled all-hands on deck to release nights.

Three decades of experience in developer tools, developer communities, and evangelizing sustainable software delivery practices. Programming, product management, and product marketing roles at Sun Microsystems, Gupta Technologies, Remedy Software, Marimba, Keynote Systems (Dynatrace), SOASTA, and BlazeMeter. I Don’t Think So! @davekarow I don’t miss release nights!

A Layered Approach to Progressive Delivery Progressive Delivery Defined 01 Examples In The Wild The Foundation: Decouple Deploy from Release The Upper Layers: Data-Driven Feedback, Automated 02 03 04 @davekarow

Progressive Delivery Defined @davekarow 01

https://blog.csanchez.org/2019/01/22/progressive-delivery-in-kubernetes-blue-green-and-canary-deployments/ @davekarow

The Roots of “Progressive Delivery” “Well, when we’re rolling out services. What we do is progressive experimentation because what really matters is the blast radius. How many people will be affected when we roll that service out and what can we learn from them?” Sam Guckenheimer, quoted in https://www.infoq.com/presentations/progressive-delivery/ @SamGuckenheimer @monkchips (James Governor) (James Governor) @davekarow

The Roots of “Progressive Delivery” @monkchips (James Governor) ...a new basket of skills and technologies concerned with modern software development, testing and deployment. @davekarow

Examples in the Wild @davekarow 02

Example: Walmart EXPO Clockwise from top: https://www.infoq.com/news/2019/05/spark-streaming-walmart/ https://www.split.io/blog/know-your-why-experimentation-and-progressive-delivery-at- walmart-grocery/ https://medium.com/walmartlabs/the-journey-of-a-b-testing-at-walmartlabs-3b9bba8e5 58f

Example: LinkedIn LiX Clockwise from top: https://engineering.linkedin.com/blog/2020/making-the-linkedin-experimentation-engin e-20x-faster https://engineering.linkedin.com/blog/2017/06/site-speed-monitoring-in-a-b-testing-an d-feature-ramp-up https://engineering.linkedin.com/blog/2020/making-the-linkedin-experimentation-engin e-20x-faster

The Foundation: Decouple Deploy from Release @davekarow 03

Four Shades of Progressive Delivery (How You Roll Matters) Approach Benefits Blue/Green Deployment Canary Release (container based) Feature Flags Feature Flags + Data, Integrated Avoid Downtime Limit The Blast Radius Limit WIP / Achieve Flow Learn During The Process https://www.split.io/blog/learn-the-four-shades-of-progressive-delivery/ Harvey Balls by Sschulte at English Wikipedia [CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)] @davekarow

Feature Flag Like a dimmer switch for changes 0% 10% 20% 50% 100% @davekarow

What a Feature Flag Looks Like In Code treatment = flags.getTreatment(“related-posts”); if (treatment == “on”) { // show related posts } else { // skip it } @davekarow

What a Feature Flag Looks Like In Code treatment = flags.getTreatment(“search-algorithm”); if (treatment == “v1”) { // use v1 of new search algorithm } else if (feature == “v2”) { // use v2 of new search algorithm } else { // use existing search algorithm } @davekarow

Decouple Deploy From Release ● Incremental Feature Development for Flow ● Testing In Production ● Kill Switch (big red button) @davekarow 1. Incremental Feature Development a. Build smaller pieces that can be completed, tested, shipped faster b. Decouple dependencies c. Dark launches, phased migrations 2. Test in Production a. Lead Time: Time between commit and validation b. Stakeholder review in prod (no surprises) before first customer sees it c. Beta Test 3. Kill Switch: the instant “big red button” a. No hotfix b. Zero minute rollback w/o scramble

The Upper Layers: Data-Driven Feedback, Automated @davekarow 04

DEPLOY @davekarow Why Automate Data-Driven Feedback? A Different Way to “Ship” Becomes Possible

DEPLOY ERROR MITIGATION @davekarow Why Automate Data-Driven Feedback? A Different Way to “Ship” Becomes Possible

DEPLOY ERROR MITIGATION MEASURE @davekarow Why Automate Data-Driven Feedback? A Different Way to “Ship” Becomes Possible

DEPLOY ERROR MITIGATION MEASURE SCALE MITIGATION @davekarow Why Automate Data-Driven Feedback? A Different Way to “Ship” Becomes Possible

DEPLOY ERROR MITIGATION MEASURE SCALE MITIGATION RELEASE @davekarow Why Automate Data-Driven Feedback? A Different Way to “Ship” Becomes Possible

Heads Up! Not Everything Is As It Seems @davekarow

Problem To Solve: Separating Signal From Noise ● Product changes ● Marketing campaigns ● Global Pandemics ● Nice Weather @davekarow

Solution: Cancel Out External Influence With a Stats Engine (Think noise cancelling headphones, but for your metrics) 50% 50% @davekarow To go beyond correlation and look for causality, science has provided us with the randomized controlled trial. By enabling the change at random, and measuring the behavior of both the exposed and unexposed traffic we are able to distribute the effects of outside factors between the two samples Thus attributing any difference in behavior to the change itself.

Decouple Deploy From Release With Feature Flags Automate Guardrails / Do-No-Harm Metrics ● Incremental Feature Development for Flow ● Testing In Production ● Kill Switch (big red button) ● Alert on Exception / Performance Early In Rollout ● “Limit The Blast Radius” w/o Manual Heroics @davekarow 1. When do you push the big red button? a. We all have dashboards and telemetry b. Global stats vs stats on each feature’s rollout c. Correlation vs Causation 2. Limit The Blast Radius a. Instead of flipping a switch and blowing everything up, what if you could only expose the new thing to a few people and detect the issue long before the rest of users are hit with it? b. Gradual rollout + stats comparison between the old and new code path c. Observing capacity to scale & predicting scaling needs based on smaller %’s of full production load

Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails / Do-No-Harm Metrics ● Incremental Feature Development for Flow ● Testing In Production ● Kill Switch (big red button) ● Alert on Exception / Performance Early In Rollout ● “Limit The Blast Radius” w/o Manual Heroics ● Iteration w/o Measurement = Feature Factory ● Direct Evidence of Our Efforts → Pride @davekarow 1. How did this release impact my customer? a. It’s very satisfying to have real metrics showing how a release made things better for customers and the business. b. Right goal, wrong implementation: thought something was going to be a big hit but it’s not moving the needle (yet). 2. Compound effect a. Agile was supposed to be about incremental improvements, but it was too hard to measure the impact and so it became just about shipping more often. b. Build-up of sub-optimal features adds bloat, dilutes customer experience, slows down eng. c. Amplifying what works and killing what doesn’t is a positive compound impact loop… more impact with less output.

Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails / Do-No-Harm Metrics ● Incremental Feature Development for Flow ● Testing In Production ● Kill Switch (big red button) ● Alert on Exception / Performance Early In Rollout ● “Limit The Blast Radius” w/o Manual Heroics ● Iteration w/o Measurement = Feature Factory ● Direct Evidence of Our Efforts → Pride ● Take Bigger Risks, Safely ● Learn Faster With Less Investment ○ Dynamic Config ○ Painted Door Test to Learn (A/B Test) Top Image Attribution: VillageHero from Ulm, Germany, CC BY-SA 2.0 , via Wikimedia Commons @davekarow 1. Do the least effort for maximum impact. Instead of building multiple versions of stuff that has to be thrown away, how can we learn faster with less investment? a. Painted door experiments b. Try 3 vs 5 options c. Jon Smart (Sooner Safer Happier): less output to achieve same or better outcome is far superior use of resources (and better for morale) 2. Take bigger risks with focused experiments a. Imperfect Foods tested longer onboarding questionnaire on the theory that it would lead to greater engagement. Wanted to be sure it didn’t lead to greater funnel fall-out. Showed $7 increase in average cart size for cohort. Rolled out to 100% after proving it safe and effective.

My POV: This is What Sustainable Software Delivery Looks Like DEPLOY ERROR MITIGATION MEASURE SCALE MITIGATION RELEASE @davekarow

Increasing functionality & company adoption Cost to build and maintain @davekarow Progressive Delivery platforms with built-in statistical engines (i.e. Experimentation Platforms) are not a new concept, but only a few companies have been willing and able to invest the amount needed to get there on their own. A number of the logos above have become Split customers after considering to the cost to build and maintain such a solution. Split’s Feature Delivery Platform provides a significantly reduced operational cost while sustaining the same techniques for increased cadence, safety and innovation.

Send an email to swag@split.io: • Subject “London Microservices Swag” • Choose one: • Socks • T-shirt @davekarow linkedin.com/in/davekarow Feel free to connect: • @davekarow • linkedin.com/in/davekarow Swag giveaway for attendees of London Microservices meetup - please send an email no later than 15-Jan-2021. Thx!