DeveloperWeek 2020 Progressive Delivery: Patterns & Benefits

Slide 1

Slide 1 text

Progressive Delivery Patterns & Benefits Dave Karow Continuous Delivery Evangelist Split.io @davekarow

Slide 2

Slide 2 text

The future is already here — it's just not very evenly distributed. William Gibson @davekarow

Slide 3

Slide 3 text

What is Progressive Delivery? Patterns In The Wild Summing It Up QR Code @davekarow

Slide 4

Slide 4 text

What is Progressive Delivery and what are the potential benefits? @davekarow @davekarow

Slide 5

Slide 5 text

Carlos Sanchez (Sr. Cloud Software Engineer @ Adobe) https://blog.csanchez.org/2019/01/22/progressive-delivery-in-kubernetes-blue-green-and-canary-deployments/ Progressive Delivery is the next step after Continuous Delivery, where new versions are deployed to a subset of users and are evaluated in terms of correctness and performance before rolling them to the totality of the users and rolled back if not matching some key metrics. @davekarow

Slide 6

Slide 6 text

Potential Benefits of Progressive Delivery Avoid Downtime Limit the Blast Radius Limit WIP / Achieve Flow Learn During the Process @davekarow

Slide 7

Slide 7 text

How You Roll Matters Approach Benefits Blue/Green Deployment Canary Release Feature Flag Rollout Feature Delivery Platform Avoid Downtime Limit The Blast Radius Limit WIP / Achieve Flow Learn During The Process https://www.split.io/blog/learn-the-four-shades-of-progressive-delivery/ Harvey Balls by Sschulte at English Wikipedia [CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)] @davekarow

Slide 8

Slide 8 text

How You Roll Matters @davekarow Approach Benefits Blue/Green Deployment Canary Release Feature Flag Rollout Feature Delivery Platform Avoid Downtime Limit The Blast Radius Limit WIP / Achieve Flow Learn During The Process @davekarow https://www.split.io/blog/learn-the-four-shades-of-progressive-delivery/ Harvey Balls by Sschulte at English Wikipedia [CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)]

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Feature Delivery Platform Capabilities @davekarow

Slide 13

Slide 13 text

Feature Delivery Platform Capabilities @davekarow

Slide 14

Slide 14 text

Feature Delivery Platform Capabilities @davekarow

Slide 15

Slide 15 text

Let’s Venture Into the Wild! Bruce Turner from AustinTX https://www.flickr.com/people/66994844@N00 [CC BY (https://creativecommons.org/licenses/by/2.0)] @davekarow

Slide 16

Slide 16 text

Booking.com @davekarow

Slide 17

Slide 17 text

Booking.com: a well-documented example of the pattern @davekarow

Slide 18

Slide 18 text

https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-a6c16c5a1185 @davekarow

Slide 19

Slide 19 text

@davekarow @davekarow

Slide 20

Slide 20 text

@davekarow

Slide 21

Slide 21 text

Feature Flag Managing exposure like a dimmer or light board 0% 10% 20% 50% 100% @davekarow

Slide 22

Slide 22 text

Booking.com’s experience with Manage: “asynchronous feature release” ● Deploying has no impact on user experience ● Deploy more frequently with less risk to business and users ● The big win is Agility @davekarow

Slide 23

Slide 23 text

@davekarow

Slide 24

Slide 24 text

Monitoring the needle in the haystack If you roll out a change to just 5% of your population ...and 20% (1 in 5) of the exposed users get an error, that’s a HUGE problem! But, what % of your total user population is getting that error? 1% @davekarow

Slide 25

Slide 25 text

@davekarow

Slide 26

Slide 26 text

@davekarow

Slide 27

Slide 27 text

Booking.com’s experience with Monitor: “Experimentation as a safety net” ● Each new feature is wrapped in its own experiment ● Allows: monitoring and stopping of individual changes ● The developer or team responsible for the feature can enable and disable it... ● ...regardless of who deployed the new code that contained it. @davekarow

Slide 28

Slide 28 text

Booking.com safety net automated: “circuit breaker” ● Active for the first three minutes of feature release ● Severe degradation → automatic abort of that feature ● Acceptable divergence from core value of local ownership and responsibility where it’s a “no brainer” that users are being negatively impacted @davekarow

Slide 29

Slide 29 text

Booking.com Circuit Breaker (Automatic Stopping) @davekarow

Slide 30

Slide 30 text

Guardrail metrics @davekarow

Slide 31

Slide 31 text

@davekarow

Slide 32

Slide 32 text

Booking.com’s experience with Experimentation: A way to validate ideas ● Measure (in a controlled manner) the impact changes have on user behaviour ● Every change has a clear objective (explicitly stated hypothesis on how it will improve user experience) ● Measuring allows validation that desired outcome is achieved @davekarow

Slide 33

Slide 33 text

Feature Flag Experimentation example @davekarow 50% 50%

Slide 34

Slide 34 text

The quicker we manage to validate new ideas the less time is wasted on things that don’t work and the more time is left to work on things that make a difference. Booking’s Big Takeaway @davekarow

Slide 35

Slide 35 text

LinkedIn XLNT/LIX @davekarow

Slide 36

Slide 36 text

● Built a targeting engine that could “split” traffic between existing and new code ● Impact analysis was by hand only (and took ~2 weeks), so nobody did it :-( Essentially just feature flags without automated feedback LinkedIn early days: a modest start for XLNT @davekarow

Slide 37

Slide 37 text

LinkedIn XLNT Today A controlled release with standardized KPI calculation launched very 5 minutes 100 releases per day 6000 metrics that can be “followed” by any stakeholder: “What releases are moving the numbers I care about?” @davekarow

Slide 38

Slide 38 text

Lessons learned at LinkedIn ● Build for scale: no more coordinating over email ● Make it trustworthy: targeting and analysis must be rock solid ● Design for diverse teams, not just data scientists Ya Xu Head of Data Science, LinkedIn Decisions Conference 10/2/2018 @davekarow

Slide 39

Slide 39 text

Step 1 Feature flags Step 2 Sensors Correlation Step 3 Stats Engine Causation “Holy Grail” Mgmt console System of record Alerting Increasing functionality & company adoption Cost to build and maintain Summing it up: The patterns are proven @davekarow Maturity hasn’t come easily or fast for the pioneers

Slide 40

Slide 40 text

Split implements these patterns as a service: split.io @davekarow

Slide 41

Slide 41 text

@davekarow

Slide 42

Slide 42 text

@davekarow

Slide 43

Slide 43 text

@davekarow

Slide 44

Slide 44 text

Checklists to DIY or Buy ● Foundational Capabilities You’ll Need ● How-To’s: Monitor & Experiment @davekarow

Slide 45

Slide 45 text

Decouple deploy from release ❏ Allow changes of exposure w/o new deploy or rollback ❏ Support targeting by UserID, attribute (population), random hash Foundational Capability #1 @davekarow

Slide 46

Slide 46 text

Automate a reliable and consistent way to answer, “Who have we exposed this to so far?” ❏ Record who hit a flag, which way they were sent, and why ❏ Confirm that targeting is working as intended ❏ Confirm that expected traffic levels are reached Foundational Capability #2 @davekarow

Slide 47

Slide 47 text

Automate a reliable and consistent way to answer, “How is it going for them (and us)?” ❏ Automate comparison of system health (errors, latency, etc…) ❏ Automate comparison of user behavior (business outcomes) ❏ Make it easy to include “Guardrail Metrics” in comparisons to avoid the local optimization trap Foundational Capability #3 @davekarow

Slide 48

Slide 48 text

Limit the blast radius of unexpected consequences so you can replace the “big bang” release night with more frequent, less stressful rollouts. Build on the foundational capabilities to: ❏ Ramp in stages, starting with dev team, then dogfooding, then % of public ❏ Monitor at feature rollout level, not just globally (vivid facts vs faint signals) ❏ Alert at the team level (build it/own it) ❏ Kill if severe degradation detected (stop the pain now, triage later) ❏ Continue to ramp up healthy features while “sick” are ramped down or killed How-To: Release Faster With Less Risk @davekarow

Slide 49

Slide 49 text

Focus precious engineering cycles on “what works” with experimentation, making statistically rigorous observations about what moves KPIs (and what doesn’t). Build on the foundational capabilities to: ❏ Target an experiment to a specific segment of users ❏ Ensure random, deterministic, persistent allocation to A/B/n variants ❏ Ingest metrics chosen before the experiment starts (not cherry-picked after) ❏ Compute statistical significance before proclaiming winners ❏ Design for diverse audiences, not just data scientists (buy-in needed to stick) How-To: Engineer for Impact (Not Output) @davekarow

Slide 50

Slide 50 text

Whatever you are, try to be a good one. William Makepeace Thackeray @davekarow

Slide 51

Slide 51 text

Progressive Delivery Resources tinyurl.com/pd4u2020 (just posted to twitter as well) @davekarow