Advanced Feature Flagging: It's All About The Data (GOTO Chicago 2020)

Feature flags deliver the control needed to decouple deploy from release but can break traditional monitoring and KPIs. The good news is that teams are using them to "kill the release night" by moving from big-bang releases to gradual releases during normal business hours. The bad news is that gradual release techniques add challenges to traditional ways of monitoring system health and user behavior. No one wants to move faster if that means less visibility and thus greater risk.

We'll look at advanced implementation techniques that marry the precision control of feature flags with automated ingest of data and statistical computation of KPIs. This allows teams to proactively identify system performance and user behavior differences between the status quo and new code. Advanced feature flagging implementations “build-in” observability to every release. When you push a feature to 5% of users, it becomes trivial to see how user and system behavior varies for those users vs. the other 95%. Teams further along this journey auto-calculate “do-no-harm" metrics, so it’s easy to detect unintended consequences of their work before ramping up to all users.

You’ll leave this session with a clear vision of how your team can achieve the same benefits, by either enhancing your in-house solution or adopting a commercial tool.

Transcript

1. Advanced Feature Flagging: It’s All About The Data
2. Continuous Delivery Evangelist DAVE KAROW
3. FEATURE FLAG REVIEW A quick review of feature flags and rollout strategies 01 02 HOW YOU MEASURE MATTERS Don’t believe everything you see 03 FLAGS + DATA = EXPERIMENTATION Attribution, calculation and analysis, automatic 24x7
4. FEATURE FLAGS
5. When you want to When you are ready DEPLOY RELEASE
6. if{} else{} STATEMENT Separates code deploy from feature release
7. SWITCH OFF ON RAMPED MULTI-VARIATE TYPES OF FLAGS
8. SPEED How quickly can we reach a decision? ROLLOUT FRAMEWORK QUALITY How confident can we be of that decision? RISK How can we minimize bad outcomes? CREDIT SQR: Balancing Speed, Quality and Risk in Online Experiments Ya Xu, et.al
9. PHASES OF ROLLOUT DEPLOY Code deployed, no exposure
10. PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes
11. PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact
12. PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact SCALE MITIGATION 50-100% Ramp Identify scaling issues RELEASE Complete rollout
13. HOW YOU MEASURE MATTERS
14. DON’T BELIEVE EVERYTHING YOU SEE... “Can’t we just change things and monitor what happens?” New Release Metrics Change
15. SYSTEMS DASHBOARD
16. FEATURE ENABLED TO 100%
17. FEATURE ENABLED TO 5% FEATURE ENABLED TO 100%
18. Both ice cream sales and shark attacks increase when the weather is hot and sunny, but they are not caused by each other They are both caused by good weather, with lots of people at the beach, both eating ice cream and swimming in the sea CORRELATION IS NOT CAUSATION
19. SYSTEMS DASHBOARD Not related to a feature change A customer was undergoing a DDoS
20. DON’T BELIEVE EVERYTHING YOU SEE... “Can’t we just change things and monitor what happens?” New Release Metrics Change
21. DON’T BELIEVE EVERYTHING YOU SEE... • Product changes • Marketing campaigns • Global Pandemics • Nice Weather New Release Metrics Change Everything else in the world
22. MEASURING CAUSALITY Control Treatment 50% 50%
23. ATTRIBUTION
24. CALCULATION
25. ANALYSIS
26. AD-HOC ANALYSIS TAGGING METRICS EXPERIMENTATION PLATFORM Homegrown or SaaS offerings provide both data collection and statistical analysis Storing feature attribution in BI database for querying Most dashboarding tools allow tagging of data for segmentation
27. FLAGS + DATA = EXPERIMENTATION
28. STATISTICAL ENGINE TELEMETRY SYSTEM TARGETING SYSTEM ANATOMY OF AN EXPERIMENTATION PLATFORM MANAGEMENT CONSOLE CREDIT Understanding Experimentation Platforms [Aijaz, Stuart, Jewkes]
29. TARGETING SYSTEM FAST RANDOM STICKY RELIABLE
30. TARGETING SYSTEM FEATURE TRAFFIC KEY ASSIGNMENT
31. “user_1234” Hashing Algorithm TRAFFIC KEY “A4F6D32” HASH VALUE Normalize (in mod 100) 19 BUCKET BETWEEN 0 AND 99 Feature Seed
32. 12 91 6 54 34 63 25 70
33. Centralize tracking across tools Ensure reliable delivery Identify and exclude malicious traffic TELEMETRY SYSTEM Receive batches of events Store in warehouse for processing WRAPPER SERVICE track (String key, String eventType, double value)
34. STATISTICAL ENGINE
35. STATISTICAL ENGINE
36. STATISTICAL ENGINE
37. MANAGEMENT CONSOLE
38. EXAMPLE IMPLEMENTATION Walmart EXPO
39. EXAMPLE IMPLEMENTATION Linked In LiX
40. EXAMPLE IMPLEMENTATION Booking.com
41. LESSONS LEARNED @ Microsoft + Google + LinkedIn https://experimentguide.com/
42. EVERY FEATURE IS AN EXPERIMENT Targeted feature rollout allows for rapid A/B tests and customer insights. KILL THE RELEASE NIGHT Decoupling deployment from release make release rituals go away. AUTOMATE DELIVERY WITH DATA Independent feature rollouts are orchestrated by data, not people.
43. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik THANKS! Let’s move to discussion! linkedin.com/in/davekarow dave.karow@split.io @davekarow