Advanced Feature Flagging: It's All About The Data (GOTO Chicago 2020)

Slide 1

Slide 1 text

Advanced Feature Flagging: It’s All About The Data

Slide 2

Slide 2 text

Continuous Delivery Evangelist DAVE KAROW

Slide 3

Slide 3 text

FEATURE FLAG REVIEW A quick review of feature ﬂags and rollout strategies 01 02 HOW YOU MEASURE MATTERS Don’t believe everything you see 03 FLAGS + DATA = EXPERIMENTATION Attribution, calculation and analysis, automatic 24x7

Slide 4

Slide 4 text

FEATURE FLAGS

Slide 5

Slide 5 text

When you want to When you are ready DEPLOY RELEASE

Slide 6

Slide 6 text

if{} else{} STATEMENT Separates code deploy from feature release

Slide 7

Slide 7 text

SWITCH OFF ON RAMPED MULTI-VARIATE TYPES OF FLAGS

Slide 8

Slide 8 text

SPEED How quickly can we reach a decision? ROLLOUT FRAMEWORK QUALITY How conﬁdent can we be of that decision? RISK How can we minimize bad outcomes? CREDIT SQR: Balancing Speed, Quality and Risk in Online Experiments Ya Xu, et.al

Slide 9

Slide 9 text

PHASES OF ROLLOUT DEPLOY Code deployed, no exposure

Slide 10

Slide 10 text

PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes

Slide 11

Slide 11 text

PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact

Slide 12

Slide 12 text

PHASES OF ROLLOUT DEPLOY Code deployed, no exposure ERROR MITIGATION 1-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact SCALE MITIGATION 50-100% Ramp Identify scaling issues RELEASE Complete rollout

Slide 13

Slide 13 text

HOW YOU MEASURE MATTERS

Slide 14

Slide 14 text

DON’T BELIEVE EVERYTHING YOU SEE... “Can’t we just change things and monitor what happens?” New Release Metrics Change

Slide 15

Slide 15 text

SYSTEMS DASHBOARD

Slide 16

Slide 16 text

FEATURE ENABLED TO 100%

Slide 17

Slide 17 text

FEATURE ENABLED TO 5% FEATURE ENABLED TO 100%

Slide 18

Slide 18 text

Both ice cream sales and shark attacks increase when the weather is hot and sunny, but they are not caused by each other They are both caused by good weather, with lots of people at the beach, both eating ice cream and swimming in the sea CORRELATION IS NOT CAUSATION

Slide 19

Slide 19 text

SYSTEMS DASHBOARD Not related to a feature change A customer was undergoing a DDoS

Slide 20

Slide 20 text

DON’T BELIEVE EVERYTHING YOU SEE... “Can’t we just change things and monitor what happens?” New Release Metrics Change

Slide 21

Slide 21 text

DON’T BELIEVE EVERYTHING YOU SEE... ● Product changes ● Marketing campaigns ● Global Pandemics ● Nice Weather New Release Metrics Change Everything else in the world

Slide 22

Slide 22 text

MEASURING CAUSALITY Control Treatment 50% 50%

Slide 23

Slide 23 text

ATTRIBUTION

Slide 24

Slide 24 text

CALCULATION

Slide 25

Slide 25 text

ANALYSIS

Slide 26

Slide 26 text

AD-HOC ANALYSIS TAGGING METRICS EXPERIMENTATION PLATFORM Homegrown or SaaS offerings provide both data collection and statistical analysis Storing feature attribution in BI database for querying Most dashboarding tools allow tagging of data for segmentation

Slide 27

Slide 27 text

FLAGS + DATA = EXPERIMENTATION

Slide 28

Slide 28 text

STATISTICAL ENGINE TELEMETRY SYSTEM TARGETING SYSTEM ANATOMY OF AN EXPERIMENTATION PLATFORM MANAGEMENT CONSOLE CREDIT Understanding Experimentation Platforms [Aijaz, Stuart, Jewkes]

Slide 29

Slide 29 text

TARGETING SYSTEM FAST RANDOM STICKY RELIABLE

Slide 30

Slide 30 text

TARGETING SYSTEM FEATURE TRAFFIC KEY ASSIGNMENT

Slide 31

Slide 31 text

“user_1234” Hashing Algorithm TRAFFIC KEY “A4F6D32” HASH VALUE Normalize (in mod 100) 19 BUCKET BETWEEN 0 AND 99 Feature Seed

Slide 32

Slide 32 text

12 91 6 54 34 63 25 70

Slide 33

Slide 33 text

Centralize tracking across tools Ensure reliable delivery Identify and exclude malicious traﬃc TELEMETRY SYSTEM Receive batches of events Store in warehouse for processing WRAPPER SERVICE track (String key, String eventType, double value)

Slide 34

Slide 34 text

STATISTICAL ENGINE

Slide 35

Slide 35 text

STATISTICAL ENGINE

Slide 36

Slide 36 text

STATISTICAL ENGINE

Slide 37

Slide 37 text

MANAGEMENT CONSOLE

Slide 38

Slide 38 text

EXAMPLE IMPLEMENTATION Walmart EXPO

Slide 39

Slide 39 text

EXAMPLE IMPLEMENTATION Linked In LiX

Slide 40

Slide 40 text

EXAMPLE IMPLEMENTATION Booking.com

Slide 41

Slide 41 text

LESSONS LEARNED @ Microsoft + Google + LinkedIn https://experimentguide.com/

Slide 42

Slide 42 text

EVERY FEATURE IS AN EXPERIMENT Targeted feature rollout allows for rapid A/B tests and customer insights. KILL THE RELEASE NIGHT Decoupling deployment from release make release rituals go away. AUTOMATE DELIVERY WITH DATA Independent feature rollouts are orchestrated by data, not people.

Slide 43

Slide 43 text

CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik THANKS! Let’s move to discussion! linkedin.com/in/davekarow [email protected] @davekarow