Product Experimentation Workshop Workshop Kickoff + Overview

Hands-On Case Study: Existing Experiment Today’s Agenda Introductions + Getting Started 1 2 2 Hands-On Lab: Creating An Experiment 3 What’s Possible When Experimentation is Built-In? 4 AMA with Split Data Scientist Lizzie Eardley 5

Introductions Dave Karow Continuous Delivery Evangelist 3 Henry Jewkes Experimentation Architect Lizzie Eardley Data Scientist + Experimentation PM

Workshop Housekeeping 4 Diving in together

Zoom Etiquette ● Mute your microphone when not speaking ● Make sure your name is displayed as your Zoom username ● Questions and comments are encouraged, we’ll stop often, but feel free to raise a hand in Zoom, DM an instructor or assistant, or simply speak up! 5

Get ready ● Follow along in Zoom 6

Get ready ● Follow along in Zoom ● Review steps in the handout 7

Get ready ● Follow along in Zoom ● Review steps in the handout ● Ask questions in Slack or Zoom 8

Get ready ● Follow along in Zoom ● Review steps in the handout ● Ask questions in Slack or Zoom ● Use Zoom reactions 9

Get ready ● Follow along in Zoom ● Review steps in the handout ● Ask questions in Slack or Zoom ● Use Zoom reactions ● Before the second Hands-On session, find your personal lab credentials in Slack 10 workshop+demoorg-***@split.io

Let’s Get Started! 11

Why We Need Experimentation 12 New Release Metrics Change “Can’t we just change things and monitor what happens?”

Problem: Separating Signal From Noise 13 New Release Metrics Change Everything else in the world ● Product changes ● Marketing campaigns ● Global Pandemics ● Nice Weather

Solution: Cancel Out External Influence With Experimentation (Think noise cancelling headphones, but for your metrics) 14 Control Treatment 50% 50%

Hands-On Case Study Session 15 Hotel Booking Travel Site

Agenda ▶ Scenario Set-Up ▶ Your Role and Current Focus ▶ One of Your Experiments ▶ Hands-On Tour ▶ Your Website: Split Reservations ▶ Split: Your Experiment ▶ Split: Metrics and Events ▶ Debrief 16

Scenario Setup ▶ Your Role as PM, Bookings: ▶ Grow User Engagement and Revenue ▶ Your Current Focus: ▶ Bookings Per User ▶ Bookings Per Platinum User ▶ Average Price Paid Per User 17

One of Your Experiments What If We Show More Info By Default? 18

Let’s Find Out! 19 Time To Go Hands-On

Open Two Browser Tabs: 20 www.splitreservations.com app.split.io

1. Log Out of Split (if currently logged in) 21 Login Page: app.split.io/login Email: train@sunflower.com Password: Sunflower1! 2. Log in to Split as: train@sunflower.com

Events ▶ Event = evidence of user behavior or system response ▶ Not always a user action: could be a system response too: ▶ Errors ▶ Response time ▶ Number of rows returned ▶ Something a user did or experienced along their journey ▶ Ideally, you already have event telemetry. ▶ If so, Split let’s you harvest it. ▶ If you need to capture new events, you can create them via our SDK or API 22

Back to Split 23 Let’s look At An Event

Debrief 24 What You Saw

Case Study Debrief 1/2 ▶ What You Saw ▶ Split: a feature flag in your code, gatekeeping code execution flow into to one or more treatments (i.e. code paths) ▶ Targeting rules: the per-split rules which determine which treatment a user is routed to at runtime ▶ Events: collected data points containing evidence of user behaviors or system responses. inspect/debug them with live tail ▶ Metrics Definition: the formula for deriving a metric from a series of events ▶ Metrics impact: the per-split dashboard showing the statistically significant differences in user and system behavior between treatments ▶ Metrics Trends + Details: a “drill-down” into the details behind a single metric on a split’s dashboard 25

Case Study Debrief 2/2 ▶ The Data Attribution Model Behind What You Saw ▶ An Impression is the record of a Treatment being served to a specific user (who got what treatment, when, and why). ▶ The user got that Impression because they passed through a Split which routed them to a specific Treatment using Targeting rules. ▶ An Event is a data point containing evidence of a user behavior or system response at a point in time. Events are not specific to one Split… they, like Metrics, are global and automatically calculated for every Split. ▶ A Metric is calculated per user*, based on Impressions and Events seen for that user, using the formula constructed in a Metric Definition. ▶ Metrics Impact is the statistical result of the comparing the distributions of Metric values across different Treatments in a Split *Per User, per Account or per any other Traffic Type 26

27 Your App Targeting Rules Feature Flags and Targeting Data Layer & Analytics Management Console SDK User Attributes (age, account balance) Visual Debrief: Targeting

28 Your App SDK Feature Flags and Targeting Data Layer & Analytics Management Console Visual Debrief: Impressions Impression Data ● Traffic Key (UUID, account ID, etc) ● Split name ● Version (Treatment) of split ● Rule used ● Timestamp Impression Data

29 Your App SDK Feature Flags and Targeting Data Layer & Analytics Management Console Visual Debrief: Events SDK Feature Flags and Targeting Data Layer & Analytics Management Console Event Data ● Via SDK, Split API, or packaged integration ○ Traffic Key ○ Event Type ○ Properties

30 Visual Debrief: Metrics impact (Attribution) Impressions + Metrics ⇓ Attribution Impression Events Metric Events

Hands-On Experiment Lab Onboarding Metrics

Log Out from the Case Study 32

Log in to Lab 33 ● https://app.split.io/login ● Credentials shared in Slack ● Message us if you need help ○ Missing credentials ○ Failed login ○ Loading spinner

Break Time Hands-On Lab is Next!

35 Let’s run an experiment! Our travel site has a problem with onboarding

Problem ▶ Losing customers during the onboarding process ▶ Customer feedback is that it takes too long ▶ Internally, we need more visibility 36

Current Onboarding Searching Trip Planning Comparisons History report 37

Current Onboarding Searching Trip Planning Comparisons History report 38

Removed ● Travel history report ○ Showed where you have visited ○ Cool feature! ○ For power users 39

Removed ● Travel history report ○ Showed where you have visited ○ Cool feature! ○ For power users ● New user survey ● Tracking ○ Time to completion ○ Progress through onboarding ○ Product interactions Added 40

From Feature 41 To Experiment

Hypothesis ▶ Removing reporting from onboarding will reduce onboarding time and increase the number of users who complete onboarding 42

Hypothesis ▶ Removing reporting from onboarding will reduce onboarding time and increase the number of users who complete onboarding Action ▶ Remove reporting from onboarding process Problem ▶ Losing customers during the onboarding process ▶ Customer feedback is that it takes too long 43

Metrics Planning 44 https://miro.com/app/boar d/o9J_lQWrXsA=/

Time to take a look! 45

Break Time “What’s Possible When Experimentation is Built-In?” is Next!

What’s Possible When Experimentation is Built-In? Uptime, Safety, Flow, and a Culture of Continuous Learning 47

What changes when the lift to do experimentation is small? 48 ● What if automated data science could help you move faster, easier, safer, instead of requiring “prioritization” to get coverage? ● What happens when you escape the limitation of impact analysis being performed on “critical” releases only?

Do you do this each time your team releases to production? 49 49

50 Can you remember a release night that went like this?

How You Release Matters (Progressive Delivery Techniques) 51 Harvey Balls by Sschulte at English Wikipedia [CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)] Approach Benefits Blue/Green Deployment Canary Release (container based) Feature Flags Feature Flags + Experimentation Avoid Downtime Limit The Blast Radius Limit WIP / Achieve Flow Learn During The Process

Problem: Detecting Early Signs of Trouble 52 Feature Exposed to 100% of users

Problem: Detecting Early Signs of Trouble 53 Feature Exposed to 100% of users Feature Enabled for 5% of users

Solution: Monitoring Guardrail Metrics Automatically 54 Control Treatment 5% 95%

Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 55 Decouple Deploy From Release With Feature Flags ● Incremental Feature Development for Flow ● Testing In Production ● Kill Switch (big red button)

Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 56 Decouple Deploy From Release With Feature Flags Automate Guardrails ● Alert on Exception / Performance Early In Rollout ● “Limit The Blast Radius” w/o Manual Heroics

Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 57 Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails ● Iteration w/o Measurement = Feature Factory 😡 ● Direct Evidence of Our Efforts → Pride 😎

Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 58 Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails ● Take Bigger Risks, Safely ● Learn Faster With Less Investment ○ Dynamic Config ○ Painted Door Test to Learn (A/B Test)

What Software Delivery Looks Like When Experimentation is Built-In 59 DEPLOY Code deployed No exposure ERROR MITIGATION 0-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact SCALE MITIGATION 50-100% Ramp Identify scaling issues RELEASE Complete rollout

Break Time AMA With Split Data Scientist Lizzie Eardley is Next!

Thank you 61