Split Product Experimentation Workshop 3-16-21

Three-hour hands-on workshop for product and dev team leaders who want to know more about setting up and interpreting experiments via Split’s feature delivery platform. We introduced experimentation concepts and best practices, walked through a case study of an existing experiment, and then dug into Split to build one of our own. The session wrapped up with an ask-me-anything (AMA) session with Lizzie Eardley, a data scientist on Split’s product management team.

Transcript

1. Product Experimentation Workshop Workshop Kickoff + Overview
2. Hands-On Case Study: Existing Experiment Today’s Agenda Introductions + Getting Started 1 2 2 Hands-On Lab: Creating An Experiment 3 What’s Possible When Experimentation is Built-In? 4 AMA with Split Data Scientist Lizzie Eardley 5
3. Introductions Dave Karow Continuous Delivery Evangelist 3 Henry Jewkes Experimentation Architect Lizzie Eardley Data Scientist + Experimentation PM
4. Workshop Housekeeping 4 Diving in together
5. Zoom Etiquette • Mute your microphone when not speaking • Make sure your name is displayed as your Zoom username • Questions and comments are encouraged, we’ll stop often, but feel free to raise a hand in Zoom, DM an instructor or assistant, or simply speak up! 5
6. Get ready • Follow along in Zoom 6
7. Get ready • Follow along in Zoom • Review steps in the handout 7
8. Get ready • Follow along in Zoom • Review steps in the handout • Ask questions in Slack or Zoom 8
9. Get ready • Follow along in Zoom • Review steps in the handout • Ask questions in Slack or Zoom • Use Zoom reactions 9
10. Get ready • Follow along in Zoom • Review steps in the handout • Ask questions in Slack or Zoom • Use Zoom reactions • Before the second Hands-On session, find your personal lab credentials in Slack 10 workshop+demoorg-***@split.io
11. Let’s Get Started! 11
12. Why We Need Experimentation 12 New Release Metrics Change “Can’t we just change things and monitor what happens?”
13. Problem: Separating Signal From Noise 13 New Release Metrics Change Everything else in the world • Product changes • Marketing campaigns • Global Pandemics • Nice Weather
14. Solution: Cancel Out External Influence With Experimentation (Think noise cancelling headphones, but for your metrics) 14 Control Treatment 50% 50%
15. Hands-On Case Study Session 15 Hotel Booking Travel Site
16. Agenda ▶ Scenario Set-Up ▶ Your Role and Current Focus ▶ One of Your Experiments ▶ Hands-On Tour ▶ Your Website: Split Reservations ▶ Split: Your Experiment ▶ Split: Metrics and Events ▶ Debrief 16
17. Scenario Setup ▶ Your Role as PM, Bookings: ▶ Grow User Engagement and Revenue ▶ Your Current Focus: ▶ Bookings Per User ▶ Bookings Per Platinum User ▶ Average Price Paid Per User 17
18. One of Your Experiments What If We Show More Info By Default? 18
19. Let’s Find Out! 19 Time To Go Hands-On
20. Open Two Browser Tabs: 20 www.splitreservations.com app.split.io
21. 1. Log Out of Split (if currently logged in) 21 Login Page: app.split.io/login Email: train@sunflower.com Password: Sunflower1! 2. Log in to Split as: train@sunflower.com
22. Events ▶ Event = evidence of user behavior or system response ▶ Not always a user action: could be a system response too: ▶ Errors ▶ Response time ▶ Number of rows returned ▶ Something a user did or experienced along their journey ▶ Ideally, you already have event telemetry. ▶ If so, Split let’s you harvest it. ▶ If you need to capture new events, you can create them via our SDK or API 22
23. Back to Split 23 Let’s look At An Event
24. Debrief 24 What You Saw
25. Case Study Debrief 1/2 ▶ What You Saw ▶ Split: a feature flag in your code, gatekeeping code execution flow into to one or more treatments (i.e. code paths) ▶ Targeting rules: the per-split rules which determine which treatment a user is routed to at runtime ▶ Events: collected data points containing evidence of user behaviors or system responses. inspect/debug them with live tail ▶ Metrics Definition: the formula for deriving a metric from a series of events ▶ Metrics impact: the per-split dashboard showing the statistically significant differences in user and system behavior between treatments ▶ Metrics Trends + Details: a “drill-down” into the details behind a single metric on a split’s dashboard 25
26. Case Study Debrief 2/2 ▶ The Data Attribution Model Behind What You Saw ▶ An Impression is the record of a Treatment being served to a specific user (who got what treatment, when, and why). ▶ The user got that Impression because they passed through a Split which routed them to a specific Treatment using Targeting rules. ▶ An Event is a data point containing evidence of a user behavior or system response at a point in time. Events are not specific to one Split… they, like Metrics, are global and automatically calculated for every Split. ▶ A Metric is calculated per user*, based on Impressions and Events seen for that user, using the formula constructed in a Metric Definition. ▶ Metrics Impact is the statistical result of the comparing the distributions of Metric values across different Treatments in a Split *Per User, per Account or per any other Traffic Type 26
27. 27 Your App Targeting Rules Feature Flags and Targeting Data Layer & Analytics Management Console SDK User Attributes (age, account balance) Visual Debrief: Targeting
28. 28 Your App SDK Feature Flags and Targeting Data Layer & Analytics Management Console Visual Debrief: Impressions Impression Data • Traffic Key (UUID, account ID, etc) • Split name • Version (Treatment) of split • Rule used • Timestamp Impression Data
29. 29 Your App SDK Feature Flags and Targeting Data Layer & Analytics Management Console Visual Debrief: Events SDK Feature Flags and Targeting Data Layer & Analytics Management Console Event Data • Via SDK, Split API, or packaged integration ◦ Traffic Key ◦ Event Type ◦ Properties
30. 30 Visual Debrief: Metrics impact (Attribution) Impressions + Metrics ⇓ Attribution Impression Events Metric Events
31. Hands-On Experiment Lab Onboarding Metrics
32. Log Out from the Case Study 32
33. Log in to Lab 33 • https://app.split.io/login • Credentials shared in Slack • Message us if you need help ◦ Missing credentials ◦ Failed login ◦ Loading spinner
34. Break Time Hands-On Lab is Next!
35. 35 Let’s run an experiment! Our travel site has a problem with onboarding
36. Problem ▶ Losing customers during the onboarding process ▶ Customer feedback is that it takes too long ▶ Internally, we need more visibility 36
37. Current Onboarding Searching Trip Planning Comparisons History report 37
38. Current Onboarding Searching Trip Planning Comparisons History report 38
39. Removed • Travel history report ◦ Showed where you have visited ◦ Cool feature! ◦ For power users 39
40. Removed • Travel history report ◦ Showed where you have visited ◦ Cool feature! ◦ For power users • New user survey • Tracking ◦ Time to completion ◦ Progress through onboarding ◦ Product interactions Added 40
41. From Feature 41 To Experiment
42. Hypothesis ▶ Removing reporting from onboarding will reduce onboarding time and increase the number of users who complete onboarding 42
43. Hypothesis ▶ Removing reporting from onboarding will reduce onboarding time and increase the number of users who complete onboarding Action ▶ Remove reporting from onboarding process Problem ▶ Losing customers during the onboarding process ▶ Customer feedback is that it takes too long 43
44. Metrics Planning 44 https://miro.com/app/boar d/o9J_lQWrXsA=/
45. Time to take a look! 45
46. Break Time “What’s Possible When Experimentation is Built-In?” is Next!
47. What’s Possible When Experimentation is Built-In? Uptime, Safety, Flow, and a Culture of Continuous Learning 47
48. What changes when the lift to do experimentation is small? 48 • What if automated data science could help you move faster, easier, safer, instead of requiring “prioritization” to get coverage? • What happens when you escape the limitation of impact analysis being performed on “critical” releases only?
49. Do you do this each time your team releases to production? 49 49
50. 50 Can you remember a release night that went like this?
51. How You Release Matters (Progressive Delivery Techniques) 51 Harvey Balls by Sschulte at English Wikipedia [CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)] Approach Benefits Blue/Green Deployment Canary Release (container based) Feature Flags Feature Flags + Experimentation Avoid Downtime Limit The Blast Radius Limit WIP / Achieve Flow Learn During The Process
52. Problem: Detecting Early Signs of Trouble 52 Feature Exposed to 100% of users
53. Problem: Detecting Early Signs of Trouble 53 Feature Exposed to 100% of users Feature Enabled for 5% of users
54. Solution: Monitoring Guardrail Metrics Automatically 54 Control Treatment 5% 95%
55. Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 55 Decouple Deploy From Release With Feature Flags • Incremental Feature Development for Flow • Testing In Production • Kill Switch (big red button)
56. Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 56 Decouple Deploy From Release With Feature Flags Automate Guardrails • Alert on Exception / Performance Early In Rollout • “Limit The Blast Radius” w/o Manual Heroics
57. Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 57 Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails • Iteration w/o Measurement = Feature Factory 😡 • Direct Evidence of Our Efforts → Pride 😎
58. Feature Flags and Experimentation: A Layered Approach to Productivity and Psychological Safety 58 Measure Release Impact Decouple Deploy From Release With Feature Flags Automate Guardrails • Take Bigger Risks, Safely • Learn Faster With Less Investment ◦ Dynamic Config ◦ Painted Door Test to Learn (A/B Test)
59. What Software Delivery Looks Like When Experimentation is Built-In 59 DEPLOY Code deployed No exposure ERROR MITIGATION 0-50% Ramp Identify bugs/crashes MEASURE Maximum Power Ramp Understand impact SCALE MITIGATION 50-100% Ramp Identify scaling issues RELEASE Complete rollout
60. Break Time AMA With Split Data Scientist Lizzie Eardley is Next!
61. Thank you 61