Share of Matches
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
With Sunglasses Control
Sunglasses
Slide 14
Slide 14 text
Share of Premium Conversion
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses
Slide 15
Slide 15 text
User Retention
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
spotify:user:chocochipset
92b2976bb15d26c9008
Slide 18
Slide 18 text
1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
Slide 19
Slide 19 text
1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
Slide 20
Slide 20 text
1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
0% 60%
Slide 21
Slide 21 text
Markets
Slide 22
Slide 22 text
Demographics
Slide 23
Slide 23 text
User Attributes
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
Login
resolveABFlags()
response(ABFlags)
ABBA
Cache AB Values
Load Cached or
Default AB Flags
Slide 26
Slide 26 text
Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);
if (isButtonPink) {
// alternate path here:
button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control
Slide 27
Slide 27 text
Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control
Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);
if (isButtonPink) {
// alternate path here:
button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 1
buffer-quality low
gallery-artist Control
pink-button 1
Variation A Control
Slide 28
Slide 28 text
Control
Variation
Slide 29
Slide 29 text
Analytics
impressions and
interactions
Slide 30
Slide 30 text
* just a personal preference
Slide 31
Slide 31 text
2. How?
Slide 32
Slide 32 text
1. Hypothesis 2. Design 3. Run 4. Analysis
Slide 33
Slide 33 text
1. Formulate
a Hypothesis
Slide 34
Slide 34 text
2. Design the Test
Slide 35
Slide 35 text
#
shuffle
plays
2.1 Target Metrics
Slide 36
Slide 36 text
# matches
Samson, 3
# replies
# walks in the park
Slide 37
Slide 37 text
2.2 Test Group
more test bandwith, less confidence, smaller effect
in business
less test bandwith, more confidence, bigger effect in
business
Small
Large
Slide 38
Slide 38 text
2.3 Duration
Slide 39
Slide 39 text
3. Run the
test
Slide 40
Slide 40 text
1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
Test A
Test A
Test B
Test B
Test A / Test B
Watch out for
conflicting tests!
Slide 41
Slide 41 text
Don’t cut them short!
Bruno Cesar 48’
Ronaldo 89’
Morata 90’ + 4’
2 -1