Growing Apps with
AB Testing in Firebase
Abid Jami
l
Technical Lead, GrocerApp
Slide 2
Slide 2 text
I am leading Growth Pod of Engineering team
at GrocerAp
p
I write codes in Kotlin, and Java for Androi
d
I am Programmer, CS Researcher, & Speaker
ABOUT ME
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
AB Testing
Slide 5
Slide 5 text
AB Testing Process
Slide 6
Slide 6 text
AB Testing Lifecycle
Slide 7
Slide 7 text
Example AB Testing
Slide 8
Slide 8 text
Example AB Testing
we have to create our own dataset. The dataset will contain the columns below:
• customer_id: the unique identifier of the customer
• segment: customer’s segment; high-value or low-value
• group: indicates whether the customer is in the test or control group
• purchase_count: # of purchases completed by the customer
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Firebase AB Testing
Slide 15
Slide 15 text
Firebase Analytics
Slide 16
Slide 16 text
Firebase Predictions
Slide 17
Slide 17 text
Firebase Predicitions
Slide 18
Slide 18 text
Setting up the experiment
Slide 19
Slide 19 text
Define your targets
Slide 20
Slide 20 text
Decide your goals
Slide 21
Slide 21 text
Create variants
Slide 22
Slide 22 text
Create variants
Slide 23
Slide 23 text
Create variants
Slide 24
Slide 24 text
Decide weights
Slide 25
Slide 25 text
Experimentation Results
Slide 26
Slide 26 text
Interpreting test results
• Powered by Google Optimize.
• Uses Bayesian inference
• Observed data is calculated directly from analytics data
• Modeled data comes from the application of our Bayesian model
Slide 27
Slide 27 text
Observed Data
•Probability to beat baseline: how likely that the metric is higher for this
variant compared to the baseline
•Percent difference from baseline: based on the median model
estimates of the metric for the variant and the baseline
•Metric ranges: the ranges where the value of the metric is most likely
to to be found.