A/B Testing with React
Eivind Arvesen and Lars Dølvik
Slide 2
Slide 2 text
o Introduction
o MatSans
• About the project
• The stack
• Hypothesis- and goal-driven development/Workflow
o Story-time
o What is A/B testing?
• react-ab-test (library)
▪ Isomorphism
o Evaluation
• Measurements (GA: scrolling and events…)
• Interpreting the numbers
o Practical demonstration
Outline
Slide 3
Slide 3 text
o MatSans is a recipe focused application
o Runs on Google Cloud Platform
o Isomorphic
o Built with React and Relay in front
o GraphQL as data query language
o Uses ElasticSearch to hold our content
o Enonic CMS to publish content
o Hypothesis- and goal-driven
development/Workflow
MatSans
Slide 4
Slide 4 text
o Tactical goal
o Action What?
o Hypothesis Why?
o Measurement How?
o Estimated effect
MatSans
Slide 5
Slide 5 text
Story-time
Slide 6
Slide 6 text
Story-time
Slide 7
Slide 7 text
Story-time
The original Variation 1 Variation 2
Slide 8
Slide 8 text
A/B-TESTING!
Slide 9
Slide 9 text
A form of statistical hypothesis testing.
What is A/B testing?
Illustration by Maxime Lorant
Slide 10
Slide 10 text
Usual steps:
● Have an idea or supposition about our solution
● Verify or falsify the hypothesis via experiment(s)
○ Collect various measurements
What is A/B testing?
Slide 11
Slide 11 text
Why?
It’s normal to be wrong.
Measuring how our development impacts our users makes us
able to make decisions based on facts!
What is A/B-testing?
Slide 12
Slide 12 text
React-AB-Test (on npm)
https://github.com/pushtell/react-ab-test
What is A/B-testing?
Slide 13
Slide 13 text
”
The only true wisdom
is in knowing you
know nothing.
Socrates - allegedly...
● Be skeptical
● Don’t measure too much
● Have a good basis for comparison
Slide 14
Slide 14 text
o Google Analytics API
o Scroll and time tracking
o Event tracking
o Tracking the correct event
o Follow a standard
o Write documentation
o Less is more(ish)
o Evaluating the numbers
Evaluation
Slide 15
Slide 15 text
o A good grounds for comparison
o Avoid noise in your data
o Supplement qualitatively...
"Lies, damned lies, and statistics"
Evaluation
Slide 16
Slide 16 text
Evaluation
What is the user’s real intent?
Slide 17
Slide 17 text
Evaluation
The original Variation 1 Variation 2
Slide 18
Slide 18 text
Evaluation
Slide 19
Slide 19 text
Evaluation
Slide 20
Slide 20 text
Let’s see how we can perform A/B-testing in practice: look at
code and run it in the browser.
https://github.com/eivind88/ab-testing-react-meetup/tree/mast
er
Demo
Slide 21
Slide 21 text
o It is not as complicated as it might seem
• The thought process is more the half the job
o Keep it simple and structured
• Follow a standard and write documentation
o Keep your experiments limited
o You don’t know it if you haven’t proven it
[email protected][email protected]
In summary