ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

Product experimentation offers great potential for product teams to meet business goals and to build stronger teams. By replacing the fear of failure with the opportunity to learn, AB testing provides a safety net that encourages teams to take risks. When product ideas come from only a few members of the team, we limit opportunities for internal creativity and leadership. Experimentation invites all team members to participate in ideation while promoting ownership and entrepreneurship.

Just a few years ago, AB testing wasn’t prevalent at Strava. In this talk, I will tell the story of how we built and scaled an experimentation culture that now runs several hundred AB tests a year. I will go into detail about the tools, technology and processes we have developed and how these changes have had a positive impact on our product teams.

Recordings & more: https://addconf.com/

Transcript

1. How Strava built and scaled a culture of experimentation Jason van der Merwe, Engineering Manager @ Strava
2. Connect athletes to what motivates them and help them find their personal best.
3. 2 billions activities 42+ million athletes 82% of our athletes live outside of the United-States
4. 4 years at Strava Born in South Africa, live in San Francisco Want to open a restaurant someday
5. you have ever had an idea of how to improve the product you’re working on. Raise your hand if ...
6. someone told you: “I don’t think that idea will work”. Raise your hand if ...
7. you thought that person was wrong. Raise your hand if ...
8. you have ever told someone “I think your idea won’t work”. Raise your hand if ...
9. is not a very scientific way to make decisions about what we’re building. “I think”
11. How can we remove bias from what we build and learn what our users actually want?
12. How can we remove bias from what we build and learn what our users actually want? A/B testing!
13. James Lind, British Naval Doctor 1747 and scurvy is a big problem 12 sailors on board contract scurvy
14. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar Half a pint of seawater 2 oranges, 1 lemon A spicy paste
15. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar Half a pint of seawater 2 oranges, 1 lemon A spicy paste
16. James Lind discovered what worked, even though he didn’t know why.
17. A/B testing can help you scientifically determine which solution best solves your user problem.
18. 5 experiments 2015 25 experiments 2016 300+ experiments 2019
19. Cohorting How we segment our users in a test Reporting Process How we know what happened in a test How can we learn as quickly as possible?
20. Cohorting 2015 Starting using Apptimize for mobile experiments 2017 Built our own server side cohorting service 2019 Replaced Apptimize by extending our server side cohorting to mobile
21. Reporting: product analytics 2015 Built a strongly typed analytics system with Protocol Buffers 2017 Moved to Segment.IO for experiment analytics 2018 Built a new analytics platform using Snowplow events
22. 2015-2018 All experiments analyzed manually by an analyst 2018 OneRing beta rolled out with Growth Team metrics 2019 OneRing contains all company metrics Reporting: business metrics
23. Reporting: business metrics
24. 2015 All code merged is expected to be robust and built to last 2017 Experimental code guidelines rolled out 2018 SlackBot built to alert on experiment status Process
25. 5 experiments 2015 25 experiments 2016 300+ experiments 2019
26. A culture of experimentation
27. 1. Encourages risk taking by providing a safety net that promotes learning 2. Increases the collaboration and creativity of the team, resulting in better ideas 3. Promotes leadership and entrepreneurship on the team
28. 1. Encourages risk taking by providing a safety net that promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
29. Any product plan is born out of a set of assumptions, which we like to masquerade as “intuition”.
31. “The mere fact that you have an idea and nothing else comes to mind and you feel a great deal of confidence — absolutely does not guarantee accuracy,” — Daniel Kahneman
33. A culture of experimentation encourages learning about whether those assumptions are correct or not before investing heavily in a final solution.
34. In a culture of experimentation the goal is learning.
35. When the goal is learning, it’s more acceptable to take risks.
36. What happens when you don’t prioritize learnings?
38. “We rushed our redesign, solving one problem but creating many others. Unfortunately, we didn’t give ourselves enough time to continue iterating and testing the redesign with a smaller percentage of our community … Quote from https://cheddar.com/media/snap-ceo-evan-spiegel-company-memo-on-2019-strategic-goals-and-profitability
39. ... As a result, we had to continue our iterations after we launched, causing a lot of frustration for our community.” — Evan Spiegel
40. “Spiegel always went on his gut rather than relying on user data like Facebook. Aging further and further away from his core audience, he misread what teens cared about.” — John Constine, TechCrunch
41. By replacing the fear of failure with the opportunity to learn, experimentation provides a safety net that encourages teams to take risks.
42. 1. Encourages risk taking by providing a safety net that promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
43. Define requirements Measure solution Implement solution Design solution
44. Product Manager Determines what we build Designer Determines how it looks and behaves Engineer Builds what has been determined Analyst Measures what we build
45. it is everyone’s responsibility to ship great product to users
46. Define user or business problem Define requirements for experiment Develop hypothesis Brainstorm potential solutions Product Manager Everyone Everyone Everyone
47. How does experimentation increase creativity and collaboration?
48. A culture of experimentation values a willingness to try ideas you might not have tried before - with the goal of learning.
49. Experimentation gives us more reasons to try new things, and fewer reasons to say no.
50. Experiments Day #1 Target metric must be a current focus for the team #2 Target at 10% users #3 Must be small enough to implement in an afternoon
52. 27,000 new follows during the test over 17 days.
53. 1. Encourages risk taking by providing a safety net that promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
54. Project Aristotle sought to answer: What makes a Google team effective?
55. Psychological safety is ‘‘a sense of confidence that the team will not embarrass, reject or punish someone for speaking up… the shared belief held by members of a team that the team is safe for interpersonal risk-taking.’’ — Amy Edmondson, Harvard Business School
56. A culture of experimentation that encourages risk taking and ideas from everyone, is a culture where psychological safety can grow and flourish.
57. A culture of experimentation is a safe place to allow team members to step up and be leaders.
58. A culture of experimentation builds stronger teams
59. What if i’m not on a cross-functional team?
60. Everyone has the right to be creative and to be a leader. Be slow to judge others’ ideas. Be quick to collaborate with your teammates. Take risks and be willing to try new things.
61. Questions? jasonvdm@strava.com strava.com/athletes/jasonvdm @jasonvdmerwe