Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

Product experimentation offers great potential for product teams to meet business goals and to build stronger teams. By replacing the fear of failure with the opportunity to learn, AB testing provides a safety net that encourages teams to take risks. When product ideas come from only a few members of the team, we limit opportunities for internal creativity and leadership. Experimentation invites all team members to participate in ideation while promoting ownership and entrepreneurship.

Just a few years ago, AB testing wasn’t prevalent at Strava. In this talk, I will tell the story of how we built and scaled an experimentation culture that now runs several hundred AB tests a year. I will go into detail about the tools, technology and processes we have developed and how these changes have had a positive impact on our product teams.

Recordings & more: https://addconf.com/

More Decks by ADDC - App Design & Development Conference

Other Decks in Technology

Transcript

  1. How Strava built and scaled a culture of experimentation Jason

    van der Merwe, Engineering Manager @ Strava
  2. Connect athletes to what motivates them and help them find

    their personal best.
  3. 2 billions activities 42+ million athletes 82% of our athletes

    live outside of the United-States
  4. 4 years at Strava Born in South Africa, live in

    San Francisco Want to open a restaurant someday
  5. you have ever had an idea of how to improve

    the product you’re working on. Raise your hand if ...
  6. someone told you: “I don’t think that idea will work”.

    Raise your hand if ...
  7. you thought that person was wrong. Raise your hand if

    ...
  8. you have ever told someone “I think your idea won’t

    work”. Raise your hand if ...
  9. is not a very scientific way to make decisions about

    what we’re building. “I think”
  10. None
  11. How can we remove bias from what we build and

    learn what our users actually want?
  12. How can we remove bias from what we build and

    learn what our users actually want? A/B testing!
  13. James Lind, British Naval Doctor 1747 and scurvy is a

    big problem 12 sailors on board contract scurvy
  14. A quart of cider 25 drops of sulfuric acid 6

    spoonfuls of vinegar Half a pint of seawater 2 oranges, 1 lemon A spicy paste
  15. A quart of cider 25 drops of sulfuric acid 6

    spoonfuls of vinegar Half a pint of seawater 2 oranges, 1 lemon A spicy paste
  16. James Lind discovered what worked, even though he didn’t know

    why.
  17. A/B testing can help you scientifically determine which solution best

    solves your user problem.
  18. 5 experiments 2015 25 experiments 2016 300+ experiments 2019

  19. Cohorting How we segment our users in a test Reporting

    Process How we know what happened in a test How can we learn as quickly as possible?
  20. Cohorting 2015 Starting using Apptimize for mobile experiments 2017 Built

    our own server side cohorting service 2019 Replaced Apptimize by extending our server side cohorting to mobile
  21. Reporting: product analytics 2015 Built a strongly typed analytics system

    with Protocol Buffers 2017 Moved to Segment.IO for experiment analytics 2018 Built a new analytics platform using Snowplow events
  22. 2015-2018 All experiments analyzed manually by an analyst 2018 OneRing

    beta rolled out with Growth Team metrics 2019 OneRing contains all company metrics Reporting: business metrics
  23. Reporting: business metrics

  24. 2015 All code merged is expected to be robust and

    built to last 2017 Experimental code guidelines rolled out 2018 SlackBot built to alert on experiment status Process
  25. 5 experiments 2015 25 experiments 2016 300+ experiments 2019

  26. A culture of experimentation

  27. 1. Encourages risk taking by providing a safety net that

    promotes learning 2. Increases the collaboration and creativity of the team, resulting in better ideas 3. Promotes leadership and entrepreneurship on the team
  28. 1. Encourages risk taking by providing a safety net that

    promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
  29. Any product plan is born out of a set of

    assumptions, which we like to masquerade as “intuition”.
  30. None
  31. “The mere fact that you have an idea and nothing

    else comes to mind and you feel a great deal of confidence — absolutely does not guarantee accuracy,” — Daniel Kahneman
  32. None
  33. A culture of experimentation encourages learning about whether those assumptions

    are correct or not before investing heavily in a final solution.
  34. In a culture of experimentation the goal is learning.

  35. When the goal is learning, it’s more acceptable to take

    risks.
  36. What happens when you don’t prioritize learnings?

  37. None
  38. “We rushed our redesign, solving one problem but creating many

    others. Unfortunately, we didn’t give ourselves enough time to continue iterating and testing the redesign with a smaller percentage of our community … Quote from https://cheddar.com/media/snap-ceo-evan-spiegel-company-memo-on-2019-strategic-goals-and-profitability
  39. ... As a result, we had to continue our iterations

    after we launched, causing a lot of frustration for our community.” — Evan Spiegel
  40. “Spiegel always went on his gut rather than relying on

    user data like Facebook. Aging further and further away from his core audience, he misread what teens cared about.” — John Constine, TechCrunch
  41. By replacing the fear of failure with the opportunity to

    learn, experimentation provides a safety net that encourages teams to take risks.
  42. 1. Encourages risk taking by providing a safety net that

    promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
  43. Define requirements Measure solution Implement solution Design solution

  44. Product Manager Determines what we build Designer Determines how it

    looks and behaves Engineer Builds what has been determined Analyst Measures what we build
  45. it is everyone’s responsibility to ship great product to users

  46. Define user or business problem Define requirements for experiment Develop

    hypothesis Brainstorm potential solutions Product Manager Everyone Everyone Everyone
  47. How does experimentation increase creativity and collaboration?

  48. A culture of experimentation values a willingness to try ideas

    you might not have tried before - with the goal of learning.
  49. Experimentation gives us more reasons to try new things, and

    fewer reasons to say no.
  50. Experiments Day #1 Target metric must be a current focus

    for the team #2 Target at 10% users #3 Must be small enough to implement in an afternoon
  51. None
  52. 27,000 new follows during the test over 17 days.

  53. 1. Encourages risk taking by providing a safety net that

    promotes learning Increases the collaboration and creativity of the team, resulting in better ideas Promotes leadership and entrepreneurship on the team 2. 3.
  54. Project Aristotle sought to answer: What makes a Google team

    effective?
  55. Psychological safety is ‘‘a sense of confidence that the team

    will not embarrass, reject or punish someone for speaking up… the shared belief held by members of a team that the team is safe for interpersonal risk-taking.’’ — Amy Edmondson, Harvard Business School
  56. A culture of experimentation that encourages risk taking and ideas

    from everyone, is a culture where psychological safety can grow and flourish.
  57. A culture of experimentation is a safe place to allow

    team members to step up and be leaders.
  58. A culture of experimentation builds stronger teams

  59. What if i’m not on a cross-functional team?

  60. Everyone has the right to be creative and to be

    a leader. Be slow to judge others’ ideas. Be quick to collaborate with your teammates. Take risks and be willing to try new things.
  61. Questions? jasonvdm@strava.com strava.com/athletes/jasonvdm @jasonvdmerwe