ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

Product experimentation offers great potential for product teams to meet business goals and to build stronger teams. By replacing the fear of failure with the opportunity to learn, AB testing provides a safety net that encourages teams to take risks. When product ideas come from only a few members of the team, we limit opportunities for internal creativity and leadership. Experimentation invites all team members to participate in ideation while promoting ownership and entrepreneurship.

Just a few years ago, AB testing wasn’t prevalent at Strava. In this talk, I will tell the story of how we built and scaled an experimentation culture that now runs several hundred AB tests a year. I will go into detail about the tools, technology and processes we have developed and how these changes have had a positive impact on our product teams.

Recordings & more: https://addconf.com/

Transcript

1. How Strava built and scaled
   a culture of experimentation
   Jason van der Merwe, Engineering Manager @ Strava
   

   View Slide

2. Connect athletes
   to what motivates them
   and help them find
   their personal best.
   

   View Slide

3. 2 billions activities
   42+ million athletes
   82% of our athletes live
   outside of the
   United-States
   

   View Slide

4. 4 years at Strava
   Born in South Africa,
   live in San Francisco
   Want to open
   a restaurant someday
   

   View Slide

5. you have ever had an idea of how
   to improve the product you’re
   working on.
   Raise your hand if ...
   

   View Slide

6. someone told you: “I don’t think
   that idea will work”.
   Raise your hand if ...
   

   View Slide

7. you thought that person
   was wrong.
   Raise your hand if ...
   

   View Slide

8. you have ever told someone
   “I think your idea won’t work”.
   Raise your hand if ...
   

   View Slide

9. is not a very scientific way to make
   decisions about what
   we’re building.
   “I think”
   

   View Slide

10. View Slide

11. How can we remove bias from
    what we build and learn what our
    users actually want?
    

    View Slide

12. How can we remove bias from
    what we build and learn what our
    users actually want?
    A/B testing!
    

    View Slide

13. James Lind,
    British Naval Doctor
    1747 and scurvy is a
    big problem
    12 sailors on board
    contract scurvy
    

    View Slide

14. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar
    Half a pint of seawater 2 oranges, 1 lemon A spicy paste
    

    View Slide

15. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar
    Half a pint of seawater 2 oranges, 1 lemon A spicy paste
    

    View Slide

16. James Lind discovered
    what worked,
    even though he didn’t
    know why.
    

    View Slide

17. A/B testing can help you
    scientifically determine which
    solution best solves your
    user problem.
    

    View Slide

18. 5
    experiments
    2015
    25
    experiments
    2016
    300+
    experiments
    2019
    

    View Slide

19. Cohorting
    How we segment our users in a test
    Reporting
    Process
    How we know what happened in a test
    How can we learn as quickly as possible?
    

    View Slide

20. Cohorting
    2015
    Starting using Apptimize for mobile experiments
    2017
    Built our own server side cohorting service
    2019
    Replaced Apptimize by extending our server side cohorting
    to mobile
    

    View Slide

21. Reporting: product analytics
    2015
    Built a strongly typed analytics system with Protocol Buffers
    2017
    Moved to Segment.IO for experiment analytics
    2018
    Built a new analytics platform using Snowplow events
    

    View Slide

22. 2015-2018
    All experiments analyzed manually by an analyst
    2018
    OneRing beta rolled out with Growth Team metrics
    2019
    OneRing contains all company metrics
    Reporting: business metrics
    

    View Slide

23. Reporting: business metrics
    

    View Slide

24. 2015
    All code merged is expected to be robust and built to last
    2017
    Experimental code guidelines rolled out
    2018
    SlackBot built to alert on experiment status
    Process
    

    View Slide

25. 5
    experiments
    2015
    25
    experiments
    2016
    300+
    experiments
    2019
    

    View Slide

26. A culture of experimentation
    

    View Slide

27. 1. Encourages risk taking by providing a
    safety net that promotes learning
    2. Increases the collaboration and creativity
    of the team, resulting in better ideas
    3. Promotes leadership
    and entrepreneurship on the team
    

    View Slide

28. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.
    

    View Slide

29. Any product plan is born out of a
    set of assumptions, which we like
    to masquerade as “intuition”.
    

    View Slide

30. View Slide

31. “The mere fact that you have an
    idea and nothing else comes to
    mind and you feel a great deal of
    confidence — absolutely does not
    guarantee accuracy,”
    — Daniel Kahneman
    

    View Slide

32. View Slide

33. A culture of experimentation
    encourages learning about
    whether those assumptions are
    correct or not before investing
    heavily in a final solution.
    

    View Slide

34. In a culture of experimentation
    the goal is learning.
    

    View Slide

35. When the goal is learning, it’s more
    acceptable to take risks.
    

    View Slide

36. What happens when you don’t
    prioritize learnings?
    

    View Slide

37. View Slide

38. “We rushed our redesign,
    solving one problem but
    creating many others.
    Unfortunately, we didn’t give
    ourselves enough time to
    continue iterating and
    testing the redesign with a
    smaller percentage of our
    community …
    Quote from https://cheddar.com/media/snap-ceo-evan-spiegel-company-memo-on-2019-strategic-goals-and-profitability
    

    View Slide

39. ... As a result, we had to
    continue our iterations after
    we launched, causing a lot of
    frustration for our
    community.”
    — Evan Spiegel
    

    View Slide

40. “Spiegel always went on his
    gut rather than relying on
    user data like Facebook.
    Aging further and further
    away from his core
    audience, he misread what
    teens cared about.”
    — John Constine,
    TechCrunch
    

    View Slide

41. By replacing the fear of failure with
    the opportunity to learn,
    experimentation provides a safety
    net that encourages teams
    to take risks.
    

    View Slide

42. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.
    

    View Slide

43. Define
    requirements
    Measure solution
    Implement solution
    Design solution
    

    View Slide

44. Product Manager
    Determines what we build
    Designer
    Determines how it looks and behaves
    Engineer
    Builds what has been determined
    Analyst
    Measures what we build
    

    View Slide

45. it is everyone’s responsibility to
    ship great product to users
    

    View Slide

46. Define user or
    business problem
    Define requirements
    for experiment
    Develop hypothesis
    Brainstorm
    potential solutions
    Product Manager Everyone
    Everyone Everyone
    

    View Slide

47. How does experimentation
    increase creativity and
    collaboration?
    

    View Slide

48. A culture of experimentation
    values a willingness to try ideas
    you might not have tried before -
    with the goal of learning.
    

    View Slide

49. Experimentation gives us more
    reasons to try new things,
    and fewer reasons to say no.
    

    View Slide

50. Experiments Day
    #1
    Target metric must be a current focus for the team
    #2
    Target at 10% users
    #3
    Must be small enough to implement in an afternoon
    

    View Slide

51. View Slide

52. 27,000 new follows during the test
    over 17 days.
    

    View Slide

53. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.
    

    View Slide

54. Project Aristotle sought to answer:
    What makes a Google team
    effective?
    

    View Slide

55. Psychological safety is ‘‘a
    sense of confidence that the
    team will not embarrass, reject
    or punish someone for
    speaking up… the shared
    belief held by members of a
    team that the team is safe for
    interpersonal risk-taking.’’
    — Amy Edmondson,
    Harvard Business School
    

    View Slide

56. A culture of experimentation that
    encourages risk taking and ideas
    from everyone, is a culture where
    psychological safety can grow
    and flourish.
    

    View Slide

57. A culture of experimentation is a
    safe place to allow team members
    to step up and be leaders.
    

    View Slide

58. A culture of experimentation
    builds stronger teams
    

    View Slide

59. What if i’m not on
    a cross-functional team?
    

    View Slide

60. Everyone has the right to be creative and
    to be a leader.
    Be slow to judge others’ ideas.
    Be quick to collaborate with your teammates.
    Take risks and be willing to try new things.
    

    View Slide

61. Questions?
    jasonvdm@strava.com
    strava.com/athletes/jasonvdm
    @jasonvdmerwe
    

    View Slide