$30 off During Our Annual Pro Sale. View Details »

ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

ADDC 2019 - Jason van der Merwe - Building and scaling a culture of AB testing

Product experimentation offers great potential for product teams to meet business goals and to build stronger teams. By replacing the fear of failure with the opportunity to learn, AB testing provides a safety net that encourages teams to take risks. When product ideas come from only a few members of the team, we limit opportunities for internal creativity and leadership. Experimentation invites all team members to participate in ideation while promoting ownership and entrepreneurship.

Just a few years ago, AB testing wasn’t prevalent at Strava. In this talk, I will tell the story of how we built and scaled an experimentation culture that now runs several hundred AB tests a year. I will go into detail about the tools, technology and processes we have developed and how these changes have had a positive impact on our product teams.

Recordings & more: https://addconf.com/

More Decks by ADDC - App Design & Development Conference

Other Decks in Technology

Transcript

  1. How Strava built and scaled
    a culture of experimentation
    Jason van der Merwe, Engineering Manager @ Strava

    View Slide

  2. Connect athletes
    to what motivates them
    and help them find
    their personal best.

    View Slide

  3. 2 billions activities
    42+ million athletes
    82% of our athletes live
    outside of the
    United-States

    View Slide

  4. 4 years at Strava
    Born in South Africa,
    live in San Francisco
    Want to open
    a restaurant someday

    View Slide

  5. you have ever had an idea of how
    to improve the product you’re
    working on.
    Raise your hand if ...

    View Slide

  6. someone told you: “I don’t think
    that idea will work”.
    Raise your hand if ...

    View Slide

  7. you thought that person
    was wrong.
    Raise your hand if ...

    View Slide

  8. you have ever told someone
    “I think your idea won’t work”.
    Raise your hand if ...

    View Slide

  9. is not a very scientific way to make
    decisions about what
    we’re building.
    “I think”

    View Slide

  10. View Slide

  11. How can we remove bias from
    what we build and learn what our
    users actually want?

    View Slide

  12. How can we remove bias from
    what we build and learn what our
    users actually want?
    A/B testing!

    View Slide

  13. James Lind,
    British Naval Doctor
    1747 and scurvy is a
    big problem
    12 sailors on board
    contract scurvy

    View Slide

  14. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar
    Half a pint of seawater 2 oranges, 1 lemon A spicy paste

    View Slide

  15. A quart of cider 25 drops of sulfuric acid 6 spoonfuls of vinegar
    Half a pint of seawater 2 oranges, 1 lemon A spicy paste

    View Slide

  16. James Lind discovered
    what worked,
    even though he didn’t
    know why.

    View Slide

  17. A/B testing can help you
    scientifically determine which
    solution best solves your
    user problem.

    View Slide

  18. 5
    experiments
    2015
    25
    experiments
    2016
    300+
    experiments
    2019

    View Slide

  19. Cohorting
    How we segment our users in a test
    Reporting
    Process
    How we know what happened in a test
    How can we learn as quickly as possible?

    View Slide

  20. Cohorting
    2015
    Starting using Apptimize for mobile experiments
    2017
    Built our own server side cohorting service
    2019
    Replaced Apptimize by extending our server side cohorting
    to mobile

    View Slide

  21. Reporting: product analytics
    2015
    Built a strongly typed analytics system with Protocol Buffers
    2017
    Moved to Segment.IO for experiment analytics
    2018
    Built a new analytics platform using Snowplow events

    View Slide

  22. 2015-2018
    All experiments analyzed manually by an analyst
    2018
    OneRing beta rolled out with Growth Team metrics
    2019
    OneRing contains all company metrics
    Reporting: business metrics

    View Slide

  23. Reporting: business metrics

    View Slide

  24. 2015
    All code merged is expected to be robust and built to last
    2017
    Experimental code guidelines rolled out
    2018
    SlackBot built to alert on experiment status
    Process

    View Slide

  25. 5
    experiments
    2015
    25
    experiments
    2016
    300+
    experiments
    2019

    View Slide

  26. A culture of experimentation

    View Slide

  27. 1. Encourages risk taking by providing a
    safety net that promotes learning
    2. Increases the collaboration and creativity
    of the team, resulting in better ideas
    3. Promotes leadership
    and entrepreneurship on the team

    View Slide

  28. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.

    View Slide

  29. Any product plan is born out of a
    set of assumptions, which we like
    to masquerade as “intuition”.

    View Slide

  30. View Slide

  31. “The mere fact that you have an
    idea and nothing else comes to
    mind and you feel a great deal of
    confidence — absolutely does not
    guarantee accuracy,”
    — Daniel Kahneman

    View Slide

  32. View Slide

  33. A culture of experimentation
    encourages learning about
    whether those assumptions are
    correct or not before investing
    heavily in a final solution.

    View Slide

  34. In a culture of experimentation
    the goal is learning.

    View Slide

  35. When the goal is learning, it’s more
    acceptable to take risks.

    View Slide

  36. What happens when you don’t
    prioritize learnings?

    View Slide

  37. View Slide

  38. “We rushed our redesign,
    solving one problem but
    creating many others.
    Unfortunately, we didn’t give
    ourselves enough time to
    continue iterating and
    testing the redesign with a
    smaller percentage of our
    community …
    Quote from https://cheddar.com/media/snap-ceo-evan-spiegel-company-memo-on-2019-strategic-goals-and-profitability

    View Slide

  39. ... As a result, we had to
    continue our iterations after
    we launched, causing a lot of
    frustration for our
    community.”
    — Evan Spiegel

    View Slide

  40. “Spiegel always went on his
    gut rather than relying on
    user data like Facebook.
    Aging further and further
    away from his core
    audience, he misread what
    teens cared about.”
    — John Constine,
    TechCrunch

    View Slide

  41. By replacing the fear of failure with
    the opportunity to learn,
    experimentation provides a safety
    net that encourages teams
    to take risks.

    View Slide

  42. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.

    View Slide

  43. Define
    requirements
    Measure solution
    Implement solution
    Design solution

    View Slide

  44. Product Manager
    Determines what we build
    Designer
    Determines how it looks and behaves
    Engineer
    Builds what has been determined
    Analyst
    Measures what we build

    View Slide

  45. it is everyone’s responsibility to
    ship great product to users

    View Slide

  46. Define user or
    business problem
    Define requirements
    for experiment
    Develop hypothesis
    Brainstorm
    potential solutions
    Product Manager Everyone
    Everyone Everyone

    View Slide

  47. How does experimentation
    increase creativity and
    collaboration?

    View Slide

  48. A culture of experimentation
    values a willingness to try ideas
    you might not have tried before -
    with the goal of learning.

    View Slide

  49. Experimentation gives us more
    reasons to try new things,
    and fewer reasons to say no.

    View Slide

  50. Experiments Day
    #1
    Target metric must be a current focus for the team
    #2
    Target at 10% users
    #3
    Must be small enough to implement in an afternoon

    View Slide

  51. View Slide

  52. 27,000 new follows during the test
    over 17 days.

    View Slide

  53. 1. Encourages risk taking by providing a
    safety net that promotes learning
    Increases the collaboration and creativity
    of the team, resulting in better ideas
    Promotes leadership
    and entrepreneurship on the team
    2.
    3.

    View Slide

  54. Project Aristotle sought to answer:
    What makes a Google team
    effective?

    View Slide

  55. Psychological safety is ‘‘a
    sense of confidence that the
    team will not embarrass, reject
    or punish someone for
    speaking up… the shared
    belief held by members of a
    team that the team is safe for
    interpersonal risk-taking.’’
    — Amy Edmondson,
    Harvard Business School

    View Slide

  56. A culture of experimentation that
    encourages risk taking and ideas
    from everyone, is a culture where
    psychological safety can grow
    and flourish.

    View Slide

  57. A culture of experimentation is a
    safe place to allow team members
    to step up and be leaders.

    View Slide

  58. A culture of experimentation
    builds stronger teams

    View Slide

  59. What if i’m not on
    a cross-functional team?

    View Slide

  60. Everyone has the right to be creative and
    to be a leader.
    Be slow to judge others’ ideas.
    Be quick to collaborate with your teammates.
    Take risks and be willing to try new things.

    View Slide

  61. Questions?
    [email protected]
    strava.com/athletes/jasonvdm
    @jasonvdmerwe

    View Slide