$30 off During Our Annual Pro Sale. View Details »

Johnny Ross_Rebuild Better with Experimentation_UX Y'all 2022

UX Y'all
September 27, 2022

Johnny Ross_Rebuild Better with Experimentation_UX Y'all 2022

UX Y'all

September 27, 2022

More Decks by UX Y'all

Other Decks in Design


  1. Rebuild Better With Experimentation Johnny Ross | UX Researcher @

  2. IDEAS

  3. Gary Vaynerchuk “Ideas are worthless without execution; execution is pointless

    with the wrong ideas.”
  4. …Test your theories using experiments! Point B: Desired Reality Point

    A: Ideas Test #1 Test #2 Test #3 Before launching costly solutions… $$$$$
  5. The Hitchhiker’s Guide to Experimentation Discovery Experiments Optimization Experiments How

    to Start Experimenting Gain early product direction about what problems to solve & what to build. Test different designs to inform how best to build the solution. Some simple tips to help you start your first experiment.
  6. Discovery Experiments Goal = Prioritize: Avoid weeks/months of effort on

    solutions that don’t resonate. Types of Tests: • Concept Videos • Interest Signups • “Fake Door” Tests Note: This is to supplement generative research; NOT replace it!!
  7. Discovery Example: What’s worth building (first)? On-Premise Waitlist View-Only Menu

  8. Discovery example: Prioritize through concept video & signups KPI TEST

    CONTROL DECISION Beta Signups & Perceived Usefulness Concept video of both solutions + Survey/Signup 500 users (restaurants) shown both options View-Only had significantly more signups & prioritized
  9. Discovery Example: Concept Videos x Beta Signups Design: • Presented

    Concept Video to 500 dine-in focused customers (restaurants). • Surveyed for usefulness, feedback, and beta interest. Results: • View-Only Menus had significantly higher signup rates (82 vs. 38 Waitlist signups). • Digital Ordering team works on View-Only Menus. • Waitlist goes to New Venture backlog.
  10. Optimization Experiments Goal = Optimize: (you don’t say!) Tweak designs

    of existing solutions to improve desired metrics. Types of Tests: • A/B Tests (code) • A/B Tests (no or low-code) Note: This is to supplement usability testing; NOT replace it!!
  11. Optimization example: Boosting adoption with low-code A/B test KPI TEST

    CONTROL DECISION Segment creation on product data pages New Create Segment button in a visible place Original button in drop down menu Productionalize new Create Segment button Old Option
  12. Optimization Example: Boost Adoption with Low-Code A/B Test Design: •

    Created “Segment” button using Pendo. • Ran with 50% of new users. Results: • Increased “Segment” adoption by 127%! Previous Insight: • Users create target “segments” to filter product data. • Users (especially new ones) rarely use the dropdown option. A: Old (Only) Option B: New Option
  13. Tips for Getting Started with Experiments Start with the end

    in mind Just get started, scale later Build & share with others Know your impact metrics and develop hypotheses on how to influence them. Focus on getting experiments out and then figure out how to scale a program. Partner with other teams to plan & run experiments. Share your results broadly!
  14. One more thing…

  15. Test Boldly

  16. Resources to Get Started Start at the End Matt Wallaert

    Testing Business Ideas Alexander Osterwalder David J. Bland Guide: How-To Create Experiments Johnny Ross | UX Researcher Link
  17. CREDITS: This presentation template was created by Slidesgo, including icons

    by Flaticon, and infographics & images by Freepik Thanks! Reach out on LinkedIn! Johnny Ross UX Research @ Pendo.io https://www.linkedin.com/in/johnny-c-ross/
  18. Icon pack: sky map