Johnny Ross_Rebuild Better with Experimentation_UX Y'all 2022

Transcript

1. Rebuild Better With Experimentation Johnny Ross | UX Researcher @ Pendo
2. IDEAS
3. Gary Vaynerchuk “Ideas are worthless without execution; execution is pointless with the wrong ideas.”
4. …Test your theories using experiments! Point B: Desired Reality Point A: Ideas Test #1 Test #2 Test #3 Before launching costly solutions… $$$$$
5. The Hitchhiker’s Guide to Experimentation Discovery Experiments Optimization Experiments How to Start Experimenting Gain early product direction about what problems to solve & what to build. Test different designs to inform how best to build the solution. Some simple tips to help you start your first experiment.
6. Discovery Experiments Goal = Prioritize: Avoid weeks/months of effort on solutions that don’t resonate. Types of Tests: • Concept Videos • Interest Signups • “Fake Door” Tests Note: This is to supplement generative research; NOT replace it!!
7. Discovery Example: What’s worth building (first)? On-Premise Waitlist View-Only Menu
8. Discovery example: Prioritize through concept video & signups KPI TEST CONTROL DECISION Beta Signups & Perceived Usefulness Concept video of both solutions + Survey/Signup 500 users (restaurants) shown both options View-Only had significantly more signups & prioritized
9. Discovery Example: Concept Videos x Beta Signups Design: • Presented Concept Video to 500 dine-in focused customers (restaurants). • Surveyed for usefulness, feedback, and beta interest. Results: • View-Only Menus had significantly higher signup rates (82 vs. 38 Waitlist signups). • Digital Ordering team works on View-Only Menus. • Waitlist goes to New Venture backlog.
10. Optimization Experiments Goal = Optimize: (you don’t say!) Tweak designs of existing solutions to improve desired metrics. Types of Tests: • A/B Tests (code) • A/B Tests (no or low-code) Note: This is to supplement usability testing; NOT replace it!!
11. Optimization example: Boosting adoption with low-code A/B test KPI TEST CONTROL DECISION Segment creation on product data pages New Create Segment button in a visible place Original button in drop down menu Productionalize new Create Segment button Old Option
12. Optimization Example: Boost Adoption with Low-Code A/B Test Design: • Created “Segment” button using Pendo. • Ran with 50% of new users. Results: • Increased “Segment” adoption by 127%! Previous Insight: • Users create target “segments” to filter product data. • Users (especially new ones) rarely use the dropdown option. A: Old (Only) Option B: New Option
13. Tips for Getting Started with Experiments Start with the end in mind Just get started, scale later Build & share with others Know your impact metrics and develop hypotheses on how to influence them. Focus on getting experiments out and then figure out how to scale a program. Partner with other teams to plan & run experiments. Share your results broadly!
14. One more thing…
15. Test Boldly
16. Resources to Get Started Start at the End Matt Wallaert Testing Business Ideas Alexander Osterwalder David J. Bland Guide: How-To Create Experiments Johnny Ross | UX Researcher Link
17. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik Thanks! Reach out on LinkedIn! Johnny Ross UX Research @ Pendo.io https://www.linkedin.com/in/johnny-c-ross/
18. Icon pack: sky map