HubSpot’s growth process and how we prioritize experiments

Transcript

1. DAVIDLYKHIM.COM HubSpot's Growth Process and How We Prioritize Experiments @DAVIDLYKHIM | DAVIDLYKHIM.COM
2. DAVIDLYKHIM.COM Imagine
 This…
3. DAVIDLYKHIM.COM Revenue Time
4. DAVIDLYKHIM.COM
5. DAVIDLYKHIM.COM Get the app that will immediately improve your productivity by 72%. SIGN UP NOW A Get the app that will immediately improve your productivity by 72%. SIGN UP NOW B
6. DAVIDLYKHIM.COM Revenue Time
7. DAVIDLYKHIM.COM So you run more experiments…
8. DAVIDLYKHIM.COM Revenue Time
9. DAVIDLYKHIM.COM What’s Happening?
10. DAVIDLYKHIM.COM Example Problems
11. DAVIDLYKHIM.COM Example Problems We get traffic to our landing page but people aren’t signing up for the product. 01
12. DAVIDLYKHIM.COM Example Problems We get traffic to our landing page but people aren’t signing up for the product. 01 People sign up for our product but they sign in once and never return. 02
13. DAVIDLYKHIM.COM Example Problems People sign up for our product but they sign in once and never return. 02 We get traffic to our landing page but people aren’t signing up for the product. 01 People use our free product, but they don’t upgrade to a paid product. 03
14. DAVIDLYKHIM.COM
15. DAVIDLYKHIM.COM What we’ll cover today Why We Run Experiments Obsession of Documentation Our Experiment Process Example Experiments
16. DAVIDLYKHIM.COM Experiments help us understand how to increase a metric which often means getting more customers and generating more revenue.
17. DAVIDLYKHIM.COM Anyone can do a tactic.
18. DAVIDLYKHIM.COM Effectiveness Time Tactic Discovered 01 Tactic Optimized 02 Tactic Adopted Widely 03 Tactic Fatigues 04 Life Cycle of Growth Tactics Source: BrianBalfour.com
19. DAVIDLYKHIM.COM The difference between experimentation and experimentation that works is having a growth process and strategy.
20. DAVIDLYKHIM.COM Without the scientific method, we’d just be “growth hacking”  — trying random things from loud blogs without learning about our users. Tal Raviv | Patreon, Product Manager PatreonHQ.com | “That’s Not a Hypothesis”
21. DAVIDLYKHIM.COM How do you avoid growth hacking?
22. DAVIDLYKHIM.COM A fine filter for high quality experiments Obsess over documentation Revisit and update documentation
23. DAVIDLYKHIM.COM
24. DAVIDLYKHIM.COM Hypothesis Objective Statement Experiment Design Predicted Outcome Success Metrics Analysis Learnings Next Steps HubSpot’s Experimentation Process
25. DAVIDLYKHIM.COM Hypothesis A statement of what you believe to be true about your customers today.
26. DAVIDLYKHIM.COM A hypothesis is not a prediction.
27. DAVIDLYKHIM.COM Example Hypothesis We believe that people are overwhelmed by all the questions in our sign up flow.
28. DAVIDLYKHIM.COM The hypothesis serves as a guardrail for brainstorming experiment ideas.
29. DAVIDLYKHIM.COM Prediction What you believe will happen if your hypothesis is true.
30. DAVIDLYKHIM.COM Example Prediction If we display social proof in the signup flow, then people will feel reassured and will be more likely to finish the signup process.
31. DAVIDLYKHIM.COM
32. DAVIDLYKHIM.COM
33. DAVIDLYKHIM.COM
34. DAVIDLYKHIM.COM
35. DAVIDLYKHIM.COM
36. DAVIDLYKHIM.COM
37. DAVIDLYKHIM.COM
38. DAVIDLYKHIM.COM If we combine the first and second screens, then people will feel like there are fewer steps and the completion rate will increase.
39. DAVIDLYKHIM.COM If we remove questions about their role and only ask for email and password, then we can maximize the number of people signing up.
40. DAVIDLYKHIM.COM If we display social proof in the sign up flow, then people will feel more reassured about the product and the completion rate will increase.
41. DAVIDLYKHIM.COM How do you expect the experiment to impact the metric? By how much? How much effort is required to align with other teams? To get buy-in? What technical challenges are there? IMPACT EFFORT
42. DAVIDLYKHIM.COM Low Effort High Effort Low Impact High Impact Maybe do it. Just do it. Don’t do it. Make time to do it.
43. DAVIDLYKHIM.COM Objective
 Statement What you hope to accomplish by running the experiment.
44. DAVIDLYKHIM.COM Example Objective
 Statement The main objective of this experiment is to increase the number of people signing up for HubSpot’s free products.
45. DAVIDLYKHIM.COM Experiment
 Design Ensure that the right type of data, and enough of it, to answer your questions as clearly as possible.
46. DAVIDLYKHIM.COM Experiment
 Design
 Includes How many variants will you test? How will the experiment variant(s) look? What sample size is needed?
47. DAVIDLYKHIM.COM Experiment Design Control: Shown to 25% If we display social proof in the sign up flow, then people will feel more reassured about the product and the completion rate will increase.
48. DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2: Shown to 25% Variant 3: Shown to 25% We will run the experiment for the CRM sign up flow for 14 days and need at least 20,000 exposures to reach statistical significance.
49. DAVIDLYKHIM.COM Predicted
 Outcome What you hope to accomplish by running the experiment.
50. DAVIDLYKHIM.COM Example Predicted
 Outcome We predict that the completion rate of the signup flow will increase by 5 percentage points. If our hypothesis is true…
51. DAVIDLYKHIM.COM Example Predicted
 Outcome We predict that the completion rate of the signup flow will increase by 5 percentage points, resulting in 300 more signups per week. At an upgrade rate of 10% and an average sales price of $20, that will result in an additional $600 in monthly recurring revenue.* If our hypothesis is true… *Fake numbers.
52. DAVIDLYKHIM.COM Peer Review
53. DAVIDLYKHIM.COM The experiment goes live!
54. DAVIDLYKHIM.COM Post-Mortem An open discussion at the conclusion of a project, where you and your team can identify and analyze all aspects related to the project.
55. DAVIDLYKHIM.COM Experiment
 Post-Mortem Why was our hypothesis and/or prediction right/ wrong? Why did the experiment turned out the way it did? (For both successes and failures.) What are follow up experiments to continue testing this hypothesis (if it’s still worth testing)?
56. DAVIDLYKHIM.COM Let’s talk about some experiments.
57. DAVIDLYKHIM.COM We believe that people are overwhelmed by all the questions in our sign up flow. If we display social proof in the signup flow, then people will feel reassured and finish signing up. Increase the number of people signing up for HubSpot’s free products. If our hypothesis is true, we expect to see a 3 percentage point improvement in signup completion, translating to 1,000 additional signups per week and $10,000 in additional MRR.* Hypothesis Objective Expected Outcome Prediction
58. DAVIDLYKHIM.COM Experiment Design Control: Shown to 25%
59. DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2: Shown to 25% Variant 3: Shown to 25%
60. DAVIDLYKHIM.COM Results Data was inconclusive.
61. DAVIDLYKHIM.COM Learnings Variations of social proof did not positively or negatively impact the signup completion rate. We should think about how different types of social proof may perform in different regions. We should think about other, more impactful ways, to improve the signup completion rate.
62. DAVIDLYKHIM.COM What were
 next steps?
63. DAVIDLYKHIM.COM The signup app has tech debt and takes 5-10 seconds to load. The signup app backend is split into two parts and sometimes the second part doesn’t load. The design of the app is inconsistent with the website style guide and may confuse the user. What’s preventing the signup completion rate from being better?
64. DAVIDLYKHIM.COM We believe that people who sign up for a HubSpot CRM Free account are getting fatigued and in some cases confused when signing up for an account. If we improve the signup load time with an improve design, the signup completion rate will improve. The main objective of this experiment is to understand whether improving the signup load time with a visually consistent design will impact ability to sign up for HubSpot. If our hypothesis is true then we predict that we will see at least a 5% lift in signup completion rate, resulting in in 10,000 more signups a month and $200,000 additional revenue. Hypothesis Objective Expected Outcome Prediction
65. DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03 04 05
66. DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03 04
67. DAVIDLYKHIM.COM Results Control
68. DAVIDLYKHIM.COM Results Variant
69. DAVIDLYKHIM.COM Learnings Reducing steps, adding visual consistency, and being transparent about the number of steps in the signup flow leads to higher conversion rate through the entire signup funnel.
70. DAVIDLYKHIM.COM
71. DAVIDLYKHIM.COM We improve our experimentation process and refine our filter.
72. DAVIDLYKHIM.COM In growth, it isn’t about winning or losing...succeeding or failing.
73. DAVIDLYKHIM.COM You win
 or
 You learn
74. DAVIDLYKHIM.COM And you grow.
75. DAVIDLYKHIM.COM Good Luck!
76. DAVIDLYKHIM.COM DAVID LY KHIM Growth Product Manager at HubSpot @DAVIDLYKHIM | DAVIDLYKHIM.COM davidlykhim.com/growth Get the full essay and growth resources: