Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HubSpot’s growth process and how we prioritize experiments

HubSpot’s growth process and how we prioritize experiments

464a90eb6ea6699a621cf138541d65f8?s=128

Product-Led Alliance

April 21, 2021
Tweet

More Decks by Product-Led Alliance

Other Decks in Education

Transcript

  1. DAVIDLYKHIM.COM HubSpot's Growth Process and How We Prioritize Experiments @DAVIDLYKHIM

    | DAVIDLYKHIM.COM
  2. DAVIDLYKHIM.COM Imagine
 This…

  3. DAVIDLYKHIM.COM Revenue Time

  4. DAVIDLYKHIM.COM

  5. DAVIDLYKHIM.COM Get the app that will immediately improve your productivity

    by 72%. SIGN UP NOW A Get the app that will immediately improve your productivity by 72%. SIGN UP NOW B
  6. DAVIDLYKHIM.COM Revenue Time

  7. DAVIDLYKHIM.COM So you run more experiments…

  8. DAVIDLYKHIM.COM Revenue Time

  9. DAVIDLYKHIM.COM What’s Happening?

  10. DAVIDLYKHIM.COM Example Problems

  11. DAVIDLYKHIM.COM Example Problems We get traffic to our landing page

    but people aren’t signing up for the product. 01
  12. DAVIDLYKHIM.COM Example Problems We get traffic to our landing page

    but people aren’t signing up for the product. 01 People sign up for our product but they sign in once and never return. 02
  13. DAVIDLYKHIM.COM Example Problems People sign up for our product but

    they sign in once and never return. 02 We get traffic to our landing page but people aren’t signing up for the product. 01 People use our free product, but they don’t upgrade to a paid product. 03
  14. DAVIDLYKHIM.COM

  15. DAVIDLYKHIM.COM What we’ll cover today Why We Run Experiments Obsession

    of Documentation Our Experiment Process Example Experiments
  16. DAVIDLYKHIM.COM Experiments help us understand how to increase a metric

    which often means getting more customers and generating more revenue.
  17. DAVIDLYKHIM.COM Anyone can do a tactic.

  18. DAVIDLYKHIM.COM Effectiveness Time Tactic Discovered 01 Tactic Optimized 02 Tactic

    Adopted Widely 03 Tactic Fatigues 04 Life Cycle of Growth Tactics Source: BrianBalfour.com
  19. DAVIDLYKHIM.COM The difference between experimentation and experimentation that works is

    having a growth process and strategy.
  20. DAVIDLYKHIM.COM Without the scientific method, we’d just be “growth hacking” 

    — trying random things from loud blogs without learning about our users. Tal Raviv | Patreon, Product Manager PatreonHQ.com | “That’s Not a Hypothesis”
  21. DAVIDLYKHIM.COM How do you avoid growth hacking?

  22. DAVIDLYKHIM.COM A fine filter for high quality experiments Obsess over

    documentation Revisit and update documentation
  23. DAVIDLYKHIM.COM

  24. DAVIDLYKHIM.COM Hypothesis Objective Statement Experiment Design Predicted Outcome Success Metrics

    Analysis Learnings Next Steps HubSpot’s Experimentation Process
  25. DAVIDLYKHIM.COM Hypothesis A statement of what you believe to be

    true about your customers today.
  26. DAVIDLYKHIM.COM A hypothesis is not a prediction.

  27. DAVIDLYKHIM.COM Example Hypothesis We believe that people are overwhelmed by

    all the questions in our sign up flow.
  28. DAVIDLYKHIM.COM The hypothesis serves as a guardrail for brainstorming experiment

    ideas.
  29. DAVIDLYKHIM.COM Prediction What you believe will happen if your hypothesis

    is true.
  30. DAVIDLYKHIM.COM Example Prediction If we display social proof in the

    signup flow, then people will feel reassured and will be more likely to finish the signup process.
  31. DAVIDLYKHIM.COM

  32. DAVIDLYKHIM.COM

  33. DAVIDLYKHIM.COM

  34. DAVIDLYKHIM.COM

  35. DAVIDLYKHIM.COM

  36. DAVIDLYKHIM.COM

  37. DAVIDLYKHIM.COM

  38. DAVIDLYKHIM.COM If we combine the first and second screens, then

    people will feel like there are fewer steps and the completion rate will increase.
  39. DAVIDLYKHIM.COM If we remove questions about their role and only

    ask for email and password, then we can maximize the number of people signing up.
  40. DAVIDLYKHIM.COM If we display social proof in the sign up

    flow, then people will feel more reassured about the product and the completion rate will increase.
  41. DAVIDLYKHIM.COM How do you expect the experiment to impact the

    metric? By how much? How much effort is required to align with other teams? To get buy-in? What technical challenges are there? IMPACT EFFORT
  42. DAVIDLYKHIM.COM Low Effort High Effort Low Impact High Impact Maybe

    do it. Just do it. Don’t do it. Make time to do it.
  43. DAVIDLYKHIM.COM Objective
 Statement What you hope to accomplish by running

    the experiment.
  44. DAVIDLYKHIM.COM Example Objective
 Statement The main objective of this experiment

    is to increase the number of people signing up for HubSpot’s free products.
  45. DAVIDLYKHIM.COM Experiment
 Design Ensure that the right type of data,

    and enough of it, to answer your questions as clearly as possible.
  46. DAVIDLYKHIM.COM Experiment
 Design
 Includes How many variants will you test?

    How will the experiment variant(s) look? What sample size is needed?
  47. DAVIDLYKHIM.COM Experiment Design Control: Shown to 25% If we display

    social proof in the sign up flow, then people will feel more reassured about the product and the completion rate will increase.
  48. DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2:

    Shown to 25% Variant 3: Shown to 25% We will run the experiment for the CRM sign up flow for 14 days and need at least 20,000 exposures to reach statistical significance.
  49. DAVIDLYKHIM.COM Predicted
 Outcome What you hope to accomplish by running

    the experiment.
  50. DAVIDLYKHIM.COM Example Predicted
 Outcome We predict that the completion rate

    of the signup flow will increase by 5 percentage points. If our hypothesis is true…
  51. DAVIDLYKHIM.COM Example Predicted
 Outcome We predict that the completion rate

    of the signup flow will increase by 5 percentage points, resulting in 300 more signups per week. At an upgrade rate of 10% and an average sales price of $20, that will result in an additional $600 in monthly recurring revenue.* If our hypothesis is true… *Fake numbers.
  52. DAVIDLYKHIM.COM Peer Review

  53. DAVIDLYKHIM.COM The experiment goes live!

  54. DAVIDLYKHIM.COM Post-Mortem An open discussion at the conclusion of a

    project, where you and your team can identify and analyze all aspects related to the project.
  55. DAVIDLYKHIM.COM Experiment
 Post-Mortem Why was our hypothesis and/or prediction right/

    wrong? Why did the experiment turned out the way it did? (For both successes and failures.) What are follow up experiments to continue testing this hypothesis (if it’s still worth testing)?
  56. DAVIDLYKHIM.COM Let’s talk about some experiments.

  57. DAVIDLYKHIM.COM We believe that people are overwhelmed by all the

    questions in our sign up flow. If we display social proof in the signup flow, then people will feel reassured and finish signing up. Increase the number of people signing up for HubSpot’s free products. If our hypothesis is true, we expect to see a 3 percentage point improvement in signup completion, translating to 1,000 additional signups per week and $10,000 in additional MRR.* Hypothesis Objective Expected Outcome Prediction
  58. DAVIDLYKHIM.COM Experiment Design Control: Shown to 25%

  59. DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2:

    Shown to 25% Variant 3: Shown to 25%
  60. DAVIDLYKHIM.COM Results Data was inconclusive.

  61. DAVIDLYKHIM.COM Learnings Variations of social proof did not positively or

    negatively impact the signup completion rate. We should think about how different types of social proof may perform in different regions. We should think about other, more impactful ways, to improve the signup completion rate.
  62. DAVIDLYKHIM.COM What were
 next steps?

  63. DAVIDLYKHIM.COM The signup app has tech debt and takes 5-10

    seconds to load. The signup app backend is split into two parts and sometimes the second part doesn’t load. The design of the app is inconsistent with the website style guide and may confuse the user. What’s preventing the signup completion rate from being better?
  64. DAVIDLYKHIM.COM We believe that people who sign up for a

    HubSpot CRM Free account are getting fatigued and in some cases confused when signing up for an account. If we improve the signup load time with an improve design, the signup completion rate will improve. The main objective of this experiment is to understand whether improving the signup load time with a visually consistent design will impact ability to sign up for HubSpot. If our hypothesis is true then we predict that we will see at least a 5% lift in signup completion rate, resulting in in 10,000 more signups a month and $200,000 additional revenue. Hypothesis Objective Expected Outcome Prediction
  65. DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03

    04 05
  66. DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03

    04
  67. DAVIDLYKHIM.COM Results Control

  68. DAVIDLYKHIM.COM Results Variant

  69. DAVIDLYKHIM.COM Learnings Reducing steps, adding visual consistency, and being transparent

    about the number of steps in the signup flow leads to higher conversion rate through the entire signup funnel.
  70. DAVIDLYKHIM.COM

  71. DAVIDLYKHIM.COM We improve our experimentation process and refine our filter.

  72. DAVIDLYKHIM.COM In growth, it isn’t about winning or losing...succeeding or

    failing.
  73. DAVIDLYKHIM.COM You win
 or
 You learn

  74. DAVIDLYKHIM.COM And you grow.

  75. DAVIDLYKHIM.COM Good Luck!

  76. DAVIDLYKHIM.COM DAVID LY KHIM Growth Product Manager at HubSpot @DAVIDLYKHIM

    | DAVIDLYKHIM.COM davidlykhim.com/growth Get the full essay and growth resources: