HubSpot’s growth process and how we prioritize experiments

Slide 1

Slide 1 text

DAVIDLYKHIM.COM HubSpot's Growth Process and How We Prioritize Experiments @DAVIDLYKHIM | DAVIDLYKHIM.COM

Slide 2

Slide 2 text

DAVIDLYKHIM.COM Imagine  This…

Slide 3

Slide 3 text

DAVIDLYKHIM.COM Revenue Time

Slide 4

Slide 4 text

DAVIDLYKHIM.COM

Slide 5

Slide 5 text

DAVIDLYKHIM.COM Get the app that will immediately improve your productivity by 72%. SIGN UP NOW A Get the app that will immediately improve your productivity by 72%. SIGN UP NOW B

Slide 6

Slide 6 text

DAVIDLYKHIM.COM Revenue Time

Slide 7

Slide 7 text

DAVIDLYKHIM.COM So you run more experiments…

Slide 8

Slide 8 text

DAVIDLYKHIM.COM Revenue Time

Slide 9

Slide 9 text

DAVIDLYKHIM.COM What’s Happening?

Slide 10

Slide 10 text

DAVIDLYKHIM.COM Example Problems

Slide 11

Slide 11 text

DAVIDLYKHIM.COM Example Problems We get traffic to our landing page but people aren’t signing up for the product. 01

Slide 12

Slide 12 text

DAVIDLYKHIM.COM Example Problems We get traffic to our landing page but people aren’t signing up for the product. 01 People sign up for our product but they sign in once and never return. 02

Slide 13

Slide 13 text

DAVIDLYKHIM.COM Example Problems People sign up for our product but they sign in once and never return. 02 We get traffic to our landing page but people aren’t signing up for the product. 01 People use our free product, but they don’t upgrade to a paid product. 03

Slide 14

Slide 14 text

DAVIDLYKHIM.COM

Slide 15

Slide 15 text

DAVIDLYKHIM.COM What we’ll cover today Why We Run Experiments Obsession of Documentation Our Experiment Process Example Experiments

Slide 16

Slide 16 text

DAVIDLYKHIM.COM Experiments help us understand how to increase a metric which often means getting more customers and generating more revenue.

Slide 17

Slide 17 text

DAVIDLYKHIM.COM Anyone can do a tactic.

Slide 18

Slide 18 text

DAVIDLYKHIM.COM Effectiveness Time Tactic Discovered 01 Tactic Optimized 02 Tactic Adopted Widely 03 Tactic Fatigues 04 Life Cycle of Growth Tactics Source: BrianBalfour.com

Slide 19

Slide 19 text

DAVIDLYKHIM.COM The difference between experimentation and experimentation that works is having a growth process and strategy.

Slide 20

Slide 20 text

DAVIDLYKHIM.COM Without the scientific method, we’d just be “growth hacking” — trying random things from loud blogs without learning about our users. Tal Raviv | Patreon, Product Manager PatreonHQ.com | “That’s Not a Hypothesis”

Slide 21

Slide 21 text

DAVIDLYKHIM.COM How do you avoid growth hacking?

Slide 22

Slide 22 text

DAVIDLYKHIM.COM A fine filter for high quality experiments Obsess over documentation Revisit and update documentation

Slide 23

Slide 23 text

DAVIDLYKHIM.COM

Slide 24

Slide 24 text

DAVIDLYKHIM.COM Hypothesis Objective Statement Experiment Design Predicted Outcome Success Metrics Analysis Learnings Next Steps HubSpot’s Experimentation Process

Slide 25

Slide 25 text

DAVIDLYKHIM.COM Hypothesis A statement of what you believe to be true about your customers today.

Slide 26

Slide 26 text

DAVIDLYKHIM.COM A hypothesis is not a prediction.

Slide 27

Slide 27 text

DAVIDLYKHIM.COM Example Hypothesis We believe that people are overwhelmed by all the questions in our sign up flow.

Slide 28

Slide 28 text

DAVIDLYKHIM.COM The hypothesis serves as a guardrail for brainstorming experiment ideas.

Slide 29

Slide 29 text

DAVIDLYKHIM.COM Prediction What you believe will happen if your hypothesis is true.

Slide 30

Slide 30 text

DAVIDLYKHIM.COM Example Prediction If we display social proof in the signup flow, then people will feel reassured and will be more likely to finish the signup process.

Slide 31

Slide 31 text

DAVIDLYKHIM.COM

Slide 32

Slide 32 text

DAVIDLYKHIM.COM

Slide 33

Slide 33 text

DAVIDLYKHIM.COM

Slide 34

Slide 34 text

DAVIDLYKHIM.COM

Slide 35

Slide 35 text

DAVIDLYKHIM.COM

Slide 36

Slide 36 text

DAVIDLYKHIM.COM

Slide 37

Slide 37 text

DAVIDLYKHIM.COM

Slide 38

Slide 38 text

DAVIDLYKHIM.COM If we combine the first and second screens, then people will feel like there are fewer steps and the completion rate will increase.

Slide 39

Slide 39 text

DAVIDLYKHIM.COM If we remove questions about their role and only ask for email and password, then we can maximize the number of people signing up.

Slide 40

Slide 40 text

DAVIDLYKHIM.COM If we display social proof in the sign up flow, then people will feel more reassured about the product and the completion rate will increase.

Slide 41

Slide 41 text

DAVIDLYKHIM.COM How do you expect the experiment to impact the metric? By how much? How much effort is required to align with other teams? To get buy-in? What technical challenges are there? IMPACT EFFORT

Slide 42

Slide 42 text

DAVIDLYKHIM.COM Low Effort High Effort Low Impact High Impact Maybe do it. Just do it. Don’t do it. Make time to do it.

Slide 43

Slide 43 text

DAVIDLYKHIM.COM Objective  Statement What you hope to accomplish by running the experiment.

Slide 44

Slide 44 text

DAVIDLYKHIM.COM Example Objective  Statement The main objective of this experiment is to increase the number of people signing up for HubSpot’s free products.

Slide 45

Slide 45 text

DAVIDLYKHIM.COM Experiment  Design Ensure that the right type of data, and enough of it, to answer your questions as clearly as possible.

Slide 46

Slide 46 text

DAVIDLYKHIM.COM Experiment  Design  Includes How many variants will you test? How will the experiment variant(s) look? What sample size is needed?

Slide 47

Slide 47 text

DAVIDLYKHIM.COM Experiment Design Control: Shown to 25% If we display social proof in the sign up flow, then people will feel more reassured about the product and the completion rate will increase.

Slide 48

Slide 48 text

DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2: Shown to 25% Variant 3: Shown to 25% We will run the experiment for the CRM sign up flow for 14 days and need at least 20,000 exposures to reach statistical significance.

Slide 49

Slide 49 text

DAVIDLYKHIM.COM Predicted  Outcome What you hope to accomplish by running the experiment.

Slide 50

Slide 50 text

DAVIDLYKHIM.COM Example Predicted  Outcome We predict that the completion rate of the signup flow will increase by 5 percentage points. If our hypothesis is true…

Slide 51

Slide 51 text

DAVIDLYKHIM.COM Example Predicted  Outcome We predict that the completion rate of the signup flow will increase by 5 percentage points, resulting in 300 more signups per week. At an upgrade rate of 10% and an average sales price of $20, that will result in an additional $600 in monthly recurring revenue.* If our hypothesis is true… *Fake numbers.

Slide 52

Slide 52 text

DAVIDLYKHIM.COM Peer Review

Slide 53

Slide 53 text

DAVIDLYKHIM.COM The experiment goes live!

Slide 54

Slide 54 text

DAVIDLYKHIM.COM Post-Mortem An open discussion at the conclusion of a project, where you and your team can identify and analyze all aspects related to the project.

Slide 55

Slide 55 text

DAVIDLYKHIM.COM Experiment  Post-Mortem Why was our hypothesis and/or prediction right/ wrong? Why did the experiment turned out the way it did? (For both successes and failures.) What are follow up experiments to continue testing this hypothesis (if it’s still worth testing)?

Slide 56

Slide 56 text

DAVIDLYKHIM.COM Let’s talk about some experiments.

Slide 57

Slide 57 text

DAVIDLYKHIM.COM We believe that people are overwhelmed by all the questions in our sign up flow. If we display social proof in the signup flow, then people will feel reassured and finish signing up. Increase the number of people signing up for HubSpot’s free products. If our hypothesis is true, we expect to see a 3 percentage point improvement in signup completion, translating to 1,000 additional signups per week and $10,000 in additional MRR.* Hypothesis Objective Expected Outcome Prediction

Slide 58

Slide 58 text

DAVIDLYKHIM.COM Experiment Design Control: Shown to 25%

Slide 59

Slide 59 text

DAVIDLYKHIM.COM Experiment Design Variant 1: Shown to 25% Variant 2: Shown to 25% Variant 3: Shown to 25%

Slide 60

Slide 60 text

DAVIDLYKHIM.COM Results Data was inconclusive.

Slide 61

Slide 61 text

DAVIDLYKHIM.COM Learnings Variations of social proof did not positively or negatively impact the signup completion rate. We should think about how different types of social proof may perform in different regions. We should think about other, more impactful ways, to improve the signup completion rate.

Slide 62

Slide 62 text

DAVIDLYKHIM.COM What were  next steps?

Slide 63

Slide 63 text

DAVIDLYKHIM.COM The signup app has tech debt and takes 5-10 seconds to load. The signup app backend is split into two parts and sometimes the second part doesn’t load. The design of the app is inconsistent with the website style guide and may confuse the user. What’s preventing the signup completion rate from being better?

Slide 64

Slide 64 text

DAVIDLYKHIM.COM We believe that people who sign up for a HubSpot CRM Free account are getting fatigued and in some cases confused when signing up for an account. If we improve the signup load time with an improve design, the signup completion rate will improve. The main objective of this experiment is to understand whether improving the signup load time with a visually consistent design will impact ability to sign up for HubSpot. If our hypothesis is true then we predict that we will see at least a 5% lift in signup completion rate, resulting in in 10,000 more signups a month and $200,000 additional revenue. Hypothesis Objective Expected Outcome Prediction

Slide 65

Slide 65 text

DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03 04 05

Slide 66

Slide 66 text

DAVIDLYKHIM.COM Experiment Design Control: Shown to 50% 01 02 03 04

Slide 67

Slide 67 text

DAVIDLYKHIM.COM Results Control

Slide 68

Slide 68 text

DAVIDLYKHIM.COM Results Variant

Slide 69

Slide 69 text

DAVIDLYKHIM.COM Learnings Reducing steps, adding visual consistency, and being transparent about the number of steps in the signup flow leads to higher conversion rate through the entire signup funnel.

Slide 70

Slide 70 text

DAVIDLYKHIM.COM

Slide 71

Slide 71 text

DAVIDLYKHIM.COM We improve our experimentation process and refine our filter.

Slide 72

Slide 72 text

DAVIDLYKHIM.COM In growth, it isn’t about winning or losing...succeeding or failing.

Slide 73

Slide 73 text

DAVIDLYKHIM.COM You win  or  You learn

Slide 74

Slide 74 text

DAVIDLYKHIM.COM And you grow.

Slide 75

Slide 75 text

DAVIDLYKHIM.COM Good Luck!

Slide 76

Slide 76 text

DAVIDLYKHIM.COM DAVID LY KHIM Growth Product Manager at HubSpot @DAVIDLYKHIM | DAVIDLYKHIM.COM davidlykhim.com/growth Get the full essay and growth resources: