Slide 1
Slide 1 text
Customer Driven
Development -
What, Why & How?
Ishan Khanna (@droidchef)
Android @ Getaround (Amsterdam)
Slide 2
Slide 2 text
LET’S DEFINE CUSTOMER
DRIVEN DEVELOPMENT
Slide 3
Slide 3 text
KEEP THE CUSTOMER AT THE
_________ OF ____________.
Slide 4
Slide 4 text
KEEP THE CUSTOMER AT THE
CENTER OF ____________.
Slide 5
Slide 5 text
KEEP THE CUSTOMER AT THE
CENTER OF EVERYTHING.
Slide 6
Slide 6 text
BY GIVING THEM THE POWER
TO SHAPE THE PRODUCT
AND
DRIVE YOUR BUSINESS
FORWARD
Slide 7
Slide 7 text
HOW YOU IMPLEMENT
THIS IN PRACTICE?
Slide 8
Slide 8 text
WELCOME A/B TESTS
Slide 9
Slide 9 text
A/B testing means “go[ing] in a
direction where your customers
are telling you that you’re adding
value to their lives.”
Slide 10
Slide 10 text
WHY SHOULD YOU DO THIS?
Slide 11
Slide 11 text
BY UNDERSTANDING YOUR
CUSTOMERS, YOU CAN GIVE
THEM WHAT THEY
ACTUALLY NEED.
Slide 12
Slide 12 text
AND… YOU CAN ALSO
DETERMINE, THE DIRECTION
FORWARD BECAUSE OF THE
DATA YOU COLLECT IN THE
PROCESS.
Slide 13
Slide 13 text
WHATEVER THE
OUTCOME OF YOUR
EXPERIMENT IS,
YOU ALWAYS WIN.
Slide 14
Slide 14 text
SIMPLY BECAUSE YOU
EITHER EARN MORE OR
YOU LEARN MORE.
Slide 15
Slide 15 text
EARNING = METRIC
MOVES IN THE
EXPECTED DIRECTION
Slide 16
Slide 16 text
LEARNING = METRIC
MOVES IN THE
UNEXPECTED DIRECTION
Slide 17
Slide 17 text
FOUNDATIONS OF
AN A/B TEST
Slide 18
Slide 18 text
● HYPOTHESIS
● TARGET AUDIENCE
● PRIMARY METRIC
● RUNTIME
● SECONDARY
METRIC(S)
Slide 19
Slide 19 text
WHAT IS A HYPOTHESIS?
Slide 20
Slide 20 text
SIMPLY PUT, IT IS
AN EDUCATED GUESS
Slide 21
Slide 21 text
EXAMPLE
BY ADDING A TOOLTIP
ON A BUTTON WE WILL
INCREASE CAR
RENTALS.
Slide 22
Slide 22 text
BASE VARIANT
It only takes a few mins
Slide 23
Slide 23 text
Changing (element tested)
from _____ to _____ will
increase/decrease
(a conversion metric).
SYNTAX
Slide 24
Slide 24 text
FORMULATING A
WINNING HYPOTHESIS
Slide 25
Slide 25 text
AIM TO ALTER
CUSTOMER BEHAVIOR
EITHER POSITIVELY OR NEGATIVELY
Slide 26
Slide 26 text
FOCUS ON DERIVING
LEARNINGS FROM THE TEST
Slide 27
Slide 27 text
BACKED BY DATA
Slide 28
Slide 28 text
TEMPLATE FOR
WRITING A WINNING
HYPOTHESIS
Slide 29
Slide 29 text
“BASED ON _______ WE BELIEVE, BY
DOING ______ WE WILL BE ABLE TO
_______. WE WILL SEE WHEN _____
MOVES IN ______ DIRECTION.”
Slide 30
Slide 30 text
“Based on our user research we
believe that by adding a tooltip, we
will be able to inform the users that
booking process is blazing fast. We
will see this when the net rentals
start increasing.”
Slide 31
Slide 31 text
WHAT ARE METRICS?
Slide 32
Slide 32 text
PRIMARY METRIC
Slide 33
Slide 33 text
Which one do you think
should be selected?
● Net Rentals Per Day
● Gross Rentals Per Day
● Cancellations Per Day
● Clicks on “Next” Button
Slide 34
Slide 34 text
Which one do you think
should be selected?
● Net Rentals Per Day
● Gross Rentals Per Day
● Cancellations Per Day
● Clicks on “Next” Button
Slide 35
Slide 35 text
Because:
● It accounts for
cancellations as well.
● If it increases, revenue
definitely increases.
Slide 36
Slide 36 text
SECONDARY METRICS
Slide 37
Slide 37 text
Which one do you think
should be selected?
● Gross Rentals Per Day
● Cancellations Per Day
● Clicks on “Next” Button
Slide 38
Slide 38 text
THEY CAN AID IN
UNDERSTANDING THE
RESULTS OF AN
EXPERIMENT BETTER
AND WITH HIGHER
CONFIDENCE.
Slide 39
Slide 39 text
CHOOSING THE RIGHT
METRICS
Slide 40
Slide 40 text
RUNTIME OF AN EXPERIMENT
Slide 41
Slide 41 text
Statistical power is the
probability of a hypothesis test
of finding an effect if there is an
effect to be found.
UNDERSTANDING POWER
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
IMPORTANCE OF CALCULATING
RUNTIME BEFORE RUNNING
THE EXPERIMENT
Slide 45
Slide 45 text
WHAT IS NOISE?
Slide 46
Slide 46 text
IMAGINE YOU COOK A DISH
AND SERVE IT TO 10 PEOPLE,
ASKING THEIR FEEDBACK
ABOUT THE TASTE.
Slide 47
Slide 47 text
ONLY 6 OF THEM TASTE
IT BUT ALL 10 PROVIDE
A FEEDBACK.
Slide 48
Slide 48 text
DO YOU THINK THE
FEEDBACK OF 4
PEOPLE WHO DIDN’T
TASTE YOUR DISH,
MATTERS?
Slide 49
Slide 49 text
NO, AND THAT IS WHAT
WE CALL NOISE.
Slide 50
Slide 50 text
HOW TO ELIMINATE
NOISE?
Slide 51
Slide 51 text
STAGES TO THE RESCUE
Slide 52
Slide 52 text
STAGES ALLOW YOU TO FILTER
OUT THE USERS WHOSE
BEHAVIOR YOU ARE ACTUALLY
INTERESTED IN GAUGING.
Slide 53
Slide 53 text
TRY TO SEGREGATE YOUR
TRAFFIC TO A POINT WHERE
YOU CAN ELIMINATE
MAXIMUM AMOUNT OF
NOISE.
Slide 54
Slide 54 text
AVOIDING STATISTICAL
GHOSTS
Slide 55
Slide 55 text
A SIGNIFICANT PROBLEM OF
PERCEIVING AN INSIGNIFICANT
RESULT AS A SIGNIFICANT RESULT
Slide 56
Slide 56 text
EXAMPLE
SCENARIO 1 SCENARIO 2 SCENARIO 3 SCENARIO 4
AFTER 200
OBSERVATIONS
INSIGNIFICANT INSIGNIFICANT SIGNIFICANT SIGNIFICANT
AFTER 500 INSIGNIFICANT SIGNIFICANT INSIGNIFICANT SIGNIFICANT
END OF
EXPERIMENT
INSIGNIFICANT SIGNIFICANT INSIGNIFICANT SIGNIFICANT
Suppose you analyze an experiment after 200 and 500 observations. There are
four things that could happen:
Assuming treatments A and B are the same and the significance level is 5%,
then at the end of the experiment, we’ll have a significant result 5% of the
time.
Slide 57
Slide 57 text
SCENARIO 1 SCENARIO 2 SCENARIO 3 SCENARIO 4
AFTER 200
OBSERVATIONS
INSIGNIFICANT INSIGNIFICANT SIGNIFICANT SIGNIFICANT
AFTER 500 INSIGNIFICANT SIGNIFICANT TRIAL STOPPED TRIAL STOPPED
END OF
EXPERIMENT
INSIGNIFICANT SIGNIFICANT SIGNIFICANT SIGNIFICANT
But suppose we stop the experiment as soon as there is a significant result.
Now look at the four things that could happen:
Slide 58
Slide 58 text
LEARNINGS
FIRST ROW EXACTLY THE
SAME AS BEFORE.
ASSUMING A AND B ARE
ACTUALLY THE SAME
WE INCREASED THE RATIO OF
SIGNIFICANT RELATIVE TO
INSIGNIFICANT RESULTS.
Slide 59
Slide 59 text
Therefore, the reported
significance level – the “percent
of the time the observed
difference is due to chance” – will
be wrong.
Slide 60
Slide 60 text
MAKE SURE TO PLAN
FOLLOW UPS
Slide 61
Slide 61 text
LET’S RECAP THE PROCESS
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
THANK YOU!
SLIDES :
http://bit.ly/dc-berlin-2019
QUESTIONS? @droidchef