Justin Cheng, Caroline Lo, Jure Leskovec / Stanford University + Pinterest
Predicting Intent Using Activity Logs
How Goal Specificity and Temporal Range Affect User Behavior
Slide 2
Slide 2 text
Ajzen (1985)
Intent precedes and predicts any future behavior
Slide 3
Slide 3 text
Intent directs how people use systems
Intent precedes and predicts any future behavior
Slide 4
Slide 4 text
Intent directs how people use systems
Intent precedes and predicts any future behavior
Slide 5
Slide 5 text
Intent prediction enables adaptive user interfaces
Intent directs how people use systems
Intent precedes and predicts any future behavior
Task-specific UI Recommendations
Slide 6
Slide 6 text
But how do we infer intent?
Slide 7
Slide 7 text
Challenge #1: can only observe user behavior
But how do we infer intent?
Slide 8
Slide 8 text
Challenge #2: intent can vary significantly
Challenge #1: can only observe user behavior
But how do we infer intent?
Slide 9
Slide 9 text
Intent varies greatly by individual…
(From our preliminary survey of Pinterest users using Mechanical Turk)
Bored, just
looking around
Gardening ideas
Searching for
car images
Finding interesting
optical illusions
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 10
Slide 10 text
…and even for the same individual.
(From our preliminary survey of Pinterest users using Mechanical Turk)
“I usually go on Pinterest to search for ideas for
decorating, furniture, parties, and I also like looking
at future stuff like weddings. I sometimes look at
travel photos too because it helps me experience
things vicariously.”
Slide 11
Slide 11 text
Categorizing intent in search and shopping
Prior work
Slide 12
Slide 12 text
Broder (2002)
Search is navigational, informational, transactional.
Categorizing intent in search and shopping
Prior work
Looking for a website, for information, or to buy something.
Slide 13
Slide 13 text
Chiou & Ting (2011); Novak, et al. (2003)
Shopping is either goal-oriented or experiential.
Search is navigational, informational, transactional.
Categorizing intent in search and shopping
Prior work
Buying something specific, or just browsing.
Slide 14
Slide 14 text
How to organize?
Bored, just
looking around
Gardening ideas
Searching for
car images
Finding interesting
optical illusions
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 15
Slide 15 text
Bored, just
looking around
Gardening ideas
Searching for
car images
Finding interesting
optical illusions
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
First attempt: goal-oriented vs. experiential?
Slide 16
Slide 16 text
Finding interesting
optical illusions
First attempt: goal-oriented vs. experiential?
Bored, just
looking around
Gardening ideas
Searching for
car images
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 17
Slide 17 text
Finding interesting
optical illusions
First attempt: goal-oriented vs. experiential?
Bored, just
looking around
Gardening ideas
Searching for
car images
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 18
Slide 18 text
Is there a better way to quantify intent?
Slide 19
Slide 19 text
Is there a better way to quantify intent?
and predict
Slide 20
Slide 20 text
A generalizable framework for intent prediction
This work
Slide 21
Slide 21 text
Large-scale survey + behavioral analysis
A generalizable framework for intent prediction
This work
Slide 22
Slide 22 text
Identifies two key dimensions of intent
Large-scale survey + behavioral analysis
A generalizable framework for intent prediction
This work
Goal specificity Temporal range
Slide 23
Slide 23 text
How does intent affect behavior?
Can we predict intent (quickly)?
1
2
3
What is intent?
Slide 24
Slide 24 text
How does intent affect behavior?
Can we predict intent (quickly)?
1
2
3
What is intent?
Slide 25
Slide 25 text
How do we define intent?
Slide 26
Slide 26 text
Using two key dimensions of goal-setting:
How do we define intent?
Slide 27
Slide 27 text
Hollenbeck & Klein (1987); Locke (1968)
Goal specificity: how well-defined is a goal?
Using two key dimensions of goal-setting:
How do we define intent?
Killing
time
Monster truck
cake recipes
Gardening
ideas
Less specific More specific
Slide 28
Slide 28 text
Austin & Vancouver (1996); Frese & Zapf (1994)
Temporal range: when will a goal be achieved?
Goal specificity: how well-defined is a goal?
Using two key dimensions of goal-setting:
How do we define intent?
Recipe for
dinner tonight
Wedding
favors
Quote of
the week
Short-term Long-term
Slide 29
Slide 29 text
Goal specificity and temporal range
Bored, just
looking around
Gardening ideas
Searching for
car images
Finding interesting
optical illusions
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes
for son’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 30
Slide 30 text
Goal specificity and temporal range
Longer-term
More specific
Less specific
Shorter-term
Bored, just
looking around
Gardening ideas
Searching for
car images
Finding interesting
optical illusions
Make a monster
truck cake
Need a quote
of the week
Wedding
bouquet ideas
Recipes for
dinner tonight
Looking for
good food
Cupcake recipes for
stepson’s birthday
Look up
a saved pin
To waste time
Stock images
for my website
Fall shoe
inspiration
Anxiety tips
Fitness help
Slide 31
Slide 31 text
Large-scale survey + behavioral analysis
Method
Slide 32
Slide 32 text
(Also in the paper: specific motivations and topics of interest)
Users surveyed on goal specificity, temporal range.
Large-scale survey + behavioral analysis
Method
“Are you visiting with a goal in mind?”
Slide 33
Slide 33 text
(Also in the paper: specific motivations and topics of interest)
Users surveyed on goal specificity, temporal range.
Large-scale survey + behavioral analysis
Method
“When are you planning to act on it (if applicable)?”
Slide 34
Slide 34 text
We then analyzed their activity in first ten minutes.
Users surveyed on goal specificity, temporal range.
Large-scale survey + behavioral analysis
Method
Slide 35
Slide 35 text
850k interactions across ~6k users
Data
views, clicks, searches
Slide 36
Slide 36 text
How does goal specificity vary among users?
% Users
0%
5%
10%
15%
20%
25%
30%
35%
Goal Specificity (1 = Not Specific, 7 = Very Specific)
1 2 3 4 5 6 7
How does intent affect behavior?
Can we predict intent (quickly)?
1
2
3
What is intent?
Slide 43
Slide 43 text
How does intent influence search?
Slide 44
Slide 44 text
(p < 10-3, Cohen’s d = 0.42)
Goal-specific users search more,
How does intent influence search?
Goal-specific
Goal-nonspecific
# searches
0.4
1.1
Slide 45
Slide 45 text
(p < 10-3, d = 0.24)
issue more complex queries,
Goal-specific users search more,
How does intent influence search?
Goal-specific
Goal-nonspecific
# words per search query
2.7
3.0
Slide 46
Slide 46 text
(p < 10-3, d = 0.42)
and start searching more quickly.
issue more complex queries,
Goal-specific users search more,
How does intent influence search?
Goal-specific
Goal-nonspecific
Time to first search (minutes)
3.0
1.9
Slide 47
Slide 47 text
How does intent affect browsing?
Do goal-specific users browse quickly? Or in detail?
Slide 48
Slide 48 text
How does intent affect browsing?
Are users with short-term goals in a hurry?
Slide 49
Slide 49 text
(p < 10-3, d = 0.15)
Goal-specific users browse less content,
How does intent influence browsing?
Goal-specific
Goal-nonspecific
Pins seen
156
144
Slide 50
Slide 50 text
(p < 10-3, effect size r = 0.35)
and in fewer categories.
Goal-specific users browse less content,
How does intent influence browsing?
Goal-specific
Goal-nonspecific
# distinct categories
10.4
8.7
Slide 51
Slide 51 text
(p < 10-3, χ2 = 20.8)
Goal-specific users spend more time per session,
and in fewer categories.
Goal-specific users browse less content,
How does intent affect browsing?
Goal-specific
Goal-nonspecific
Sessions ≥ 30 mins
41.8%
48.0%
Slide 52
Slide 52 text
(p < 10-3, χ2 = 13.7)
as do users with short-term goals!
Goal-specific users spend more time per session,
and in fewer categories.
Goal-specific users browse less content,
How does intent affect browsing?
Short-term
Long-term
Sessions ≥ 30 mins
41.7%
48.4%
Slide 53
Slide 53 text
How does intent influence user retention?
Does goal specificity increase the likelihood of return visits?
Slide 54
Slide 54 text
How does intent influence user retention?
Do long-term goals increase the likelihood of return visits?
Slide 55
Slide 55 text
(p < 10-3, χ2 = 16.0 comparing day 0 and day 3)
Goal-specific users are less likely to return soon,
How does intent influence user retention?
Returns
49%
53%
57%
Days since initial visit
Goal-nonspecific
Goal-specific
0 1 2 3
Slide 56
Slide 56 text
(n.s.)
but temporal range doesn’t affect future visits.
Goal-specific users are less likely to return soon,
How does intent influence user retention?
Slide 57
Slide 57 text
Intent × Behavior
Goal Specificity Temporal Range
Searching
Browsing
Saving
Time spent
Return visits
Slide 58
Slide 58 text
Intent × Behavior
Goal Specificity Temporal Range
Searching ˛*
Browsing ▼*
Saving
Time spent ˛*
Return visits ▼*
How does intent influence recipe-finding?
Case study
Slide 61
Slide 61 text
(p < 0.01, d = 0.30)
Users with short-term goals examine recipes closely,
How does intent influence recipe-finding?
Case study
Short-term
Long-term
Recipe closeups
0.9
1.8
Slide 62
Slide 62 text
(p < 0.01, d = 0.28)
and tend to be looking to make main courses.
Users with short-term goals examine recipes closely,
How does intent influence recipe-finding?
Case study
Short-term
Long-term
Recipes with meat or seafood
27%
42%
Slide 63
Slide 63 text
How does intent affect behavior?
Can we predict intent (quickly)?
1
2
3
What is intent?
Slide 64
Slide 64 text
Can we predict intent of a user session?
Slide 65
Slide 65 text
Can we predict intent in the first ten minutes?
Is the user acting in the long-term?
Is the user goal-specific?
Slide 66
Slide 66 text
Can we predict intent in the first ten minutes?
Demographics Current activity Historical activity
gender,
age,
…
searches,
views,
…
past searches,
past views,
…
Slide 67
Slide 67 text
Can we predict intent in the first ten minutes?
Demographics
Historical Activity
Current Activity
0.00 0.20 0.40 0.60 0.80
0.72
0.62
0.54
0.78
0.67
0.56
Goal Specificity Temporal Range
AUC
+
+
Slide 68
Slide 68 text
Can we predict intent in the first ten minutes?
Demographics
Historical Activity
Current Activity
0.00 0.20 0.40 0.60 0.80
0.72
0.62
0.54
0.78
0.67
0.56
Goal Specificity Temporal Range
AUC
+
+
Slide 69
Slide 69 text
Can we predict intent in the first ten minutes?
Demographics
Historical Activity
Current Activity
0.00 0.20 0.40 0.60 0.80
0.72
0.62
0.54
0.78
0.67
0.56
Goal Specificity Temporal Range
AUC
+
+
Slide 70
Slide 70 text
Can we predict intent in the first ten minutes?
Slide 71
Slide 71 text
Can we predict intent in the first ten minutes?
first minute?
Slide 72
Slide 72 text
Can we predict intent in the first ten minutes?
30 seconds?
first minute?
Slide 73
Slide 73 text
Can we predict intent in the first ten minutes?
30 seconds (or less)?
first minute?
a majority of users have made 0 pins, closeups, searches
Slide 74
Slide 74 text
How quickly can we predict intent?
AUC
0.6
0.65
0.7
0.75
0.8
Observation Period
0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
Goal Specificity
Temporal Range
Slide 75
Slide 75 text
How quickly can we predict intent?
AUC
0.6
0.65
0.7
0.75
0.8
Observation Period
0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
Goal Specificity
Temporal Range
Slide 76
Slide 76 text
While 10 minutes is sufficient for predicting intent…
AUC
0.6
0.65
0.7
0.75
0.8
Observation Period
0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
Goal Specificity
Temporal Range
Slide 77
Slide 77 text
…intent can be predicted much more quickly…
AUC
0.6
0.65
0.7
0.75
0.8
Observation Period
0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
Goal Specificity
Temporal Range
Slide 78
Slide 78 text
…and can be predicted even before a session begins
AUC
0.6
0.65
0.7
0.75
0.8
Observation Period
0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
Goal Specificity
Temporal Range
Slide 79
Slide 79 text
High-level intent can be predicted within minutes
from low-level behavioral signals.
Slide 80
Slide 80 text
How do these findings apply?
Slide 81
Slide 81 text
Designing for browsing vs. doing
How do these findings apply?
task-based interfaces?
prioritizing newness?
Slide 82
Slide 82 text
Situational recommendations and suggestions
Designing for browsing vs. doing
How do these findings apply?
detecting changing intent
specific or diverse?
Slide 83
Slide 83 text
Method adaptable to other content-based services
Situational recommendations and suggestions
Designing for browsing vs. doing
How do these findings apply?
Yelp/Zomato? Instagram?
Slide 84
Slide 84 text
Justin Cheng @jcccf, Caroline Lo @csuen, Jure Leskovec @jure / Stanford University + Pinterest
http://bit.ly/pinterest-paper
Predicting Intent Using Activity Logs
How Goal Specificity and Temporal Range Affect User Behavior