Predicting Intent Using Activity Logs

People have different intents in using online platforms. They may be trying to accomplish specific, short-term goals, or less well-defined, longer-term goals. While understanding user intent is fundamental to the design and personalization of online platforms, little is known about how intent varies across individuals, or how it relates to their behavior. Here, we develop a framework for understanding intent in terms of goal specificity and temporal range. Our methodology combines survey-based methodology with an observational analysis of user activity. Applying this framework to Pinterest, we surveyed nearly 6000 users to quantify their intent, and then studied their subsequent behavior on the web site. We find that goal specificity is bimodal – users tend to be either strongly goal-specific or goal-nonspecific. Goal-specific users search more and consume less content in greater detail than goal-nonspecific users: they spend more time using Pinterest, but are less likely to return in the near future. Users with short-term goals are also more focused and more likely to refer to past saved content than users with long-term goals, but less likely to save content for the future. Further, intent can vary by demographic, and with the topic of interest. Last, we show that user’s intent and activity are intimately related by building a model that can predict a user’s intent for using Pinterest after observing their activity for only two minutes. Altogether, this work shows how intent can be predicted from user behavior.

Presented at WWW 2017.

Transcript

1. Justin Cheng, Caroline Lo, Jure Leskovec / Stanford University + Pinterest Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior
2. Ajzen (1985) Intent precedes and predicts any future behavior
3. Intent directs how people use systems Intent precedes and predicts any future behavior
4. Intent directs how people use systems Intent precedes and predicts any future behavior
5. Intent prediction enables adaptive user interfaces Intent directs how people use systems Intent precedes and predicts any future behavior Task-specific UI Recommendations
6. But how do we infer intent?
7. Challenge #1: can only observe user behavior But how do we infer intent?
8. Challenge #2: intent can vary significantly Challenge #1: can only observe user behavior But how do we infer intent?
9. Intent varies greatly by individual… (From our preliminary survey of Pinterest users using Mechanical Turk) Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
10. …and even for the same individual. (From our preliminary survey of Pinterest users using Mechanical Turk) “I usually go on Pinterest to search for ideas for decorating, furniture, parties, and I also like looking at future stuff like weddings. I sometimes look at travel photos too because it helps me experience things vicariously.”
11. Categorizing intent in search and shopping Prior work
12. Broder (2002) Search is navigational, informational, transactional. Categorizing intent in search and shopping Prior work Looking for a website, for information, or to buy something.
13. Chiou & Ting (2011); Novak, et al. (2003) Shopping is either goal-oriented or experiential. Search is navigational, informational, transactional. Categorizing intent in search and shopping Prior work Buying something specific, or just browsing.
14. How to organize? Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
15. Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help First attempt: goal-oriented vs. experiential?
16. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored, just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
17. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored, just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
18. Is there a better way to quantify intent?
19. Is there a better way to quantify intent? and predict
20. A generalizable framework for intent prediction This work
21. Large-scale survey + behavioral analysis A generalizable framework for intent prediction This work
22. Identifies two key dimensions of intent Large-scale survey + behavioral analysis A generalizable framework for intent prediction This work Goal specificity Temporal range
23. How does intent affect behavior? Can we predict intent (quickly)? 1 2 3 What is intent?
24. How does intent affect behavior? Can we predict intent (quickly)? 1 2 3 What is intent?
25. How do we define intent?
26. Using two key dimensions of goal-setting: How do we define intent?
27. Hollenbeck & Klein (1987); Locke (1968) Goal specificity: how well-defined is a goal? Using two key dimensions of goal-setting: How do we define intent? Killing time Monster truck cake recipes Gardening ideas Less specific More specific
28. Austin & Vancouver (1996); Frese & Zapf (1994) Temporal range: when will a goal be achieved? Goal specificity: how well-defined is a goal? Using two key dimensions of goal-setting: How do we define intent? Recipe for
 dinner tonight Wedding favors Quote of
 the week Short-term Long-term
29. Goal specificity and temporal range Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
30. Goal specificity and temporal range Longer-term More specific Less specific Shorter-term Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for stepson’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
31. Large-scale survey + behavioral analysis Method
32. (Also in the paper: specific motivations and topics of interest) Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “Are you visiting with a goal in mind?”
33. (Also in the paper: specific motivations and topics of interest) Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “When are you planning to act on it (if applicable)?”
34. We then analyzed their activity in first ten minutes. Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method
35. 850k interactions across ~6k users Data views, clicks, searches
36. How does goal specificity vary among users? % Users 0% 5% 10% 15% 20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
37. Goal specificity is bimodal % Users 0% 5% 10% 15% 20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
38. Goal specificity is bimodal % Users 0% 5% 10% 15% 20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7 Goal-nonspecific Goal-specific
39. How does temporal range vary among users? % Users 0% 5% 10% 15% 20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
40. Temporal range varies significantly % Users 0% 5% 10% 15% 20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
41. Goal specificity correlates with temporal range % Users 0% 10% 20% 30% 40% 50% 60% Short-Term Mid-Term Long-Term Unsure/NA Goal-Specific Goal-Nonspecific (0-2 days) (3-7 days) (>7 days)
42. How does intent affect behavior? Can we predict intent (quickly)? 1 2 3 What is intent?
43. How does intent influence search?
44. (p < 10-3, Cohen’s d = 0.42) Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific # searches 0.4 1.1
45. (p < 10-3, d = 0.24) issue more complex queries, Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific # words per search query 2.7 3.0
46. (p < 10-3, d = 0.42) and start searching more quickly. issue more complex queries, Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific Time to first search (minutes) 3.0 1.9
47. How does intent affect browsing? Do goal-specific users browse quickly? Or in detail?
48. How does intent affect browsing? Are users with short-term goals in a hurry?
49. (p < 10-3, d = 0.15) Goal-specific users browse less content, How does intent influence browsing? Goal-specific Goal-nonspecific Pins seen 156 144
50. (p < 10-3, effect size r = 0.35) and in fewer categories. Goal-specific users browse less content, How does intent influence browsing? Goal-specific Goal-nonspecific # distinct categories 10.4 8.7
51. (p < 10-3, χ2 = 20.8) Goal-specific users spend more time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Goal-specific Goal-nonspecific Sessions ≥ 30 mins 41.8% 48.0%
52. (p < 10-3, χ2 = 13.7) as do users with short-term goals! Goal-specific users spend more time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Short-term Long-term Sessions ≥ 30 mins 41.7% 48.4%
53. How does intent influence user retention? Does goal specificity increase the likelihood of return visits?
54. How does intent influence user retention? Do long-term goals increase the likelihood of return visits?
55. (p < 10-3, χ2 = 16.0 comparing day 0 and day 3) Goal-specific users are less likely to return soon, How does intent influence user retention? Returns 49% 53% 57% Days since initial visit Goal-nonspecific Goal-specific 0 1 2 3
56. (n.s.) but temporal range doesn’t affect future visits. Goal-specific users are less likely to return soon, How does intent influence user retention?
57. Intent × Behavior Goal Specificity Temporal Range Searching Browsing Saving Time spent Return visits
58. Intent × Behavior Goal Specificity Temporal Range Searching ˛* Browsing ▼* Saving Time spent ˛* Return visits ▼*
59. Intent × Behavior Goal Specificity Temporal Range Searching ˛* Browsing ▼* ˛* Saving ˛* Time spent ˛* ▼* Return visits ▼*
60. How does intent influence recipe-finding? Case study
61. (p < 0.01, d = 0.30) Users with short-term goals examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipe closeups 0.9 1.8
62. (p < 0.01, d = 0.28) and tend to be looking to make main courses. Users with short-term goals examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipes with meat or seafood 27% 42%
63. How does intent affect behavior? Can we predict intent (quickly)? 1 2 3 What is intent?
64. Can we predict intent of a user session?
65. Can we predict intent in the first ten minutes? Is the user acting in the long-term? Is the user goal-specific?
66. Can we predict intent in the first ten minutes? Demographics Current activity Historical activity gender, age, … searches, views, … past searches, past views, …
67. Can we predict intent in the first ten minutes? Demographics Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
68. Can we predict intent in the first ten minutes? Demographics Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
69. Can we predict intent in the first ten minutes? Demographics Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
70. Can we predict intent in the first ten minutes?
71. Can we predict intent in the first ten minutes? first minute?
72. Can we predict intent in the first ten minutes? 30 seconds? first minute?
73. Can we predict intent in the first ten minutes? 30 seconds (or less)? first minute? a majority of users have made 0 pins, closeups, searches
74. How quickly can we predict intent? AUC 0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
75. How quickly can we predict intent? AUC 0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
76. While 10 minutes is sufficient for predicting intent… AUC 0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
77. …intent can be predicted much more quickly… AUC 0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
78. …and can be predicted even before a session begins AUC 0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
79. High-level intent can be predicted within minutes from low-level behavioral signals.
80. How do these findings apply?
81. Designing for browsing vs. doing How do these findings apply? task-based interfaces? prioritizing newness?
82. Situational recommendations and suggestions Designing for browsing vs. doing How do these findings apply? detecting changing intent specific or diverse?
83. Method adaptable to other content-based services Situational recommendations and suggestions Designing for browsing vs. doing How do these findings apply? Yelp/Zomato? Instagram?
84. Justin Cheng @jcccf, Caroline Lo @csuen, Jure Leskovec @jure / Stanford University + Pinterest http://bit.ly/pinterest-paper Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior