Predicting Intent Using Activity Logs

Predicting Intent Using Activity Logs

People have different intents in using online platforms. They may be trying to accomplish specific, short-term goals, or less well-defined, longer-term goals. While understanding user intent is fundamental to the design and personalization of online platforms, little is known about how intent varies across individuals, or how it relates to their behavior. Here, we develop a framework for understanding intent in terms of goal specificity and temporal range. Our methodology combines survey-based methodology with an observational analysis of user activity. Applying this framework to Pinterest, we surveyed nearly 6000 users to quantify their intent, and then studied their subsequent behavior on the web site. We find that goal specificity is bimodal – users tend to be either strongly goal-specific or goal-nonspecific. Goal-specific users search more and consume less content in greater detail than goal-nonspecific users: they spend more time using Pinterest, but are less likely to return in the near future. Users with short-term goals are also more focused and more likely to refer to past saved content than users with long-term goals, but less likely to save content for the future. Further, intent can vary by demographic, and with the topic of interest. Last, we show that user’s intent and activity are intimately related by building a model that can predict a user’s intent for using Pinterest after observing their activity for only two minutes. Altogether, this work shows how intent can be predicted from user behavior.

Presented at WWW 2017.

8480b47e733a040fba07c32da414b0e0?s=128

Justin Cheng

April 05, 2017
Tweet

Transcript

  1. Justin Cheng, Caroline Lo, Jure Leskovec / Stanford University +

    Pinterest Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior
  2. Ajzen (1985) Intent precedes and predicts any future behavior

  3. Intent directs how people use systems Intent precedes and predicts

    any future behavior
  4. Intent directs how people use systems Intent precedes and predicts

    any future behavior
  5. Intent prediction enables adaptive user interfaces Intent directs how people

    use systems Intent precedes and predicts any future behavior Task-specific UI Recommendations
  6. But how do we infer intent?

  7. Challenge #1: can only observe user behavior But how do

    we infer intent?
  8. Challenge #2: intent can vary significantly Challenge #1: can only

    observe user behavior But how do we infer intent?
  9. Intent varies greatly by individual… (From our preliminary survey of

    Pinterest users using Mechanical Turk) Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  10. …and even for the same individual. (From our preliminary survey

    of Pinterest users using Mechanical Turk) “I usually go on Pinterest to search for ideas for decorating, furniture, parties, and I also like looking at future stuff like weddings. I sometimes look at travel photos too because it helps me experience things vicariously.”
  11. Categorizing intent in search and shopping Prior work

  12. Broder (2002) Search is navigational, informational, transactional. Categorizing intent in

    search and shopping Prior work Looking for a website, for information, or to buy something.
  13. Chiou & Ting (2011); Novak, et al. (2003) Shopping is

    either goal-oriented or experiential. Search is navigational, informational, transactional. Categorizing intent in search and shopping Prior work Buying something specific, or just browsing.
  14. How to organize? Bored, just looking around Gardening ideas Searching

    for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  15. Bored, just looking around Gardening ideas Searching for car images

    Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help First attempt: goal-oriented vs. experiential?
  16. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored,

    just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  17. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored,

    just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  18. Is there a better way to quantify intent?

  19. Is there a better way to quantify intent? and predict

  20. A generalizable framework for intent prediction This work

  21. Large-scale survey + behavioral analysis A generalizable framework for intent

    prediction This work
  22. Identifies two key dimensions of intent Large-scale survey + behavioral

    analysis A generalizable framework for intent prediction This work Goal specificity Temporal range
  23. How does intent affect behavior? Can we predict intent (quickly)?

    1 2 3 What is intent?
  24. How does intent affect behavior? Can we predict intent (quickly)?

    1 2 3 What is intent?
  25. How do we define intent?

  26. Using two key dimensions of goal-setting: How do we define

    intent?
  27. Hollenbeck & Klein (1987); Locke (1968) Goal specificity: how well-defined

    is a goal? Using two key dimensions of goal-setting: How do we define intent? Killing time Monster truck cake recipes Gardening ideas Less specific More specific
  28. Austin & Vancouver (1996); Frese & Zapf (1994) Temporal range:

    when will a goal be achieved? Goal specificity: how well-defined is a goal? Using two key dimensions of goal-setting: How do we define intent? Recipe for
 dinner tonight Wedding favors Quote of
 the week Short-term Long-term
  29. Goal specificity and temporal range Bored, just looking around Gardening

    ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  30. Goal specificity and temporal range Longer-term More specific Less specific

    Shorter-term Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for stepson’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  31. Large-scale survey + behavioral analysis Method

  32. (Also in the paper: specific motivations and topics of interest)

    Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “Are you visiting with a goal in mind?”
  33. (Also in the paper: specific motivations and topics of interest)

    Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “When are you planning to act on it (if applicable)?”
  34. We then analyzed their activity in first ten minutes. Users

    surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method
  35. 850k interactions across ~6k users Data views, clicks, searches

  36. How does goal specificity vary among users? % Users 0%

    5% 10% 15% 20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
  37. Goal specificity is bimodal % Users 0% 5% 10% 15%

    20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
  38. Goal specificity is bimodal % Users 0% 5% 10% 15%

    20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7 Goal-nonspecific Goal-specific
  39. How does temporal range vary among users? % Users 0%

    5% 10% 15% 20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
  40. Temporal range varies significantly % Users 0% 5% 10% 15%

    20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
  41. Goal specificity correlates with temporal range % Users 0% 10%

    20% 30% 40% 50% 60% Short-Term Mid-Term Long-Term Unsure/NA Goal-Specific Goal-Nonspecific (0-2 days) (3-7 days) (>7 days)
  42. How does intent affect behavior? Can we predict intent (quickly)?

    1 2 3 What is intent?
  43. How does intent influence search?

  44. (p < 10-3, Cohen’s d = 0.42) Goal-specific users search

    more, How does intent influence search? Goal-specific Goal-nonspecific # searches 0.4 1.1
  45. (p < 10-3, d = 0.24) issue more complex queries,

    Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific # words per search query 2.7 3.0
  46. (p < 10-3, d = 0.42) and start searching more

    quickly. issue more complex queries, Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific Time to first search (minutes) 3.0 1.9
  47. How does intent affect browsing? Do goal-specific users browse quickly?

    Or in detail?
  48. How does intent affect browsing? Are users with short-term goals

    in a hurry?
  49. (p < 10-3, d = 0.15) Goal-specific users browse less

    content, How does intent influence browsing? Goal-specific Goal-nonspecific Pins seen 156 144
  50. (p < 10-3, effect size r = 0.35) and in

    fewer categories. Goal-specific users browse less content, How does intent influence browsing? Goal-specific Goal-nonspecific # distinct categories 10.4 8.7
  51. (p < 10-3, χ2 = 20.8) Goal-specific users spend more

    time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Goal-specific Goal-nonspecific Sessions ≥ 30 mins 41.8% 48.0%
  52. (p < 10-3, χ2 = 13.7) as do users with

    short-term goals! Goal-specific users spend more time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Short-term Long-term Sessions ≥ 30 mins 41.7% 48.4%
  53. How does intent influence user retention? Does goal specificity increase

    the likelihood of return visits?
  54. How does intent influence user retention? Do long-term goals increase

    the likelihood of return visits?
  55. (p < 10-3, χ2 = 16.0 comparing day 0 and

    day 3) Goal-specific users are less likely to return soon, How does intent influence user retention? Returns 49% 53% 57% Days since initial visit Goal-nonspecific Goal-specific 0 1 2 3
  56. (n.s.) but temporal range doesn’t affect future visits. Goal-specific users

    are less likely to return soon, How does intent influence user retention?
  57. Intent × Behavior Goal Specificity Temporal Range Searching Browsing Saving

    Time spent Return visits
  58. Intent × Behavior Goal Specificity Temporal Range Searching ˛* Browsing

    ▼* Saving Time spent ˛* Return visits ▼*
  59. Intent × Behavior Goal Specificity Temporal Range Searching ˛* Browsing

    ▼* ˛* Saving ˛* Time spent ˛* ▼* Return visits ▼*
  60. How does intent influence recipe-finding? Case study

  61. (p < 0.01, d = 0.30) Users with short-term goals

    examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipe closeups 0.9 1.8
  62. (p < 0.01, d = 0.28) and tend to be

    looking to make main courses. Users with short-term goals examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipes with meat or seafood 27% 42%
  63. How does intent affect behavior? Can we predict intent (quickly)?

    1 2 3 What is intent?
  64. Can we predict intent of a user session?

  65. Can we predict intent in the first ten minutes? Is

    the user acting in the long-term? Is the user goal-specific?
  66. Can we predict intent in the first ten minutes? Demographics

    Current activity Historical activity gender, age, … searches, views, … past searches, past views, …
  67. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  68. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  69. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  70. Can we predict intent in the first ten minutes?

  71. Can we predict intent in the first ten minutes? first

    minute?
  72. Can we predict intent in the first ten minutes? 30

    seconds? first minute?
  73. Can we predict intent in the first ten minutes? 30

    seconds (or less)? first minute? a majority of users have made 0 pins, closeups, searches
  74. How quickly can we predict intent? AUC 0.6 0.65 0.7

    0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  75. How quickly can we predict intent? AUC 0.6 0.65 0.7

    0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  76. While 10 minutes is sufficient for predicting intent… AUC 0.6

    0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  77. …intent can be predicted much more quickly… AUC 0.6 0.65

    0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  78. …and can be predicted even before a session begins AUC

    0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  79. High-level intent can be predicted within minutes from low-level behavioral

    signals.
  80. How do these findings apply?

  81. Designing for browsing vs. doing How do these findings apply?

    task-based interfaces? prioritizing newness?
  82. Situational recommendations and suggestions Designing for browsing vs. doing How

    do these findings apply? detecting changing intent specific or diverse?
  83. Method adaptable to other content-based services Situational recommendations and suggestions

    Designing for browsing vs. doing How do these findings apply? Yelp/Zomato? Instagram?
  84. Justin Cheng @jcccf, Caroline Lo @csuen, Jure Leskovec @jure /

    Stanford University + Pinterest http://bit.ly/pinterest-paper Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior