Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Predicting Intent Using Activity Logs

Predicting Intent Using Activity Logs

People have different intents in using online platforms. They may be trying to accomplish specific, short-term goals, or less well-defined, longer-term goals. While understanding user intent is fundamental to the design and personalization of online platforms, little is known about how intent varies across individuals, or how it relates to their behavior. Here, we develop a framework for understanding intent in terms of goal specificity and temporal range. Our methodology combines survey-based methodology with an observational analysis of user activity. Applying this framework to Pinterest, we surveyed nearly 6000 users to quantify their intent, and then studied their subsequent behavior on the web site. We find that goal specificity is bimodal – users tend to be either strongly goal-specific or goal-nonspecific. Goal-specific users search more and consume less content in greater detail than goal-nonspecific users: they spend more time using Pinterest, but are less likely to return in the near future. Users with short-term goals are also more focused and more likely to refer to past saved content than users with long-term goals, but less likely to save content for the future. Further, intent can vary by demographic, and with the topic of interest. Last, we show that user’s intent and activity are intimately related by building a model that can predict a user’s intent for using Pinterest after observing their activity for only two minutes. Altogether, this work shows how intent can be predicted from user behavior.

Presented at WWW 2017.

Justin Cheng

April 05, 2017
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. Justin Cheng, Caroline Lo, Jure Leskovec / Stanford University +

    Pinterest Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior
  2. Intent prediction enables adaptive user interfaces Intent directs how people

    use systems Intent precedes and predicts any future behavior Task-specific UI Recommendations
  3. Challenge #2: intent can vary significantly Challenge #1: can only

    observe user behavior But how do we infer intent?
  4. Intent varies greatly by individual… (From our preliminary survey of

    Pinterest users using Mechanical Turk) Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  5. …and even for the same individual. (From our preliminary survey

    of Pinterest users using Mechanical Turk) “I usually go on Pinterest to search for ideas for decorating, furniture, parties, and I also like looking at future stuff like weddings. I sometimes look at travel photos too because it helps me experience things vicariously.”
  6. Broder (2002) Search is navigational, informational, transactional. Categorizing intent in

    search and shopping Prior work Looking for a website, for information, or to buy something.
  7. Chiou & Ting (2011); Novak, et al. (2003) Shopping is

    either goal-oriented or experiential. Search is navigational, informational, transactional. Categorizing intent in search and shopping Prior work Buying something specific, or just browsing.
  8. How to organize? Bored, just looking around Gardening ideas Searching

    for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  9. Bored, just looking around Gardening ideas Searching for car images

    Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help First attempt: goal-oriented vs. experiential?
  10. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored,

    just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  11. Finding interesting optical illusions First attempt: goal-oriented vs. experiential? Bored,

    just looking around Gardening ideas Searching for car images Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  12. Identifies two key dimensions of intent Large-scale survey + behavioral

    analysis A generalizable framework for intent prediction This work Goal specificity Temporal range
  13. Hollenbeck & Klein (1987); Locke (1968) Goal specificity: how well-defined

    is a goal? Using two key dimensions of goal-setting: How do we define intent? Killing time Monster truck cake recipes Gardening ideas Less specific More specific
  14. Austin & Vancouver (1996); Frese & Zapf (1994) Temporal range:

    when will a goal be achieved? Goal specificity: how well-defined is a goal? Using two key dimensions of goal-setting: How do we define intent? Recipe for
 dinner tonight Wedding favors Quote of
 the week Short-term Long-term
  15. Goal specificity and temporal range Bored, just looking around Gardening

    ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for son’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  16. Goal specificity and temporal range Longer-term More specific Less specific

    Shorter-term Bored, just looking around Gardening ideas Searching for car images Finding interesting optical illusions Make a monster truck cake Need a quote of the week Wedding bouquet ideas Recipes for dinner tonight Looking for good food Cupcake recipes for stepson’s birthday Look up a saved pin To waste time Stock images for my website Fall shoe inspiration Anxiety tips Fitness help
  17. (Also in the paper: specific motivations and topics of interest)

    Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “Are you visiting with a goal in mind?”
  18. (Also in the paper: specific motivations and topics of interest)

    Users surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method “When are you planning to act on it (if applicable)?”
  19. We then analyzed their activity in first ten minutes. Users

    surveyed on goal specificity, temporal range. Large-scale survey + behavioral analysis Method
  20. How does goal specificity vary among users? % Users 0%

    5% 10% 15% 20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
  21. Goal specificity is bimodal % Users 0% 5% 10% 15%

    20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7
  22. Goal specificity is bimodal % Users 0% 5% 10% 15%

    20% 25% 30% 35% Goal Specificity (1 = Not Specific, 7 = Very Specific) 1 2 3 4 5 6 7 Goal-nonspecific Goal-specific
  23. How does temporal range vary among users? % Users 0%

    5% 10% 15% 20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
  24. Temporal range varies significantly % Users 0% 5% 10% 15%

    20% 25% 30% 35% Short-Term Mid-Term Long-Term Unsure/NA (0-2 days) (3-7 days) (>7 days)
  25. Goal specificity correlates with temporal range % Users 0% 10%

    20% 30% 40% 50% 60% Short-Term Mid-Term Long-Term Unsure/NA Goal-Specific Goal-Nonspecific (0-2 days) (3-7 days) (>7 days)
  26. (p < 10-3, Cohen’s d = 0.42) Goal-specific users search

    more, How does intent influence search? Goal-specific Goal-nonspecific # searches 0.4 1.1
  27. (p < 10-3, d = 0.24) issue more complex queries,

    Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific # words per search query 2.7 3.0
  28. (p < 10-3, d = 0.42) and start searching more

    quickly. issue more complex queries, Goal-specific users search more, How does intent influence search? Goal-specific Goal-nonspecific Time to first search (minutes) 3.0 1.9
  29. (p < 10-3, d = 0.15) Goal-specific users browse less

    content, How does intent influence browsing? Goal-specific Goal-nonspecific Pins seen 156 144
  30. (p < 10-3, effect size r = 0.35) and in

    fewer categories. Goal-specific users browse less content, How does intent influence browsing? Goal-specific Goal-nonspecific # distinct categories 10.4 8.7
  31. (p < 10-3, χ2 = 20.8) Goal-specific users spend more

    time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Goal-specific Goal-nonspecific Sessions ≥ 30 mins 41.8% 48.0%
  32. (p < 10-3, χ2 = 13.7) as do users with

    short-term goals! Goal-specific users spend more time per session, and in fewer categories. Goal-specific users browse less content, How does intent affect browsing? Short-term Long-term Sessions ≥ 30 mins 41.7% 48.4%
  33. (p < 10-3, χ2 = 16.0 comparing day 0 and

    day 3) Goal-specific users are less likely to return soon, How does intent influence user retention? Returns 49% 53% 57% Days since initial visit Goal-nonspecific Goal-specific 0 1 2 3
  34. (n.s.) but temporal range doesn’t affect future visits. Goal-specific users

    are less likely to return soon, How does intent influence user retention?
  35. Intent × Behavior Goal Specificity Temporal Range Searching ˛* Browsing

    ▼* ˛* Saving ˛* Time spent ˛* ▼* Return visits ▼*
  36. (p < 0.01, d = 0.30) Users with short-term goals

    examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipe closeups 0.9 1.8
  37. (p < 0.01, d = 0.28) and tend to be

    looking to make main courses. Users with short-term goals examine recipes closely, How does intent influence recipe-finding? Case study Short-term Long-term Recipes with meat or seafood 27% 42%
  38. Can we predict intent in the first ten minutes? Is

    the user acting in the long-term? Is the user goal-specific?
  39. Can we predict intent in the first ten minutes? Demographics

    Current activity Historical activity gender, age, … searches, views, … past searches, past views, …
  40. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  41. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  42. Can we predict intent in the first ten minutes? Demographics

    Historical Activity Current Activity 0.00 0.20 0.40 0.60 0.80 0.72 0.62 0.54 0.78 0.67 0.56 Goal Specificity Temporal Range AUC + +
  43. Can we predict intent in the first ten minutes? 30

    seconds (or less)? first minute? a majority of users have made 0 pins, closeups, searches
  44. How quickly can we predict intent? AUC 0.6 0.65 0.7

    0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  45. How quickly can we predict intent? AUC 0.6 0.65 0.7

    0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  46. While 10 minutes is sufficient for predicting intent… AUC 0.6

    0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  47. …intent can be predicted much more quickly… AUC 0.6 0.65

    0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  48. …and can be predicted even before a session begins AUC

    0.6 0.65 0.7 0.75 0.8 Observation Period 0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m Goal Specificity Temporal Range
  49. Designing for browsing vs. doing How do these findings apply?

    task-based interfaces? prioritizing newness?
  50. Situational recommendations and suggestions Designing for browsing vs. doing How

    do these findings apply? detecting changing intent specific or diverse?
  51. Method adaptable to other content-based services Situational recommendations and suggestions

    Designing for browsing vs. doing How do these findings apply? Yelp/Zomato? Instagram?
  52. Justin Cheng @jcccf, Caroline Lo @csuen, Jure Leskovec @jure /

    Stanford University + Pinterest http://bit.ly/pinterest-paper Predicting Intent Using Activity Logs How Goal Specificity and Temporal Range Affect User Behavior