Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Predicting Intent Using Activity Logs

Predicting Intent Using Activity Logs

People have different intents in using online platforms. They may be trying to accomplish specific, short-term goals, or less well-defined, longer-term goals. While understanding user intent is fundamental to the design and personalization of online platforms, little is known about how intent varies across individuals, or how it relates to their behavior. Here, we develop a framework for understanding intent in terms of goal specificity and temporal range. Our methodology combines survey-based methodology with an observational analysis of user activity. Applying this framework to Pinterest, we surveyed nearly 6000 users to quantify their intent, and then studied their subsequent behavior on the web site. We find that goal specificity is bimodal – users tend to be either strongly goal-specific or goal-nonspecific. Goal-specific users search more and consume less content in greater detail than goal-nonspecific users: they spend more time using Pinterest, but are less likely to return in the near future. Users with short-term goals are also more focused and more likely to refer to past saved content than users with long-term goals, but less likely to save content for the future. Further, intent can vary by demographic, and with the topic of interest. Last, we show that user’s intent and activity are intimately related by building a model that can predict a user’s intent for using Pinterest after observing their activity for only two minutes. Altogether, this work shows how intent can be predicted from user behavior.

Presented at WWW 2017.

Justin Cheng

April 05, 2017
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. Justin Cheng, Caroline Lo, Jure Leskovec / Stanford University + Pinterest
    Predicting Intent Using Activity Logs
    How Goal Specificity and Temporal Range Affect User Behavior

    View full-size slide

  2. Ajzen (1985)
    Intent precedes and predicts any future behavior

    View full-size slide

  3. Intent directs how people use systems
    Intent precedes and predicts any future behavior

    View full-size slide

  4. Intent directs how people use systems
    Intent precedes and predicts any future behavior

    View full-size slide

  5. Intent prediction enables adaptive user interfaces
    Intent directs how people use systems
    Intent precedes and predicts any future behavior
    Task-specific UI Recommendations

    View full-size slide

  6. But how do we infer intent?

    View full-size slide

  7. Challenge #1: can only observe user behavior
    But how do we infer intent?

    View full-size slide

  8. Challenge #2: intent can vary significantly
    Challenge #1: can only observe user behavior
    But how do we infer intent?

    View full-size slide

  9. Intent varies greatly by individual…
    (From our preliminary survey of Pinterest users using Mechanical Turk)
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Finding interesting
    optical illusions
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  10. …and even for the same individual.
    (From our preliminary survey of Pinterest users using Mechanical Turk)
    “I usually go on Pinterest to search for ideas for
    decorating, furniture, parties, and I also like looking
    at future stuff like weddings. I sometimes look at
    travel photos too because it helps me experience
    things vicariously.”

    View full-size slide

  11. Categorizing intent in search and shopping
    Prior work

    View full-size slide

  12. Broder (2002)
    Search is navigational, informational, transactional.
    Categorizing intent in search and shopping
    Prior work
    Looking for a website, for information, or to buy something.

    View full-size slide

  13. Chiou & Ting (2011); Novak, et al. (2003)
    Shopping is either goal-oriented or experiential.
    Search is navigational, informational, transactional.
    Categorizing intent in search and shopping
    Prior work
    Buying something specific, or just browsing.

    View full-size slide

  14. How to organize?
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Finding interesting
    optical illusions
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  15. Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Finding interesting
    optical illusions
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help
    First attempt: goal-oriented vs. experiential?

    View full-size slide

  16. Finding interesting
    optical illusions
    First attempt: goal-oriented vs. experiential?
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  17. Finding interesting
    optical illusions
    First attempt: goal-oriented vs. experiential?
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  18. Is there a better way to quantify intent?

    View full-size slide

  19. Is there a better way to quantify intent?
    and predict

    View full-size slide

  20. A generalizable framework for intent prediction
    This work

    View full-size slide

  21. Large-scale survey + behavioral analysis
    A generalizable framework for intent prediction
    This work

    View full-size slide

  22. Identifies two key dimensions of intent
    Large-scale survey + behavioral analysis
    A generalizable framework for intent prediction
    This work
    Goal specificity Temporal range

    View full-size slide

  23. How does intent affect behavior?
    Can we predict intent (quickly)?
    1
    2
    3
    What is intent?

    View full-size slide

  24. How does intent affect behavior?
    Can we predict intent (quickly)?
    1
    2
    3
    What is intent?

    View full-size slide

  25. How do we define intent?

    View full-size slide

  26. Using two key dimensions of goal-setting:
    How do we define intent?

    View full-size slide

  27. Hollenbeck & Klein (1987); Locke (1968)
    Goal specificity: how well-defined is a goal?
    Using two key dimensions of goal-setting:
    How do we define intent?
    Killing
    time
    Monster truck
    cake recipes
    Gardening
    ideas
    Less specific More specific

    View full-size slide

  28. Austin & Vancouver (1996); Frese & Zapf (1994)
    Temporal range: when will a goal be achieved?
    Goal specificity: how well-defined is a goal?
    Using two key dimensions of goal-setting:
    How do we define intent?
    Recipe for

    dinner tonight
    Wedding
    favors
    Quote of

    the week
    Short-term Long-term

    View full-size slide

  29. Goal specificity and temporal range
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Finding interesting
    optical illusions
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes
    for son’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  30. Goal specificity and temporal range
    Longer-term
    More specific
    Less specific
    Shorter-term
    Bored, just
    looking around
    Gardening ideas
    Searching for
    car images
    Finding interesting
    optical illusions
    Make a monster
    truck cake
    Need a quote
    of the week
    Wedding
    bouquet ideas
    Recipes for
    dinner tonight
    Looking for
    good food
    Cupcake recipes for
    stepson’s birthday
    Look up
    a saved pin
    To waste time
    Stock images
    for my website
    Fall shoe
    inspiration
    Anxiety tips
    Fitness help

    View full-size slide

  31. Large-scale survey + behavioral analysis
    Method

    View full-size slide

  32. (Also in the paper: specific motivations and topics of interest)
    Users surveyed on goal specificity, temporal range.
    Large-scale survey + behavioral analysis
    Method
    “Are you visiting with a goal in mind?”

    View full-size slide

  33. (Also in the paper: specific motivations and topics of interest)
    Users surveyed on goal specificity, temporal range.
    Large-scale survey + behavioral analysis
    Method
    “When are you planning to act on it (if applicable)?”

    View full-size slide

  34. We then analyzed their activity in first ten minutes.
    Users surveyed on goal specificity, temporal range.
    Large-scale survey + behavioral analysis
    Method

    View full-size slide

  35. 850k interactions across ~6k users
    Data
    views, clicks, searches

    View full-size slide

  36. How does goal specificity vary among users?
    % Users
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    35%
    Goal Specificity (1 = Not Specific, 7 = Very Specific)
    1 2 3 4 5 6 7

    View full-size slide

  37. Goal specificity is bimodal
    % Users
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    35%
    Goal Specificity (1 = Not Specific, 7 = Very Specific)
    1 2 3 4 5 6 7

    View full-size slide

  38. Goal specificity is bimodal
    % Users
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    35%
    Goal Specificity (1 = Not Specific, 7 = Very Specific)
    1 2 3 4 5 6 7
    Goal-nonspecific Goal-specific

    View full-size slide

  39. How does temporal range vary among users?
    % Users
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    35%
    Short-Term Mid-Term Long-Term Unsure/NA
    (0-2 days) (3-7 days) (>7 days)

    View full-size slide

  40. Temporal range varies significantly
    % Users
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    35%
    Short-Term Mid-Term Long-Term Unsure/NA
    (0-2 days) (3-7 days) (>7 days)

    View full-size slide

  41. Goal specificity correlates with temporal range
    % Users
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    Short-Term Mid-Term Long-Term Unsure/NA
    Goal-Specific
    Goal-Nonspecific
    (0-2 days) (3-7 days) (>7 days)

    View full-size slide

  42. How does intent affect behavior?
    Can we predict intent (quickly)?
    1
    2
    3
    What is intent?

    View full-size slide

  43. How does intent influence search?

    View full-size slide

  44. (p < 10-3, Cohen’s d = 0.42)
    Goal-specific users search more,
    How does intent influence search?
    Goal-specific
    Goal-nonspecific
    # searches
    0.4
    1.1

    View full-size slide

  45. (p < 10-3, d = 0.24)
    issue more complex queries,
    Goal-specific users search more,
    How does intent influence search?
    Goal-specific
    Goal-nonspecific
    # words per search query
    2.7
    3.0

    View full-size slide

  46. (p < 10-3, d = 0.42)
    and start searching more quickly.
    issue more complex queries,
    Goal-specific users search more,
    How does intent influence search?
    Goal-specific
    Goal-nonspecific
    Time to first search (minutes)
    3.0
    1.9

    View full-size slide

  47. How does intent affect browsing?
    Do goal-specific users browse quickly? Or in detail?

    View full-size slide

  48. How does intent affect browsing?
    Are users with short-term goals in a hurry?

    View full-size slide

  49. (p < 10-3, d = 0.15)
    Goal-specific users browse less content,
    How does intent influence browsing?
    Goal-specific
    Goal-nonspecific
    Pins seen
    156
    144

    View full-size slide

  50. (p < 10-3, effect size r = 0.35)
    and in fewer categories.
    Goal-specific users browse less content,
    How does intent influence browsing?
    Goal-specific
    Goal-nonspecific
    # distinct categories
    10.4
    8.7

    View full-size slide

  51. (p < 10-3, χ2 = 20.8)
    Goal-specific users spend more time per session,
    and in fewer categories.
    Goal-specific users browse less content,
    How does intent affect browsing?
    Goal-specific
    Goal-nonspecific
    Sessions ≥ 30 mins
    41.8%
    48.0%

    View full-size slide

  52. (p < 10-3, χ2 = 13.7)
    as do users with short-term goals!
    Goal-specific users spend more time per session,
    and in fewer categories.
    Goal-specific users browse less content,
    How does intent affect browsing?
    Short-term
    Long-term
    Sessions ≥ 30 mins
    41.7%
    48.4%

    View full-size slide

  53. How does intent influence user retention?
    Does goal specificity increase the likelihood of return visits?

    View full-size slide

  54. How does intent influence user retention?
    Do long-term goals increase the likelihood of return visits?

    View full-size slide

  55. (p < 10-3, χ2 = 16.0 comparing day 0 and day 3)
    Goal-specific users are less likely to return soon,
    How does intent influence user retention?
    Returns
    49%
    53%
    57%
    Days since initial visit
    Goal-nonspecific
    Goal-specific
    0 1 2 3

    View full-size slide

  56. (n.s.)
    but temporal range doesn’t affect future visits.
    Goal-specific users are less likely to return soon,
    How does intent influence user retention?

    View full-size slide

  57. Intent × Behavior
    Goal Specificity Temporal Range
    Searching
    Browsing
    Saving
    Time spent
    Return visits

    View full-size slide

  58. Intent × Behavior
    Goal Specificity Temporal Range
    Searching ˛*
    Browsing ▼*
    Saving
    Time spent ˛*
    Return visits ▼*

    View full-size slide

  59. Intent × Behavior
    Goal Specificity Temporal Range
    Searching ˛*
    Browsing ▼* ˛*
    Saving ˛*
    Time spent ˛* ▼*
    Return visits ▼*

    View full-size slide

  60. How does intent influence recipe-finding?
    Case study

    View full-size slide

  61. (p < 0.01, d = 0.30)
    Users with short-term goals examine recipes closely,
    How does intent influence recipe-finding?
    Case study
    Short-term
    Long-term
    Recipe closeups
    0.9
    1.8

    View full-size slide

  62. (p < 0.01, d = 0.28)
    and tend to be looking to make main courses.
    Users with short-term goals examine recipes closely,
    How does intent influence recipe-finding?
    Case study
    Short-term
    Long-term
    Recipes with meat or seafood
    27%
    42%

    View full-size slide

  63. How does intent affect behavior?
    Can we predict intent (quickly)?
    1
    2
    3
    What is intent?

    View full-size slide

  64. Can we predict intent of a user session?

    View full-size slide

  65. Can we predict intent in the first ten minutes?
    Is the user acting in the long-term?
    Is the user goal-specific?

    View full-size slide

  66. Can we predict intent in the first ten minutes?
    Demographics Current activity Historical activity
    gender,
    age,

    searches,
    views,

    past searches,
    past views,

    View full-size slide

  67. Can we predict intent in the first ten minutes?
    Demographics
    Historical Activity
    Current Activity
    0.00 0.20 0.40 0.60 0.80
    0.72
    0.62
    0.54
    0.78
    0.67
    0.56
    Goal Specificity Temporal Range
    AUC
    +
    +

    View full-size slide

  68. Can we predict intent in the first ten minutes?
    Demographics
    Historical Activity
    Current Activity
    0.00 0.20 0.40 0.60 0.80
    0.72
    0.62
    0.54
    0.78
    0.67
    0.56
    Goal Specificity Temporal Range
    AUC
    +
    +

    View full-size slide

  69. Can we predict intent in the first ten minutes?
    Demographics
    Historical Activity
    Current Activity
    0.00 0.20 0.40 0.60 0.80
    0.72
    0.62
    0.54
    0.78
    0.67
    0.56
    Goal Specificity Temporal Range
    AUC
    +
    +

    View full-size slide

  70. Can we predict intent in the first ten minutes?

    View full-size slide

  71. Can we predict intent in the first ten minutes?
    first minute?

    View full-size slide

  72. Can we predict intent in the first ten minutes?
    30 seconds?
    first minute?

    View full-size slide

  73. Can we predict intent in the first ten minutes?
    30 seconds (or less)?
    first minute?
    a majority of users have made 0 pins, closeups, searches

    View full-size slide

  74. How quickly can we predict intent?
    AUC
    0.6
    0.65
    0.7
    0.75
    0.8
    Observation Period
    0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
    Goal Specificity
    Temporal Range

    View full-size slide

  75. How quickly can we predict intent?
    AUC
    0.6
    0.65
    0.7
    0.75
    0.8
    Observation Period
    0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
    Goal Specificity
    Temporal Range

    View full-size slide

  76. While 10 minutes is sufficient for predicting intent…
    AUC
    0.6
    0.65
    0.7
    0.75
    0.8
    Observation Period
    0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
    Goal Specificity
    Temporal Range

    View full-size slide

  77. …intent can be predicted much more quickly…
    AUC
    0.6
    0.65
    0.7
    0.75
    0.8
    Observation Period
    0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
    Goal Specificity
    Temporal Range

    View full-size slide

  78. …and can be predicted even before a session begins
    AUC
    0.6
    0.65
    0.7
    0.75
    0.8
    Observation Period
    0s 30s 1m 2m 3m 4m 5m 10m 15m 30m 45m 60m 180m
    Goal Specificity
    Temporal Range

    View full-size slide

  79. High-level intent can be predicted within minutes
    from low-level behavioral signals.

    View full-size slide

  80. How do these findings apply?

    View full-size slide

  81. Designing for browsing vs. doing
    How do these findings apply?
    task-based interfaces?
    prioritizing newness?

    View full-size slide

  82. Situational recommendations and suggestions
    Designing for browsing vs. doing
    How do these findings apply?
    detecting changing intent
    specific or diverse?

    View full-size slide

  83. Method adaptable to other content-based services
    Situational recommendations and suggestions
    Designing for browsing vs. doing
    How do these findings apply?
    Yelp/Zomato? Instagram?

    View full-size slide

  84. Justin Cheng @jcccf, Caroline Lo @csuen, Jure Leskovec @jure / Stanford University + Pinterest
    http://bit.ly/pinterest-paper
    Predicting Intent Using Activity Logs
    How Goal Specificity and Temporal Range Affect User Behavior

    View full-size slide