What Web Search Behaviors Lead to Online Purchase Satisfaction? (EN)

What Web Search Behaviors Lead to
Online Purchase Satisfaction?
Yuki Yanagida, Makoto P. Kato (University of Tsukuba)
Yuka Kawada, Takehiro Yamamoto, Hiroaki Ohshima
(University of Hyogo)
Sumio Fujita (Yahoo Japan Corporation)
15th ACM Web Science Conference (WebSci 2023)

View Slide

Satisfying people purchasing products is an important issue in EC
• “Post-purchase” satisfaction improves to be continuously use of EC sites [1]
• Customer satisfaction can have a big impact since 41% of people shop
online at least once a week [2]
[1] Gustafsson et al. The effects of customer satisfaction, relationship commitment dimensions, and triggers on customer retention. Journal of
marketing, 69(4), 2005.
[2] PwC. December 2021 global consumer insights pulse survey. https://www.pwc.com/gx/en/industries/consumer-markets/consumer-
insights-survey/archive/consumer-insights-survey-december-2021.html, (accessed 2022-11-28).
Background 2

View Slide

We investigate the relationship between
information-seeking behavior on the web and post-purchase satisfaction
• The findings of this study enables us to:
‒ Predict post-purchase satisfaction from web search behavior for non-reviewing users
• Also addressed in this study
Purpose 3
Examine the relationship between web search behavior and review ratings on EC site
⊃
≡
Def
Subset
Data Satisfaction
Studied
search behavior
Existing post-purchase
satisfaction study
Product or user
characteristics
“Post-purchase”
satisfaction
❌
Existing search behavior
study
Logs, subject experiment
“Search”
satisfaction
“EC” search
This study
Logs, product or user
characteristics
“Post-purchase”
satisfaction
“Web” search
Comparison to existing studies

View Slide

• RQ1: Are web search behaviors different for SAT (satisfied) and DSAT
(dissatisfied) users?
A Within a week prior to the purchase, SAT users more frequently searched for a wider
range of product-related information
• RQ2: Are web search behaviors different depending on post-purchase
satisfaction and product/user characteristics?
A SAT users searched with more specific queries prior to purchase when:
• They are relatively familiar with web search
• They are purchasing a relatively expensive product
• RQ3: What is the relationship between query terms and satisfaction?
A Users looking for opinions of others are more likely to be satisfied with their purchase
RQs and Their Answers 4

View Slide

We estimate the search intent of each query
to characterize web search behaviors
Analysis Flow 5
Query
EOS R5 price
popular camera
night photography
weather in Tokyo
Query log
Query Intent
EOS R5 price TF (narrow)
popular camera DM (broad)
night photography Other
weather in Tokyo Other
Query Intent
night photography DM (broad)
Rule-based estimation
Estimation by
weak supervision
Classify queries
=
Query frequency calculation
for each time segment
Time
Frequency
Web queries are enormous
→ Identify each intent systematically based on an existing taxonomy
SAT
DSAT
We sampled logs from a major commercial search engine and EC sites
(The same account was used in these services)

View Slide

We want to estimate the search intent of each query
→ Create product dictionary and classify queries based on rules
• TF: considering purchase and
searching narrowly
‒ Corresponds to Target Finding in [3]
‒ It contains a product name, a brand name,
or model number in camera
• DM: searching broadly than TF
‒ Corresponds to Decision Making in [3]
‒ Condition TF is not met but contains
“camera”
• Other
Rule-based Estimation 6
Query Intent
Related to camera but classified as Other
[3] Su et al. User intent, behaviour, and perceived satisfaction in product search. WSDM 2018.
e.g. : Camera category
Search intent was defined
based on previous research [3]
Analyze differences in search behavior by intent
We use a weak supervision for words that cannot
be classified by the rule-based approach
Measure

View Slide

• RQ1: Are web search behaviors different for SAT (satisfied) and DSAT
(dissatisfied) users?
A Within a week prior to the purchase, SAT users more frequently searched for a wider
range of product-related information
A

View Slide

Temporal distribution of TF (narrow) queries
RQ1: Are Web Search Behaviors Different for SAT and DSAT Users? 8
0
0.0001
0.0002
0.0003
0.0004
0.0005
(-4w
,-3w
]
(-3w
,-2w
]
(-2w
,-1w
]
(-1w
,0]
[0,+
1w
)
[+
1w
,+
2w
)
[+
2w
,+
3w
)
[+
3w
,+
4w
)
Search frequency
Time segment
SAT
DSAT
DM (broad) was similar
Within a week prior to the
purchase, SAT users more
searched with TF (narrow)
SAT users looked for a broader
range of information
Hypothesize
Normalized frequency by
total number of searches
e.g. [0,+1w): the time segment
within a week of purchase
Purchase
6/Mar 10/Mar 13/Mar 17/Mar
Purchase
Time
Align and to count

View Slide

Comparison of Query Term Entropy 9
We compared query term entropy prior to the purchase
to examine whether SAT users looked for a broader range of information
SAT users used more diverse query terms than
DSAT users for both TF and DM queries
0
0.2
0.4
0.6
0.8
1
1.2
TF (narrow) DM (broad)
Average Entropy
5.8
6
6.2
6.4
6.6
6.8
7
Other
SAT
DSAT
Intent

View Slide

A
• RQ2: Are web search behaviors different depending on post-purchase
satisfaction and product/user characteristics?
A SAT users searched with more specific queries prior to purchase when:
• They are relatively familiar with web search
• They are purchasing a relatively expensive product
• Search intent
‒ TF (narrow): Narrower search
‒ DM (broad): Broader search than TF

View Slide

RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? 11
Search Frequency
Temporal distribution of TF (narrow) queries for each
combination of product price and search frequency levels
SAT
DSAT
Search Frequency
Purchase
Purchase
Low-price
Low-search-
frequency
High-price
Low-search-
frequency
High-price
High-search-
frequency
Low-price
High-search-
frequency
Purchase
Purchase

View Slide

Temporal distribution of TF (narrow) queries for each
RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? 12
SAT
DSAT
High-price
High-search-
frequency Purchase
If they are familiar with web search, they might turn to conduct
web search about a specific product to lower the purchasing risk
Only when the product price and search frequency
were high, SAT users searched more frequently with
TF queries prior to the purchase

View Slide

A
A
• RQ3: What is the relationship between query terms and satisfaction?
A Users looking for opinions of others are more likely to be satisfied with their purchase

View Slide

• Clustering query
‒ Averaged the embeddings of words in TF
and DM queries used by each user and
obtained a vector representing each user
‒ Normalized each cluster by the number
of searches for purchase-related terms
• Compare satisfaction with each cluster
‒ C3 has the highest percentage of satisfaction
‒ C2 has the lowest percentage of satisfaction
RQ3: What Is The Relationship between Query Terms and Satisfaction? 14
purchase,
store, etc.
coupon,
sale, etc.
warranty,
repair, etc.
Cluster (C)
recommen-
dation, etc.
reputation,
etc.
latest,
2017, etc.
searched
with years
looked for
reviews
Purchase-related
word group
looked for
suggestion
visited
EC sites
average
cluster
C1 C2 C3 C4 C5
Users looking for opinions of others are more
likely to be satisfied with their purchase
Suggest

View Slide

• Prediction by temporal distribution of search intents and query words
‒ Frequency of search intents
‒ Bag of Words created from the queries
• No characteristic weights were seen
‒ Averaged word embeddings from the queries
‒ Additional pre-training BERT by the queries
• Ordered TF or DM queries chronologically
and split by [SEP]
Satisfaction Prediction 15
0.531
0.559
0.530 0.529
0.559
0.548
0.533
0.48
0.5
0.52
0.54
0.56
0.58
Accuracy
Frequency of search intents
Frequency of search intents + product and user features
Bag of Words
Word embeddings
Word embeddings + product and user features
BERT
BERT + product and user features
Accuracy of each model
① BERT has the highest accuracy
• Time series of queries may be useful for predicting
② The product and user features
improve accuracy
• BERT does not learn enough word/the features
combinations due to lack of training data
①
②
0

View Slide

• Implication
‒ SAT users searched for products more frequently and diversely before the purchase
‒ In the marketing research literature, an increase in the amount of search leads to
customer satisfaction indirectly [4]
Implication 16
Suggest
It should encourage a user to search
more frequently and diversely before purchasing
[4] Punj and Staelin. A model of consumer information search behavior for new automobiles. Journal of consumer research, 9(4), 1983.
popular camera
EOS R5 reputation
Nikon camera
Typical
SAT user
Typical
DSAT user
EOS R5 sale
camera sale
Purchase
Search timeline
Canon camera popular
Recommend queries
e.g.

View Slide

• Within a week prior to the purchase, SAT users more frequently
searched for a wider range of product-related information
• SAT users searched with more specific queries prior to purchase when:
‒ They are relatively familiar with web search
‒ They are purchasing a relatively expensive product
• Users looking for opinions of others are more likely to be satisfied with
their purchase
Summary 17
Main findings
We investigate the relationship between
information-seeking behavior on the web and post-purchase satisfaction
Examine the relationship between Web search behavior and review ratings on EC site
⊃
Subset
≡
Def

View Slide

Appendix

View Slide

Related Work: Customer Satisfaction 19
• Relationship between the level of interest in the product and
“post-purchase” satisfaction [5]
• Relationship between perceived product quality and “post-purchase”
satisfaction [6]
• User interest and product quality affect “post-purchase” satisfaction
• Whereas the effect of web information-seeking behaviors on the post-
purchase satisfaction has not been widely explored yet
Satisfaction
Positive
correlation
Product
quality
Satisfaction
Positive
correlation
Level of
interest
[5] Richins et al. Post-purchase product satisfaction: Incorporating the effects of involvement and time. Journal of Business Research, 23(2), 1991.
[6] Tsiotsou. The role of perceived product quality and overall satisfaction on purchase intentions. International journal of consumer studies, 30(2), 2006.

View Slide

• Relationship between search behaviors in EC sites and satisfaction
with search results [3]
‒ They also showed that searches directly related to purchasing can be classified into
two types of intent
• The relationship between search intent and "search result"
satisfaction was examined, but not "post-purchase" satisfaction
[3] Su et al. User intent, behaviour, and perceived satisfaction in product search. WSDM 2018.
Related Work: Search Behaviors Related to Purchasing 20
Search sessions on EC sites
Display
search results
Satisfaction with search results
Satisfied
e.g.
Query
popular camera
night photography
Query
EOS spec
EOS R5 price
EOS R5 used
TF
(narrow)
DM
(broad)
Dissatisfied
Intent to consider purchasing
and search narrowly
Intent to search products broadly
EOS: Camera Brand

View Slide

• Yahoo! JAPAN Search: logs of users routinely using the search engine
(≡ who conducted web search at least 10 days in each month)
‒ Sampled 13,882 users and 48,758,880 queries
• Yahoo! JAPAN Shopping: Purchase log and Reviews
‒ Out of 5-point scale, reviews with a rating of three or lower tend to include negative
comments so purchase with a 4 or 5 rating was defined as satisfactory purchase
‒ Num. of purchases: 15,277
• Since the same account was used in these services, we could identify
the same user's web search behavior and purchases
Dataset 21

View Slide

• 32 categories were grouped into 5 meta categories:
Appliance, Audio, Beauty, Gadgets, and Outdoor
• We grouped 32 categories into 5 meta categories and analyzed about
fifteen thousand purchases
Data Statistics 22
Search log Purchase log Reviews
Service Yahoo! JAPAN Search Yahoo! JAPAN Shopping
Period 2016 - 2017 2016 - 2018
Num. of users 13,882
Num. of purchases N/A SAT: 12,620; DSAT: 2,657
Num. of queries 48,758,880 N/A
Ave. query length 1.70 N/A
Ave. session length 3.03 N/A

View Slide

We use a weak supervision approach for
words that cannot be classified by the rule-based approach
• We want to identify query related to camera
• Identify words related to camera from queries classified as “Other”
‒ Weak supervision classifies them as DM (broad) because the word is not included in
the product dictionary
Intent Estimation by Weak Supervision 23
Query Intent
Query Intent
night photography DM (broad)
Classify
Words related to camera but classified as Other
=

View Slide

Pseudo positive: query between model
number, product name, or “camera”
Detail of Weak Supervision 24
Search session Session probably not
related to camera
Tokyo sightseeing
Mt. Fuji history
camera for beginners
how to choose a lens
EOS R5 spec
Intent estimated
by rule-base
DM (broad)
Other
TF (narrow)
Other
Other
Search timeline
Probably related
to camera
Pseudo negative: queries for sessions where
TF or DM never appear
• We used logistic regression to predict
‒ Prediction results with support vector machines, etc. were almost the same
• Sampled 100 queries for each meta category and take a majority vote with three annotators
‒ Accuracy: 0.91, F1 score: 0.53
• Predict camera accessories, etc. as positive

View Slide

How to Normalize Search Frequency 25
Intent ・・・ (-1w, 0] [0, +1w) ・・・
TF (narrow) 6 3
DM (broad) 4 2
Other 10 5
Num. of searches user u
Normalized by
the total search frequency
Intent ・・・ (-1w, 0] [0, +1w) ・・・
TF (narrow) 6/30 3/30
DM (broad) 4/30 2/30
Other 10/30 5/30
Normalize
: 30
Average the value of each cell across
all users and graph the resulting values
𝑢!,#
𝑢∗,∗
Normalized by time
segment t and intent i
Sum of

View Slide

RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? (DM) 26
Search Frequency
Temporal distribution of DM (broad) queries for each
SAT
DSAT
Search Frequency
Purchase
Purchase
Low-price
Low-search-
frequency
High-price
Low-search-
frequency
High-price
High-search-
frequency
Low-price
High-search-
frequency
Purchase
Purchase

View Slide

Temporal distribution of DM (broad) queries for each
RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? (DM) 27
Purchase
Only when the product price and search frequency
were low, DSAT users searched more frequently
with DM queries posterior to the purchase
When customers are dissatisfied with their purchase
and the product price is low, they might turn to find
alternatives to the purchased product

View Slide

Compare satisfaction with each cluster
Characteristics of Each Cluster 28
Users looking for opinions of others are
more likely to be satisfied with their purchase
C3 has the highest percentage of satisfaction
C2 has the lowest percentage of satisfaction
Ave. Num. of
Search TF (narrow)
Ave. Num. of
Search DM (broad)
Ratio of SAT
Ave. Price
Cluster (C)
C1 C2 C3 C4 C5

Cluster (C)
searched
with years
looked for
reviews
looked for
suggestion
visited
EC sites
average
cluster
C1 C2 C3 C4 C5
purchase,
store, etc.
coupon,
sale, etc.
warranty,
repair, etc.
recommen-
dation, etc.
reputation,
etc.
latest,
2017, etc.
Purchase-related
word group

View Slide

SAT DSAT
S 2(1f c2(-,
1: 0e.l ceme,2, .0-blem, (,q3(07
0e. (0, b37(,g, 5 00 ,27
2: (eC-m A), (eC-m B), m (l -0de0
3c2(-,, 1ell(,g, 12-0e, (eC-m C)
1h-..(,g, (eC-m D), .30ch 1e
(0e2 (le0 X), (eC-m E), b--)(,g
3: c-m. 0(1-,, d(ffe0e,ce, beg(,,e0
27.e, h-5 2- ch--1e, m )e0
4: .0(ce, (,e6.e,1(4e, f-0 1-,g, c-12
5: l(gh2, 1(8e, 1.ec, 5e(gh2
6: 1(l4e0, 5h(2e, 0ed, c-l-0, bl c)
7: 1 le, .-(,2, c m. (g,, c-3.-,
8: W-M, e4 l3 2(-,, 0e4(e5, 0e.32 2(-,
9: 2015, 2016, 2017, ,e5-27.e, l 2e12
10: h-5 2- 31e, 1e23.
11: 0ec-mme,d, .-.3l 0, 0 ,)(,g
Te0m g0-3.
0.033 0.042
0.173 0.180
0.072 0.061
0.099 0.130
0.025 0.019
0.043 0.042
0.024 0.028
0.115 0.123
0.070 0.070
0.021 0.017
0.325 0.287
DIFF
-0.009
-0.007
0.012
-0.031
0.006
0.001
-0.004
-0.009
-0.000
0.004
0.038
The Difference of Query Terms 29
SAT users searched for
suggestions
DSAT users
searched for price

View Slide

-0.2
-0.1
0
0.1
0.2
0.3
0.4
(-4w
,-3w
]
(-3w
,-2w
]
(-2w
,-1w
]
(-1w
,0]
[0, +
1w
)
[+
1w
, +
2w
)
[+
2w
, +
3w
)
[+
3w
, +
4w
)
Price
Search
Frequency
Weight
Product and user features and time segment for each intent
TF (narrow)
DM (broad)
Product and user features *
*
*
*
• Prediction by temporal distribution of search intents
‒ We used Random Forest and Logistic
Regression to indicate feature weights
‒ Accuracy using
product and user features: 0.56
• = The level of price and search frequency
Prediction by Temporal Distribution of Search Intents 30
Weights in Logistic Regression
(* indicates that the factor is statistically significant)
We try to predict (explain) satisfaction based on differences
in the temporal distribution of search intent and query words
Accuracy is not high, but the effects of some factors are statistically significant

View Slide

Limitation 31
• The limitations of our study
‒ it does not take into account the information actually collected by users
• More detailed relationship between user behavior and purchase satisfaction will be the
subject of future research

View Slide

What Web Search Behaviors Lead to Online Purchase Satisfaction? (EN)

What Web Search Behaviors Lead to Online Purchase Satisfaction? (EN)

YANAGIDA Yuki

More Decks by YANAGIDA Yuki

Other Decks in Research

Featured

Transcript