What Web Search Behaviors Lead to Online Purchase Satisfaction? Yuki Yanagida, Makoto P. Kato (University of Tsukuba) Yuka Kawada, Takehiro Yamamoto, Hiroaki Ohshima (University of Hyogo) Sumio Fujita (Yahoo Japan Corporation) 15th ACM Web Science Conference (WebSci 2023)

Satisfying people purchasing products is an important issue in EC • “Post-purchase” satisfaction improves to be continuously use of EC sites [1] • Customer satisfaction can have a big impact since 41% of people shop online at least once a week [2] [1] Gustafsson et al. The effects of customer satisfaction, relationship commitment dimensions, and triggers on customer retention. Journal of marketing, 69(4), 2005. [2] PwC. December 2021 global consumer insights pulse survey. https://www.pwc.com/gx/en/industries/consumer-markets/consumer- insights-survey/archive/consumer-insights-survey-december-2021.html, (accessed 2022-11-28). Background 2

We investigate the relationship between information-seeking behavior on the web and post-purchase satisfaction • The findings of this study enables us to: ‒ Predict post-purchase satisfaction from web search behavior for non-reviewing users • Also addressed in this study Purpose 3 Examine the relationship between web search behavior and review ratings on EC site ⊃ ≡ Def Subset Data Satisfaction Studied search behavior Existing post-purchase satisfaction study Product or user characteristics “Post-purchase” satisfaction ❌ Existing search behavior study Logs, subject experiment “Search” satisfaction “EC” search This study Logs, product or user characteristics “Post-purchase” satisfaction “Web” search Comparison to existing studies

• RQ1: Are web search behaviors different for SAT (satisfied) and DSAT (dissatisfied) users? A Within a week prior to the purchase, SAT users more frequently searched for a wider range of product-related information • RQ2: Are web search behaviors different depending on post-purchase satisfaction and product/user characteristics? A SAT users searched with more specific queries prior to purchase when: • They are relatively familiar with web search • They are purchasing a relatively expensive product • RQ3: What is the relationship between query terms and satisfaction? A Users looking for opinions of others are more likely to be satisfied with their purchase RQs and Their Answers 4

We estimate the search intent of each query to characterize web search behaviors Analysis Flow 5 Query EOS R5 price popular camera night photography weather in Tokyo Query log Query Intent EOS R5 price TF (narrow) popular camera DM (broad) night photography Other weather in Tokyo Other Query Intent EOS R5 price TF (narrow) popular camera DM (broad) night photography DM (broad) weather in Tokyo Other Rule-based estimation Estimation by weak supervision Classify queries = Query frequency calculation for each time segment Time Frequency Web queries are enormous → Identify each intent systematically based on an existing taxonomy SAT DSAT We sampled logs from a major commercial search engine and EC sites (The same account was used in these services)

We want to estimate the search intent of each query → Create product dictionary and classify queries based on rules • TF: considering purchase and searching narrowly ‒ Corresponds to Target Finding in [3] ‒ It contains a product name, a brand name, or model number in camera • DM: searching broadly than TF ‒ Corresponds to Decision Making in [3] ‒ Condition TF is not met but contains “camera” • Other Rule-based Estimation 6 Query Intent EOS R5 price TF (narrow) popular camera DM (broad) night photography Other weather in Tokyo Other Related to camera but classified as Other [3] Su et al. User intent, behaviour, and perceived satisfaction in product search. WSDM 2018. e.g. : Camera category Search intent was defined based on previous research [3] Analyze differences in search behavior by intent We use a weak supervision for words that cannot be classified by the rule-based approach Measure

• RQ1: Are web search behaviors different for SAT (satisfied) and DSAT (dissatisfied) users? A Within a week prior to the purchase, SAT users more frequently searched for a wider range of product-related information A RQs and Their Answers 7

Temporal distribution of TF (narrow) queries RQ1: Are Web Search Behaviors Different for SAT and DSAT Users? 8 0 0.0001 0.0002 0.0003 0.0004 0.0005 (-4w ,-3w ] (-3w ,-2w ] (-2w ,-1w ] (-1w ,0] [0,+ 1w ) [+ 1w ,+ 2w ) [+ 2w ,+ 3w ) [+ 3w ,+ 4w ) Search frequency Time segment SAT DSAT DM (broad) was similar Within a week prior to the purchase, SAT users more searched with TF (narrow) SAT users looked for a broader range of information Hypothesize Normalized frequency by total number of searches e.g. [0,+1w): the time segment within a week of purchase Purchase 6/Mar 10/Mar 13/Mar 17/Mar Purchase Time Align and to count

Comparison of Query Term Entropy 9 We compared query term entropy prior to the purchase to examine whether SAT users looked for a broader range of information SAT users used more diverse query terms than DSAT users for both TF and DM queries 0 0.2 0.4 0.6 0.8 1 1.2 TF (narrow) DM (broad) Average Entropy 5.8 6 6.2 6.4 6.6 6.8 7 Other SAT DSAT Intent

A • RQ2: Are web search behaviors different depending on post-purchase satisfaction and product/user characteristics? A SAT users searched with more specific queries prior to purchase when: • They are relatively familiar with web search • They are purchasing a relatively expensive product RQs and Their Answers 10 • Search intent ‒ TF (narrow): Narrower search ‒ DM (broad): Broader search than TF

RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? 11 Search Frequency Temporal distribution of TF (narrow) queries for each combination of product price and search frequency levels SAT DSAT Search Frequency Purchase Purchase Low-price Low-search- frequency High-price Low-search- frequency High-price High-search- frequency Low-price High-search- frequency Purchase Purchase

Temporal distribution of TF (narrow) queries for each combination of product price and search frequency levels RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? 12 SAT DSAT High-price High-search- frequency Purchase If they are familiar with web search, they might turn to conduct web search about a specific product to lower the purchasing risk Only when the product price and search frequency were high, SAT users searched more frequently with TF queries prior to the purchase

A A • RQ3: What is the relationship between query terms and satisfaction? A Users looking for opinions of others are more likely to be satisfied with their purchase RQs and Their Answers 13

• Clustering query ‒ Averaged the embeddings of words in TF and DM queries used by each user and obtained a vector representing each user ‒ Normalized each cluster by the number of searches for purchase-related terms • Compare satisfaction with each cluster ‒ C3 has the highest percentage of satisfaction ‒ C2 has the lowest percentage of satisfaction RQ3: What Is The Relationship between Query Terms and Satisfaction? 14 purchase, store, etc. coupon, sale, etc. warranty, repair, etc. Cluster (C) recommen- dation, etc. reputation, etc. latest, 2017, etc. searched with years looked for reviews Purchase-related word group looked for suggestion visited EC sites average cluster C1 C2 C3 C4 C5 Users looking for opinions of others are more likely to be satisfied with their purchase Suggest

• Prediction by temporal distribution of search intents and query words ‒ Frequency of search intents ‒ Bag of Words created from the queries • No characteristic weights were seen ‒ Averaged word embeddings from the queries ‒ Additional pre-training BERT by the queries • Ordered TF or DM queries chronologically and split by [SEP] Satisfaction Prediction 15 0.531 0.559 0.530 0.529 0.559 0.548 0.533 0.48 0.5 0.52 0.54 0.56 0.58 Accuracy Frequency of search intents Frequency of search intents + product and user features Bag of Words Word embeddings Word embeddings + product and user features BERT BERT + product and user features Accuracy of each model ① BERT has the highest accuracy • Time series of queries may be useful for predicting ② The product and user features improve accuracy • BERT does not learn enough word/the features combinations due to lack of training data ① ② 0

• Implication ‒ SAT users searched for products more frequently and diversely before the purchase ‒ In the marketing research literature, an increase in the amount of search leads to customer satisfaction indirectly [4] Implication 16 Suggest It should encourage a user to search more frequently and diversely before purchasing [4] Punj and Staelin. A model of consumer information search behavior for new automobiles. Journal of consumer research, 9(4), 1983. popular camera EOS R5 reputation Nikon camera Typical SAT user Typical DSAT user EOS R5 sale camera sale Purchase Search timeline Canon camera popular Recommend queries e.g.

• Within a week prior to the purchase, SAT users more frequently searched for a wider range of product-related information • SAT users searched with more specific queries prior to purchase when: ‒ They are relatively familiar with web search ‒ They are purchasing a relatively expensive product • Users looking for opinions of others are more likely to be satisfied with their purchase Summary 17 Main findings We investigate the relationship between information-seeking behavior on the web and post-purchase satisfaction Examine the relationship between Web search behavior and review ratings on EC site ⊃ Subset ≡ Def

Appendix

Related Work: Customer Satisfaction 19 • Relationship between the level of interest in the product and “post-purchase” satisfaction [5] • Relationship between perceived product quality and “post-purchase” satisfaction [6] • User interest and product quality affect “post-purchase” satisfaction • Whereas the effect of web information-seeking behaviors on the post- purchase satisfaction has not been widely explored yet Satisfaction Positive correlation Product quality Satisfaction Positive correlation Level of interest [5] Richins et al. Post-purchase product satisfaction: Incorporating the effects of involvement and time. Journal of Business Research, 23(2), 1991. [6] Tsiotsou. The role of perceived product quality and overall satisfaction on purchase intentions. International journal of consumer studies, 30(2), 2006.

• Relationship between search behaviors in EC sites and satisfaction with search results [3] ‒ They also showed that searches directly related to purchasing can be classified into two types of intent • The relationship between search intent and "search result" satisfaction was examined, but not "post-purchase" satisfaction [3] Su et al. User intent, behaviour, and perceived satisfaction in product search. WSDM 2018. Related Work: Search Behaviors Related to Purchasing 20 Search sessions on EC sites Display search results Satisfaction with search results Satisfied e.g. Query popular camera night photography Query EOS spec EOS R5 price EOS R5 used TF (narrow) DM (broad) Dissatisfied Intent to consider purchasing and search narrowly Intent to search products broadly EOS: Camera Brand

• Yahoo! JAPAN Search: logs of users routinely using the search engine (≡ who conducted web search at least 10 days in each month) ‒ Sampled 13,882 users and 48,758,880 queries • Yahoo! JAPAN Shopping: Purchase log and Reviews ‒ Out of 5-point scale, reviews with a rating of three or lower tend to include negative comments so purchase with a 4 or 5 rating was defined as satisfactory purchase ‒ Num. of purchases: 15,277 • Since the same account was used in these services, we could identify the same user's web search behavior and purchases Dataset 21

• 32 categories were grouped into 5 meta categories: Appliance, Audio, Beauty, Gadgets, and Outdoor • We grouped 32 categories into 5 meta categories and analyzed about fifteen thousand purchases Data Statistics 22 Search log Purchase log Reviews Service Yahoo! JAPAN Search Yahoo! JAPAN Shopping Period 2016 - 2017 2016 - 2018 Num. of users 13,882 Num. of purchases N/A SAT: 12,620; DSAT: 2,657 Num. of queries 48,758,880 N/A Ave. query length 1.70 N/A Ave. session length 3.03 N/A

We use a weak supervision approach for words that cannot be classified by the rule-based approach • We want to identify query related to camera • Identify words related to camera from queries classified as “Other” ‒ Weak supervision classifies them as DM (broad) because the word is not included in the product dictionary Intent Estimation by Weak Supervision 23 Query Intent EOS R5 price TF (narrow) popular camera DM (broad) night photography Other weather in Tokyo Other Query Intent EOS R5 price TF (narrow) popular camera DM (broad) night photography DM (broad) weather in Tokyo Other Classify Words related to camera but classified as Other =

Pseudo positive: query between model number, product name, or “camera” Detail of Weak Supervision 24 Search session Session probably not related to camera Tokyo sightseeing Mt. Fuji history camera for beginners how to choose a lens EOS R5 spec Intent estimated by rule-base DM (broad) Other TF (narrow) Other Other Search timeline Probably related to camera Pseudo negative: queries for sessions where TF or DM never appear • We used logistic regression to predict ‒ Prediction results with support vector machines, etc. were almost the same • Sampled 100 queries for each meta category and take a majority vote with three annotators ‒ Accuracy: 0.91, F1 score: 0.53 • Predict camera accessories, etc. as positive

How to Normalize Search Frequency 25 Intent ・・・ (-1w, 0] [0, +1w) ・・・ TF (narrow) 6 3 DM (broad) 4 2 Other 10 5 Num. of searches user u Normalized by the total search frequency Intent ・・・ (-1w, 0] [0, +1w) ・・・ TF (narrow) 6/30 3/30 DM (broad) 4/30 2/30 Other 10/30 5/30 Normalize : 30 Average the value of each cell across all users and graph the resulting values 𝑢!,# 𝑢∗,∗ Normalized by time segment t and intent i Sum of

RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? (DM) 26 Search Frequency Temporal distribution of DM (broad) queries for each combination of product price and search frequency levels SAT DSAT Search Frequency Purchase Purchase Low-price Low-search- frequency High-price Low-search- frequency High-price High-search- frequency Low-price High-search- frequency Purchase Purchase

Temporal distribution of DM (broad) queries for each combination of product price and search frequency levels RQ2: Are Web Search Behaviors Different Depending on Post-purchase Satisfaction and Product/user Characteristics? (DM) 27 Purchase Only when the product price and search frequency were low, DSAT users searched more frequently with DM queries posterior to the purchase When customers are dissatisfied with their purchase and the product price is low, they might turn to find alternatives to the purchased product

Compare satisfaction with each cluster Characteristics of Each Cluster 28 Users looking for opinions of others are more likely to be satisfied with their purchase C3 has the highest percentage of satisfaction C2 has the lowest percentage of satisfaction Ave. Num. of Search TF (narrow) Ave. Num. of Search DM (broad) Ratio of SAT Ave. Price Cluster (C) C1 C2 C3 C4 C5 Cluster (C) searched with years looked for reviews looked for suggestion visited EC sites average cluster C1 C2 C3 C4 C5 purchase, store, etc. coupon, sale, etc. warranty, repair, etc. recommen- dation, etc. reputation, etc. latest, 2017, etc. Purchase-related word group

SAT DSAT S 2(1f c2(-, 1: 0e.l ceme,2, .0-blem, (,q3(07 0e. (0, b37(,g, 5 00 ,27 2: (eC-m A), (eC-m B), m (l -0de0 3c2(-,, 1ell(,g, 12-0e, (eC-m C) 1h-..(,g, (eC-m D), .30ch 1e (0e2 (le0 X), (eC-m E), b--)(,g 3: c-m. 0(1-,, d(ffe0e,ce, beg(,,e0 27.e, h-5 2- ch--1e, m )e0 4: .0(ce, (,e6.e,1(4e, f-0 1-,g, c-12 5: l(gh2, 1(8e, 1.ec, 5e(gh2 6: 1(l4e0, 5h(2e, 0ed, c-l-0, bl c) 7: 1 le, .-(,2, c m. (g,, c-3.-, 8: W-M, e4 l3 2(-,, 0e4(e5, 0e.32 2(-, 9: 2015, 2016, 2017, ,e5-27.e, l 2e12 10: h-5 2- 31e, 1e23. 11: 0ec-mme,d, .-.3l 0, 0 ,)(,g Te0m g0-3. 0.033 0.042 0.173 0.180 0.072 0.061 0.099 0.130 0.025 0.019 0.043 0.042 0.024 0.028 0.115 0.123 0.070 0.070 0.021 0.017 0.325 0.287 DIFF -0.009 -0.007 0.012 -0.031 0.006 0.001 -0.004 -0.009 -0.000 0.004 0.038 The Difference of Query Terms 29 SAT users searched for suggestions DSAT users searched for price

-0.2 -0.1 0 0.1 0.2 0.3 0.4 (-4w ,-3w ] (-3w ,-2w ] (-2w ,-1w ] (-1w ,0] [0, + 1w ) [+ 1w , + 2w ) [+ 2w , + 3w ) [+ 3w , + 4w ) Price Search Frequency Weight Product and user features and time segment for each intent TF (narrow) DM (broad) Product and user features * * * * • Prediction by temporal distribution of search intents ‒ We used Random Forest and Logistic Regression to indicate feature weights ‒ Accuracy using product and user features: 0.56 • = The level of price and search frequency Prediction by Temporal Distribution of Search Intents 30 Weights in Logistic Regression (* indicates that the factor is statistically significant) We try to predict (explain) satisfaction based on differences in the temporal distribution of search intent and query words Accuracy is not high, but the effects of some factors are statistically significant

Limitation 31 • The limitations of our study ‒ it does not take into account the information actually collected by users • More detailed relationship between user behavior and purchase satisfaction will be the subject of future research