Slide 1

Slide 1 text

Entity Ranking by Learning and Inferring Pairwise Preferences from User Reviews AIRS2017 Shinryo Uchida, Takehiro Yamamoto, Makoto P. Kato, Hiroaki Ohshima, Katsumi Tanaka Kyoto University, Japan 1

Slide 2

Slide 2 text

2 Which camera is more suitable for capturing a sport scene? Canon EOS 6D EF24-105 Price: $1,500 Sensor size: 35mm Shutter speed: 1/4000 - 30sec ISO: 100 - 102400 Focal length: 24 - 105mm Weight: 1,205g Attributes Nikon D7200 18-300 Price: $1,500 Sensor size: APS-C Shutter speed: 1/8000 - 30sec ISO: 100 - 102400 Focal length: 18 - 300mm Weight: 1,225g Attributes

Slide 3

Slide 3 text

Definition Search Attribute − An attribute that a user can know its value before using the product − E.g., size, weight, ISO, price, … Experience Attribute − An attribute that a user cannot know its value unless they use the product − E.g., capturing sport scene, portability, usability, … 3 Nelson, P. (1970). Information and consumer behavior. The Journal of Political Economy, 311-329.

Slide 4

Slide 4 text

Motivation  Difficult to predict the quality of an experience attribute from search attributes − especially for non-experts 4 Which camera is best for capturing beautiful photo? ? Sensor:APS-C Sensor:APS-H Sensor:35mm

Slide 5

Slide 5 text

Learning to rank entities in terms of a given experience attribute based on their search attributes Research Goal = (0.6, 0.4, 0.2, 0.7, … ) = (0.3, 0.2, 0.2, 0.4, … ) = (0.8, 0.6, 0.5, 0.7, … ) Size Weight Rsl. F-value Search Attributes o(1) o(2) o(3) = 0.2 = 0.8 = 0.5 Rank Portability fq (o(2)) fq (o(1)) fq (o(3)) e.g. q = Portability (Experience Attribute) 1st 2nd 3rd 5

Slide 6

Slide 6 text

Use pairwise preferences extracted from user reviews Basic Approach >Portable Pairwise preference >Portable >Portable … Reviews Training data (o(1) , o(2)) (o(1) , o(3)) (o(3) , o(4)) … Ranking SVM Extract Learn Represent entities by search attributes = (0.6, 0.4, 0.2, 0.7, … ) Size Weight Rsl. F-value o(1) : 6

Slide 7

Slide 7 text

Pairwise Preference  Pairwise Preference(obj1 >attr obj2 ) − Order of objects in terms of a certain attribute − Extracted from comparative sentences written in user reviews 7 Extracted >Portable Pairwise pref. User Reviews is more portable than Comparative sentence

Slide 8

Slide 8 text

Challenge  Few pairwise preferences in user reviews − Lack of training data 8 Extracted >Portable Pairwise pref. User Reviews is more portable than Comparative sentence >Portable >Portable >Portable >Portable

Slide 9

Slide 9 text

Our Approach: Inferring Preferences Infer pairwise preferences based on attribute dependencies to gain the size of training data 9 Attribute Dep. Weight →− Portable >Weight Pairwise Pref. >Size Portable Size →− Portable Infer

Slide 10

Slide 10 text

Attribute Dependency  Attribute Dependency(attr1 →[+|−] attr2) − Dependency between two attributes − Extracted from resultative conjunctions written in user reviews Attribute dep. Weight →− Portable Extracted User Reviews Resultative is portable because it is light 10

Slide 11

Slide 11 text

Inferring Preferences Infer pairwise preferences based on attribute dependencies to gain the size of training data 11 Attribute Dep. Weight →− Portability >Weight Pairwise Pref. >Size Portable Size →− Portability Reviews Extracted Infer

Slide 12

Slide 12 text

Confidence of Preferences  Motivation − Inferred pairwise pref. may not be reliable  Approach − Assign confidence to pairwise preferences 12 Portable Pairwise pref. explicitly written in reviews >Portable , 0.9 , 0.8 , 0.3 , 0.2 confidence score

Slide 13

Slide 13 text

Confidence of Preferences 13 Attribute Dep. Weight →− Portable >Weight Pairwise Pref.

Slide 14

Slide 14 text

Confidence of Preferences 14 Attribute Dep. Weight →− Portable >Weight Pairwise Pref.

Slide 15

Slide 15 text

Confidence of Preferences 15 Attribute Dep. Weight →− Portable >Weight Pairwise Pref.

Slide 16

Slide 16 text

Confidence of Preferences 16 Attribute Dep. Weight →− Portable >Weight Pairwise Pref.

Slide 17

Slide 17 text

Method Overview Reviews Training data (o(1) , o(2), 1.0) (o(1) , o(3), 0.8) (o(4) , o(3), 0.6) … Extracted Learn By their search attributes Represented Attribute dep. Weight →− Portability Size →− Portability (o(1) , o(4), 0.4) (o(5) , o(1), 0.3) (o(2) , o(3), 0.2) … Fuzzy Ranking SVM >Portability Pairwise pref. >Portability >Portability … 1.0 0.8 0.6 0.4 0.3 >Portability Inferred pref. >Portability >Portability … 0.4 0.3 0.2 Infer Increase the size of training data by inference Confidence for pairwise pref. 17

Slide 18

Slide 18 text

Fuzzy Ranking SVM 18 minimize: subject to: minimize: subject to: Ranking SVM [Joachim2002] Fuzzy Ranking SVM (Fuzzy SVM + Ranking SVM) minimize: subject to: Fuzzy SVM [Lin2002] Training data: { ( xi , xj , si ) } weight of training data Ranking SVM Ranking SVM weight of training data (confidence)

Slide 19

Slide 19 text

Experiments Dataset Evaluation Metric Category # Entities # Reviews # Attributes Example experience attributes Cameras 688 15,473 47 Usability, feeling of hold Smartphones 624 33,731 186 Easy-to-hold, comfort Headphones 2,229 13,117 304 Clearness, richness collected from Kakaku.com • Accuracy of learnt pairwise preferences evaluated by leave-one-out cross validation Ground Truth • Pairwise preferences written in reviews 20

Slide 20

Slide 20 text

Baselines and Proposed Methods 20  Ranking by Frequency (Freq) − # of reviews that contain the query (= experience attr.)  Ranking by Regression (Reg) − Frequency as dependent variable  Without inference and confidence (L2R) − only uses pairwise preferences written in reviews  Without confidence (L2R+Inf) − Infers preferences but confidences are always set to 1.0  Inference and confidence (L2R+Inf+Conf) Baselines Proposed Methods

Slide 21

Slide 21 text

 Learning to rank by pairwise preferences outperformed baselines − Frequency of experience attributes in reviews are not useful for ranking Experimental Results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Cameras Smartphones Headphones Macro Average Freq Reg L2R L2R+Infer L2R+Infer+Conf baselines proposed 22

Slide 22

Slide 22 text

 The inference (L2R+Infer(+Conf)) improved the performance of learning to rank (L2R) − Although the effect is small for headphones (few attribute dependencies found) Experimental Results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Cameras Smartphones Headphones Macro Average Freq Reg L2R L2R+Infer L2R+Infer+Conf baselines proposed 23

Slide 23

Slide 23 text

Summary  Learning to rank entities in terms of a given experience attribute based on search attributes − Helpful to non-experts  Key Ideas − Learning to rank by pairwise preferences − Inferring pairwise preferences - To gain the size of training data − Confidence for preferences - Fuzzy Ranking SVM 23