APWeb2012発表資料

%JTQVUFETFOUFODFTVHHFTUJPOUPXBSET DSFEJCJMJUZPSJFOUFE8FCTFBSDI :VTVLF:BNBNPUP ,ZPUP6OJWFSTJUZ [email protected]

Q. *TUIF8FCDSFEJCMF

4FSJPVTTJUVBUJPOPG8FCDSFEJCJMJUZ ※2 S. Nakamura et al., Trustworthiness analysis of Web
search results (ECDL 2007) -BSHFBNPVOUPG TVTQJDJPVT8FCJOGP .BOZQFPQMFUSVTU 8FCTFBSDISFTVMUT 57 43 % % Credible Non-credible Questionnaire on Web search engine More than 50% of medical HP are unauthorized(*1) I can trust Google! ※1 E. Sillence et al., Trust and mistrust of online health sites (CHI 2004)

$PX4FBSDIDSFEJCJMJUZPSJFOUFE8FCTFBSDIFOHJOF 7JTVBMJ[FT TDPSFTPG8FCQBHFT POJNQPSUBOU DSFEJCJMJUZDSJUFSJB 1. 2. +VTJDF%JFU(PPEPS#BE 3. 4.
F⒎FDUJWFEJFUNFUIPE 4LJQCSFBLGBTUGPSEJFU )$(EJFUJTUIFCFTUXBZ %JFUMBCPSBUPSZ 3DJH5DQN &RYHUDJH 7\SLFDOLW\ 5HSXWDWLRQ )UHVKQHVV 3DJH5DQN &RYHUDJH 7\SLFDOLW\ 5HSXWDWLRQ )UHVKQHVV 3DJH5DQN &RYHUDJH 7\SLFDOLW\ 5HSXWDWLRQ )UHVKQHVV 3DJH5DQN &RYHUDJH 7\SLFDOLW\ 5HSXWDWLRQ )UHVKQHVV 3DJH5DQN &RYHUDJH 7\SLFDOLW\ 5HSXWDWLRQ )UHVKQHVV TFBSDI Coverage and reputation are important! * Enhancing credibility judgment of Web search results, ACM CHI 2011. 2$BO$PX4FBSDI IFMQVTFST "/PUQFSGFDU

&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU 1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM )JHI GBNJMJBSJUZ .VDI JOUFSFTU 'PSRVFSZ 4FMFDUTDSFEJCMF
8FCQBHFT 4IBMMPX DSFEJCJMJUZ KVEHNFOU -PX GBNJMJBSJUZ -JUUMFJOUFSFTU 'PSRVFSZ User 4USJDUDSFEJCJMJUZ KVEHNFOU 5 (FUTTVTQJDJPVT QBHFT XJUIPVUOPUJDF

&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU 1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM )JHI GBNJMJBSJUZ .VDI JOUFSFTU 'PSRVFSZ (FUTDSFEJCMF
8FCQBHFT 4IBMMPX DSFEJCJMJUZ KVEHNFOU -PX GBNJMJBSJUZ -JUUMFJOUFSFTU 'PSRVFSZ User 4USJDUDSFEJCJMJUZ KVEHNFOU 6 (FUTTVTQJDJPVT QBHFT XJUIPVUOPUJDF 'PSVTJOH$PX4FBSDI NPSFFGGFDUJWFMZ *U`TOFDFTTBSZUPNBLFVTFSTNPSFDBSFGVM JOTFBSDIQSPDFTT

3FTFBSDIHPBM %JTQVUFETFOUFODFTVHHFTUJPO GPSTFBSDIRVFSZ .BLFVTFSTNPSFDBSFGVMBCPVUDSFEJCJMJUZKVEHNFOU

effective diet method %JTQVUFETFOUFODFTVHHFTUJPOJOTFBSDIQSPDFTT Disputed sentences about “effective diet method”
5IFBULJOT EJFUJTNPSFFGGFDUJWFUIBOPUIFSEJFUNFUIPET )$(EJFUNFUIPEJTRVJUFFGGFDUJWF )JHIQSPUFJOEJFUTBSFUIFNPTUFGGFDUJWF I didn’t know that HCG diet is suspicious!

4ZTUFNGMPX *OQVUBRVFSZ &YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC 3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE %JTQVUFETFOUFODFTVHHFTUJPO

&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Core idea 6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM There is no evidence that more
effective than other diets available the Raw Food Diet is

&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Core idea 6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM There is no evidence that more
effective than other diets available the Raw Food Diet is 4JHOBMQBUUFSO %JTQVUFETFOUFODF

&YBNQMFTPGMJOHVJTUJDQBUUFSO lno proof that ~ lmistakenly believe that ~ lno
evidence that ~ lit is not true that ~ lit is no clear that ~ lmisconception that ~ lmyth that ~ lit is denied that ~ lit is speculated that ~ * Rob Ennals et. al, What is disputed on the Web (WICOW２０10) 6TFlUIBUDMBVTFTzUPEFUFDUEJTQVUFT

TU TUFQFYUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Query: effective diet + “no evidence that” is
no evidence that Alone causes weight loss. Some weight loss … the use of diet supplements 0CUBJO8FCTFBSDISFTVMUTGSPNTFBSDIFOHJOF &YUSBDUEJTQVUFETFOUFODFTVTJOHEFMJNJUFST “the use of diet supplements alone causes weight loss” (“.”, “,”, “and”, “but”, “where”, etc.)

4ZTUFNGMPX *OQVUBRVFSZ &YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC 3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE %JTQVUFETFOUFODFTVHHFTUJPO

OE TUFQSBOLJOHPGEJTQVUFETFOUFODFT rank(s | q) = Pr(s | q) ∝
Pr(s) ((1− λ)Pr(t | M c t∈q ∏ )+ λ Pr(t | M s )) 2VFSZ MJLFMJIPPE MBOHVBHF NPEFM ᶄ )PX GSFRVFOUMZ EPFT B EJTQVUFE TFOUFODF BQQFBS ᶃ )PX NBOZ RVFSZ UFSNT BSF DPOUBJOFE JO UIF TFOUFODF 4NPPUIJOHGBDUPS 2VFSZMJLFMJIPPE -FOHUI T GSFR U Given sentence s & query q , )PXUPDBMDVMBUF

5ZQJDBMJUZPGEJTQVUFETFOUFODFT 1S TFOUFODF ≈ TFOUFODF UZQJDBMJUZ 6TF5FYU3BOL GPSDBMDVMBUJPO Typ(s t
) = Sim text (s i ,s j )⋅Typ(s i ) ∑ Rada Mihalcea et. al., TextRank: Brinking Order into Texts (EMNLP2004) )$(EJFUJTFGGFDUJWF 4LJQQJOHCSFBLGBTU XPSLTXFMM &GGFDUJWFEJFUJT)$( )$(EJFUJTUIF NPTUFGGFDUJWFXBZ 1FPQMFMJLF)$( BTEJFUXFMM *GBUFYUIBTNPSFTJNJMBSUFYUT JUDPVMECFNPSFUZQJDBM

0VUQVUFYBNQMFT

&WBMVBUJPO Relevance )PXSFMFWBOUBSFTVHHFTUFEEJTQVUFETFOUFODFTUPBRVFSZ Alertingness )PXDSJUJDBM XJMMVTFSTCFDPNFBCPVU8FCTFBSDISFTVMUTGPSBRVFSZ XIFOMPPLJOHBUUIFTVHHFTUFETFOUFODFT Usefulness )PXVTFGVM BSFUIFTVHHFTUFETFOUFODFTUPTFBSDIGPS8FCQBHFT
BCPVUDSFEJCJMJUZFWJEFODFT Metrics These are effective for enhancing my credibility judgment?

&WBMVBUJPONFUIPE 5IFTZTUFNHFOFSBUFE TFOUFODFTGPSBRVFSZVTJOHBNFUIPE 5IF TFOUFODFTXFSFTVHHFTUFE BTTVTQJDJPVTUP FWBMVBUPS
5IFFWBMVBUPSTKVEHFEUIFNXJUIZFTPSOPPOUIFDSJUFSJB 4UFQT XFSFEPOFGPS RVFSJFTPOUIFNFUIPET $PNQBSFENFUIPET CZVTJOHFWBMVBUPST 0VSNFUIPE UZQJDBMJUZ MJLFMJIPPE :BIPP,FZXPSETVHHFTUJPO 5ZQJDBMJUZ 5%4 vs. Some people doubt the following sentences about effective diets Some people doubt the following sentences about effective diets Some people doubt the following sentences about effective diets

2VFSZTFU Category Query Controversy dinosaurs extinction, global warming, earthquake cause,
plastic recycling, renewable energy Health effective diet, cancer treatment, mobile phone health, vaccine side effects, caffeine overdose Misunderstanding light bulb inventor, telephone inventor, civil war reason, caesar salad name Food coffee health, grapefruit seed extract benefit, potato poison Finance forex risk, mortgage refinancing, personal debt reduction

3FMFWBODF PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.80 0.70 0.69 0.60 0.67 0.64 1.00 0.93
0.94 0 0.2 0.4 0.6 0.8 1 r@1 r@3 r@5 Ours TYP Yahoo ˔ 2VFSZMPH CBTFE Yahoo IBT UIF IJHIFTU SFMFWBODF ˔ Ours TYP ˰ RVFSZMJLFMJIPPE NPEFM XPSLT XFMM 33% improved

"MFSUJOHOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.70 0.63 0.65 0.50 0.57 0.51 0.05 0.07
0.09 0 0.2 0.4 0.6 0.8 1 a@1 a@3 a@5 Ours TYP Yahoo 8JUIOurs & TYP BUMFBTU50% PGTVHHFTUFETFOUFODFT DBMMFEVTFST`BUUFOUJPOUPDSFEJCJMJUZKVEHNFOU

6TFGVMOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.35 0.45 0.51 0.45 0.52 0.51 0.20 0.23
0.27 0 0.2 0.4 0.6 0.8 1 u@1 u@3 u@5 Ours TYP Yahoo ˔ Ours TYP BSF CFUUFS UIBO Yahoo ˔ Ours is less useful to search for evidences than TYP?

8IZXBTPVSNFUIPEKVEHFEBTMFTTVTFGVM Evaluator’s comment 5IFTVHHFTUFEEJTQVUFEQISBTFTUIFNTFMWFT XFSFPGUFOVTFGVMBTFWJEFODFPSDMVFTUPKVEHF UIFDSFEJCJMJUZPG8FCQBHFTBCPVUHJWFO RVFSJFT4P*KVEHFEUIBU*EJEO`UOFFEUPTFBSDI GPS8FCQBHFTVTJOHUIFN A. .PTUTVHHFTUFEEJTQVUFEQISBTFTXFSF
VTFGVMDMVFTJOUIFNTFMWFT 0VSNFUIPEJTFGGFDUJWFFOPVHIUPNBLF VTFSTNPSFDSJUJDBMGPSDSFEJCJMJUZKVEHNFOU Conclusion

&SSPSBOBMZTJT Error type Query Suggested disputed sentence Non-relevant potato poison
potatoes are fattening (5) Ambiguous global warming global warming was the cause (1) Grammatical error plastic recycling shipping plastics for recycling (3) Difficult to understand telephone inventor agb was not the inventor (2) A. 'BJMVSFUPFTUJNBUFSFMFWBODF B. 'BJMVSFUPTQMJUTFOUFODFT C. 5IFFWBMVBUPSTIBEOPJEFBTBCPVUUFSNT Why? A B C B

$PODMVTJPO 5PEBZ`TUBML %JTQVUFETFOUFODFTVHHFTUJPO - More sophisticated disputed sentence extraction -
Consideration of sentence comprehensibility - Evidence search to judge credibility of disputed sentences 5PNBLFVTFSTNPSFDSJUJDBMGPSDSFEJCJMJUZKVEHNFOU UIFTZTUFNTVHHFTUTTVTQJDJPVTTFOUFODFT $SFEJCJMJUZPSJFOUFE8FCTFBSDI 'VUVSFXPSL with Thank you so much for your listening, 謝謝！

ޮՌతͳμΠΤοτ $POWFOUJPOBMRVFSZTVHHFTUJPO Suggest disputed sentences about search keyword 4VHHFTUSFMFWBOULFZXPSETUPRVFSZ 8PSLTXFMMJGVTFSTDBOOPUTFBSDIXFMM
ʢ/PUSFMBUFEUPDSFEJCJMJUZKVEHNFOUʣ &GGFDUJWFEJFU &GGFDUJWFEJFUGPPE &GGFDUJWFEJFUNFUIPE &GGFDUJWFEJFUFYFSDJTF .PTU&GGFDUJWFEJFU &GGFDUJWFEJFUXBMLJOH &GGFDUJWFEJFUNFEJDJOF Related keywords to “Effective diet”

2VFTUJPOOBJSFJOFYQFSJNFOUBMFWBMVBUJPO lUsefulness

APWeb2012発表資料

APWeb2012発表資料

Y. Yamamoto

More Decks by Y. Yamamoto

Other Decks in Research

Featured

Transcript

%JTQVUFETFOUFODFTVHHFTUJPOUPXBSET DSFEJCJMJUZPSJFOUFE8FCTFBSDI :VTVLF:BNBNPUP ,ZPUP6OJWFSTJUZ [email protected]

Q. *TUIF8FCDSFEJCMF

4FSJPVTTJUVBUJPOPG8FCDSFEJCJMJUZ ※2 S. Nakamura et al., Trustworthiness analysis of Web

$PX4FBSDIDSFEJCJMJUZPSJFOUFE8FCTFBSDIFOHJOF 7JTVBMJ[FT TDPSFTPG8FCQBHFT POJNQPSUBOU DSFEJCJMJUZDSJUFSJB 1. 2. +VTJDF%JFU(PPEPS#BE 3. 4.

&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU 1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM )JHI GBNJMJBSJUZ .VDI JOUFSFTU 'PSRVFSZ 4FMFDUTDSFEJCMF

&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU 1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM )JHI GBNJMJBSJUZ .VDI JOUFSFTU 'PSRVFSZ (FUTDSFEJCMF

3FTFBSDIHPBM %JTQVUFETFOUFODFTVHHFTUJPO GPSTFBSDIRVFSZ .BLFVTFSTNPSFDBSFGVMBCPVUDSFEJCJMJUZKVEHNFOU

effective diet method %JTQVUFETFOUFODFTVHHFTUJPOJOTFBSDIQSPDFTT Disputed sentences about “effective diet method”

4ZTUFNGMPX *OQVUBRVFSZ &YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC 3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE %JTQVUFETFOUFODFTVHHFTUJPO

&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Core idea 6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM There is no evidence that more

&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Core idea 6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM There is no evidence that more

&YBNQMFTPGMJOHVJTUJDQBUUFSO lno proof that ~ lmistakenly believe that ~ lno

TU TUFQFYUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC Query: effective diet + “no evidence that” is

4ZTUFNGMPX *OQVUBRVFSZ &YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC 3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE %JTQVUFETFOUFODFTVHHFTUJPO

OE TUFQSBOLJOHPGEJTQVUFETFOUFODFT rank(s | q) = Pr(s | q) ∝

5ZQJDBMJUZPGEJTQVUFETFOUFODFT 1S TFOUFODF ≈ TFOUFODF UZQJDBMJUZ 6TF5FYU3BOL GPSDBMDVMBUJPO Typ(s t

0VUQVUFYBNQMFT

&WBMVBUJPO Relevance )PXSFMFWBOUBSFTVHHFTUFEEJTQVUFETFOUFODFTUPBRVFSZ Alertingness )PXDSJUJDBM XJMMVTFSTCFDPNFBCPVU8FCTFBSDISFTVMUTGPSBRVFSZ XIFOMPPLJOHBUUIFTVHHFTUFETFOUFODFT Usefulness )PXVTFGVM BSFUIFTVHHFTUFETFOUFODFTUPTFBSDIGPS8FCQBHFT

&WBMVBUJPONFUIPE 5IFTZTUFNHFOFSBUFE TFOUFODFTGPSBRVFSZVTJOHBNFUIPE 5IF TFOUFODFTXFSFTVHHFTUFE BTTVTQJDJPVTUP FWBMVBUPS

2VFSZTFU Category Query Controversy dinosaurs extinction, global warming, earthquake cause,

3FMFWBODF PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.80 0.70 0.69 0.60 0.67 0.64 1.00 0.93

"MFSUJOHOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.70 0.63 0.65 0.50 0.57 0.51 0.05 0.07

6TFGVMOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ 0.35 0.45 0.51 0.45 0.52 0.51 0.20 0.23

8IZXBTPVSNFUIPEKVEHFEBTMFTTVTFGVM Evaluator’s comment 5IFTVHHFTUFEEJTQVUFEQISBTFTUIFNTFMWFT XFSFPGUFOVTFGVMBTFWJEFODFPSDMVFTUPKVEHF UIFDSFEJCJMJUZPG8FCQBHFTBCPVUHJWFO RVFSJFT4PKVEHFEUIBUEJEO`UOFFEUPTFBSDI GPS8FCQBHFTVTJOHUIFN A. .PTUTVHHFTUFEEJTQVUFEQISBTFTXFSF

&SSPSBOBMZTJT Error type Query Suggested disputed sentence Non-relevant potato poison

$PODMVTJPO 5PEBZ`TUBML %JTQVUFETFOUFODFTVHHFTUJPO - More sophisticated disputed sentence extraction -

ޮՌతͳμΠΤοτ $POWFOUJPOBMRVFSZTVHHFTUJPO Suggest disputed sentences about search keyword 4VHHFTUSFMFWBOULFZXPSETUPRVFSZ 8PSLTXFMMJGVTFSTDBOOPUTFBSDIXFMM

2VFTUJPOOBJSFJOFYQFSJNFOUBMFWBMVBUJPO lUsefulness