Slide 1
Slide 1 text
%JTQVUFETFOUFODFTVHHFTUJPOUPXBSET
DSFEJCJMJUZPSJFOUFE8FCTFBSDI
:VTVLF:BNBNPUP
,ZPUP6OJWFSTJUZ
[email protected]
Slide 2
Slide 2 text
Q. *TUIF8FCDSFEJCMF
Slide 3
Slide 3 text
4FSJPVTTJUVBUJPOPG8FCDSFEJCJMJUZ
※2 S. Nakamura et al., Trustworthiness analysis of Web search results (ECDL 2007)
-BSHFBNPVOUPG
TVTQJDJPVT8FCJOGP
.BOZQFPQMFUSVTU
8FCTFBSDISFTVMUT
57
43
%
%
Credible
Non-credible
Questionnaire on Web search engine
More than 50% of medical
HP are unauthorized(*1)
I can trust
Google!
※1 E. Sillence et al., Trust and mistrust of online health sites (CHI 2004)
Slide 4
Slide 4 text
$PX4FBSDIDSFEJCJMJUZPSJFOUFE8FCTFBSDIFOHJOF
7JTVBMJ[FT TDPSFTPG8FCQBHFT
POJNQPSUBOU DSFEJCJMJUZDSJUFSJB
1.
2. +VTJDF%JFU(PPEPS#BE
3.
4.
F⒎FDUJWFEJFUNFUIPE
4LJQCSFBLGBTUGPSEJFU
)$(EJFUJTUIFCFTUXBZ
%JFUMBCPSBUPSZ
3DJH5DQN
&RYHUDJH 7\SLFDOLW\
5HSXWDWLRQ
)UHVKQHVV
3DJH5DQN
&RYHUDJH 7\SLFDOLW\
5HSXWDWLRQ
)UHVKQHVV
3DJH5DQN
&RYHUDJH 7\SLFDOLW\
5HSXWDWLRQ
)UHVKQHVV
3DJH5DQN
&RYHUDJH 7\SLFDOLW\
5HSXWDWLRQ
)UHVKQHVV
3DJH5DQN
&RYHUDJH 7\SLFDOLW\
5HSXWDWLRQ
)UHVKQHVV
TFBSDI
Coverage and
reputation are
important!
* Enhancing credibility judgment of Web search results, ACM CHI 2011.
2$BO$PX4FBSDI IFMQVTFST
"/PUQFSGFDU
Slide 5
Slide 5 text
&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU
1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU
OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM
)JHI GBNJMJBSJUZ
.VDI JOUFSFTU
'PSRVFSZ
4FMFDUTDSFEJCMF
8FCQBHFT
4IBMMPX
DSFEJCJMJUZ
KVEHNFOU
-PX GBNJMJBSJUZ
-JUUMFJOUFSFTU
'PSRVFSZ
User
4USJDUDSFEJCJMJUZ
KVEHNFOU
5
(FUTTVTQJDJPVT
QBHFT
XJUIPVUOPUJDF
Slide 6
Slide 6 text
&MBCPSBUJPOMFWFMUPDSFEJCJMJUZKVEHNFOU
1FPQMF XJUIPVUFYQFSUJTFPSJOUFSFTUDBOOPU
OPUJDFTVTQJDJPVTJOGPSNBUJPOTPXFMM
)JHI GBNJMJBSJUZ
.VDI JOUFSFTU
'PSRVFSZ
(FUTDSFEJCMF
8FCQBHFT
4IBMMPX
DSFEJCJMJUZ
KVEHNFOU
-PX GBNJMJBSJUZ
-JUUMFJOUFSFTU
'PSRVFSZ
User
4USJDUDSFEJCJMJUZ
KVEHNFOU
6
(FUTTVTQJDJPVT
QBHFT
XJUIPVUOPUJDF
'PSVTJOH$PX4FBSDI NPSFFGGFDUJWFMZ
*U`TOFDFTTBSZUPNBLFVTFSTNPSFDBSFGVM
JOTFBSDIQSPDFTT
Slide 7
Slide 7 text
3FTFBSDIHPBM
%JTQVUFETFOUFODFTVHHFTUJPO
GPSTFBSDIRVFSZ
.BLFVTFSTNPSFDBSFGVMBCPVUDSFEJCJMJUZKVEHNFOU
Slide 8
Slide 8 text
effective diet method
%JTQVUFETFOUFODFTVHHFTUJPOJOTFBSDIQSPDFTT
Disputed sentences about “effective diet method”
5IFBULJOT EJFUJTNPSFFGGFDUJWFUIBOPUIFSEJFUNFUIPET
)$(EJFUNFUIPEJTRVJUFFGGFDUJWF
)JHIQSPUFJOEJFUTBSFUIFNPTUFGGFDUJWF
I didn’t know
that HCG diet
is suspicious!
Slide 9
Slide 9 text
4ZTUFNGMPX
*OQVUBRVFSZ
&YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC
3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH
UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE
%JTQVUFETFOUFODFTVHHFTUJPO
Slide 10
Slide 10 text
&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC
Core idea
6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM
There is no evidence that
more effective than other diets available
the Raw Food Diet is
Slide 11
Slide 11 text
&YUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC
Core idea
6TFMJOHVJTUJDQBUUFSOTPGEJTQVUFTJHOBM
There is no evidence that
more effective than other diets available
the Raw Food Diet is
4JHOBMQBUUFSO
%JTQVUFETFOUFODF
Slide 12
Slide 12 text
&YBNQMFTPGMJOHVJTUJDQBUUFSO
lno proof that ~
lmistakenly believe that ~
lno evidence that ~
lit is not true that ~
lit is no clear that ~
lmisconception that ~
lmyth that ~
lit is denied that ~
lit is speculated that ~
* Rob Ennals et. al, What is disputed on the Web (WICOW２０10)
6TFlUIBUDMBVTFTzUPEFUFDUEJTQVUFT
Slide 13
Slide 13 text
TU TUFQFYUSBDUJOHEJTQVUFETFOUFODFTGSPNUIF8FC
Query: effective diet + “no evidence that”
is no evidence that
Alone causes weight loss. Some weight loss …
the use of diet supplements
0CUBJO8FCTFBSDISFTVMUTGSPNTFBSDIFOHJOF
&YUSBDUEJTQVUFETFOUFODFTVTJOHEFMJNJUFST
“the use of diet supplements
alone causes weight loss”
(“.”, “,”, “and”, “but”, “where”, etc.)
Slide 14
Slide 14 text
4ZTUFNGMPX
*OQVUBRVFSZ
&YUSBDU EJTQVUFETFOUFODFDBOEJEBUFTGSPN8FC
3BOL UIFDBOEJEBUFTCZDPOTJEFSJOH
UIFJSUZQJDBMJUZRVFSZMJLFMJIPPE
%JTQVUFETFOUFODFTVHHFTUJPO
Slide 15
Slide 15 text
OE TUFQSBOLJOHPGEJTQVUFETFOUFODFT
rank(s | q) = Pr(s | q)
∝ Pr(s) ((1− λ)Pr(t | M
c
t∈q
∏ )+ λ Pr(t | M
s
))
2VFSZ MJLFMJIPPE MBOHVBHF NPEFM
ᶄ )PX GSFRVFOUMZ EPFT B EJTQVUFE TFOUFODF BQQFBS
ᶃ )PX NBOZ RVFSZ UFSNT BSF DPOUBJOFE JO UIF TFOUFODF
4NPPUIJOHGBDUPS 2VFSZMJLFMJIPPE
-FOHUI T
GSFR U
Given sentence s & query q ,
)PXUPDBMDVMBUF
Slide 16
Slide 16 text
5ZQJDBMJUZPGEJTQVUFETFOUFODFT
1S TFOUFODF
≈ TFOUFODF
UZQJDBMJUZ
6TF5FYU3BOL GPSDBMDVMBUJPO
Typ(s
t
) = Sim
text
(s
i
,s
j
)⋅Typ(s
i
)
∑
Rada Mihalcea et. al., TextRank: Brinking Order into Texts (EMNLP2004)
)$(EJFUJTFGGFDUJWF
4LJQQJOHCSFBLGBTU
XPSLTXFMM
&GGFDUJWFEJFUJT)$(
)$(EJFUJTUIF
NPTUFGGFDUJWFXBZ
1FPQMFMJLF)$(
BTEJFUXFMM
*GBUFYUIBTNPSFTJNJMBSUFYUT
JUDPVMECFNPSFUZQJDBM
Slide 17
Slide 17 text
0VUQVUFYBNQMFT
Slide 18
Slide 18 text
&WBMVBUJPO
Relevance
)PXSFMFWBOUBSFTVHHFTUFEEJTQVUFETFOUFODFTUPBRVFSZ
Alertingness
)PXDSJUJDBM XJMMVTFSTCFDPNFBCPVU8FCTFBSDISFTVMUTGPSBRVFSZ
XIFOMPPLJOHBUUIFTVHHFTUFETFOUFODFT
Usefulness
)PXVTFGVM BSFUIFTVHHFTUFETFOUFODFTUPTFBSDIGPS8FCQBHFT
BCPVUDSFEJCJMJUZFWJEFODFT
Metrics
These are effective
for enhancing
my credibility judgment?
Slide 19
Slide 19 text
&WBMVBUJPONFUIPE
5IFTZTUFNHFOFSBUFE TFOUFODFTGPSBRVFSZVTJOHBNFUIPE
5IF TFOUFODFTXFSFTVHHFTUFE BTTVTQJDJPVTUP FWBMVBUPS
5IFFWBMVBUPSTKVEHFEUIFNXJUIZFTPSOPPOUIFDSJUFSJB
4UFQT XFSFEPOFGPS RVFSJFTPOUIFNFUIPET
$PNQBSFENFUIPET CZVTJOHFWBMVBUPST
0VSNFUIPE UZQJDBMJUZ MJLFMJIPPE
:BIPP,FZXPSETVHHFTUJPO
5ZQJDBMJUZ 5%4
vs.
Some people doubt the following sentences about effective diets Some people doubt the following sentences about effective diets
Some people doubt the following sentences about effective diets
Slide 20
Slide 20 text
2VFSZTFU
Category Query
Controversy
dinosaurs extinction, global warming,
earthquake cause, plastic recycling,
renewable energy
Health
effective diet, cancer treatment,
mobile phone health, vaccine side effects,
caffeine overdose
Misunderstanding
light bulb inventor, telephone inventor,
civil war reason, caesar salad name
Food
coffee health, grapefruit seed extract benefit,
potato poison
Finance
forex risk, mortgage refinancing,
personal debt reduction
Slide 21
Slide 21 text
3FMFWBODF PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ
0.80
0.70 0.69
0.60
0.67 0.64
1.00
0.93 0.94
0
0.2
0.4
0.6
0.8
1
r@1 r@3 r@5
Ours TYP Yahoo
˔ 2VFSZMPH CBTFE Yahoo IBT UIF IJHIFTU SFMFWBODF
˔ Ours TYP ˰ RVFSZMJLFMJIPPE NPEFM XPSLT XFMM
33% improved
Slide 22
Slide 22 text
"MFSUJOHOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ
0.70
0.63 0.65
0.50
0.57
0.51
0.05 0.07 0.09
0
0.2
0.4
0.6
0.8
1
a@1 a@3 a@5
Ours TYP Yahoo
8JUIOurs & TYP
BUMFBTU50% PGTVHHFTUFETFOUFODFT
DBMMFEVTFST`BUUFOUJPOUPDSFEJCJMJUZKVEHNFOU
Slide 23
Slide 23 text
6TFGVMOFTT PGTVHHFTUFETFOUFODFTGPSJOQVURVFSZ
0.35
0.45
0.51
0.45
0.52 0.51
0.20
0.23
0.27
0
0.2
0.4
0.6
0.8
1
u@1 u@3 u@5
Ours TYP Yahoo
˔ Ours TYP BSF CFUUFS UIBO Yahoo
˔ Ours is less useful to search for evidences than TYP?
Slide 24
Slide 24 text
8IZXBTPVSNFUIPEKVEHFEBTMFTTVTFGVM
Evaluator’s comment
5IFTVHHFTUFEEJTQVUFEQISBTFTUIFNTFMWFT
XFSFPGUFOVTFGVMBTFWJEFODFPSDMVFTUPKVEHF
UIFDSFEJCJMJUZPG8FCQBHFTBCPVUHJWFO
RVFSJFT4P*KVEHFEUIBU*EJEO`UOFFEUPTFBSDI
GPS8FCQBHFTVTJOHUIFN
A. .PTUTVHHFTUFEEJTQVUFEQISBTFTXFSF
VTFGVMDMVFTJOUIFNTFMWFT
0VSNFUIPEJTFGGFDUJWFFOPVHIUPNBLF
VTFSTNPSFDSJUJDBMGPSDSFEJCJMJUZKVEHNFOU
Conclusion
Slide 25
Slide 25 text
&SSPSBOBMZTJT
Error type Query Suggested disputed sentence
Non-relevant potato poison potatoes are fattening (5)
Ambiguous global warming global warming was the cause (1)
Grammatical error plastic recycling shipping plastics for recycling (3)
Difficult to understand telephone inventor agb was not the inventor (2)
A. 'BJMVSFUPFTUJNBUFSFMFWBODF
B. 'BJMVSFUPTQMJUTFOUFODFT
C. 5IFFWBMVBUPSTIBEOPJEFBTBCPVUUFSNT
Why?
A
B
C
B
Slide 26
Slide 26 text
$PODMVTJPO
5PEBZ`TUBML
%JTQVUFETFOUFODFTVHHFTUJPO
- More sophisticated disputed sentence extraction
- Consideration of sentence comprehensibility
- Evidence search to judge credibility of disputed sentences
5PNBLFVTFSTNPSFDSJUJDBMGPSDSFEJCJMJUZKVEHNFOU
UIFTZTUFNTVHHFTUTTVTQJDJPVTTFOUFODFT
$SFEJCJMJUZPSJFOUFE8FCTFBSDI
'VUVSFXPSL
with
Thank you so much for your listening, 謝謝！
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
ޮՌతͳμΠΤοτ
$POWFOUJPOBMRVFSZTVHHFTUJPO
Suggest disputed sentences about search keyword
4VHHFTUSFMFWBOULFZXPSETUPRVFSZ
8PSLTXFMMJGVTFSTDBOOPUTFBSDIXFMM
ʢ/PUSFMBUFEUPDSFEJCJMJUZKVEHNFOUʣ
&GGFDUJWFEJFU
&GGFDUJWFEJFUGPPE
&GGFDUJWFEJFUNFUIPE
&GGFDUJWFEJFUFYFSDJTF
.PTU&GGFDUJWFEJFU
&GGFDUJWFEJFUXBMLJOH
&GGFDUJWFEJFUNFEJDJOF
Related keywords to “Effective diet”
Slide 29
Slide 29 text
2VFTUJPOOBJSFJOFYQFSJNFOUBMFWBMVBUJPO
lUsefulness