Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Dialogue Natural Language Inference
Search
Scatter Lab Inc.
April 03, 2020
Research
2.4k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Dialogue Natural Language Inference
Scatter Lab Inc.
April 03, 2020
Transcript
23 slides
Dialogue Natural Language Inference Sean Welleck et al., ACL’19 ࢿࠁ
(ML Research Scientist, Pingpong)
ݾର ݾର 1. Dialogue Consistency and NLI 2. Dialogue NLI
Dataset 1. Triple Generation 2. Triple Annotation 3. Re-ranking with NLI 4. Evaluation 1. On Dialogue NLI 2. On Consistency in Dialogue
Dialogue Consistency and NLI Dialogue Consistency and NLI
Dialogue Consistency and NLI Dialogue Consistency and NLI • ചীࢲ
࠺ੌҙࢿ • ࢚ਵ۽ ൞ӈೞա ೠߣ ߊࢤೞݶ ఋѺ ఀ • Semanticೠ ޙਸ ݅٘ח ֢۱݅ਵ۽ח ೧Ѿ ࠛо • Natural Language Inference (NLI) • NLU, sentence representation ١ NLP ߈ਸ ੜೞӝ ਤೠ ࣻױਵ۽ॄ જ • NLI ݽ؛ downstream task ࢿמ ೱ࢚ী ӝৈ
Dialogue Consistency and NLI Dialogue Consistency and NLI • ಕܰࣗա:
ޙ ഋక۽ അ. • ചীࢲ ੌҙࢿ • ӝࠄਵ۽ Persona consistency • ֤ܻਵ۽ ߓغח ݈ ইפۄب э ࢎۈ ݈ೡ Ѫ э ঋ ޙ • : ച ղীࢲ ೠ ࢎۈ ೠ ف ݈ ߓغח • : Ӓ ࢎۈ ಕܰࣗա৬ ߓغח P = {p1 , …, pm } (uA i , uA j ) (uA i , pA k )
Dialogue Consistency and NLI
Dialogue NLI Dataset Dialogue NLI Dataset
Dialogue NLI Dataset Dialogue NLI Dataset • ߊച-ಕܰࣗա , ಕܰࣗա-ಕܰࣗա
हਵ۽ ܖয • ߊച-ߊച हب ನೣغয ਵա प ೞ ঋ (ui , pj ) (pi , pj ) (ui , uj )
Triple Generation Dialogue NLI Dataset • Triple • PersonaChatীࢲ ಕܰࣗա
ޙҗ ߊച ੌࠗ۽ Triple ۨ࠶ • ՙܻ Entailment, Neutral, Contradiction కӦ • Tripleਸ ӝળਵ۽ E, N, Cܳ ݅ٚ! • Entailment: э Tripleী ࣘೞח ف ޙՙܻ • Neutral, Contradiction: 3о ߑߨ (e1 , r, e2 ) (u, p), (p, p)
Neutral Pairs Dialogue NLI Dataset • Miscellaneous utterance যו Tripleীب
ࣘೞ ঋח ߊച ৬ ಕܰࣗա ޙ ҙ҅ח Neutral • Persona pairing Ground truth ಕܰࣗաՙܻח ࠂغѢա ݽࣽغ ঋחח ઁ ೞী э Tripleਸ ҕਬೞ ঋ ח ಕܰࣗաՙܻ Ҋ, ೞਤ ޙٜՙܻب • Relation swap ࢲ۽ ة݀ੋ ࢎपਸ աఋղח ҙ҅ ী ࣘೞח ޙٜՙܻ u p (r, r′ )
Contradiction Pairs Dialogue NLI Dataset • Relation swap ࢲ۽ ݽࣽغח
ҙ҅ ী ࣘೞח ޙٜՙܻ • Entity swap Triple ীࢲ ೧ࢲ о عਸ ٸ ݽࣽغח ҃ ف Tripleী ࣘೞח ޙٜ ՙܻ • Numeric Tripleী ನೣػ ंܳ ܲ ं۽ ߄Լࢲ ٜ݅য ޙҗ ਗې Tripleী ؍ ޙٜਸ (r, r′ ) (e1 , r, e2 ) e2 → e′ 2 (e1 , r, e′ 2 )
Triple Annotation Dialogue NLI Dataset • ಕܰࣗա ޙ →
<category> <relation> <category> ex) <person> have_pet <animal> relation , entity ա, entityח schemaী হਵݶ ੑ۱ • ٜ݅য Triple۽ ࠙ܨ ف ઑѤ ೞաܳ ݅ೞݶ 1. о sub-string 2. (e1 , r, e2 ) ∈ ℛ ∈ ℰ u ∈ U u ∈ (e1 , r, e2 ) e2 u sim(u, p) ≥ τ
Statistics Dialogue NLI Dataset • Gold-standard test set: test set
ۨ࠶ ݏҊ ೠ ࢎۈ 3ݺ 2ݺ ࢚ੋ ࢠ݅ ݽ Ѫ
Dialogue NLI Dataset
Re-ranking with NLI Re-ranking with NLI
Consistent Dialogue Agents via NLI Re-ranking with NLI •
ߊച ஏী NLI ݽ؛ ஏ Ѿҗ ഝਊ NLI ݽ؛ Contradictionۄ ౸ױೠ റࠁח confidence݅ఀ ಕօ౭ܳ ષ ࢜۽ ࣻ۽ Re-ranking
Evaluation Evaluation
On Dialogue NLI Evaluation • InferSent, ESIM ف ݽ؛ ࢎਊ
On Consistency in Dialogue Evaluation • ݽ؛ • ച ݽ؛:
Key-value memory networkܳ PersonaChatਵ۽ ण • NLI ݽ؛: ESIMਸ Dialogue NLI۽ ण • ಣо ࣇ • PersonaChatীࢲ Triple ী ೧ೞח ߊച ܳ Ҋ agent ಕܰࣗաী ী ࣘೞח ޙ ਵݶ ܳ ਵ۽ р • Entailment ޙ 10ѐ, Contradiction ޙ 10ѐ, ޙ 10ѐܳ റࠁ۽ م • ݫܼ • Hits@k, Entail@k, Contradict@k (e1 , r, e2 ) u (e1 , r, e2 ) u
Evaluation
Result Evaluation
Human Evaluation Evaluation • ParlAIܳ ా೧ w/o re-rankingҗ w/ re-rankingਸ
࠺Ү • ಣо ୋب • ݽ؛ ݃ա ಕܰࣗաܳ ੜ ߸೮חо? (1~5) • ݽ؛ п ߊചо ಕܰࣗա৬ ੌҙغחо? (0, 1) • ݽ؛ п ߊചо ݽ؛ ߊച, ݽ؛ ಕܰࣗա৬ ݽࣽغחо? (0, 1)
хࢎפ✌ ୶о ޙ ژח ҾӘೠ ݶ ઁٚ ইې োۅ۽
োۅ ࣁਃ! ࢿࠁ (ML Research Scientist, Pingpong)
[email protected]
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
institutdelelevage
PRO
0
160
AI創薬の壁
kanojikajino
1
1.3k
PHTalks Bengaluru - SSRF When All Else Fails
dk999
0
1.2k
マニアックセッション 〜次世代のメールプロトコルの斜め読み(2026年版)
hirachan
0
520
全国町字単位空き家率推定データver1.0データ仕様
microbaseinc
0
340
SAM3を用いたコマ・吹き出しの 領域検出と分割構造からの読み順推定
kzmssk
0
230
Des solutions pour pâturer davantage en hiver
institutdelelevage
PRO
0
130
La génomique au service de la fromageabilité du lait grâce aux spectres MIR
institutdelelevage
PRO
0
230
A Preliminary Performance Evaluation of Cassandra Accord
sira
0
110
JPA2026_NetworkTutorial_JunKashihara
junkashihara
0
280
進化?迷走?CasualConc ファミリーアプリの現在地 @ 英語コーパス学会 2026
casualconc
0
160
ハードウェア研究で国際トップ会議を目指す!IROS 2027での論文採択を目指して
ayatokanada
7
4.2k
Featured
See All Featured
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
WCS-LA-2024
lcolladotor
0
1k
My Coaching Mixtape
mlcsv
0
390
Are puppies a ranking factor?
jonoalderson
2
4.2k
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
minecr
1
480
Making Projects Easy
brettharned
120
6.9k
What Being in a Rock Band Can Teach Us About Real World SEO
427marketing
0
1.1k
The #1 spot is gone: here's how to win anyway
tamaranovitovic
4
1.2k
Reality Check: Gamification 10 Years Later
codingconduct
0
2.4k
Building Adaptive Systems
keathley
44
3.3k
Build your cross-platform service in a week with App Engine
jlugia
234
19k
Between Models and Reality
mayunak
4
590