Dialogue Natural Language Inference

Transcript

1. Dialogue Natural Language Inference Sean Welleck et al., ACL’19 ੢ࢿࠁ (ML Research Scientist, Pingpong)
2. ݾର ݾର 1. Dialogue Consistency and NLI 2. Dialogue NLI Dataset 1. Triple Generation 2. Triple Annotation 3. Re-ranking with NLI 4. Evaluation 1. On Dialogue NLI 2. On Consistency in Dialogue
3. Dialogue Consistency and NLI Dialogue Consistency and NLI
4. Dialogue Consistency and NLI Dialogue Consistency and NLI • ؀ചীࢲ੄ ࠺ੌҙࢿ • ࢚؀੸ਵ۽ ൞ӈೞա ೠߣ ߊࢤೞݶ ఋѺ੉ ఀ • Semanticೠ ޙ੢ਸ ݅٘ח ֢۱݅ਵ۽ח ೧Ѿ ࠛо • Natural Language Inference (NLI) • NLU, sentence representation ١ NLP ੹߈ਸ ੜೞӝ ਤೠ ࣻױਵ۽ॄ જ਺ • NLI ݽ؛੉ downstream task ࢿמ ೱ࢚ী ӝৈ
5. Dialogue Consistency and NLI Dialogue Consistency and NLI • ಕܰࣗա: ޙ੢ ૘೤ ഋక۽ ಴അ. • ؀ചীࢲ੄ ੌҙࢿ • ӝࠄ੸ਵ۽ Persona consistency • ֤ܻ੸ਵ۽ ߓ஖غח ݈੉ ইפۄب э਷ ࢎۈ੉ ݈ೡ Ѫ э૑ ঋ਷ ޙ੢ • : ؀ച ղীࢲ ೠ ࢎۈ੉ ೠ ف ݈੉ ߓ஖غח૑ • : Ӓ ࢎۈ੄ ಕܰࣗա৬ ߓ஖غח૑ P = {p1 , …, pm } (uA i , uA j ) (uA i , pA k )
6. Dialogue Consistency and NLI
7. Dialogue NLI Dataset Dialogue NLI Dataset
8. Dialogue NLI Dataset Dialogue NLI Dataset • ߊച-ಕܰࣗա , ಕܰࣗա-ಕܰࣗա हਵ۽ ੉ܖয૗ • ߊച-ߊച हب ನೣغয ੓ਵա प೷਷ ೞ૑ ঋ਺ (ui , pj ) (pi , pj ) (ui , uj )
9. Triple Generation Dialogue NLI Dataset • Triple • PersonaChatীࢲ ಕܰࣗա ޙ੢җ ߊച ੌࠗ۽ Triple ۨ੉࠶ • ՙܻ Entailment, Neutral, Contradiction కӦ • Tripleਸ ӝળਵ۽ E, N, Cܳ ݅ٚ׮! • Entailment: э਷ Tripleী ࣘೞח ف ޙ੢ՙܻ ૟૑਺ • Neutral, Contradiction: 3о૑ ߑߨ (e1 , r, e2 ) (u, p), (p, p)
10. Neutral Pairs Dialogue NLI Dataset • Miscellaneous utterance
 যו Tripleীب ࣘೞ૑ ঋח ߊച ৬ ಕܰࣗա ޙ੢ ੄ ҙ҅ח Neutral • Persona pairing
 Ground truth ಕܰࣗաՙܻח ઺ࠂغѢա ݽࣽغ૑ ঋח׮ח ੹ઁ ೞী э਷ Tripleਸ ҕਬೞ૑ ঋ ח ಕܰࣗաՙܻ ૟૙Ҋ, ೞਤ ޙ੢ٜՙܻب ૟૑਺ • Relation swap
 ࢲ۽ ة݀੸ੋ ࢎपਸ աఋղח ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ u p (r, r′ )
11. Contradiction Pairs Dialogue NLI Dataset • Relation swap
 ࢲ۽ ݽࣽغח ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ • Entity swap
 Triple ীࢲ ೧ࢲ о عਸ ٸ ݽࣽغח ҃਋ ف Tripleী ࣘೞח ޙ੢ٜ ՙܻ ૟૑਺ • Numeric
 Tripleী ನೣػ ं੗ܳ ׮ܲ ं੗۽ ߄Լࢲ ٜ݅য૓ ޙ੢җ ਗې Tripleী ੓؍ ޙ੢ٜਸ ૟૑਺ (r, r′ ) (e1 , r, e2 ) e2 → e′ 2 (e1 , r, e′ 2 )
12. Triple Annotation Dialogue NLI Dataset • ಕܰࣗա ޙ੢ → 
 <category> <relation> <category>
 ex) <person> have_pet <animal>
 relation , entity ੉ա, entityח schemaী হਵݶ ૒੽ ੑ۱ • ٜ݅য૓ Triple۽ ࠙ܨ
 ف ઑѤ ઺ ೞաܳ ݅઒ೞݶ 
 1. о ੄ sub-string
 2. (e1 , r, e2 ) ∈ ℛ ∈ ℰ u ∈ U u ∈ (e1 , r, e2 ) e2 u sim(u, p) ≥ τ
13. Statistics Dialogue NLI Dataset • Gold-standard test set: test set ۨ੉࠶੉ ݏ׮Ҋ ೠ ࢎۈ੉ 3ݺ ઺ 2ݺ ੉࢚ੋ ࢠ೒݅ ݽ਷ Ѫ
14. Dialogue NLI Dataset
15. Re-ranking with NLI Re-ranking with NLI
16. Consistent Dialogue Agents via NLI Re-ranking with NLI • ׮਺ ߊച ৘ஏী NLI ݽ؛੄ ৘ஏ Ѿҗ ഝਊ
 NLI ݽ؛੉ Contradiction੉ۄ ౸ױೠ റࠁח confidence݅ఀ ಕօ౭ܳ ષ
 
 
 ࢜۽਍ ੼ࣻ۽ Re-ranking
17. Evaluation Evaluation
18. On Dialogue NLI Evaluation • InferSent, ESIM ف ݽ؛ ࢎਊ
19. On Consistency in Dialogue Evaluation • ݽ؛ • ؀ച ݽ؛: Key-value memory networkܳ PersonaChatਵ۽ ೟ण • NLI ݽ؛: ESIMਸ Dialogue NLI۽ ೟ण • ಣо ࣇ • PersonaChatীࢲ Triple ী ೧׼ೞח ߊച ܳ ଺Ҋ agent ಕܰࣗաী ী ࣘೞח ޙ੢੉ ੓ਵݶ ܳ ੿׹ਵ۽ р઱ • Entailment ޙ੢ 10ѐ, Contradiction ޙ੢ 10ѐ, ੐੄ ޙ੢ 10ѐܳ റࠁ۽ م • ݫ౟ܼ • Hits@k, Entail@k, Contradict@k (e1 , r, e2 ) u (e1 , r, e2 ) u
20. Evaluation
21. Result Evaluation
22. Human Evaluation Evaluation • ParlAIܳ ా೧ w/o re-rankingҗ w/ re-rankingਸ ࠺Ү • ಣо ୋب • ݽ؛੉ ঴݃ա ಕܰࣗաܳ ੜ ؀߸೮חо? (1~5) • ݽ؛੄ п ߊചо ಕܰࣗա৬ ੌҙغחо? (0, 1) • ݽ؛੄ п ߊചо ݽ؛੄ ੉੹ ߊച, ഑਷ ݽ؛ ಕܰࣗա৬ ݽࣽغחо? (0, 1)
23. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽ োۅ ઱ࣁਃ! ੢ࢿࠁ (ML Research Scientist, Pingpong) Email.seongbo@scatterlab.co.kr