Dialogue Natural Language Inference

Dialogue Natural Language Inference

A42dd3541cd40296dcd8a5e6b4a01bef?s=128

Scatter Lab Inc.

April 03, 2020
Tweet

Transcript

  1. Dialogue Natural Language Inference Sean Welleck et al., ACL’19 ੢ࢿࠁ

    (ML Research Scientist, Pingpong)
  2. ݾର ݾର 1. Dialogue Consistency and NLI 2. Dialogue NLI

    Dataset 1. Triple Generation 2. Triple Annotation 3. Re-ranking with NLI 4. Evaluation 1. On Dialogue NLI 2. On Consistency in Dialogue
  3. Dialogue Consistency and NLI Dialogue Consistency and NLI

  4. Dialogue Consistency and NLI Dialogue Consistency and NLI • ؀ചীࢲ੄

    ࠺ੌҙࢿ • ࢚؀੸ਵ۽ ൞ӈೞա ೠߣ ߊࢤೞݶ ఋѺ੉ ఀ • Semanticೠ ޙ੢ਸ ݅٘ח ֢۱݅ਵ۽ח ೧Ѿ ࠛо • Natural Language Inference (NLI) • NLU, sentence representation ١ NLP ੹߈ਸ ੜೞӝ ਤೠ ࣻױਵ۽ॄ જ਺ • NLI ݽ؛੉ downstream task ࢿמ ೱ࢚ী ӝৈ
  5. Dialogue Consistency and NLI Dialogue Consistency and NLI • ಕܰࣗա:

    ޙ੢ ૘೤ ഋక۽ ಴അ. • ؀ചীࢲ੄ ੌҙࢿ • ӝࠄ੸ਵ۽ Persona consistency • ֤ܻ੸ਵ۽ ߓ஖غח ݈੉ ইפۄب э਷ ࢎۈ੉ ݈ೡ Ѫ э૑ ঋ਷ ޙ੢ • : ؀ച ղীࢲ ೠ ࢎۈ੉ ೠ ف ݈੉ ߓ஖غח૑ • : Ӓ ࢎۈ੄ ಕܰࣗա৬ ߓ஖غח૑ P = {p1 , …, pm } (uA i , uA j ) (uA i , pA k )
  6. Dialogue Consistency and NLI

  7. Dialogue NLI Dataset Dialogue NLI Dataset

  8. Dialogue NLI Dataset Dialogue NLI Dataset • ߊച-ಕܰࣗա , ಕܰࣗա-ಕܰࣗա

    हਵ۽ ੉ܖয૗ • ߊച-ߊച हب ನೣغয ੓ਵա प೷਷ ೞ૑ ঋ਺ (ui , pj ) (pi , pj ) (ui , uj )
  9. Triple Generation Dialogue NLI Dataset • Triple • PersonaChatীࢲ ಕܰࣗա

    ޙ੢җ ߊച ੌࠗ۽ Triple ۨ੉࠶ • ՙܻ Entailment, Neutral, Contradiction కӦ • Tripleਸ ӝળਵ۽ E, N, Cܳ ݅ٚ׮! • Entailment: э਷ Tripleী ࣘೞח ف ޙ੢ՙܻ ૟૑਺ • Neutral, Contradiction: 3о૑ ߑߨ (e1 , r, e2 ) (u, p), (p, p)
  10. Neutral Pairs Dialogue NLI Dataset • Miscellaneous utterance
 যו Tripleীب

    ࣘೞ૑ ঋח ߊച ৬ ಕܰࣗա ޙ੢ ੄ ҙ҅ח Neutral • Persona pairing
 Ground truth ಕܰࣗաՙܻח ઺ࠂغѢա ݽࣽغ૑ ঋח׮ח ੹ઁ ೞী э਷ Tripleਸ ҕਬೞ૑ ঋ ח ಕܰࣗաՙܻ ૟૙Ҋ, ೞਤ ޙ੢ٜՙܻب ૟૑਺ • Relation swap
 ࢲ۽ ة݀੸ੋ ࢎपਸ աఋղח ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ u p (r, r′ )
  11. Contradiction Pairs Dialogue NLI Dataset • Relation swap
 ࢲ۽ ݽࣽغח

    ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ • Entity swap
 Triple ীࢲ ೧ࢲ о عਸ ٸ ݽࣽغח ҃਋ ف Tripleী ࣘೞח ޙ੢ٜ ՙܻ ૟૑਺ • Numeric
 Tripleী ನೣػ ं੗ܳ ׮ܲ ं੗۽ ߄Լࢲ ٜ݅য૓ ޙ੢җ ਗې Tripleী ੓؍ ޙ੢ٜਸ ૟૑਺ (r, r′ ) (e1 , r, e2 ) e2 → e′ 2 (e1 , r, e′ 2 )
  12. Triple Annotation Dialogue NLI Dataset • ಕܰࣗա ޙ੢ → 


    <category> <relation> <category>
 ex) <person> have_pet <animal>
 relation , entity ੉ա, entityח schemaী হਵݶ ૒੽ ੑ۱ • ٜ݅য૓ Triple۽ ࠙ܨ
 ف ઑѤ ઺ ೞաܳ ݅઒ೞݶ 
 1. о ੄ sub-string
 2. (e1 , r, e2 ) ∈ ℛ ∈ ℰ u ∈ U u ∈ (e1 , r, e2 ) e2 u sim(u, p) ≥ τ
  13. Statistics Dialogue NLI Dataset • Gold-standard test set: test set

    ۨ੉࠶੉ ݏ׮Ҋ ೠ ࢎۈ੉ 3ݺ ઺ 2ݺ ੉࢚ੋ ࢠ೒݅ ݽ਷ Ѫ
  14. Dialogue NLI Dataset

  15. Re-ranking with NLI Re-ranking with NLI

  16. Consistent Dialogue Agents via NLI Re-ranking with NLI • ׮਺

    ߊച ৘ஏী NLI ݽ؛੄ ৘ஏ Ѿҗ ഝਊ
 NLI ݽ؛੉ Contradiction੉ۄ ౸ױೠ റࠁח confidence݅ఀ ಕօ౭ܳ ષ
 
 
 ࢜۽਍ ੼ࣻ۽ Re-ranking
  17. Evaluation Evaluation

  18. On Dialogue NLI Evaluation • InferSent, ESIM ف ݽ؛ ࢎਊ

  19. On Consistency in Dialogue Evaluation • ݽ؛ • ؀ച ݽ؛:

    Key-value memory networkܳ PersonaChatਵ۽ ೟ण • NLI ݽ؛: ESIMਸ Dialogue NLI۽ ೟ण • ಣо ࣇ • PersonaChatীࢲ Triple ী ೧׼ೞח ߊച ܳ ଺Ҋ agent ಕܰࣗաী ী ࣘೞח ޙ੢੉ ੓ਵݶ ܳ ੿׹ਵ۽ р઱ • Entailment ޙ੢ 10ѐ, Contradiction ޙ੢ 10ѐ, ੐੄ ޙ੢ 10ѐܳ റࠁ۽ م • ݫ౟ܼ • Hits@k, Entail@k, Contradict@k (e1 , r, e2 ) u (e1 , r, e2 ) u
  20. Evaluation

  21. Result Evaluation

  22. Human Evaluation Evaluation • ParlAIܳ ా೧ w/o re-rankingҗ w/ re-rankingਸ

    ࠺Ү • ಣо ୋب • ݽ؛੉ ঴݃ա ಕܰࣗաܳ ੜ ؀߸೮חо? (1~5) • ݽ؛੄ п ߊചо ಕܰࣗա৬ ੌҙغחо? (0, 1) • ݽ؛੄ п ߊചо ݽ؛੄ ੉੹ ߊച, ഑਷ ݽ؛ ಕܰࣗա৬ ݽࣽغחо? (0, 1)
  23. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽

    োۅ ઱ࣁਃ! ੢ࢿࠁ (ML Research Scientist, Pingpong) Email.seongbo@scatterlab.co.kr