Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dialogue Natural Language Inference

Dialogue Natural Language Inference

Scatter Lab Inc.

April 03, 2020
Tweet

More Decks by Scatter Lab Inc.

Other Decks in Research

Transcript

  1. ݾର ݾର 1. Dialogue Consistency and NLI 2. Dialogue NLI

    Dataset 1. Triple Generation 2. Triple Annotation 3. Re-ranking with NLI 4. Evaluation 1. On Dialogue NLI 2. On Consistency in Dialogue
  2. Dialogue Consistency and NLI Dialogue Consistency and NLI • ؀ചীࢲ੄

    ࠺ੌҙࢿ • ࢚؀੸ਵ۽ ൞ӈೞա ೠߣ ߊࢤೞݶ ఋѺ੉ ఀ • Semanticೠ ޙ੢ਸ ݅٘ח ֢۱݅ਵ۽ח ೧Ѿ ࠛо • Natural Language Inference (NLI) • NLU, sentence representation ١ NLP ੹߈ਸ ੜೞӝ ਤೠ ࣻױਵ۽ॄ જ਺ • NLI ݽ؛੉ downstream task ࢿמ ೱ࢚ী ӝৈ
  3. Dialogue Consistency and NLI Dialogue Consistency and NLI • ಕܰࣗա:

    ޙ੢ ૘೤ ഋక۽ ಴അ. • ؀ചীࢲ੄ ੌҙࢿ • ӝࠄ੸ਵ۽ Persona consistency • ֤ܻ੸ਵ۽ ߓ஖غח ݈੉ ইפۄب э਷ ࢎۈ੉ ݈ೡ Ѫ э૑ ঋ਷ ޙ੢ • : ؀ച ղীࢲ ೠ ࢎۈ੉ ೠ ف ݈੉ ߓ஖غח૑ • : Ӓ ࢎۈ੄ ಕܰࣗա৬ ߓ஖غח૑ P = {p1 , …, pm } (uA i , uA j ) (uA i , pA k )
  4. Dialogue NLI Dataset Dialogue NLI Dataset • ߊച-ಕܰࣗա , ಕܰࣗա-ಕܰࣗա

    हਵ۽ ੉ܖয૗ • ߊച-ߊച हب ನೣغয ੓ਵա प೷਷ ೞ૑ ঋ਺ (ui , pj ) (pi , pj ) (ui , uj )
  5. Triple Generation Dialogue NLI Dataset • Triple • PersonaChatীࢲ ಕܰࣗա

    ޙ੢җ ߊച ੌࠗ۽ Triple ۨ੉࠶ • ՙܻ Entailment, Neutral, Contradiction కӦ • Tripleਸ ӝળਵ۽ E, N, Cܳ ݅ٚ׮! • Entailment: э਷ Tripleী ࣘೞח ف ޙ੢ՙܻ ૟૑਺ • Neutral, Contradiction: 3о૑ ߑߨ (e1 , r, e2 ) (u, p), (p, p)
  6. Neutral Pairs Dialogue NLI Dataset • Miscellaneous utterance
 যו Tripleীب

    ࣘೞ૑ ঋח ߊച ৬ ಕܰࣗա ޙ੢ ੄ ҙ҅ח Neutral • Persona pairing
 Ground truth ಕܰࣗաՙܻח ઺ࠂغѢա ݽࣽغ૑ ঋח׮ח ੹ઁ ೞী э਷ Tripleਸ ҕਬೞ૑ ঋ ח ಕܰࣗաՙܻ ૟૙Ҋ, ೞਤ ޙ੢ٜՙܻب ૟૑਺ • Relation swap
 ࢲ۽ ة݀੸ੋ ࢎपਸ աఋղח ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ u p (r, r′ )
  7. Contradiction Pairs Dialogue NLI Dataset • Relation swap
 ࢲ۽ ݽࣽغח

    ҙ҅ ী ࣘೞח ޙ੢ٜՙܻ ૟૑਺ • Entity swap
 Triple ীࢲ ೧ࢲ о عਸ ٸ ݽࣽغח ҃਋ ف Tripleী ࣘೞח ޙ੢ٜ ՙܻ ૟૑਺ • Numeric
 Tripleী ನೣػ ं੗ܳ ׮ܲ ं੗۽ ߄Լࢲ ٜ݅য૓ ޙ੢җ ਗې Tripleী ੓؍ ޙ੢ٜਸ ૟૑਺ (r, r′ ) (e1 , r, e2 ) e2 → e′ 2 (e1 , r, e′ 2 )
  8. Triple Annotation Dialogue NLI Dataset • ಕܰࣗա ޙ੢ → 


    <category> <relation> <category>
 ex) <person> have_pet <animal>
 relation , entity ੉ա, entityח schemaী হਵݶ ૒੽ ੑ۱ • ٜ݅য૓ Triple۽ ࠙ܨ
 ف ઑѤ ઺ ೞաܳ ݅઒ೞݶ 
 1. о ੄ sub-string
 2. (e1 , r, e2 ) ∈ ℛ ∈ ℰ u ∈ U u ∈ (e1 , r, e2 ) e2 u sim(u, p) ≥ τ
  9. Statistics Dialogue NLI Dataset • Gold-standard test set: test set

    ۨ੉࠶੉ ݏ׮Ҋ ೠ ࢎۈ੉ 3ݺ ઺ 2ݺ ੉࢚ੋ ࢠ೒݅ ݽ਷ Ѫ
  10. Consistent Dialogue Agents via NLI Re-ranking with NLI • ׮਺

    ߊച ৘ஏী NLI ݽ؛੄ ৘ஏ Ѿҗ ഝਊ
 NLI ݽ؛੉ Contradiction੉ۄ ౸ױೠ റࠁח confidence݅ఀ ಕօ౭ܳ ષ
 
 
 ࢜۽਍ ੼ࣻ۽ Re-ranking
  11. On Consistency in Dialogue Evaluation • ݽ؛ • ؀ച ݽ؛:

    Key-value memory networkܳ PersonaChatਵ۽ ೟ण • NLI ݽ؛: ESIMਸ Dialogue NLI۽ ೟ण • ಣо ࣇ • PersonaChatীࢲ Triple ী ೧׼ೞח ߊച ܳ ଺Ҋ agent ಕܰࣗաী ী ࣘೞח ޙ੢੉ ੓ਵݶ ܳ ੿׹ਵ۽ р઱ • Entailment ޙ੢ 10ѐ, Contradiction ޙ੢ 10ѐ, ੐੄ ޙ੢ 10ѐܳ റࠁ۽ م • ݫ౟ܼ • Hits@k, Entail@k, Contradict@k (e1 , r, e2 ) u (e1 , r, e2 ) u
  12. Human Evaluation Evaluation • ParlAIܳ ా೧ w/o re-rankingҗ w/ re-rankingਸ

    ࠺Ү • ಣо ୋب • ݽ؛੉ ঴݃ա ಕܰࣗաܳ ੜ ؀߸೮חо? (1~5) • ݽ؛੄ п ߊചо ಕܰࣗա৬ ੌҙغחо? (0, 1) • ݽ؛੄ п ߊചо ݽ؛੄ ੉੹ ߊച, ഑਷ ݽ؛ ಕܰࣗա৬ ݽࣽغחо? (0, 1)