Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network

Transcript

1. 스캐터랩(ScatterLab) ੌ࢚؀ച ੋҕ૑מ Technical Seminar: Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network 백영민 Dialogue System Machine Learning Engineer
2. #1. Introduction
3. !3 Conversational AI ࢎۈҗ Open-domain topicਵ۽ ੗োझۣҊ ૑ࣘ੸ੋ ؀ചܳ ೡ ࣻ ੓ח AI
4. • ࢎۈ਷ ؀ചೡ ٸ, ৈ۞ ੄ઓҙ҅(੄޷੸, ӝמ੸)ܳ Ҋ۰ೠ׮. !4 Introduction #1 Human Conversation A: ցޖ ৮߷೮য ৮੹ ୶ୌ! B: ೵.. ƀƀ աب оҊर׮ ৈ೯૑? ਺ध੼? …
5. • ࢎۈ਷ ؀ചೡ ٸ, ৈ۞ ੄ઓҙ҅(੄޷੸, ӝמ੸)ܳ Ҋ۰ೠ׮. !5 Introduction #1 Human Conversation B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮੹ ୶ୌ! B: ೵.. ƀƀ աب оҊर׮ ৈ೯૑? ਺ध੼? …
6. • ࢎۈ਷ ؀ചೡ ٸ, ৈ۞ ੄ઓҙ҅(੄޷੸, ӝמ੸)ܳ Ҋ۰ೠ׮. !6 Introduction #1 Human Conversation A: য়ט ੼बী ஠ಌܚীࢲ ೫ߡѢ ݡ঻য!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮੹ ୶ୌ! B: ೵.. ƀƀ աب оҊर׮ ਺ध੼!
7. • ࢎۈ਷ ؀ചೡ ٸ, ৈ۞ ੄ઓҙ҅(੄޷੸, ӝמ੸)ܳ Ҋ۰ೠ׮. !7 Introduction #1 Human Conversation A: য়ט ੼बী ஠ಌܚীࢲ ೫ߡѢ ݡ঻য!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮੹ ୶ୌ! B: ೵.. ƀƀ աب оҊर׮ ਺ध੼! A: Ѣӝ оݶ ԙ ؊࠶ߡѢ ࣁ౟ܳ ݡযঠ೧!
8. • ؀ചഋ ੋఠಕ੉झীࢲ ࢎۈ੉ ইצ ஹೊఠ(ࠈ)੉ ਬ੷৬ ࣗాೞח ࢲ࠺झ • ୭Ӕ োҳ زೱ • Open Domain topic ীࢲ ࢎۈҗ ૑ࣘ੸੉Ҋ ੗োझۣѱ ؀ചೡ ࣻ ੓ח Chatbot • Data Driven Approach: ୷੸ػ ؘ੉ఠܳ ߄ఔਵ۽ ٜ݅য૓ Chatbot • Retrieval-Based • Generation-Based !8 Introduction #1 Conversational AI - Chatbot
9. • ৈ۞ ؘ੉ఠٜਸ ੉ਊೞৈ ݽ؛ਸ ೟ण • Retrieval-Based Approach: • ޷ܻ ੿೧૓ റࠁ ׹߸ٜ ઺ о੢ જ਷ ׹߸ਸ ࢶఖೞ੗! • അ੤ pingpong੉ ࢎਊೞҊ ੓ח ߑध • Single-Turn/Multi-Turn • Generation-Based Approach: • ؀ച ؘ੉ఠٜ۽ ࠗఠ ಁఢਸ ೟णೞৈ ੉ܳ ߄ఔਵ۽ ࢜۽਍ ׹߸ਸ ࢤࢿೞ੗! !9 Introduction #1 Conversational AI - Data Driven Approach
10. • ࢎۈ਷ ؀ചೡ ٸ, ৈ۞ ੄ઓҙ҅(੄޷੸, ӝמ੸)ܳ Ҋ۰ೠ׮. • ੉੹ ؀ചীࢲ ৈ۞ ױਤ(ױয, ҳ ޙ੢)ٜী ੄ઓೞৈ ߈਽ೠ׮. • п ੄ઓҙ҅ܳ Ҋ۰ೡ ࣻ ੓ח ݽ؛(chat bot)ਸ ٜ݅੗! • Single-Turn: ੉੹ ߊച݅ ࠁҊ ؀׹ਸ ࢶఖ - context߂ ੄ઓҙ҅ܳ ߈৔ೞӝ ൨ٜ׮ • Multi-Turn: ੉੹ Nѐ੄ ߊചܳ ࠁҊ ؀׹ਸ ࢶఖ - context߂ ੄ઓҙ҅ܳ ߈৔ !10 Introduction #1 Conversational AI - Human Conversation
11. #2. Recent Work
12. !12 Multi-turn Retrieval Nѐ੄ ੉੹ ߊചܳ ా೧ റࠁ૑ীࢲ ੸੺ೠ ׹ਸ ࢶఖ
13. • ৈ۞ ఢ੄ ߊചٜਸ RNN(Recurrent Neural Network)ܳ ੉ਊೞৈ encoding • ׮নೠ ੄ઓҙ҅ܳ ౵ঈೞӝী ੸੺ೞ૑ ঋ׮. - ҳઑ ࢚੄ ೠ҅ • ঩਷ ੄ઓҙ҅(಴ݶ੸ਵ۽ ࠁ੉ח textual relevance - زੌೠ ױয, ਬࢎೠ ױয ١)݅ ౵ঈ оמ • Ө਷ ੄ઓҙ҅(coreference, long-term dependency ١)ী ஂড !13 Recent work #2 RNN(Recurrent Neural Network)
14. !14 Transformer self-attention݅ਸ ੉ਊೠ ࢜۽਍ architecture
15. • ౠ੿ ࠗ࠙(ױয, ҳ, ޙ੢ ١)ਸ “૘઺(attend)”ೞৈ Ѿҗܳ ب୹ೞח ߑध (Query - Key) • ୡӝীח RNN੄ ޙઁ(long-term dependency)ܳ ೧Ѿ೧઱ӝ ਤ೧ ࢎਊ • “Attention is All you need” - Attention ݅ਵ۽ب(Self-attention) જ਷ Ѿҗܳ յ ࣻ ੓਺ਸ ࠁ੐ • BERT/GPT ١ ୭Ӕ SOTA ݽ؛ٜ੉ ؀ࠗ࠙ ࢎਊ !15 Recent work #2 Attention
16. !16 Recent work #2 Attention A о E, F, G, H ܳ attend ೞৈ ࢜۽਍ A’ ࢤࢿ Query Key
17. !17 Recent work #2 Attention B о E, F, G, H ܳ attend ೞৈ ࢜۽਍ B’ ࢤࢿ Query Key
18. !18 Recent work #2 Attention C о E, F, G, H ܳ attend ೞৈ ࢜۽਍ C’ ࢤࢿ Query Key
19. • Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ -> ੗ӝ ੗नী ؀ೠ attention !19 Recent work #2 Transformer - Self Attention
20. • Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ -> ੗ӝ ੗नী ؀ೠ attention !20 Recent work #2 Transformer - Self Attention A о A, B, C, D ܳ attend ೞৈ ࢜۽਍ A’ ࢤࢿ Query & Key
21. • Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ -> ੗ӝ ੗नী ؀ೠ attention !21 Recent work #2 Transformer - Self Attention B о A, B, C, D ܳ attend ೞৈ ࢜۽਍ B’ ࢤࢿ Query & Key
22. • Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ -> ੗ӝ ੗नী ؀ೠ attention !22 Recent work #2 Transformer - Self Attention C о A, B, C, D ܳ attend ೞৈ ࢜۽਍ C’ ࢤࢿ Query & Key
23. !23 Recent work #2 Transformer - Self Attention ޙ੢ী ؀ೠ Ө਷ ੉೧
24. !24 Recent work #2 Transformer - Self Attention Layerо ऺৈтࣻ۾ ࠂ೤੸ੋ ੄޷/ҙ҅ܳ ݽ؛݂
25. #3. Method
26. !26 Deep Attention Matching Network Transformerܳ ੉ਊೠ Multi-turn retrieval model
27. • ಽҊ੗ೞח ޙઁ: Multi-trun retrieval • Data: Multi-turn ؀ച ؘ੉ఠࣇ (c, r, y) • c: n-1ѐ੄ context ߊച • r: response റࠁ • y: label (0, 1) • Ubuntu Corpus V1, Douban Conversational Corpus • g(c, r) -> yܳ ࣻ೯ೞח g(model)ਸ ೟णೞ੗! !27 Model Architecture #3 Problem
28. !28 Model Architecture #3 Overview
29. !29 Model Architecture #3 Input • Input utterance • Context utterance: • Reply utterance: • Embedding: • п ױযٜী ؀೧ d(=200)ରਗ embedding • Pre-trained word2vec ੉ਊ ui = [wui ,k ]nui −1 k=0 , nui : maxword(context) r = [wui ,k ]nr −1 t=0 , nr : maxword(reply)
30. !30 Model Architecture #3 Representation • Stacked Self-Attention • L ѐ੄ transformer blockਸ ੉ਊ • п transformer outputਸ ੷੢(ա઺ী ੉ਊ) • iߣ૩ context utterance੄ output • Reply utterance੄ output • ׮নೠ ױਤ(ױয, ҳ, ޙ੢)੄ ੄ઓҙ҅ܳ ౵ঈೞӝ ਤೣ • ױয ੄ઓҙ҅ח ࠺Ү੸ ծ਷ layer • ҳ, ޙ੢ ੄ઓҙ҅ח ࠺Ү੸ ֫਷ layer [U0 i , . . . UL i ] [R0, . . . RL]
31. !31 Model Architecture #3 Matching Cross Attention Match Self Attention Match
32. !32 Model Architecture #3 Aggregation
33. !33 Model Architecture #3 Aggregation
34. !34 Model Architecture #3 Aggregation & Loss Last Linear Layer Loss
35. !35 Model Architecture #3 Summary Input
36. !36 Model Architecture #3 Summary
37. #4. Result & Conclusion
38. !38 Result ف ؘ੉ఠࣇীࢲ ݽف SOTA!
39. !39 Result #4 Result
40. !40 Result #4 Result - ࠺Ү੸ Turnࣻо ੸ਸ ٸח ੿ഛبо ઑӘ ڄয૑૑݅ ੹୓੸ਵ۽ ੌ੿ೞ׮. - ޙ੢੄ ӡ੉о ӡࣻ۾(ನೣೠ ੿ࠁо ݆ਸࣻ۾) stacked layer੄ ബҗܳ ੜ ߉ח׮. - self-attention layerܳ ऺਸ ࣻ۾(~5) ੿ഛبо ֫ই઎׮. - 5ѐ ऺӝ۽ Ѿ੿ - ӡ੉о ૣ਷ utteranceী ؀ೠ ੿ഛبח ծ׮ - о૑Ҋ੓ח ੿ࠁо ੸ӝ ٸޙ
41. !41 Discussion അ੤ ਋ܻ ߑधҗ੄ ࠺Ү
42. !42 Discussion #4 Vs BERT DAM BERT
43. !43 Discussion #4 Vs BERT • п Utterance, Replyܳ زੌೠ stacked self- attentionী пп ాҗदఇ - п ޙ੢׼ representationਸ ঳ਸ ࣻ ੓਺ • п Layer Ѿҗܳ ݽف ੉ਊ • (U1,r), (U2,r), (U3,r)…җ੄ attentionਸ ஏ੿ റ ೠߣ ؊ Ѿҗܳ ೤஖ח җ੿(conv 3d)ܳ Ѣஜ - ৈ۞ utterance, replyী ಌઉ੓ח ੄ઓҙ҅ܳ ౵ঈೡ ࣻ ੓ ਸө? (֤ޙীࢲ ୊਺ ઁद೮؍ ޙઁ੼ਸ ೧Ѿೞ૑ ޅೞ ח ו՝..) • ҅࢑۝੉ ੸਺ + RMMҗ э਷ modelਸ ݅ٚ׮ݶ script replyী ؀ೠ representationਸ ޷ܻ ҅࢑ ೧ ֬ਸ ࣻ ੓਺ • ݽٚ Utterance, Replyܳ ೞա੄ inputਵ۽ ೤஖Ҋ BERT inputਵ۽ ࢎਊ - п ޙ੢׼ representationਸ ঳ਸ ࣻ হ਺ • ݃૑݄ Layer੄ Ѿҗ݅ ੉ਊ • ݽٚ Utterance, Replyр੄ ੄ઓҙ҅ܳ attentionਵ ۽ ૒੽੸ਵ۽ modelingೡ ࣻ ੓਺ - ೞ૑݅ ؘ੉ఠ੄ ন੉ ݆૑ ঋ׮ݶ ੗ਬبо ցޖ ֫ইࢲ ೟णೞӝ ൨ٜ ૑ ঋਸө? • ҅࢑۝੉ ݆਺ + script replyী ؀ೠ replyܳ ޷ܻ ݅ ٜ ࣻ হ਺(п ޙ੢׼ representationਸ ঳ਸ ࣻ হਵ޲ ۽) - ࢲ࠺झ द ޙઁо ࢤӡࣻب..? DAM BERT
44. !44 ו՛੼ ޖ঺ਸ וԕա?
45. • Transformerী ؀೧ ׮द ೠ ߣ Өѱ ࢤп೧ࠅ ࣻ ੓ח ӝഥ - self attention੄ ӝמী ؀೧ • Multi-turn replyী ؀ೠ ࢜۽਍ दبٜী ؀ೠ ו՝ • അ੤ प೷઺ੋ ߑߨ(BERT)ী ؀೧ غجইࠅ ࣻ ੓঻؍ ӝഥ !45 Discussion #4 ו՛੼
46. !46 Thank you хࢎ೤פ׮.