Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Multi-Turn Response Selection for Chatbots with...
Search
Scatter Lab Inc.
June 05, 2019
Research
2.3k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network
Scatter Lab Inc.
June 05, 2019
Transcript
46 slides
스캐터랩(ScatterLab) ੌ࢚ച ੋҕמ Technical Seminar: Multi-Turn Response Selection for Chatbots
with Deep Attention Matching Network 백영민 Dialogue System Machine Learning Engineer
#1. Introduction
!3 Conversational AI ࢎۈҗ Open-domain topicਵ۽ োझۣҊ ࣘੋ ചܳ ೡ
ࣻ ח AI
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !4 Introduction
#1 Human Conversation A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ৈ೯? ध? …
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !5 Introduction
#1 Human Conversation B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ৈ೯? ध? …
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !6 Introduction
#1 Human Conversation A: য়ט बী ಌܚীࢲ ೫ߡѢ ݡয!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ध!
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !7 Introduction
#1 Human Conversation A: য়ט बী ಌܚীࢲ ೫ߡѢ ݡয!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ध! A: Ѣӝ оݶ ԙ ؊࠶ߡѢ ࣁܳ ݡযঠ೧!
• ചഋ ੋఠಕझীࢲ ࢎۈ ইצ ஹೊఠ(ࠈ) ਬ৬ ࣗాೞח ࢲ࠺झ •
୭Ӕ োҳ زೱ • Open Domain topic ীࢲ ࢎۈҗ ࣘҊ োझۣѱ ചೡ ࣻ ח Chatbot • Data Driven Approach: ୷ػ ؘఠܳ ߄ఔਵ۽ ٜ݅য Chatbot • Retrieval-Based • Generation-Based !8 Introduction #1 Conversational AI - Chatbot
• ৈ۞ ؘఠٜਸ ਊೞৈ ݽ؛ਸ ण • Retrieval-Based Approach: •
ܻ ೧ റࠁ ߸ٜ о જ ߸ਸ ࢶఖೞ! • അ pingpong ࢎਊೞҊ ח ߑध • Single-Turn/Multi-Turn • Generation-Based Approach: • ച ؘఠٜ۽ ࠗఠ ಁఢਸ णೞৈ ܳ ߄ఔਵ۽ ࢜۽ ߸ਸ ࢤࢿೞ! !9 Introduction #1 Conversational AI - Data Driven Approach
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. •
ചীࢲ ৈ۞ ױਤ(ױয, ҳ ޙ)ٜী ઓೞৈ ߈ೠ. • п ઓҙ҅ܳ Ҋ۰ೡ ࣻ ח ݽ؛(chat bot)ਸ ٜ݅! • Single-Turn: ߊച݅ ࠁҊ ਸ ࢶఖ - context߂ ઓҙ҅ܳ ߈ೞӝ ൨ٜ • Multi-Turn: Nѐ ߊചܳ ࠁҊ ਸ ࢶఖ - context߂ ઓҙ҅ܳ ߈ !10 Introduction #1 Conversational AI - Human Conversation
#2. Recent Work
!12 Multi-turn Retrieval Nѐ ߊചܳ ా೧ റࠁীࢲ ೠ ਸ
ࢶఖ
• ৈ۞ ఢ ߊചٜਸ RNN(Recurrent Neural Network)ܳ ਊೞৈ encoding •
নೠ ઓҙ҅ܳ ঈೞӝী ೞ ঋ. - ҳઑ ࢚ ೠ҅ • ઓҙ҅(ݶਵ۽ ࠁח textual relevance - زੌೠ ױয, ਬࢎೠ ױয ١)݅ ঈ оמ • Ө ઓҙ҅(coreference, long-term dependency ١)ী ஂড !13 Recent work #2 RNN(Recurrent Neural Network)
!14 Transformer self-attention݅ਸ ਊೠ ࢜۽ architecture
• ౠ ࠗ࠙(ױয, ҳ, ޙ ١)ਸ “(attend)”ೞৈ Ѿҗܳ بೞח ߑध
(Query - Key) • ୡӝীח RNN ޙઁ(long-term dependency)ܳ ೧Ѿ೧ӝ ਤ೧ ࢎਊ • “Attention is All you need” - Attention ݅ਵ۽ب(Self-attention) જ Ѿҗܳ յ ࣻ ਸ ࠁ • BERT/GPT ١ ୭Ӕ SOTA ݽ؛ٜ ࠗ࠙ ࢎਊ !15 Recent work #2 Attention
!16 Recent work #2 Attention A о E, F, G,
H ܳ attend ೞৈ ࢜۽ A’ ࢤࢿ Query Key
!17 Recent work #2 Attention B о E, F, G,
H ܳ attend ೞৈ ࢜۽ B’ ࢤࢿ Query Key
!18 Recent work #2 Attention C о E, F, G,
H ܳ attend ೞৈ ࢜۽ C’ ࢤࢿ Query Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !19 Recent work #2 Transformer - Self Attention
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !20 Recent work #2 Transformer - Self Attention A о A, B, C, D ܳ attend ೞৈ ࢜۽ A’ ࢤࢿ Query & Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !21 Recent work #2 Transformer - Self Attention B о A, B, C, D ܳ attend ೞৈ ࢜۽ B’ ࢤࢿ Query & Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !22 Recent work #2 Transformer - Self Attention C о A, B, C, D ܳ attend ೞৈ ࢜۽ C’ ࢤࢿ Query & Key
!23 Recent work #2 Transformer - Self Attention ޙী ೠ
Ө ೧
!24 Recent work #2 Transformer - Self Attention Layerо ऺৈтࣻ۾
ࠂੋ /ҙ҅ܳ ݽ؛݂
#3. Method
!26 Deep Attention Matching Network Transformerܳ ਊೠ Multi-turn retrieval model
• ಽҊೞח ޙઁ: Multi-trun retrieval • Data: Multi-turn ച ؘఠࣇ
(c, r, y) • c: n-1ѐ context ߊച • r: response റࠁ • y: label (0, 1) • Ubuntu Corpus V1, Douban Conversational Corpus • g(c, r) -> yܳ ࣻ೯ೞח g(model)ਸ णೞ! !27 Model Architecture #3 Problem
!28 Model Architecture #3 Overview
!29 Model Architecture #3 Input • Input utterance • Context
utterance: • Reply utterance: • Embedding: • п ױযٜী ೧ d(=200)ରਗ embedding • Pre-trained word2vec ਊ ui = [wui ,k ]nui −1 k=0 , nui : maxword(context) r = [wui ,k ]nr −1 t=0 , nr : maxword(reply)
!30 Model Architecture #3 Representation • Stacked Self-Attention • L
ѐ transformer blockਸ ਊ • п transformer outputਸ (աী ਊ) • iߣ૩ context utterance output • Reply utterance output • নೠ ױਤ(ױয, ҳ, ޙ) ઓҙ҅ܳ ঈೞӝ ਤೣ • ױয ઓҙ҅ח ࠺Ү ծ layer • ҳ, ޙ ઓҙ҅ח ࠺Ү ֫ layer [U0 i , . . . UL i ] [R0, . . . RL]
!31 Model Architecture #3 Matching Cross Attention Match Self Attention
Match
!32 Model Architecture #3 Aggregation
!33 Model Architecture #3 Aggregation
!34 Model Architecture #3 Aggregation & Loss Last Linear Layer
Loss
!35 Model Architecture #3 Summary Input
!36 Model Architecture #3 Summary
#4. Result & Conclusion
!38 Result ف ؘఠࣇীࢲ ݽف SOTA!
!39 Result #4 Result
!40 Result #4 Result - ࠺Ү Turnࣻо ਸ ٸח ഛبо
ઑӘ ڄয݅ ਵ۽ ੌೞ. - ޙ ӡо ӡࣻ۾(ನೣೠ ࠁо ݆ਸࣻ۾) stacked layer ബҗܳ ੜ ߉ח. - self-attention layerܳ ऺਸ ࣻ۾(~5) ഛبо ֫ই. - 5ѐ ऺӝ۽ Ѿ - ӡо ૣ utteranceী ೠ ഛبח ծ - оҊח ࠁо ӝ ٸޙ
!41 Discussion അ ܻ ߑधҗ ࠺Ү
!42 Discussion #4 Vs BERT DAM BERT
!43 Discussion #4 Vs BERT • п Utterance, Replyܳ زੌೠ
stacked self- attentionী пп ాҗदఇ - п ޙ representationਸ ਸ ࣻ • п Layer Ѿҗܳ ݽف ਊ • (U1,r), (U2,r), (U3,r)…җ attentionਸ ஏ റ ೠߣ ؊ Ѿҗܳ ח җ(conv 3d)ܳ Ѣஜ - ৈ۞ utterance, replyী ಌઉח ઓҙ҅ܳ ঈೡ ࣻ ਸө? (֤ޙীࢲ ઁद೮؍ ޙઁਸ ೧Ѿೞ ޅೞ ח ו՝..) • ҅ + RMMҗ э modelਸ ݅ٚݶ script replyী ೠ representationਸ ܻ ҅ ೧ ֬ਸ ࣻ • ݽٚ Utterance, Replyܳ ೞա inputਵ۽ Ҋ BERT inputਵ۽ ࢎਊ - п ޙ representationਸ ਸ ࣻ হ • ݄݃ Layer Ѿҗ݅ ਊ • ݽٚ Utterance, Replyр ઓҙ҅ܳ attentionਵ ۽ ਵ۽ modelingೡ ࣻ - ೞ݅ ؘఠ ন ݆ ঋݶ ਬبо ցޖ ֫ইࢲ णೞӝ ൨ٜ ঋਸө? • ҅ ݆ + script replyী ೠ replyܳ ܻ ݅ ٜ ࣻ হ(п ޙ representationਸ ਸ ࣻ হਵ ۽) - ࢲ࠺झ द ޙઁо ࢤӡࣻب..? DAM BERT
!44 ו՛ ޖਸ וԕա?
• Transformerী ೧ द ೠ ߣ Өѱ ࢤп೧ࠅ ࣻ ח
ӝഥ - self attention ӝמী ೧ • Multi-turn replyী ೠ ࢜۽ दبٜী ೠ ו՝ • അ पੋ ߑߨ(BERT)ী ೧ غجইࠅ ࣻ ؍ ӝഥ !45 Discussion #4 ו՛
!46 Thank you хࢎפ.
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
超効率化への挑戦:1bit LLMの現状と展望
yumaichikawa
0
1k
Empreinte carbone des ovins : une approche globale (technique, économique, environnementale) – 4 années de résultats avec LIFE GREEN SHEEP
institutdelelevage
PRO
0
150
VLMの推論を高速化する視覚トークン削減の仕組み
tattaka
2
440
Kunai: Toward a Verifier-Safe Layered DSL for Nested Encapsulation Packet Filtering on eBPF
takehaya
1
210
ハードウェア研究で国際トップ会議を目指す!IROS 2027での論文採択を目指して
ayatokanada
7
4.2k
JPA2026_NetworkTutorial_JunKashihara
junkashihara
0
280
【令和7年度版】内航船 VS 外航船 早見表
anikisencyo
0
220
Hitting the buffers
beale
0
410
横浜市長(山中氏)の言動にかかる第三者による調査報告書
sishi2026
0
680
宛先から考えるデジタルアーカイブ:――ジャック・デリダの〈痕跡〉と三輪田米山日記
ktanishima
0
100
XDPerf: A High-Performance Traffic Generator Built with WASM and eBPF
takehaya
1
400
Kaggle|AI Agent Security — しくじり先生、俺みたいになるな
pomcho555
1
140
Featured
See All Featured
Jess Joyce - The Pitfalls of Following Frameworks
techseoconnect
PRO
1
480
Paper Plane
katiecoart
PRO
6
55k
Building Applications with DynamoDB
mza
96
7.3k
Collaborative Software Design: How to facilitate domain modelling decisions
baasie
1
380
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
akashhashmi
0
550
Context Engineering - Making Every Token Count
addyosmani
9
1.3k
How Fast Is Fast Enough? [PerfNow 2025]
tammyeverts
3
980
YesSQL, Process and Tooling at Scale
rocio
174
15k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
greggifford
PRO
0
400
A better future with KSS
kneath
240
18k
Visual Storytelling: How to be a Superhuman Communicator
reverentgeek
2
750
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
31
3.4k