Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Multi-Turn Response Selection for Chatbots with...
Search
Scatter Lab Inc.
June 05, 2019
Research
2.3k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network
Scatter Lab Inc.
June 05, 2019
Transcript
46 slides
스캐터랩(ScatterLab) ੌ࢚ച ੋҕמ Technical Seminar: Multi-Turn Response Selection for Chatbots
with Deep Attention Matching Network 백영민 Dialogue System Machine Learning Engineer
#1. Introduction
!3 Conversational AI ࢎۈҗ Open-domain topicਵ۽ োझۣҊ ࣘੋ ചܳ ೡ
ࣻ ח AI
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !4 Introduction
#1 Human Conversation A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ৈ೯? ध? …
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !5 Introduction
#1 Human Conversation B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ৈ೯? ध? …
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !6 Introduction
#1 Human Conversation A: য়ט बী ಌܚীࢲ ೫ߡѢ ݡয!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ध!
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. !7 Introduction
#1 Human Conversation A: য়ט बী ಌܚীࢲ ೫ߡѢ ݡয!! B: য় Ѣӝ যٿয?? A: ցޖ ৮߷೮য ৮ ୶ୌ! B: .. ƀƀ աب оҊर ध! A: Ѣӝ оݶ ԙ ؊࠶ߡѢ ࣁܳ ݡযঠ೧!
• ചഋ ੋఠಕझীࢲ ࢎۈ ইצ ஹೊఠ(ࠈ) ਬ৬ ࣗాೞח ࢲ࠺झ •
୭Ӕ োҳ زೱ • Open Domain topic ীࢲ ࢎۈҗ ࣘҊ োझۣѱ ചೡ ࣻ ח Chatbot • Data Driven Approach: ୷ػ ؘఠܳ ߄ఔਵ۽ ٜ݅য Chatbot • Retrieval-Based • Generation-Based !8 Introduction #1 Conversational AI - Chatbot
• ৈ۞ ؘఠٜਸ ਊೞৈ ݽ؛ਸ ण • Retrieval-Based Approach: •
ܻ ೧ റࠁ ߸ٜ о જ ߸ਸ ࢶఖೞ! • അ pingpong ࢎਊೞҊ ח ߑध • Single-Turn/Multi-Turn • Generation-Based Approach: • ച ؘఠٜ۽ ࠗఠ ಁఢਸ णೞৈ ܳ ߄ఔਵ۽ ࢜۽ ߸ਸ ࢤࢿೞ! !9 Introduction #1 Conversational AI - Data Driven Approach
• ࢎۈ ചೡ ٸ, ৈ۞ ઓҙ҅(, ӝמ)ܳ Ҋ۰ೠ. •
ചীࢲ ৈ۞ ױਤ(ױয, ҳ ޙ)ٜী ઓೞৈ ߈ೠ. • п ઓҙ҅ܳ Ҋ۰ೡ ࣻ ח ݽ؛(chat bot)ਸ ٜ݅! • Single-Turn: ߊച݅ ࠁҊ ਸ ࢶఖ - context߂ ઓҙ҅ܳ ߈ೞӝ ൨ٜ • Multi-Turn: Nѐ ߊചܳ ࠁҊ ਸ ࢶఖ - context߂ ઓҙ҅ܳ ߈ !10 Introduction #1 Conversational AI - Human Conversation
#2. Recent Work
!12 Multi-turn Retrieval Nѐ ߊചܳ ా೧ റࠁীࢲ ೠ ਸ
ࢶఖ
• ৈ۞ ఢ ߊചٜਸ RNN(Recurrent Neural Network)ܳ ਊೞৈ encoding •
নೠ ઓҙ҅ܳ ঈೞӝী ೞ ঋ. - ҳઑ ࢚ ೠ҅ • ઓҙ҅(ݶਵ۽ ࠁח textual relevance - زੌೠ ױয, ਬࢎೠ ױয ١)݅ ঈ оמ • Ө ઓҙ҅(coreference, long-term dependency ١)ী ஂড !13 Recent work #2 RNN(Recurrent Neural Network)
!14 Transformer self-attention݅ਸ ਊೠ ࢜۽ architecture
• ౠ ࠗ࠙(ױয, ҳ, ޙ ١)ਸ “(attend)”ೞৈ Ѿҗܳ بೞח ߑध
(Query - Key) • ୡӝীח RNN ޙઁ(long-term dependency)ܳ ೧Ѿ೧ӝ ਤ೧ ࢎਊ • “Attention is All you need” - Attention ݅ਵ۽ب(Self-attention) જ Ѿҗܳ յ ࣻ ਸ ࠁ • BERT/GPT ١ ୭Ӕ SOTA ݽ؛ٜ ࠗ࠙ ࢎਊ !15 Recent work #2 Attention
!16 Recent work #2 Attention A о E, F, G,
H ܳ attend ೞৈ ࢜۽ A’ ࢤࢿ Query Key
!17 Recent work #2 Attention B о E, F, G,
H ܳ attend ೞৈ ࢜۽ B’ ࢤࢿ Query Key
!18 Recent work #2 Attention C о E, F, G,
H ܳ attend ೞৈ ࢜۽ C’ ࢤࢿ Query Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !19 Recent work #2 Transformer - Self Attention
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !20 Recent work #2 Transformer - Self Attention A о A, B, C, D ܳ attend ೞৈ ࢜۽ A’ ࢤࢿ Query & Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !21 Recent work #2 Transformer - Self Attention B о A, B, C, D ܳ attend ೞৈ ࢜۽ B’ ࢤࢿ Query & Key
• Self-Attention: • Query, Key, Valueܳ ݽف زੌೞѱ ೣ ->
ӝ नী ೠ attention !22 Recent work #2 Transformer - Self Attention C о A, B, C, D ܳ attend ೞৈ ࢜۽ C’ ࢤࢿ Query & Key
!23 Recent work #2 Transformer - Self Attention ޙী ೠ
Ө ೧
!24 Recent work #2 Transformer - Self Attention Layerо ऺৈтࣻ۾
ࠂੋ /ҙ҅ܳ ݽ؛݂
#3. Method
!26 Deep Attention Matching Network Transformerܳ ਊೠ Multi-turn retrieval model
• ಽҊೞח ޙઁ: Multi-trun retrieval • Data: Multi-turn ച ؘఠࣇ
(c, r, y) • c: n-1ѐ context ߊച • r: response റࠁ • y: label (0, 1) • Ubuntu Corpus V1, Douban Conversational Corpus • g(c, r) -> yܳ ࣻ೯ೞח g(model)ਸ णೞ! !27 Model Architecture #3 Problem
!28 Model Architecture #3 Overview
!29 Model Architecture #3 Input • Input utterance • Context
utterance: • Reply utterance: • Embedding: • п ױযٜী ೧ d(=200)ରਗ embedding • Pre-trained word2vec ਊ ui = [wui ,k ]nui −1 k=0 , nui : maxword(context) r = [wui ,k ]nr −1 t=0 , nr : maxword(reply)
!30 Model Architecture #3 Representation • Stacked Self-Attention • L
ѐ transformer blockਸ ਊ • п transformer outputਸ (աী ਊ) • iߣ૩ context utterance output • Reply utterance output • নೠ ױਤ(ױয, ҳ, ޙ) ઓҙ҅ܳ ঈೞӝ ਤೣ • ױয ઓҙ҅ח ࠺Ү ծ layer • ҳ, ޙ ઓҙ҅ח ࠺Ү ֫ layer [U0 i , . . . UL i ] [R0, . . . RL]
!31 Model Architecture #3 Matching Cross Attention Match Self Attention
Match
!32 Model Architecture #3 Aggregation
!33 Model Architecture #3 Aggregation
!34 Model Architecture #3 Aggregation & Loss Last Linear Layer
Loss
!35 Model Architecture #3 Summary Input
!36 Model Architecture #3 Summary
#4. Result & Conclusion
!38 Result ف ؘఠࣇীࢲ ݽف SOTA!
!39 Result #4 Result
!40 Result #4 Result - ࠺Ү Turnࣻо ਸ ٸח ഛبо
ઑӘ ڄয݅ ਵ۽ ੌೞ. - ޙ ӡо ӡࣻ۾(ನೣೠ ࠁо ݆ਸࣻ۾) stacked layer ബҗܳ ੜ ߉ח. - self-attention layerܳ ऺਸ ࣻ۾(~5) ഛبо ֫ই. - 5ѐ ऺӝ۽ Ѿ - ӡо ૣ utteranceী ೠ ഛبח ծ - оҊח ࠁо ӝ ٸޙ
!41 Discussion അ ܻ ߑधҗ ࠺Ү
!42 Discussion #4 Vs BERT DAM BERT
!43 Discussion #4 Vs BERT • п Utterance, Replyܳ زੌೠ
stacked self- attentionী пп ాҗदఇ - п ޙ representationਸ ਸ ࣻ • п Layer Ѿҗܳ ݽف ਊ • (U1,r), (U2,r), (U3,r)…җ attentionਸ ஏ റ ೠߣ ؊ Ѿҗܳ ח җ(conv 3d)ܳ Ѣஜ - ৈ۞ utterance, replyী ಌઉח ઓҙ҅ܳ ঈೡ ࣻ ਸө? (֤ޙীࢲ ઁद೮؍ ޙઁਸ ೧Ѿೞ ޅೞ ח ו՝..) • ҅ + RMMҗ э modelਸ ݅ٚݶ script replyী ೠ representationਸ ܻ ҅ ೧ ֬ਸ ࣻ • ݽٚ Utterance, Replyܳ ೞա inputਵ۽ Ҋ BERT inputਵ۽ ࢎਊ - п ޙ representationਸ ਸ ࣻ হ • ݄݃ Layer Ѿҗ݅ ਊ • ݽٚ Utterance, Replyр ઓҙ҅ܳ attentionਵ ۽ ਵ۽ modelingೡ ࣻ - ೞ݅ ؘఠ ন ݆ ঋݶ ਬبо ցޖ ֫ইࢲ णೞӝ ൨ٜ ঋਸө? • ҅ ݆ + script replyী ೠ replyܳ ܻ ݅ ٜ ࣻ হ(п ޙ representationਸ ਸ ࣻ হਵ ۽) - ࢲ࠺झ द ޙઁо ࢤӡࣻب..? DAM BERT
!44 ו՛ ޖਸ וԕա?
• Transformerী ೧ द ೠ ߣ Өѱ ࢤп೧ࠅ ࣻ ח
ӝഥ - self attention ӝמী ೧ • Multi-turn replyী ೠ ࢜۽ दبٜী ೠ ו՝ • അ पੋ ߑߨ(BERT)ী ೧ غجইࠅ ࣻ ؍ ӝഥ !45 Discussion #4 ו՛
!46 Thank you хࢎפ.
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
HackSick vol.7 LT資料【LLMアーキテクチャ入門・事前学習時の躓き所解説】 スパースなAttention・状態空間モデル
rikkabotan7
0
240
最先端NLP勉強会2026 論文紹介:Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 paper)
kogoro
4
970
AI創薬の壁
kanojikajino
1
1.3k
MIRU2026 チュートリアル講演2:三次元データ処理の動向
nnchiba
8
5.6k
Adapter les bergeries aux fortes chaleurs
institutdelelevage
PRO
0
180
ICAITD 2026 Oral Presentation
rikkabotan7
0
150
[最先端NLP勉強会2026] Checklists Are Better Than Reward Models For Aligning Language Models
nzw0301
1
350
SLAMはどこまで解決されたのか?
tomonom
0
1.7k
横浜市長(山中氏)の言動にかかる第三者による調査報告書
sishi2026
0
680
Model Predictive Control for Uncertain Systems: Robust & Data-Driven Designs
kohlerjohannes
0
220
20261101GISA-FOSS4G
tosseto
1
160
HAKARI-Bench - 実運用視点での情報検索モデル評価ベンチマーク
hotchpotch
1
870
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
23k
Darren the Foodie - Storyboard
khoart
PRO
5
4.1k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
akashhashmi
0
550
Leveraging Curiosity to Care for An Aging Population
cassininazir
1
570
Marketing to machines
jonoalderson
1
6k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
Code Reviewing Like a Champion
maltzj
529
40k
Neural Spatial Audio Processing for Sound Field Analysis and Control
skoyamalab
0
620
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
12
1.4k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
68
60k
Rails Girls Zürich Keynote
gr2m
96
14k
GraphQLとの向き合い方2022年版
quramy
50
15k