Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Real-Time Open-Domain Question Answering with D...
Search
Scatter Lab Inc.
July 17, 2019
Research
940
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index
Scatter Lab Inc.
July 17, 2019
Transcript
18 slides
Real-Time Open-Domain Question Answering With Dense-Sparse Phrase Index 스캐터랩(ScatterLab) Machine
Learning Engineer 김준성 ੌ࢚ച ੋҕמ
Motivation Query When is the Christmas? Christmas Jejus Christmas Tree
Christmas Eve … Document Retrieval Reading Comprehension ࣻ ࢤਸ ୷ೞೞח ೯ࢎо दػ Ѫ 4ࣁӝ ޖ۵ࠗఠ. दীח ܻझ݃झ զ о ഛೞѱ ೧ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ оؘ ೞܖо ࢶغ . 12ਘ 25ੌ ܻझ݃झ۽ ܻ Ѫ AD 354֙, റ. 12ਘ 25ੌ
Motivation Query When is the Christmas? Christmas Jejus Christmas Tree
Christmas Eve … Document Retrieval Reading Comprehension ࣻ ࢤਸ ୷ೞೞח ೯ࢎо दػ Ѫ 4ࣁӝ ޖ۵ࠗఠ. दীח ܻझ݃झ զ о ഛೞѱ ೧ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ оؘ ೞܖо ࢶغ . 12ਘ 25ੌ ܻझ݃झ۽ ܻ Ѫ AD 354֙, റ. 12ਘ 25ੌ TF-IDF Searching SQuAD
Motivation Query When is the Christmas? Christmas Jejus Christmas Tree
Christmas Eve … Document Retrieval Reading Comprehension ࣻ ࢤਸ ୷ೞೞח ೯ࢎо दػ Ѫ 4ࣁӝ ޖ۵ࠗఠ. दীח ܻझ݃झ զ о ഛೞѱ ೧ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ оؘ ೞܖо ࢶغ . 12ਘ 25ੌ ܻझ݃झ۽ ܻ Ѫ AD 354֙, റ. 12ਘ 25ੌ TF-IDF Searching SQuAD Error Propagation Ӓܻझ݃झ ࣛ۽ … X
Motivation Query When is the Christmas? Christmas Jejus Christmas Tree
Christmas Eve … Document Retrieval Reading Comprehension ࣻ ࢤਸ ୷ೞೞח ೯ࢎо दػ Ѫ 4ࣁӝ ޖ۵ࠗఠ. दীח ܻझ݃झ զ о ഛೞѱ ೧ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ оؘ ೞܖо ࢶغ . 12ਘ 25ੌ ܻझ݃झ۽ ܻ Ѫ AD 354֙, റ. 12ਘ 25ੌ TF-IDF Searching SQuAD Super Slow Document Size ࣻরѤ WIKi Model Size BERT-Large 10-100 sec/query ࢲ࠺झೡ ࣻ হ
Motivation Super Slow SQuAD-BERT ܻझ݃झח ઁঠ | ࣻ ࢤਸ ୷ೞೞח..
ܻझ݃झח ઁঠ | ӝةҮ ੋٜীѱ য о… ܻझ݃झח ઁঠ | োੋٜ о ೠ…. Ҋ ೞח Articleਸ Caching ೡ ࣻ হӝ ٸޙী Query ী ٮۄࢲ ݒߣ ҅೧ঠೣ 12ਘ 25ੌ
Models Query Embedding [CLS] What day is it the Christmas?
Query Encoding
Models Christmas is an annual festival commemorating the birth of
Jesus Christ, observed on December 25.. Phrase Encoding
Models Christmas is an annual festival commemorating the birth of
Jesus Christ, observed on December 25 Start Span Embed End Span Embed December 25 Phrase Encoding
Models Christmas is an annual festival commemorating the birth of
Jesus Christ, observed on December 25 Start Span Embed End Span Embed annual festival commemorating the birth of Jesus Christ Phrase Encoding
Sparse Encoding Models ੌ߈ੋ Document Retrieval + Reading Comprehension झఋੌ
ইצ ߄۽ Query -> Span ՙܻ Distribution Matching ೣ (RMM э?)
Sparse Encoding Models START EMBED / END EMBED / COHERENCE
EMBED = DENSE EMBEDDING
Sparse Encoding Models ৵ Coherenceо ৵ ٜযо??
Models Dense Encoding Token Embed dimਸ 4١࠙ ೧ࢲ 1,2,3,4 ۽
ա׃ Start_embed1, End_embed2, dot[Start_embed3, End_embed4] Concat
Sparse Encoding 2-Gram TF-IDFܳ ਊ೧ࢲ Sparse Embeddingਸ ݅ٞ + Paragraph-Level
Sparse Embedding ب ؊ೣ Models
ੌ߈ਵ۽ח ݽٚ Document ৬ ࠺Ү೧ࢲ argmaxܳ ೞח ߑߨ ࢎਊ Training
Training Efficiency
Training Training Efficiency ೞ݅ ܻח ݏח ௪ܻী ೧ࢲ Positive ೞѱ
णೞݶ ؽ (RMMীࢲ Positive Sample) ઑӘ؊ ࣻਵ۽ ܻೞݶ ؊ ബਯਵ۽ Loss ܳ ҅ೡ ࣻ
Result SQuAD 1.1 BERTࠁח ࢿמ 8% ب ծѪ ࢎप ೞ݅
प ࢲ࠺झ۽ ٜ݅য ݽ؛ীࢲח ӝઓ SOTA (DrQA) ࠺ ࢿמ 4% ֫Ҋ ࣘبח DrQA 58ߓ ࡅܰҊ BERTࠁ 6000ߓ ب ࡅܴ
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
マニアックセッション 〜次世代のメールプロトコルの斜め読み(2026年版)
hirachan
0
520
20261101GISA-FOSS4G
tosseto
1
160
Santé et fertilité des sols : ce que nous disent les pratiques dans les systèmes de polyculture-élevage
institutdelelevage
PRO
0
240
La longévité des troupeaux laitiers: état des lieux et perspectives
institutdelelevage
PRO
0
140
Physical AIでモデリングはどう変わるか/How Physical AI Will Change Modeling
stktu
0
350
Développer des solutions de réduction des émissions de méthane entérique : 1ers résultats Méthane 2030
institutdelelevage
PRO
0
290
Evaluation génomique des femelles laitières croisées : comprendre la méthode pour accompagner les éleveurs à son utilisation
institutdelelevage
PRO
0
230
Who’s Behind the Divide?
sirmatrixpage
1
1.6k
ふとした出会いで生まれたSkillが、 社内利用1位になるまで
mikimhk
22
25k
論文読み会 SNLP2026 Tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment
s_mizuki_nlp
0
350
モデル評価を効率的にする方法
ftakahashi
0
250
進化?迷走?CasualConc ファミリーアプリの現在地 @ 英語コーパス学会 2026
casualconc
0
160
Featured
See All Featured
The Cult of Friendly URLs
andyhume
79
7.1k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
360
31k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.9k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
33
5.8k
Discover your Explorer Soul
emna__ayadi
2
1.3k
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
360
A designer walks into a library…
pauljervisheath
211
25k
Principles of Awesome APIs and How to Build Them.
keavy
128
18k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
128
56k
AI: The stuff that nobody shows you
jnunemaker
PRO
12
1.3k
Scaling GitHub
holman
464
140k
The #1 spot is gone: here's how to win anyway
tamaranovitovic
4
1.2k