Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-Time Open-Domain Question Answering with D...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Scatter Lab Inc. Scatter Lab Inc.
July 17, 2019
Research
930
0

Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

July 17, 2019

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.5k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
Harness Engineering and Al Agent
Avatar for Kazuki Inamura kzinmr
3
1.9k
HAKARI-Bench - 実運用視点での情報検索モデル評価ベンチマーク
Avatar for Yuichi Tateno hotchpotch
1
730
LA-Bench 2025：実験指示から 実行可能手順を生成するためのデータセット/LA-Bench 2025: A Dataset for Generating Executable Experimental Procedures from Experimental Instructions
Avatar for Shota Kato stktu
0
150
Using our influence and power for patient safety
Avatar for Helen Bevan helenbevan
0
410
XDPerf: A High-Performance Traffic Generator Built with WASM and eBPF
Avatar for Takeru Hayasaka takehaya
1
290
多様なデータを許容し学習し続ける模倣学習 / Advanced Imitation Learning for VLA
Avatar for PRIN Lab prinlab
0
290
最先端NLP勉強会2026 論文紹介：Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 paper)
Avatar for Kogoro kogoro
4
590
VLMの推論を高速化する視覚トークン削減の仕組み
Avatar for tattaka tattaka
2
300
Claude Code × autoresearch 実践
Avatar for 数理の弾丸 mathbullet
0
270
LLM の Attention 機構まとめ — 数式・計算量・メモリ
Avatar for プワー puwaer
8
2.6k
Cross-Media Information Spaces and Architectures
Avatar for Beat Signer signer
PRO
0
360
OWASP AISVS - C7
Avatar for shieLL shiell
1
640

Featured

See All Featured
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.5k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
270
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
230
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.4k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
600
My Coaching Mixtape
Avatar for mlcsv mlcsv
0
310
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
10
1.3k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
331
25k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.8k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.4k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
52k

Transcript

  1. Real-Time Open-Domain Question Answering With Dense-Sparse Phrase Index 스캐터랩(ScatterLab) Machine

    Learning Engineer 김준성 ੌ࢚؀ച ੋҕ૑מ

  2. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ

  3. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD

  4. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Error Propagation Ӓܻझ݃झ ࣛ۽؀୏ … X

  5. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Super Slow Document Size ࣻরѤ੄ WIKi Model Size BERT-Large 10-100 sec/query ࢲ࠺झೡ ࣻ হ਺

  6. Motivation Super Slow SQuAD-BERT ௼ܻझ݃झח ঱ઁঠ | ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח..

    ௼ܻझ݃झח ঱ઁঠ | ӝةҮ ੋٜীѱ ੓য о੢… ௼ܻझ݃झח ঱ઁঠ | োੋٜ੄ о੢ ೥ೠ…. ଺Ҋ੗ ೞח Articleਸ Caching ೡ ࣻ হӝ ٸޙী Query ী ٮۄࢲ ݒߣ ҅࢑೧ঠೣ 12ਘ 25ੌ

  7. Models Query Embedding [CLS] What day is it the Christmas?

    Query Encoding

  8. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25.. Phrase Encoding

  9. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed December 25 Phrase Encoding

  10. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed annual festival commemorating the birth of Jesus Christ Phrase Encoding

  11. Sparse Encoding Models ੌ߈੸ੋ Document Retrieval + Reading Comprehension झఋੌ੉

    ইצ ߄۽ Query -> Span ՙܻ Distribution Matching ೣ (RMM э਷?)

  12. Sparse Encoding Models START EMBED / END EMBED / COHERENCE

    EMBED = DENSE EMBEDDING

  13. Sparse Encoding Models ৵ Coherenceо ৵ ٜযо૑??

  14. Models Dense Encoding Token Embed dimਸ 4١࠙ ೧ࢲ 1,2,3,4 ۽

    ա׃ Start_embed1, End_embed2, dot[Start_embed3, End_embed4] Concat

  15. Sparse Encoding 2-Gram TF-IDFܳ ੉ਊ೧ࢲ Sparse Embeddingਸ ݅ٞ + Paragraph-Level

    Sparse Embedding ب ؊ೣ Models

  16. ੌ߈੸ਵ۽ח ݽٚ Document ৬ ࠺Ү೧ࢲ argmaxܳ ೞח ߑߨ ࢎਊ Training

    Training Efficiency

  17. Training Training Efficiency ೞ૑݅ ਋ܻח ݏח ௪ܻী ؀೧ࢲ Positive ೞѱ

    ೟णೞݶ ؽ (RMMীࢲ Positive Sample) ઑӘ؊ ࣻ೟੸ਵ۽ ੿ܻೞݶ ؊ ബਯ੸ਵ۽ Loss ܳ ҅࢑ೡ ࣻ ੓਺

  18. Result SQuAD 1.1 BERTࠁ׮ח ࢿמ੉ 8% ੿ب ծ਷Ѫ੉ ࢎप ೞ૑݅

    प ࢲ࠺झ۽ ٜ݅য૓ ݽ؛ীࢲח
 ӝઓ SOTA (DrQA) ؀࠺ ࢿמ਷ 4% ֫Ҋ
 ࣘبח DrQA 58ߓ ࡅܰҊ BERTࠁ׮ 6000ߓ ੿ب ࡅܴ