Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-Time Open-Domain Question Answering with D...

Avatar for Scatter Lab Inc. Scatter Lab Inc.
July 17, 2019
Research
910
0

Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

July 17, 2019

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
1.9k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.4k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
羽田新ルート運用6年の検証
Avatar for マン点 1manken
0
170
nlp2026 In-Context Learningに基づく経路案内のための地理的知識の活用方法に関する検討
Avatar for Takashi INUI takashiinui
0
100
SAKURAONE: An Open Ethernet-based AI HPC System And Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
1
450
進学校の生徒にはア行の苗字が多いのか
Avatar for Takashi Ozeki ozekinote
0
470
[IR Reading 2026春 論文紹介] LLM-based Listwise Reranking under the Effect of Positional Bias (ECIR 2026) /IR-Reading-2026-Spring
Avatar for Kohei Shinden koheishinden
PRO
0
220
「行ける・行けない表」による地域公共交通の性能評価
Avatar for 萬創社 bansousha
0
170
AIエージェント時代のLLM-jpモデルのあるべき姿
Avatar for Kouta Nakayama k141303
0
510
東京大学工学部計数工学科、計数工学特別講義の説明資料
Avatar for KIKUCHI Shunsuke kikuzo
0
550
オーストリア流 都市の公共交通サービス水準評価@公共交通オープンデータ最前線2026
Avatar for Traffic Brain trafficbrain
0
200
typst の使い方：言語学を研究する学生のために
Avatar for Tomoya gitomochang
0
510
Φ-Sat-2のAutoEncoderによる情報圧縮系論文
Avatar for SatAI.challenge satai
4
850
研究室単位での自律的 IPv6接続性確立に向けたAS共同運用モデルの提案と実証
Avatar for Hiroki (REO) Kashiwazaki reokashiwa
PRO
0
150

Featured

See All Featured
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
370
A Soul's Torment
Avatar for Nat Ma seathinner
6
3.1k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
27k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
460
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.4k
Fireside Chat
Avatar for paigeccino paigeccino
42
4k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
650
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.7k
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
Avatar for Kenny Baas-Schwegler baasie
0
610
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
10k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
340

Transcript

  1. Real-Time Open-Domain Question Answering With Dense-Sparse Phrase Index 스캐터랩(ScatterLab) Machine

    Learning Engineer 김준성 ੌ࢚؀ച ੋҕ૑מ

  2. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ

  3. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD

  4. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Error Propagation Ӓܻझ݃झ ࣛ۽؀୏ … X

  5. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Super Slow Document Size ࣻরѤ੄ WIKi Model Size BERT-Large 10-100 sec/query ࢲ࠺झೡ ࣻ হ਺

  6. Motivation Super Slow SQuAD-BERT ௼ܻझ݃झח ঱ઁঠ | ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח..

    ௼ܻझ݃झח ঱ઁঠ | ӝةҮ ੋٜীѱ ੓য о੢… ௼ܻझ݃झח ঱ઁঠ | োੋٜ੄ о੢ ೥ೠ…. ଺Ҋ੗ ೞח Articleਸ Caching ೡ ࣻ হӝ ٸޙী Query ী ٮۄࢲ ݒߣ ҅࢑೧ঠೣ 12ਘ 25ੌ

  7. Models Query Embedding [CLS] What day is it the Christmas?

    Query Encoding

  8. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25.. Phrase Encoding

  9. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed December 25 Phrase Encoding

  10. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed annual festival commemorating the birth of Jesus Christ Phrase Encoding

  11. Sparse Encoding Models ੌ߈੸ੋ Document Retrieval + Reading Comprehension झఋੌ੉

    ইצ ߄۽ Query -> Span ՙܻ Distribution Matching ೣ (RMM э਷?)

  12. Sparse Encoding Models START EMBED / END EMBED / COHERENCE

    EMBED = DENSE EMBEDDING

  13. Sparse Encoding Models ৵ Coherenceо ৵ ٜযо૑??

  14. Models Dense Encoding Token Embed dimਸ 4١࠙ ೧ࢲ 1,2,3,4 ۽

    ա׃ Start_embed1, End_embed2, dot[Start_embed3, End_embed4] Concat

  15. Sparse Encoding 2-Gram TF-IDFܳ ੉ਊ೧ࢲ Sparse Embeddingਸ ݅ٞ + Paragraph-Level

    Sparse Embedding ب ؊ೣ Models

  16. ੌ߈੸ਵ۽ח ݽٚ Document ৬ ࠺Ү೧ࢲ argmaxܳ ೞח ߑߨ ࢎਊ Training

    Training Efficiency

  17. Training Training Efficiency ೞ૑݅ ਋ܻח ݏח ௪ܻী ؀೧ࢲ Positive ೞѱ

    ೟णೞݶ ؽ (RMMীࢲ Positive Sample) ઑӘ؊ ࣻ೟੸ਵ۽ ੿ܻೞݶ ؊ ബਯ੸ਵ۽ Loss ܳ ҅࢑ೡ ࣻ ੓਺

  18. Result SQuAD 1.1 BERTࠁ׮ח ࢿמ੉ 8% ੿ب ծ਷Ѫ੉ ࢎप ೞ૑݅

    प ࢲ࠺झ۽ ٜ݅য૓ ݽ؛ীࢲח
 ӝઓ SOTA (DrQA) ؀࠺ ࢿמ਷ 4% ֫Ҋ
 ࣘبח DrQA 58ߓ ࡅܰҊ BERTࠁ׮ 6000ߓ ੿ب ࡅܴ