Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-Time Open-Domain Question Answering with D...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Scatter Lab Inc. Scatter Lab Inc.
July 17, 2019
Research
910
0

Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

July 17, 2019

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
1.9k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.4k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.5k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
typst の使い方：言語学を研究する学生のために
Avatar for Tomoya gitomochang
0
470
Research Engineerという仕事 / Research Engineering: Bridging Research and Business
Avatar for chck chck
1
220
通時的な類似度行列に基づく単語の意味変化の分析
Avatar for hajime kiyama rudorudo11
0
320
Apache Gravitinoで実現する Icebergカタログ統合とアクセスの一元化
Avatar for matsumooon matsumooon
0
300
SAKURAONE: An Open Ethernet-based AI HPC System And Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
1
390
Using our influence and power for patient safety
Avatar for Helen Bevan helenbevan
0
370
コーディングエージェントとABNを再考
Avatar for Hironobu Fujiyoshi hf149
2
730
進学校の生徒にはア行の苗字が多いのか
Avatar for Takashi Ozeki ozekinote
0
460
世界モデルにおける分布外データ対応の方法論
Avatar for Hidehisa Arai koukyo1994
7
2.2k
YOLO26_ Key Architectural Enhancements and Performance Benchmarking for Real-Time Object Detection
Avatar for SatAI.challenge satai
3
830
COFFEE-Japan PROJECT Impact Report（海ノ向こうコーヒー）
Avatar for 坂ノ途中 ontheslope
0
2k
老舗ものづくり企業でリサーチが変革を起こすまで - 三菱重工DXの実践
Avatar for kazu skydats
0
200

Featured

See All Featured
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.5k
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.7k
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
Avatar for Aleyda Solis aleyda
1
2k
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
150
Building the Perfect Custom Keyboard
Avatar for Naoto Takai takai
2
800
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
210
Jess Joyce - The Pitfalls of Following Frameworks
Avatar for Tech SEO Connect techseoconnect
PRO
1
170
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.3k
Kristin Tynski - Automating Marketing Tasks With AI
Avatar for Tech SEO Connect techseoconnect
PRO
0
280
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
187
22k
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.3k

Transcript

  1. Real-Time Open-Domain Question Answering With Dense-Sparse Phrase Index 스캐터랩(ScatterLab) Machine

    Learning Engineer 김준성 ੌ࢚؀ച ੋҕ૑מ

  2. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ

  3. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD

  4. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Error Propagation Ӓܻझ݃झ ࣛ۽؀୏ … X

  5. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree

    Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Super Slow Document Size ࣻরѤ੄ WIKi Model Size BERT-Large 10-100 sec/query ࢲ࠺झೡ ࣻ হ਺

  6. Motivation Super Slow SQuAD-BERT ௼ܻझ݃झח ঱ઁঠ | ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח..

    ௼ܻझ݃झח ঱ઁঠ | ӝةҮ ੋٜীѱ ੓য о੢… ௼ܻझ݃झח ঱ઁঠ | োੋٜ੄ о੢ ೥ೠ…. ଺Ҋ੗ ೞח Articleਸ Caching ೡ ࣻ হӝ ٸޙী Query ী ٮۄࢲ ݒߣ ҅࢑೧ঠೣ 12ਘ 25ੌ

  7. Models Query Embedding [CLS] What day is it the Christmas?

    Query Encoding

  8. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25.. Phrase Encoding

  9. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed December 25 Phrase Encoding

  10. Models Christmas is an annual festival commemorating the birth of

    Jesus Christ, observed on December 25 Start Span Embed End Span Embed annual festival commemorating the birth of Jesus Christ Phrase Encoding

  11. Sparse Encoding Models ੌ߈੸ੋ Document Retrieval + Reading Comprehension झఋੌ੉

    ইצ ߄۽ Query -> Span ՙܻ Distribution Matching ೣ (RMM э਷?)

  12. Sparse Encoding Models START EMBED / END EMBED / COHERENCE

    EMBED = DENSE EMBEDDING

  13. Sparse Encoding Models ৵ Coherenceо ৵ ٜযо૑??

  14. Models Dense Encoding Token Embed dimਸ 4١࠙ ೧ࢲ 1,2,3,4 ۽

    ա׃ Start_embed1, End_embed2, dot[Start_embed3, End_embed4] Concat

  15. Sparse Encoding 2-Gram TF-IDFܳ ੉ਊ೧ࢲ Sparse Embeddingਸ ݅ٞ + Paragraph-Level

    Sparse Embedding ب ؊ೣ Models

  16. ੌ߈੸ਵ۽ח ݽٚ Document ৬ ࠺Ү೧ࢲ argmaxܳ ೞח ߑߨ ࢎਊ Training

    Training Efficiency

  17. Training Training Efficiency ೞ૑݅ ਋ܻח ݏח ௪ܻী ؀೧ࢲ Positive ೞѱ

    ೟णೞݶ ؽ (RMMীࢲ Positive Sample) ઑӘ؊ ࣻ೟੸ਵ۽ ੿ܻೞݶ ؊ ബਯ੸ਵ۽ Loss ܳ ҅࢑ೡ ࣻ ੓਺

  18. Result SQuAD 1.1 BERTࠁ׮ח ࢿמ੉ 8% ੿ب ծ਷Ѫ੉ ࢎप ೞ૑݅

    प ࢲ࠺झ۽ ٜ݅য૓ ݽ؛ীࢲח
 ӝઓ SOTA (DrQA) ؀࠺ ࢿמ਷ 4% ֫Ҋ
 ࣘبח DrQA 58ߓ ࡅܰҊ BERTࠁ׮ 6000ߓ ੿ب ࡅܴ