Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Transcript

1. Real-Time Open-Domain Question Answering With Dense-Sparse Phrase Index 스캐터랩(ScatterLab) Machine Learning Engineer 김준성 ੌ࢚؀ച ੋҕ૑מ
2. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ
3. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD
4. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Error Propagation Ӓܻझ݃झ ࣛ۽؀୏ … X
5. Motivation Query When is the Christmas? Christmas Jejus Christmas Tree Christmas Eve … Document Retrieval Reading Comprehension ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח ೯ࢎо द੘ػ Ѫ਷ 4ࣁӝ ޖ۵ࠗఠ৓׮. ׼दীח ௼ܻझ݃झ੄ զ૞ о ੿ഛೞѱ ੿೧૑૑ ঋওӝ ٸޙী 1ਘ 6ੌ, 3ਘ 21ੌ, 12ਘ 25ੌ о਍ؘ ೞܖо ࢶ੿غ঻ ׮. 12ਘ 25ੌ੉ ௼ܻझ݃झ۽ ੗ܻ੟਷ Ѫ਷ AD 354֙, ੉റ੉׮. 12ਘ 25ੌ TF-IDF Searching SQuAD Super Slow Document Size ࣻরѤ੄ WIKi Model Size BERT-Large 10-100 sec/query ࢲ࠺झೡ ࣻ হ਺
6. Motivation Super Slow SQuAD-BERT ௼ܻझ݃झח ঱ઁঠ | ৘ࣻ੄ ఍ࢤਸ ୷ೞೞח.. ௼ܻझ݃झח ঱ઁঠ | ӝةҮ ੋٜীѱ ੓য о੢… ௼ܻझ݃झח ঱ઁঠ | োੋٜ੄ о੢ ೥ೠ…. ଺Ҋ੗ ೞח Articleਸ Caching ೡ ࣻ হӝ ٸޙী Query ী ٮۄࢲ ݒߣ ҅࢑೧ঠೣ 12ਘ 25ੌ
7. Models Query Embedding [CLS] What day is it the Christmas? Query Encoding
8. Models Christmas is an annual festival commemorating the birth of Jesus Christ, observed on December 25.. Phrase Encoding
9. Models Christmas is an annual festival commemorating the birth of Jesus Christ, observed on December 25 Start Span Embed End Span Embed December 25 Phrase Encoding
10. Models Christmas is an annual festival commemorating the birth of Jesus Christ, observed on December 25 Start Span Embed End Span Embed annual festival commemorating the birth of Jesus Christ Phrase Encoding
11. Sparse Encoding Models ੌ߈੸ੋ Document Retrieval + Reading Comprehension झఋੌ੉ ইצ ߄۽ Query -> Span ՙܻ Distribution Matching ೣ (RMM э਷?)
12. Sparse Encoding Models START EMBED / END EMBED / COHERENCE EMBED = DENSE EMBEDDING
13. Sparse Encoding Models ৵ Coherenceо ৵ ٜযо૑??
14. Models Dense Encoding Token Embed dimਸ 4١࠙ ೧ࢲ 1,2,3,4 ۽ ա׃ Start_embed1, End_embed2, dot[Start_embed3, End_embed4] Concat
15. Sparse Encoding 2-Gram TF-IDFܳ ੉ਊ೧ࢲ Sparse Embeddingਸ ݅ٞ + Paragraph-Level Sparse Embedding ب ؊ೣ Models
16. ੌ߈੸ਵ۽ח ݽٚ Document ৬ ࠺Ү೧ࢲ argmaxܳ ೞח ߑߨ ࢎਊ Training Training Efficiency
17. Training Training Efficiency ೞ૑݅ ਋ܻח ݏח ௪ܻী ؀೧ࢲ Positive ೞѱ ೟णೞݶ ؽ (RMMীࢲ Positive Sample) ઑӘ؊ ࣻ೟੸ਵ۽ ੿ܻೞݶ ؊ ബਯ੸ਵ۽ Loss ܳ ҅࢑ೡ ࣻ ੓਺
18. Result SQuAD 1.1 BERTࠁ׮ח ࢿמ੉ 8% ੿ب ծ਷Ѫ੉ ࢎप ೞ૑݅ प ࢲ࠺झ۽ ٜ݅য૓ ݽ؛ীࢲח
 ӝઓ SOTA (DrQA) ؀࠺ ࢿמ਷ 4% ֫Ҋ
 ࣘبח DrQA 58ߓ ࡅܰҊ BERTࠁ׮ 6000ߓ ੿ب ࡅܴ