Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Scatter Lab Inc.
August 07, 2020
Research
0
1.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Scatter Lab Inc.
August 07, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
scatterlab
0
1.3k
scatterlab
0
1.1k
scatterlab
0
1.1k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.3k
scatterlab
0
1.3k
scatterlab
0
1.3k
Other Decks in Research
See All in Research
j20232
0
550
ianozsvald
1
490
yuichiroh
0
110
waptech
5
560
noguhiro2002
1
550
tomo_otamot
0
200
diracdiego
1
760
usaito
4
1.6k
interdax
0
120
future_tamaki
0
620
takurodadada
0
660
butsugiri
1
170
Featured
See All Featured
shpigford
165
19k
davidbonilla
69
3.5k
trishagee
20
2.1k
colly
66
3k
moore
125
21k
ufuk
56
5.4k
orderedlist
PRO
329
35k
qrush
285
18k
swwweet
206
6.8k
smashingmag
283
47k
gr2m
83
11k
philhawksworth
190
17k
Transcript
MLࣁա S6E3 Approximate Nearest Neighbor Negative Contrastive Learning for
Dense Text Retrieval ӣળࢿ ML Research Scientist, Pingpong
ݾର ݾର 1. Introduction 1. ޙઁ 2. ӝઓ ӝߨ
ೠ҅ 2. Approach 1. Ӕ ߑߨ ࣗѐ 2. ࠺زӝ ण ܖ౯ 3. Experiment 1. प ࢸ҅ 2. प Ѿҗ 3. ҳഅ ࣁࠗࢎ೦
• ࠄ ֤ޙীࢲ Ҿӓਵ۽ ಽҊ ೞח ޙઁח Open-Domain Question Answering
(QA) పझ • Open-Domain QAח যڃ بݫੋী Ҵೠغয ঋ ޙਸ ؍ਸ ٸ, ࠁਬೞҊ ח (~1M+) ޙࢲٜ оؘ ನೣغয ח ਸ ח పझ۽ ೡ ࣻ णפ. • ܳ ٜݶ ਤఃೖ٣ইী ઓೞח ݽٚ ޙࢲܳ ଵઑೡ ࣻ ח о ೞী “ఋ֢झח ݻಌࣃ ࢤݺܳ લয?” ী ೠ ਸ ח Ѫ ੑפ. ޙઁ [1/2]
• ٩۞ ӝ߈ ݽ؛ਸ ਊ೧ࢲ ࠁ ഛೠ ਸ ਸ ࣻ
݅, ݽٚ ޙࢲ(+Nর)ী ೧ োਸ ࣻ೯ೞח Ѫ ݒ ࠺ബਯ Ҋ, पदр ࢲ࠺झо ࠛоמೞח ೠ҅ णפ. • ӝઓ োҳٜ ࣘب ೠ҅ਸ ӓࠂೞӝ ਤ೧ ѱ فо stage ۽ ܻ࠙ೞৈ ޙઁܳ ಽҊ ೞणפ • 1. Document Retrieval: য ী ೧ࢲ ҙ۲ ח ޙࢲٜਸ ח ױ҅ • 2. Reading Comprehension: য ী ೠ ҳੋ ਸ ҙ۲ ޙࢲܳ ଵઑೞৈ بೞח ݽ؛ • য়ט ࣗѐ೧ ܾ٘ ֤ޙ Document Retrieval ࢿמ ೱ࢚ী ҙೠ ߑߨਸ ઁউפ. ޙઁ [2/2]
• ӝઓ ࠗ࠙ োҳীࢲח Document Retrieval ী Lexical Feature ܳ
۽ ࢎਊೞणפ. • द) BM25, TF-IDF, Keyword Matching ١١ (Elastic Search ػ ӝמ) • ೞ݅ ۞ೠ ߑߨ ೣ୷ (Semantic)ܳ ೧ೞҊ ҙ۲ػ ߸ਸ ਸ ࣻח হणפ. • द) Q. ־о పठۄ ঠ? -> (పठۄ, ) ਵ۽ Ѩ࢝೧ب ف ఃਕ٘ܳ ನೣೞח ޙࢲܳ ਸ ࣻ হ.. ӝઓ ߑߨ ೠ҅ [1/3]
• ୭Ӕ োҳٜ(Lee et al., 2019; Guu et al., 2020;
Seo et al. 2019) ৬ ޙࢲܳ BERTܳ ਊ೧ Representation ਵ۽ അೞৈ ࠁ Semantic ೠ ࠁܳ ನೡ ࣻ ח ߑߨਸ ઁউೞ. • ۞ೠ ߑߨٜ BI-Encoder ҳઑ ݽ؛ਸ ࢎਊೞݴ, In-Batch Negative ۽ णਸ ࣻ೯פ. • ण ৮ܐػ റীח Document Encoderܳ ਊ೧ࢲ ܻ ޙࢲٜਸ encoding ೧ ֬ • Inference दীח ݅ BERT۽ Representation ਸ ҅ೞҊ FAISS ৬ э Approximate Nearest Neighbor Search ోਸ ਊ೧ ߄۽ Representation җ оө Top-Kѐ ޙࢲܳ ӝઓ ߑߨ ೠ҅ [2/3]
Bi-encoder ޙࢲ
णߑߨ: In-Batch Negative Q1 D1 Q2 D2 Q3 D3 Q4
D4 ण ؘఠࣇ
णߑߨ: In-Batch Negative Q1 D1 Q2 D2 Q3 D3 Q4
D4 ण ؘఠࣇ Q: (4, 512) D: (4, 512)
णߑߨ: In-Batch Negative Q1 D1 Q2 D2 Q3 D3 Q4
D4 ण ؘఠࣇ Q: (4, 512) D: (4, 512) Q ⋅ DT -> (4,4)
णߑߨ: In-Batch Negative Q1 Q2 Q3 Q4 D1 D2 D3
D4 Q1 D1 Q2 D2 Q3 D3 Q4 D4 ण ؘఠࣇ Q: (4, 512) D: (4, 512) Q ⋅ DT -> (4,4)
णߑߨ: In-Batch Negative Q1 Q2 Q3 Q4 D1 D2 D3
D4 Q1 D1 Q2 D2 Q3 D3 Q4 D4 ण ؘఠࣇ Q: (4, 512) D: (4, 512) 0.5 0.6 0.4 0.7 0.2 0.1 0.2 0.1 0.2 0.1 0.3 0.1 0.2 0.1 0.1 0.1 Softmax Q ⋅ DT п Row ߹۽ Softmaxܳ ஂೣ -> (4,4)
णߑߨ: In-Batch Negative Q1 Q2 Q3 Q4 D1 D2 D3
D4 Q1 D1 Q2 D2 Q3 D3 Q4 D4 ण ؘఠࣇ Q: (4, 512) D: (4, 512) 0.99 0.99 0.01 0.99 0.99 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 Q ⋅ DT ण ݾ: п Row ীࢲ غח ޙࢲо ઁੌ ֫ чਸ ыب۾ -> (4,4)
• ח Dense Retrieval ݽ؛ਸ णೡ ٸ ࢎਊೞח In-Batch Negativeী
ޙઁо ਸ פ. • In-Batch Negative ण ߑߨ যוب ਬࢎೠ ޙࢲٜਸ ୶ܻחؘীח ਬബೞ݅, ҙ۲ ח ޙࢲܳ ഛೞѱ ఐ࢝ೞӝীח Ӕࠄੋ ೠ҅о ਸ Ѫۄח оࢸਸ ࣁפ. • ৵ջೞݶ ৮ ҙ۲ হח റࠁٜ ী, ҙ۲ ח ೞա ޙࢲܳ ࡳب۾ णೞח Ѫҗ ҙ۲ࢿ ח റࠁٜ ীࢲ ҙ۲ ח ೞա ޙࢲܳ ࡳب۾ णೞח Ѫ ܰӝ ٸޙੑפ. ӝઓ ߑߨ ೠ҅ [2/3]
• negative sample ٜ representation ਸ t-SNEਵ۽ दпചೞৈ ࠙ࢳਸ ࣻ೯ೞणפ.
• ӝઓী ۽ ࢎਊೞ؍ Random, BM25 ӝ߈ Negative ٜ पઁ Relevant Document ৬ ࠙ನ ରо ब೮ • ژೠ Random Negative ۽ णػ ݽ؛۽ Dense Retrieval ਸ ࣻ೯द, पઁ ҙ۲ ޙࢲٜਸ நೞ ޅ೮. ӝઓ ߑߨ ೠ҅ [2/3]
• negative sample ٜ representation ਸ t-SNEਵ۽ दпചೞৈ ࠙ࢳਸ ࣻ೯ೞणפ.
• ӝઓী ۽ ࢎਊೞ؍ Random, BM25 ӝ߈ Negative ٜ पઁ Relevant Document ৬ ࠙ನ ରо ब೮ • ژೠ Random Negative ۽ णػ ݽ؛۽ Dense Retrieval ਸ ࣻ೯द, पઁ ҙ۲ ޙࢲٜਸ நೞ ޅ೮. ӝઓ ߑߨ ೠ҅ [2/3] “ উীࢲ ޤо ҙ۲ ޙࢲջ!” ೠ Ѫب णਸ ࣻ೯೧ঠ ೠ!
• ࠄ ֤ޙীࢲח णद ࢎਊغח negative sampleਸ ࡳח ࢜۽ ߑߨਸ
ઁউפ • Approximate nearest neighbor Negative Contrastive Estimation(ANCE) • ण р ݽ؛ retrieval ػ Ѿҗܳ ਊ೧ࢲ য۰ negative sampleਸ ݅٘ח ߑߨੑפ. • ࠺زӝਵ۽ faiss index ܳ N step ݃ সؘೞҊ, negative sample ਸ ࣘਵ۽ јनפ Approach
Approach
• ಣо పझ TREC 2019 Deep Learning Track ܳ ࢎਊೞणפ.
• Ѩ࢝ ূ Bing ਵ۽ ٜযৡ ߔ݅ѐ ࢚ ী ೧ࢲ ҙ۲ػ ޙࢲо ۨ࠶݂ غয ח ؘఠࣇ • ؘఠࣇਸ ࢶఖೠ ਬ۽ Ҋ, ୭नҊ, о അपੋ ࢚ടਸ ੜ ߈೮ӝ ⮶ޙী ࢎਊ೮Ҋ ח ӝࣿೞणפ. • ಣо ݫܼ MRRҗ Recall@1k, NDCGܳ ࢎਊೞणפ. • ࠗ࠙ ࢿמ Retrieval ী ೠ ࢿמਸ ஏೞҊ, ୶оਵ۽ য 100ѐ candidate ղীࢲ DR ݽ؛ਸ ਊ೧ ҙ۲ػ ޙࢲٜਸ Rerank ೞח מ۱ب э Ѩૐೞणפ. (ীࢲ RerankۄҊ ա৬ ח ࠗ࠙) • DPRҗ زੌೞѱ, بݫੋ ઁೠ হח QAؘఠࣇੋ OpenQA task ؘఠࣇਵ۽ب ಣоܳ ࣻ೯ೞणפ. ಣо ߑध Top-Nউী पઁ۽ ܻо ఋѶ ೞח passage о ನೣغয ח ইצ ಣоೞח ݫܼਸ ࢎਊೞणפ Experiment
Experiment
• ӝઓ ߑߨ BM25۽ Document Retrieval ࣻ೯റ, BERT ۽ Reranking
ೞח Two-Stage ߑߨਸ ࢎਊೞणפ • Inference दী ୨ 1.42 ୡ Ѧ۷णפ. • ߈ݶী ࠄ ֤ޙ ANN ӝ߈ Dense Retrieval ਸ ࢎਊ೮ӝ ٸޙী ࠁ ࡅܲ ࣘب Inference о оמפ. -> Inference दী 11.6ms ߆ী Ѧܻ ঋ. Ӓۢীب Two-Stage ࠁ ֫ ࢿמਸ ࠁৈષ Experiment
• Dense Retrievalਸ In-Batch Negative ߑधਵ۽݅ ण ೞח Ѫ ೠ҅
࠙ݺ ઓೠ • റࠁٜ р ࢶࣽਤܳ Ѿೞח מ۱ ࠗೞ. • ण җীࢲ ഁтܻח റࠁ ޙࢲٜ աৢ Ѫਸ о೧ࢲ, о оӰب۾ णਸ ೧ঠ ೠ. • ܳ ਤ೧ࢲ ण җীࢲ ୶ۿҗ زੌೞѱ ANN indexing ਸ ࣻ೯ೞҊ, negative ٜਸ retrieval۽ ࡳ ח ߑߨਸ ઁউೠ. ӒܻҊ ܳ ࠺زӝਵ۽ ࣻ೯ೞৈࢲ োࣘੋ णਸ ೡ ࣻ ب۾ ೠ • प Ѿҗ ઁউೞח ण ߑध पઁ పझীࢲ ࠁ ࣻೠ ࢿਸ ࠁৈ. • Ѩ࢝ Retrieval పझ৬, Open-Domain QAীࢲ Document Retrieval ࢿמਸ ಣоೞ Conclusion
• https://codertimo.github.io/2020/07/20/ANN-negative-contrastive-learning/ ଵҊܐ