Share
Embed
Research Paper Introduction in IR Reading 2022 Fall
IR Reading 2022 Fall: https://sigir.jp/post/2022-11-12-irreading_2022fall/
Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799
Takuya Asano
November 12, 2022
Transcript
20 slides
-
Takuya Asano (takuya-a) Distill-VQ: Learning Retrieval Oriented Vector Quantization By
Distilling Knowledge from Dense Embeddings IR Reading 2022 Fall -
Vector Search and ANN Background • ຒΊࠐΈΛͬͨϕΫτϧݕࡧ͕͕͖͍ͬͯͯΔ • ݕࡧΤϯδϯɺਪનγεςϜͳͲ •
ΫΤϦͱจॻͷຒΊࠐΈͷྨࣅʹΑͬͯจॻΛબ • େنͳϕΫτϧݕࡧʹ͓͍ͯɺ ۙࣅ࠷ۙ୳ࡧʢANNʣ͕Ωʔύʔπ • ࣮ੈքʹ͓͍ͯɺઢܗ୳ࡧݱ࣮తͰͳ͍ • ɾϝϞϦ༻ྔɾਫ਼ͷτϨʔυΦϑΛ࣮ݱ -
Vector Quantization (VQ) Background • ANN ͷͨΊͷσʔλߏͱΞϧΰϦζϜ • ϕΫτϧू߹Λ K
ݸͷηϯτϩΠυͰද͢Δ • ࣄલʹɺϕΫτϧू߹͔ΒηϯτϩΠυΛܭࢉ • ϕΫτϧΛΤϯίʔυ͢Δͱ͖ʹɺ࠷͍ۙηϯτϩΠυΛٻΊɺͦͷIDͷΈΛه͢Δ • ͱͷϕΫτϧΛID͚ͩͰූ߸ԽͰ͖ΔͷͰίϯύΫτʹ • KΛେ͖͘͢Δͱۙࣅਫ਼্͕͍͕ͬͯ͘ɺͦͷͿΜ͘ͳΓɺϝϞϦ༻ྔେ͖͍ -
Product Quantization (PQ) Background • ϕΫτϧͷ࣍ݩΛ M ݸʹׂͯ͠ɺͦΕͧΕ Vector Quantization
͢Δ • ͦΕͧΕͷදϕΫτϧͷू߹ΛίʔυϒοΫͱݺͿ • ೖྗϕΫτϧΛMݸʹׂ͠ɺͦΕͧΕίʔυϒοΫͷத͔Β࠷͍ۙදϕ ΫτϧΛ୳͠ɺͦͷIDΛه͢Δ • MݸͷIDͷΈͰೖྗϕΫτϧΛූ߸Խ • ϝϞϦޮۙࣅਫ਼Α͍ -
Inverted File (IVF) Background • సஔΠϯσοΫεΛิॿσʔλߏͱͯ͠ར༻ • ͍ۙϕΫτϧΛసஔϦετʹ·ͱΊΔ • ࠪ͢Δཁૉ͕গͳ͍ͷͰߴ
• ࠷ॳʹૈ͍ྔࢠԽΛߦ͍ɺసஔϦετΛऔಘ • సஔϦετΛࠪͯ͠࠷ۙͷϕΫτϧΛܭࢉ H. Jégou, M. Douze and C. Schmid, "Product Quantization for Nearest Neighbor Search," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 1, pp. 117-128, Jan. 2011, doi: 10.1109/TPAMI.2010.57. -
Distill-VQ Summary • IVF ͱ PQ Λซ༻ͨ͠ϕΫτϧྔࢠԽʹΑΓ ANN Λߦ͏ •
ରরֶश (contrastive learning) ͰҎԼΛ࠷దԽ͢Δ • IVF ͷηϯτϩΠυ • PQ ͷίʔυϒοΫ • ΫΤϦຒΊࠐΈͷΤϯίʔμʔ • Α͘܇࿅͞ΕͨີͳຒΊࠐΈΛڭࢣɺ্هͷίϯϙʔωϯτΛੜెͱֶͯ͠श • ෳͷσʔληοτɺෳͷλεΫͰ SOTA -
Distill-VQ: Workflow Method • ࣄલ४උ • ͯ͢ͷจॻͷຒΊࠐΈΛܭࢉʢDistill-VQ Ͱݻఆʣ • จॻຒΊࠐΈ͔ΒɺIVF
ͱ PQ ΛॳظԽʢηϯτϩΠυͷܭࢉʣ • ڭࢣείΞͷܭࢉͷͨΊʹɺΑ͘܇࿅͞ΕͨΫΤϦΤϯίʔμʔΛ४උ • ͜ͷΫΤϦΤϯίʔμʔΛͬͯΫΤϦຒΊࠐΈΛܭࢉ -
Distill-VQ: Workflow Method 1. ΫΤϦຒΊࠐΈΛܭࢉ Shitao Xiao, Zheng Liu, Weihao
Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Workflow Method 2. ࣄલ४උͨ͠จॻຒΊࠐΈ ͔ΒαϯϓϦϯά Shitao Xiao, Zheng Liu,
Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Workflow Method 3. IVF ΛͬͨੜెείΞΛܭࢉ Shitao Xiao, Zheng Liu,
Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Workflow Method 4. PQ ΛͬͨੜెείΞΛܭࢉ Shitao Xiao, Zheng Liu,
Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Workflow Method 5. ڭࢣείΞΛܭࢉ Shitao Xiao, Zheng Liu, Weihao
Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Workflow Method 6. ੜెείΞͱڭࢣείΞͷ ྨࣅΛܭࢉ͠ɺϞσϧΛߋ৽ Shitao Xiao, Zheng Liu,
Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Distill-VQ: Detailed Algorithm Method • ֶशΞϧΰϦζϜ • L4: จॻίϨΫγϣϯ D
͔ΒީิจॻΛαϯϓϦϯά • L5: ڭࢣͷείΞΛܭࢉ ɹɹ • L6: IVF ͱ PQ ΛͬͯੜెͷείΞΛܭࢉ ɹɹ • L7: IVFɺPQɺΫΤϦΤϯίʔμʔΛֶश • f: similarity function Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Experiment Settings Experiments • σʔληοτ • MS MARCO Passage retrieval
• Bing Search ͷΫΤϦ • Natural Questions (NQ) • Google Search ͷΫΤϦ • ϕʔεϥΠϯ • طଘͷϕΫτϧྔࢠԽख๏ (IVFPQ, IVFOPQ, ScaNN) • ࠷ۙͷಉֶ࣌शख๏ (Poeem, JPQ, RepCONC) Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Experiment Settings Experiments • Distill-VQ ͷڭࢣϞσϧͱͯ͠ɺΑ͘܇࿅͞Εͨ2छྨͷΤϯίʔμʔΛࢼͨ͠ • AR2-G • CoCondenser
• ͜ΕΒͷϞσϧ MS MARCO ͱ NQ Ͱ࠷ accurate • จॻຒΊࠐΈ • ϑΣΞʹൺֱ͢ΔͨΊʹͯ͢ͷख๏Ͱಉ͡ͷΛ༻ -
Overall Performance Experiments • ݕࡧ࣭ͷΠϯύΫτΛطଘख๏ͱൺֱ • Ұ؏ͯ͠༗ҙʹߴ͍ੑೳˍSOTA • 2छྨͷΤϯίʔμʔ AR2-G,
CoCondenser • 2छྨͷσʔληοτ MS MARCO, NQ Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Explorations of Knowledge Distillation Experiments • Distill-VQ Ͱɺsimilarity function ɺจ
ॻαϯϓϦϯάํ๏ʹબͷ༨͕͋Δͷ Ͱɺม͑ͯΈ࣮ͯݧ • ϥϯΩϯάΛߟྀͨ͠ similarity function (KL-Div, ListNet, RankNet) ͷ΄͏͕ੑೳ ͕ߴ͍ • όοναϯϓϦϯάͱ Top-K ͷΈ߹Θͤ (IB + Top-K) ɺϥϕϧ͖σʔλΛͬ ͨ߹ (GT) ΑΓߴੑೳʢʂʣ Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Efficiency and Retrieval Quality Experiments • ͱ࠶ݱͷτϨʔυΦϑΛɺFAISS ͷΦϦδφϧͷ IVFOPQ ͱൺֱ
• ͯ͢ͷઃఆͰ IVFOPQ Λ্ճͬͨ Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. 2022. Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1513–1523. https://doi.org/10.1145/3477495.3531799 -
Personal Impressions • ϥϕϧ͖ͭσʔλ͕ͳͯ͘ɺANN ͷੑೳΛ্Ͱ͖Δख๏ͱͯ͠ɺେมڵຯਂ͔ͬͨ • σʔλߏɾΞϧΰϦζϜม͑ͣʹద༻Ͱ͖ΔͷͰɺΫΤϦॲཧͷѱӨڹͳ͍ • MS MARCO
Passage ͳͲͰ༗ҙͳੑೳ্͕֬ೝ͞ΕͨͷͰ༗ • ࣮ΞϓϦέʔγϣϯʹద༻͢Δ͜ͱΛߟ͑ΔͱɺIVFɾPQɾΫΤϦΤϯίʔμʔͷ࠶ֶशͲ͏͢ Δ͔͕ؾʹͳΔ • ANN ΠϯσοΫεશମΛ࡞Γͳ͓͠ʹͳΔͱࢥ͏ͷͰɺֶशʹ͔͔Δ࣌ؒؾʹͳΔ • ʢߋ৽͕ͳ͍ɺ੩తͳΠϯσοΫεͰ͋Εͳ͍͕ɺͦͷΑ͏ͳΞϓϦέʔγϣϯكʣ