Share
Embed
[輪講資料] One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Hayato Tsukagoshi
December 18, 2023
Transcript
34 slides
-
One Embedder, Any Task: Instruction-Finetuned Text Embeddings D1, Graduate School
of Informatics, Nagoya University, Japan Hayato Tsukagoshi Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih Noah A. Smith, Luke Zettlemoyer, Tao Yu ACL 2023 Findings https://aclanthology.org/2023. fi ndings-acl.71/ -
•จຒΊࠐΈOut Of Domain (OOD) ʹऑ͍ •ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧ InstructORΛఏҊ • λεΫɾυϝΠϯ͝ͱʹҟͳΔࢦࣔ ͰจຒΊࠐΈΛੜՄೳ
•ଟ༷ͳσʔληοτऩू →ࢦࣔΛΞϊςʔτ (MEDI dataset) →ࢦࣔ+จΛຒΊࠐΈදݱʹ →ରরֶशͰϞσϧΛ܇࿅ •MTEBͳͲෳͷϕϯνϚʔΫͰฏۉͯ͠ੑೳ্ ֓ཁ 2 -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 5 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 6 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ ۙ͘ʹ ϕΫτϧؒͷڑ͕ ҙຯతͳؔΛදݱ -
•จຒΊࠐΈOut Of Domain (OOD) ʹऑ͍ •ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧ InstructORΛఏҊ • Instruction-based Omnifarious
Representations • จ͝ͱʹࢦࣔΛม͑ଟ༷ͳຒΊࠐΈΛੜ •ࢦ͕ࣔ͋Δ͜ͱͰOODͷੑೳ্ ख๏֓ཁ •ଟ༷ͳσʔληοτऩू →ࢦࣔΛΞϊςʔτ (MEDI dataset) →ࢦࣔ+จΛຒΊࠐΈදݱʹ →ରরֶशͰϞσϧΛ܇࿅ ख๏֓ཁ 7 -
•ൺֱతγϯϓϧͳରরֶशʹΑΔ܇࿅ •ࢦࣔ+จΛ·Δ͝ͱϞσϧʹೖྗ͢Δ • ࢦࣔΛߟྀͨ͠จຒΊࠐΈΛੜ ܇࿅खॱ 1. ࢦࣔͱจͷϖΞ(x, Ix, y, Iy)Λ༻ҙ
2. Ex = E(Ix⊕x), Ey = E(Iy⊕y)ͷΑ͏ʹຒΊࠐΉ 3. ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹରরֶश •ෛྫͱͯ͠in-batch negativesΛར༻ InstructOR: ܇࿅खॱ 8 Ix⊕x Iy⊕y Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ ଛࣦؔ -
•InstructORͷ܇࿅ʹ ࢦࣔͱจ͕ϖΞʹͳͬͨσʔληοτ͕ඞཁ •طଘσʔληοτΛ܇࿅༻ʹ౷߹ (ܭ300ݸ) • Super-NaturalInstructions (super-NI) • Sentence Transformers
embedding data MEDI: Multitask Embedding Data with Instructions 10 -
•InstructORͷ܇࿅ʹ ࢦࣔͱจ͕ϖΞʹͳͬͨσʔληοτ͕ඞཁ •طଘσʔληοτΛ܇࿅༻ʹ౷߹ (ܭ300ݸ) • Super-NaturalInstructions (super-NI) • Sentence Transformers
embedding data •super-NIࢦࣔͱจ͕ϖΞʹͳ͍ͬͯΔ͕ ਖ਼ྫɾෛྫ͕ଘࡏ͠ͳ͍ → Sentence-T5ͰจຒΊࠐΈΛੜ (ࢦࣔͳ͠) → ྨࣅΛͬͯਖ਼ෛྫϖΞΛࣗಈੜ MEDI: Multitask Embedding Data with Instructions 11 -
•Seq2Seqܥͷσʔληοτϥϕϧ͕ͳ͍ͷͰ͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜख๏ 13
ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗࣅ͍ͯΔ͕ग़ྗࣅ͍ͯͳ͍ͱߴείΞ -
•Seq2Seqܥͷσʔληοτϥϕϧ͕ͳ͍ͷͰ͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜख๏ 14
ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗࣅ͍ͯΔ͕ग़ྗࣅ͍ͯͳ͍ͱߴείΞ -
•Seq2Seqܥͷσʔληοτϥϕϧ͕ͳ͍ͷͰ͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜख๏ 15
ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗࣅ͍ͯΔ͕ग़ྗࣅ͍ͯͳ͍ͱߴείΞ -
•Seq2Seqܥͷσʔληοτϥϕϧ͕ͳ͍ͷͰ͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜख๏ 16
ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ ग़ྗจͷྨࣅ ೖྗࣅ͍ͯΔ͕ग़ྗࣅ͍ͯͳ͍ͱߴείΞ hard negative తͳཱͪҐஔ -
•ࢦࣔʹجͮ͘จຒΊࠐΈͷͨΊʹɺࢦ͕ࣔඞཁ(વ) •ҎԼͷใͱςϯϓϨʔτʹج͍ͮͯσʔληοτ͝ͱʹࢦࣔΛ࡞ •Text Type • ຒΊࠐΉจ͕Ͳ͏͍͏ͷ͔ • QAͷqueryͳΒʮ࣭จʯ • passageͳΒʮจॻʯ
•Task Objective (optional) • ຒΊࠐΈ͕ԿʹΘΕΔ͔ •Domain (optional) • ʮχϡʔεʯͳͲจͷυϝΠϯ MEDI: ࢦࣔͷߏங 17 -
•ࢦࣔҎԼͷςϯϓϨʔτ͔Βߏங •“Represent The (DOMAIN) TEXT TYPE for TASK OBJECTIVE:.” •࣮ࡍͷࢦ͕ࣔҎԼ
• ॏཁ: จͷλΠϓʹΑͬͯࢦࣔͰຒΊࠐΈํΛม͑ΒΕΔ MEDI: ࢦࣔͷߏங 18 ্දจதͷද͔Βൈਮͨ͠ͷ -
•ଟ༷ͳσʔληοτͷจϖΞ͔Β ͳΔେنͳσʔληοτ •ྨࣅ͕ରশ/ඇରশͳͷ͕ ͍ࠞͬͯ͟Δ • ྫ: ݕࡧͷquery—passage •ΞελϦεΫ(*)͕͍͍ͭͯΔ σʔληοτtest setΛؚΉ
• ҙ: NQ, HotPotQAޙड़͢ΔMTEB ʹؚ·Ε͓ͯΓɺϦʔΫ͍ͯ͠Δ MEDI: σʔληοτͷ༁ 19 -
•ࢦࣔʹैͬͯλεΫΛղ͚ΔΑ͏ʹLMΛtuning (Instruction Tuning) •zero-shotͰ৭ʑͳੜܥλεΫ͕͏·͘ղ͚Δ •InstructORFLANͷຒΊࠐΈ൛ͱݴ͑Δ •FLANFinetuned Language Net ͷུΒ͍͠🧐 Wei+:
Finetuned Language Models Are Zero-Shot Learners, ICLR ’22 ؔ࿈ݚڀ: FLAN 20 Flan-T5, Flan-UL2 ΊͪΌͪ͘Όڧ͍ϞσϧͳͷͰ͓͢͢Ίʂ(͜ͷ࣌Ͱ·ͩڧ͍) -
•ෳͷϕϯνϚʔΫͰൣʹੑೳΛධՁ • MTEB: จຒΊࠐΈͷେنϕϯνϚʔΫ • Billboard (ʹΑΔධՁ): ੜ͞Εͨཁͱਖ਼ղཁͱͷྨࣅͷ૬ؔ • Prompt
Retrieval: ੜʹ࠷దͳϓϩϯϓτͷݕࡧ ධՁ࣮ݧ 22 -
•58ͷσʔληοτɺ112ͷݴޠ͔Β ͳΔจຒΊࠐΈͷϕϯνϚʔΫ •STSΛ͡Ί8छྨͷλεΫ͕͋Δ •(༨ஊ) MTEBͷ࣮ݧ݁Ռ͔Β จຒΊࠐΈͰσΧ͍Ϟσϧ ͷํ͕ڧ͍͕ݟ͑Δ •ධՁ؍: InstructOR͕ ͲΕ͘Β͍ྑ͍จຒΊࠐΈΛੜͰ͖Δ͔
•ಛʹɺࢦࣔΛՃ͢Δ͜ͱͰ طଘख๏ΑΓੑೳ্͕͢Δ͔͕ؾʹͳΔ Muennigho ff +: MTEB: Massive Text Embedding Benchmark, arXiv ’22 ධՁ࣮ݧ: Massive Text Embedding Benchmark (MTEB) 23 Sentence-BERT ͷஶऀͷ Nils Reimers ͕last authorͷproject -
•ੜϞσϧɾධՁࢦඪͷํΛ ొͰ͖ΔϦʔμʔϘʔυ(ϑϨʔϜϫʔΫ) • ֤ࢦඪΛΞϯαϯϒϧͨ͠ࢦඪࣗಈࢉग़ • ੜϞσϧͷ։ൃͱධՁΛ૬ޓʹଅਐ •ධՁ؍: ੜจ—ਖ਼ղཁͷྨࣅ͕ ͲΕ͘Β͍ਓؒධՁͱ͍͔ۙ •ੜจͱ֤ਖ਼ղཁͱͷcosྨࣅͷ
࠷େͱਓखධՁͱͷϐΞιϯ૬ؔ •3ͭͷσʔληοτͰͦΕͧΕ૬ؔΛࢉग़ɺฏۉΛධՁʹ Kasai+: Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand, NAACL ’22 ධՁ࣮ݧ: Billboard (ͰͷཁͷྨࣅධՁ) 24 -
•In-Context Learningςετ࣌ͷfew-shotࣄྫΛબͿඞཁ͕͋Δ • ࣄલʹগʹߜΔͷେมɾͲΕ͕͍͍͔Θ͔Βͳ͍ •few-shotࣄྫΛຒΊࠐΈʹมˠςετࣄྫʹ͍ۙࣄྫΛೖΕͯੑೳ্ •ධՁ؍: ੜ͕ͲΕ͚ͩ͏·͘ग़དྷΔ͔ • ੑೳͷد༩͢Δྑ͍few-shotࣄྫΛ࣋ͬͯདྷΕΔ͔Ͳ͏͔͕ධՁई Su+:
Selective Annotation Makes Language Models Better Few-Shot Learners, arXiv ’22 ධՁ࣮ݧ: Prompt Retrieval 25 ίϝϯτ: LLM࣌ͷSentEvalతͳཱͪҐஔʹݟ͑Δ (SentEval: ຒΊࠐΈϕʔεͷઢܗྨثΛֶशɾͦͷਫ਼ͰධՁ) ੜGPT-J -
SimCSE: ҟͳΔDropoutΛద༻ͨ͠ಉ͡จΛਖ਼ྫ or ؚҙؔͷจϖΞΛਖ਼ྫͱͨ͠ରরֶश Contriever: ϕΫτϧݕࡧʹ͓͚ΔରরֶशͷݶքΛௐࠪɼMS MARCOͰͷ fi ne-tuningͰੑೳ্֬ೝ GTR:
ScalingͰจຒΊࠐΈͰϕΫτϧݕࡧੑೳ & ൚Խੑೳ্Λ֬ೝ coCondenser: ࣄલ܇࿅࣌ʹίʔύεґଘͷଛࣦΛՃ͑Retrieverͷ܇࿅Λؤ݈ʹ&ੑೳ্ [00] Sentence-T5: T5ΛNLI/QAσʔλͰରরֶशʹΑΔ fi ne-tuning → scaling lawͷௐࠪ + SentGLUEͰධՁ SGPT: GPTΛใݕࡧλεΫʹར༻ɼCross-EncoderܗࣜͱBi-Encoderܗࣜͷ྆ํΛBEIRͰ࣮ݧ Gao+: SimCSE: Simple Contrastive Learning of Sentence Embeddings, EMNLP ’21 Izacard+: Unsupervised Dense Information Retrieval with Contrastive Learning, TMLR ’23 Ni+: Large Dual Encoders Are Generalizable Retrievers, EMNLP ’22 Gao+: Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval, ACL ’22 Ni+: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models, CoRR ’21 Muennigho ff : SGPT: GPT Sentence Embeddings for Semantic Search, arXiv ’22 ධՁ࣮ݧ: ൺֱख๏ 26 -
•GTRΛ fi ne-tuning • Թύϥϝʔλ: 0.01 • ֶश: 2e-5 •
Optimizer: AdamW ϛχόονͷαϯϓϦϯάख๏ •֤ϛχόον୯Ұͷσʔληοτ͔ΒͳΔ • λεΫσʔληοτͷҧ͍Λֶश͠ͳ͍Α͏ʹ͢Δ ධՁ࣮ݧ: InstructORͷ܇࿅ઃఆ 27 in-batch negatives ʹΑΔѱӨڹͷݮ -
•ࢦ͕ࣔ͋Δ͜ͱͰOODʹର͢Δੑೳ্ •ൃදऀίϝϯτ • ࢦࣔͳ͠Ͱ fi ne-tuningͨ͠GTRͱͷൺֱ͕ཉ͍͠ • ࢦࣔͷޮՌ͔ɺMEDIͷଟ༷ੑͷ͓ӄ͔֬৴͕࣋ͯͳ͍ ࢦࣔͷޮՌ: Domain
Shift 32 ܇࿅࣌ʹΈ͍ͯͳ͍ υϝΠϯͷσʔληοτ -
•ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧInstructORΛఏҊ •ࢦࣔΛՃ͑Δ͜ͱͰطଘख๏ΑΓߴੑೳ ࢥͬͨ͜ͱɾؾʹͳΔ͜ͱ •ࢦࣔνϡʔχϯά͞ΕͨϞσϧ(Flan-T5ͱ͔)Λ ϕʔεʹͨ࣌͠ͷੑೳͷมԽ • ࢦࣔʹै͏ೳྗ͕͋ΔϞσϧͷํ͕ڧ͘ͳΓͦ͏ • GTRInstructORຊʹࢦࣔʹै͍͑ͯΔʁ(ICLΔͳΒDecoderܕʁ) •MEDI
datasetͷࢦ͕ࣔςϯϓϨʔτʹج͍͍ͮͯΔͷͰunnatural • ࢦࣔΛͬͱଟ༷͔ͭࣗવʹͰ͖ͨΒΑΓຒΊࠐΈͷ࣭Λ্͛ΒΕͦ͏ ·ͱΊ 34