[輪講資料] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Slide 1

Slide 1 text

One Embedder, Any Task:  Instruction-Finetuned Text Embeddings D1, Graduate School of Informatics, Nagoya University, Japan Hayato Tsukagoshi Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih  Noah A. Smith, Luke Zettlemoyer, Tao Yu ACL 2023 Findings  https://aclanthology.org/2023. fi ndings-acl.71/

Slide 2

Slide 2 text

•จຒΊࠐΈ͸Out Of Domain (OOD) ʹऑ͍ •ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧ  InstructORΛఏҊ • λεΫɾυϝΠϯ͝ͱʹҟͳΔࢦࣔ  ͰจຒΊࠐΈΛੜ੒Մೳ •ଟ༷ͳσʔληοτऩू  →ࢦࣔΛΞϊςʔτ (MEDI dataset)  →ࢦࣔ+จΛຒΊࠐΈදݱʹ  →ରরֶशͰϞσϧΛ܇࿅ •MTEBͳͲෳ਺ͷϕϯνϚʔΫͰฏۉͯ͠ੑೳ޲্ ֓ཁ 2

Slide 3

Slide 3 text

•࠷ۙจΛquery, passageͰ৔߹෼͚ͯ͠ຒΊࠐΉϞσϧ͕ڧ͍(E5) • LLM+ICLͷ༂ਐ΋͋ΓɺInstructOR͸ͦͷࣗવͳൃలʹࢥ͑Δ • ʮͲ͏΍ͬͯจΛຒΊࠐΉ͔ʯΛϢʔβ͕ૢ࡞Ͱ͖Δ࣌୅͕དྷͨ •ݸਓతʹ͔ͳΓ޷͖ͳ࿩ ໔੹ࣄ߲ •εϥΠυதͷਤද͸֤εϥΠυͰݴٴ͞Ε͍ͯΔ࿦จ͔ΒͷҾ༻Ͱ͢ •࿦จதͷ਺ࣜͱ͸ҟͳΔจࣈΛ࢖͍ͬͯΔ৔߹͕͋Γ·͢ બఆཧ༝ 3

Slide 4

Slide 4 text

•܇࿅ख๏ •MEDI dataset •ධՁ࣮ݧ ໨࣍ 4

Slide 5

Slide 5 text

•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 5 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...]

Slide 6

Slide 6 text

•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 6 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ͸ ۙ͘ʹ෼෍ ϕΫτϧؒͷڑ཭͕  ҙຯతͳؔ܎Λදݱ

Slide 7

Slide 7 text

•จຒΊࠐΈ͸Out Of Domain (OOD) ʹऑ͍ •ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧ  InstructORΛఏҊ • Instruction-based Omnifarious Representations • จ͝ͱʹࢦࣔΛม͑ଟ༷ͳຒΊࠐΈΛੜ੒ •ࢦ͕ࣔ͋Δ͜ͱͰOOD΁ͷੑೳ΋޲্ ख๏֓ཁ •ଟ༷ͳσʔληοτऩू  →ࢦࣔΛΞϊςʔτ (MEDI dataset)  →ࢦࣔ+จΛຒΊࠐΈදݱʹ  →ରরֶशͰϞσϧΛ܇࿅ ख๏֓ཁ 7

Slide 8

Slide 8 text

•ൺֱతγϯϓϧͳରরֶशʹΑΔ܇࿅ •ࢦࣔ+จΛ·Δ͝ͱϞσϧʹೖྗ͢Δ • ࢦࣔΛߟྀͨ͠จຒΊࠐΈΛੜ੒ ܇࿅खॱ 1. ࢦࣔͱจͷϖΞ(x, Ix, y, Iy)Λ༻ҙ 2. Ex = E(Ix⊕x), Ey = E(Iy⊕y)ͷΑ͏ʹຒΊࠐΉ 3. ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹରরֶश •ෛྫͱͯ͠in-batch negativesΛར༻ InstructOR: ܇࿅खॱ 8 Ix⊕x Iy⊕y Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ ଛࣦؔ਺

Slide 9

Slide 9 text

•܇࿅ख๏ •MEDI dataset •ධՁ࣮ݧ ໨࣍ 9

Slide 10

Slide 10 text

•InstructORͷ܇࿅ʹ͸  ࢦࣔͱจ͕ϖΞʹͳͬͨσʔληοτ͕ඞཁ •طଘσʔληοτΛ܇࿅༻ʹ౷߹ (ܭ300ݸ) • Super-NaturalInstructions (super-NI) • Sentence Transformers embedding data MEDI: Multitask Embedding Data with Instructions 10

Slide 11

Slide 11 text

•InstructORͷ܇࿅ʹ͸  ࢦࣔͱจ͕ϖΞʹͳͬͨσʔληοτ͕ඞཁ •طଘσʔληοτΛ܇࿅༻ʹ౷߹ (ܭ300ݸ) • Super-NaturalInstructions (super-NI) • Sentence Transformers embedding data •super-NI͸ࢦࣔͱจ͕ϖΞʹͳ͍ͬͯΔ͕  ਖ਼ྫɾෛྫ͕ଘࡏ͠ͳ͍ → Sentence-T5ͰจຒΊࠐΈΛੜ੒ (ࢦࣔͳ͠)  → ྨࣅ౓Λ࢖ͬͯਖ਼ෛྫϖΞΛࣗಈੜ੒ MEDI: Multitask Embedding Data with Instructions 11

Slide 12

Slide 12 text

•ςΩετ෼ྨܥͷσʔληοτ͔Βͷ࡞੒͸݁ߏָ खॱ 1. σʔληοτ͔Βೖྗจಉ࢜ͷྨࣅ౓͕ߴ͍จϖΞΛબͿ 2. ςΩετ෼ྨʹ͓͚Δϥϕϧ͕ಉ͡ͳΒਖ਼ྫɾҧ͏ͳΒෛྫʹ͢Δ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜ੒ख๏ 12

Slide 13

Slide 13 text

•Seq2Seqܥͷσʔληοτ͸ϥϕϧ͕ͳ͍ͷͰ޻෉͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷΋Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜ੒ख๏ 13 ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗ͸ࣅ͍ͯΔ͕ग़ྗ͸ࣅ͍ͯͳ͍ͱߴείΞ

Slide 14

Slide 14 text

•Seq2Seqܥͷσʔληοτ͸ϥϕϧ͕ͳ͍ͷͰ޻෉͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷΋Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜ੒ख๏ 14 ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗ͸ࣅ͍ͯΔ͕ग़ྗ͸ࣅ͍ͯͳ͍ͱߴείΞ

Slide 15

Slide 15 text

•Seq2Seqܥͷσʔληοτ͸ϥϕϧ͕ͳ͍ͷͰ޻෉͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷΋Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜ੒ख๏ 15 ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗ͸ࣅ͍ͯΔ͕ग़ྗ͸ࣅ͍ͯͳ͍ͱߴείΞ

Slide 16

Slide 16 text

•Seq2Seqܥͷσʔληοτ͸ϥϕϧ͕ͳ͍ͷͰ޻෉͕͍Δ •ਖ਼ྫ༻ͷείΞSpos ͱෛྫ༻ͷείΞSneg Λ༻ҙ •࠷΋Spos ͕ߴ͍ϖΞΛਖ਼ྫɺSneg ͕ߴ͍ϖΞΛෛྫʹ MEDI: ਖ਼ྫɾෛྫϖΞͷࣗಈੜ੒ख๏ 16 ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗͱग़ྗ͕ڞʹྨࣅ͍ͯ͠ΔͱߴείΞ ೖྗจͷྨࣅ౓ ग़ྗจͷྨࣅ౓ ೖྗ͸ࣅ͍ͯΔ͕ग़ྗ͸ࣅ͍ͯͳ͍ͱߴείΞ hard negative  తͳཱͪҐஔ

Slide 17

Slide 17 text

•ࢦࣔʹجͮ͘จຒΊࠐΈͷͨΊʹ͸ɺࢦ͕ࣔඞཁ(౰વ) •ҎԼͷ৘ใͱςϯϓϨʔτʹج͍ͮͯσʔληοτ͝ͱʹࢦࣔΛ࡞੒ •Text Type • ຒΊࠐΉจ͕Ͳ͏͍͏΋ͷ͔ • QAͷqueryͳΒʮ࣭໰จʯ • passageͳΒʮจॻʯ •Task Objective (optional) • ຒΊࠐΈ͕Կʹ࢖ΘΕΔ͔ •Domain (optional) • ʮχϡʔεʯͳͲจͷυϝΠϯ MEDI: ࢦࣔͷߏங 17

Slide 18

Slide 18 text

•ࢦࣔ͸ҎԼͷςϯϓϨʔτ͔Βߏங •“Represent The (DOMAIN) TEXT TYPE for TASK OBJECTIVE:.” •࣮ࡍͷࢦ͕ࣔҎԼ • ॏཁ: จͷλΠϓʹΑͬͯࢦࣔͰຒΊࠐΈํΛม͑ΒΕΔ MEDI: ࢦࣔͷߏங 18 ্ද͸࿦จதͷද͔Βൈਮͨ͠΋ͷ

Slide 19

Slide 19 text

•ଟ༷ͳσʔληοτͷจϖΞ͔Β  ͳΔେن໛ͳσʔληοτ •ྨࣅ౓͕ରশ/ඇରশͳ΋ͷ͕  ͍ࠞͬͯ͟Δ • ྫ: ݕࡧͷquery—passage •ΞελϦεΫ(*)͕͍͍ͭͯΔ  σʔληοτ͸test setΛؚΉ • ஫ҙ:  NQ, HotPotQA͸ޙड़͢ΔMTEB  ʹؚ·Ε͓ͯΓɺϦʔΫ͍ͯ͠Δ MEDI: σʔληοτͷ಺༁ 19

Slide 20

Slide 20 text

•ࢦࣔʹैͬͯλεΫΛղ͚ΔΑ͏ʹLMΛtuning (Instruction Tuning) •zero-shotͰ৭ʑͳੜ੒ܥλεΫ͕͏·͘ղ͚Δ •InstructOR͸FLANͷຒΊࠐΈ൛ͱ΋ݴ͑Δ •FLAN͸Finetuned Language Net ͷུΒ͍͠🧐 Wei+: Finetuned Language Models Are Zero-Shot Learners, ICLR ’22 ؔ࿈ݚڀ: FLAN 20 Flan-T5, Flan-UL2 ͸ΊͪΌͪ͘Όڧ͍ϞσϧͳͷͰ͓͢͢Ίʂ(͜ͷ࣌୅Ͱ΋·ͩڧ͍)

Slide 21

Slide 21 text

•܇࿅ख๏ •MEDI dataset •ධՁ࣮ݧ ໨࣍ 21

Slide 22

Slide 22 text

•ෳ਺ͷϕϯνϚʔΫͰ޿ൣʹੑೳΛධՁ • MTEB: จຒΊࠐΈͷେن໛ϕϯνϚʔΫ • Billboard (ʹΑΔධՁ): ੜ੒͞Εͨཁ໿ͱਖ਼ղཁ໿ͱͷྨࣅ౓ͷ૬ؔ • Prompt Retrieval: ੜ੒ʹ࠷దͳϓϩϯϓτͷݕࡧ ධՁ࣮ݧ 22

Slide 23

Slide 23 text

•58ͷσʔληοτɺ112ͷݴޠ͔Β  ͳΔจຒΊࠐΈͷϕϯνϚʔΫ •STSΛ͸͡Ί8छྨͷλεΫ͕͋Δ •(༨ஊ) MTEBͷ࣮ݧ݁Ռ͔Β  จຒΊࠐΈͰ΋σΧ͍Ϟσϧ  ͷํ͕ڧ͍܏޲͕ݟ͑Δ •ධՁ؍఺: InstructOR͕  ͲΕ͘Β͍ྑ͍จຒΊࠐΈΛੜ੒Ͱ͖Δ͔ •ಛʹɺࢦࣔΛ෇Ճ͢Δ͜ͱͰ  طଘख๏ΑΓੑೳ͕޲্͢Δ͔͕ؾʹͳΔ Muennigho ff +: MTEB: Massive Text Embedding Benchmark, arXiv ’22 ධՁ࣮ݧ: Massive Text Embedding Benchmark (MTEB) 23 Sentence-BERT ͷஶऀͷ Nils Reimers ͕last authorͷproject

Slide 24

Slide 24 text

•ੜ੒ϞσϧɾධՁࢦඪͷ૒ํΛ  ొ࿥Ͱ͖ΔϦʔμʔϘʔυ(ϑϨʔϜϫʔΫ) • ֤ࢦඪΛΞϯαϯϒϧͨ͠ࢦඪ΋ࣗಈࢉग़ • ੜ੒Ϟσϧͷ։ൃͱධՁΛ૬ޓʹଅਐ •ධՁ؍఺: ੜ੒จ—ਖ਼ղཁ໿ͷྨࣅ౓͕  ͲΕ͘Β͍ਓؒධՁͱ͍͔ۙ •ੜ੒จͱ֤ਖ਼ղཁ໿ͱͷcosྨࣅ౓ͷ  ࠷େ஋ͱਓखධՁͱͷϐΞιϯ૬ؔ •3ͭͷσʔληοτͰͦΕͧΕ૬ؔ܎਺Λࢉग़ɺฏۉΛධՁ஋ʹ Kasai+: Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand, NAACL ’22 ධՁ࣮ݧ: Billboard (Ͱͷཁ໿ͷྨࣅ౓ධՁ) 24

Slide 25

Slide 25 text

•In-Context Learning͸ςετ࣌ͷfew-shotࣄྫΛબͿඞཁ͕͋Δ • ࣄલʹগ਺ʹߜΔͷ͸େมɾͲΕ͕͍͍͔Θ͔Βͳ͍ •few-shotࣄྫΛຒΊࠐΈʹม׵ˠςετࣄྫʹ͍ۙࣄྫΛೖΕͯੑೳ޲্ •ධՁ؍఺: ੜ੒͕ͲΕ͚ͩ͏·͘ग़དྷΔ͔ • ੑೳͷد༩͢Δྑ͍few-shotࣄྫΛ࣋ͬͯདྷΕΔ͔Ͳ͏͔͕ධՁई౓ Su+: Selective Annotation Makes Language Models Better Few-Shot Learners, arXiv ’22 ධՁ࣮ݧ: Prompt Retrieval 25 ίϝϯτ: LLM࣌୅ͷSentEvalతͳཱͪҐஔʹݟ͑Δ (SentEval: ຒΊࠐΈϕʔεͷઢܗ෼ྨثΛֶशɾͦͷਫ਼౓ͰධՁ) ੜ੒͸GPT-J

Slide 26

Slide 26 text

SimCSE: ҟͳΔDropoutΛద༻ͨ͠ಉ͡จΛਖ਼ྫ or ؚҙؔ܎ͷจϖΞΛਖ਼ྫͱͨ͠ରরֶश Contriever: ϕΫτϧݕࡧʹ͓͚ΔରরֶशͷݶքΛௐࠪɼMS MARCOͰͷ fi ne-tuningͰੑೳ޲্֬ೝ GTR: ScalingͰจຒΊࠐΈͰ΋ϕΫτϧݕࡧੑೳ & ൚Խੑೳ޲্Λ֬ೝ coCondenser: ࣄલ܇࿅࣌ʹίʔύεґଘͷଛࣦΛՃ͑Retrieverͷ܇࿅Λؤ݈ʹ&ੑೳ޲্ [00] Sentence-T5: T5ΛNLI/QAσʔλͰରরֶशʹΑΔ fi ne-tuning → scaling lawͷௐࠪ + SentGLUEͰධՁ SGPT: GPTΛ৘ใݕࡧλεΫʹར༻ɼCross-EncoderܗࣜͱBi-Encoderܗࣜͷ྆ํΛBEIRͰ࣮ݧ Gao+: SimCSE: Simple Contrastive Learning of Sentence Embeddings, EMNLP ’21 Izacard+: Unsupervised Dense Information Retrieval with Contrastive Learning, TMLR ’23 Ni+: Large Dual Encoders Are Generalizable Retrievers, EMNLP ’22 Gao+: Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval, ACL ’22 Ni+: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models, CoRR ’21 Muennigho ff : SGPT: GPT Sentence Embeddings for Semantic Search, arXiv ’22 ධՁ࣮ݧ: ൺֱख๏ 26

Slide 27

Slide 27 text

•GTRΛ fi ne-tuning • Թ౓ύϥϝʔλ: 0.01 • ֶश཰: 2e-5 • Optimizer: AdamW ϛχόονͷαϯϓϦϯάख๏ •֤ϛχόον͸୯Ұͷσʔληοτ͔ΒͳΔ • λεΫ΍σʔληοτͷҧ͍Λֶश͠ͳ͍Α͏ʹ͢Δ ධՁ࣮ݧ: InstructORͷ܇࿅ઃఆ 27 in-batch negatives  ʹΑΔѱӨڹͷ௿ݮ

Slide 28

Slide 28 text

•InstructOR͸λεΫʹΑΒͣߴੑೳ • SimCSE͸STSͰ͸ߴੑೳ͕ͩଞͷλεΫͰ௿Ίͷੑೳ • GTR΍Contrierver͸ݕࡧͰߴੑೳ͕ͩSTS͕௿Ί •ࢦࣔΛ༩͑Δ͜ͱͰ֤λεΫʹదͨ͠ຒΊࠐΈΛੜ੒Ͱ͖͍ͯΔ ࣮ݧ݁Ռ: খ͞ΊͷϞσϧ 28 ্ද͸࿦จதͷදͷҰ෦Λൈਮͨ͠΋ͷ

Slide 29

Slide 29 text

•େ͖ΊͷϞσϧͰ΋ࢦࣔΛ༻͍ͨඍௐ੔Ͱੑೳ޲্ •GTR, InstructOR͸335M→1.5BͰੑೳ͕গ্͔͕͍ͬͯ͠͠ͳ͍ (58.4→58.8) • InstructORͷϕʔεϞσϧͷGTR͕ͪΐ͍ऑͦ͏ʁ ࣮ݧ݁Ռ: େ͖ΊͷϞσϧ 29 ্ද͸࿦จதͷදͷҰ෦Λൈਮͨ͠΋ͷ

Slide 30

Slide 30 text

•ࢦࣔͳ͠ͰඇରশͳจϖΞ(query & passageͳͲ)Λֶश͢Δͱੑೳ௿Լ • ࢦࣔΛೖΕΔ͜ͱͰΉ͠Ζੑೳ͕޲্͢Δ •super-NI͕͋Δ͜ͱͰࢦࣔʹର͢Δؤ݈ੑ͕޲্ ෼ੳ: ࢦࣔͷ༗ແʹΑΔੑೳͷมԽ & ؤ݈ੑ 30

Slide 31

Slide 31 text

•InstructOR͸ࢦࣔͷৄࡉ౓Λ্͛Δ΄Ͳੑೳ޲্ •ϞσϧύϥϝʔλΛ૿΍͢΄Ͳੑೳ޲্ • ৳ͼํ͸େ͖͘͸ͳ͍ ෼ੳ: ࢦࣔͷৄࡉ౓ & ϞσϧαΠζʹΑΔੑೳͷมԽ 31 ৳ͼํ͕ऑ͍ͷͰScalingͱ  ߹Θͤͯผͷ޻෉͕͍Δ͔΋

Slide 32

Slide 32 text

•ࢦ͕ࣔ͋Δ͜ͱͰOODʹର͢Δੑೳ΋޲্ •ൃදऀίϝϯτ • ࢦࣔͳ͠Ͱ fi ne-tuningͨ͠GTRͱͷൺֱ͕ཉ͍͠ • ࢦࣔͷޮՌ͔ɺMEDIͷଟ༷ੑͷ͓ӄ͔֬৴͕࣋ͯͳ͍ ࢦࣔͷޮՌ: Domain Shift 32 ܇࿅࣌ʹΈ͍ͯͳ͍  υϝΠϯͷσʔληοτ

Slide 33

Slide 33 text

•ຒΊࠐΈΛT-SNEͰ2࣍ݩʹ •Ͳͷࢦ͔ࣔ͸ෆ໌ɺଟ෼ˣ •ࢦࣔΛ࢖͏͜ͱͰʮจߏ଄ʯ  Ͱ͸ͳ͘ʮײ৘ʯΛදͨ͠  ҐஔʹҠಈ͍ͯ͠Δ • ʮͲͷ؍఺ͰจΛຒΊࠐΉ͔ʯ  ΛϢʔβ͕ૢ࡞Ͱ͖͍ͯΔ ఆੑධՁ: ՄࢹԽ 33 ࢦࣔ͋Γ ࢦࣔͳ͠

Slide 34

Slide 34 text

•ࢦࣔʹैͬͯจΛຒΊࠐΉϞσϧInstructORΛఏҊ •ࢦࣔΛՃ͑Δ͜ͱͰطଘख๏ΑΓߴੑೳ ࢥͬͨ͜ͱɾؾʹͳΔ͜ͱ •ࢦࣔνϡʔχϯά͞ΕͨϞσϧ(Flan-T5ͱ͔)Λ  ϕʔεʹͨ࣌͠ͷੑೳͷมԽ • ࢦࣔʹै͏ೳྗ͕͋ΔϞσϧͷํ͕ڧ͘ͳΓͦ͏ • GTR΍InstructOR͸ຊ౰ʹࢦࣔʹै͍͑ͯΔʁ(ICL΍ΔͳΒDecoderܕ͸ʁ) •MEDI datasetͷࢦ͕ࣔςϯϓϨʔτʹج͍͍ͮͯΔͷͰunnatural • ࢦࣔΛ΋ͬͱଟ༷͔ͭࣗવʹͰ͖ͨΒΑΓຒΊࠐΈͷ࣭Λ্͛ΒΕͦ͏ ·ͱΊ 34