RAG × 評価で掴む！応答精度向上レシピ

勉強会「苦しんで学ぶLLM」の発表資料
https://digeon.connpass.com/event/363110/

Transcript

1. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ ৄࡉࢿྉ ໦ϊຊ࿨໵ גࣜձࣾ Digeon 2025 ೥ 8 ݄ 2 ೔ Contents 1 ͸͡Ίʹ 3 2 جૅ஌ࣝฤ 4 2.1 RAG γεςϜͱ͸Կ͔ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.1 ैདྷͷ LLM ͷݶք . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.2 RAG ͕ղܾ͢Δ໰୊ . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.3 RAG ͷશମΞʔΩςΫνϟ . . . . . . . . . . . . . . . . . . . . . 5 2.2 ϕΫτϧͱྨࣅ౓ͷجૅʢஈ֊తઆ໌ʣ . . . . . . . . . . . . . . . . . . 5 2.2.1 Level 0ɿ࠲ඪฏ໘ͰͷϕΫτϧʢߴߍ਺ֶϨϕϧʣ . . . . . . . . 5 2.2.2 Level 1ɿ಺ੵͱίαΠϯྨࣅ౓ʢཧ޻ܥ 1 ೥Ϩϕϧʣ . . . . . . 5 2.2.3 Level 2ɿߴ࣍ݩۭؒͰͷ௚ײతཧղ . . . . . . . . . . . . . . . . . 7 2.3 Embedding ͷຊ࣭ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3.1 ʮ୯ޠ → ਺஋ϕΫτϧʯͷҙຯ . . . . . . . . . . . . . . . . . . 7 2.3.2 Word2Vec→BERT→ ݱ୅ख๏ͷਐԽ . . . . . . . . . . . . . . . . 8 2.3.3 ͳͥҙຯతྨࣅੑ͕ܭࢉͰ͖Δͷ͔ . . . . . . . . . . . . . . . . . 8 2.4 Transformer ͷجૅʢRAG ཧղʹඞཁͳ෦෼ʣ . . . . . . . . . . . . . . 9 2.4.1 Attention ػߏͷ௚ײతઆ໌ . . . . . . . . . . . . . . . . . . . . . 9 2.4.2 ͳͥจ຺ΛཧղͰ͖Δͷ͔ . . . . . . . . . . . . . . . . . . . . . . 17 3 ݕࡧٕज़ৄઆ 17 3.1 ϕΫτϧݕࡧͷ਺ֶతجૅͱ࣮૷ . . . . . . . . . . . . . . . . . . . . . . 17 3.1.1 ϕΫτϧݕࡧͷ׬શཧղ . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.2 ϕΫτϧݕࡧͷ௕ॴͱ୹ॴ . . . . . . . . . . . . . . . . . . . . . . 18 3.2 BM25 ͷৄࡉղઆͱ࣮૷ . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.1 BM25 ͷ਺ֶతఆٛͱ௚ײతཧղ . . . . . . . . . . . . . . . . . . 19 3.2.2 BM25 ͷ۩ମతܭࢉྫ . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2.3 BM25 ͷ௕ॴͱ୹ॴ . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3 ϋΠϒϦουݕࡧ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3.1 ͳͥϋΠϒϦουݕࡧ͕ඞཁ͔ . . . . . . . . . . . . . . . . . . . 21 3.3.2 ϋΠϒϦουݕࡧͷ࣮૷ํࣜ . . . . . . . . . . . . . . . . . . . . . 22 3.4 Embedding ʹΑΔυΩϡϝϯτબ୒ͷྲྀΕ . . . . . . . . . . . . . . . . . 24 3.4.1 લॲཧεςοϓ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1
2. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 2 3.4.2 ݕࡧ࣌ͷॲཧϑϩʔ . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.3 νϟϯΫ෼ׂͱݕࡧઓུ . . . . . . . . . . . . . . . . . . . . . . . 24 4 ධՁख๏ 25 4.1 ධՁͷॏཁੑͱ՝୊ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1.1 ͳͥ RAG γεςϜͷධՁ͕ࠔ೉͔ . . . . . . . . . . . . . . . . . 25 4.1.2 ධՁͷ؍఺ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 ݕࡧੑೳͷධՁࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.1 ఻౷తͳ IR ࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.2 ϥϯΩϯάධՁࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3 ੜ੒඼࣭ͷධՁࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3.1 ࢀরϕʔεධՁ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3.2 ҙຯతධՁࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 RAG ಛ༗ͷධՁࢦඪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.1 ஧࣮ੑʢFaithfulnessʣ . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.2 ౴Ҋଥ౰ੑʢAnswer Relevancyʣ . . . . . . . . . . . . . . . . . . 29 4.4.3 ίϯςΩετద߹ੑʢContext Relevancyʣ . . . . . . . . . . . . . 30 4.4.4 LLM-as-a-Judge ͷݶքͱ஫ҙ఺ . . . . . . . . . . . . . . . . . . . 31 4.4.5 ൃలతͳධՁ؍఺ . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 5 ࣮ࡍʹಈ͔ͤΔίʔυྫ 32 5.1 ؀ڥߏங . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.1.1 Google Colab Ͱͷ࣮ߦखॱ . . . . . . . . . . . . . . . . . . . . . . 32 5.2 แׅత RAG ධՁσϞɿݕࡧ͔Βੜ੒ɺධՁ·Ͱ . . . . . . . . . . . . . 32 5.3 ຊষͷ·ͱΊͱ࣍ͷεςοϓ . . . . . . . . . . . . . . . . . . . . . . . . . 44 6 ࢀߟจݙɾิ଍ࢿྉ 46
3. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 3 1 ͸͡Ίʹ ຊࢿྉ͸ɺRAGʢRetrieval-Augmented GenerationʣγεςϜͷཧ࿦͔ΒධՁ·Ͱɺ࣮ ફతͳ஌ࣝͱٕज़Λशಘ͢Δ͜ͱΛ໨తͱ͍ͯ͠·͢ɻൃදऀࣗ਎͕ࠓճॳΊͯษڧ ͢Δ಺༰Ͱ͋ΔͨΊɺৄࡉࢿྉͱͯ͠ྫɾ௚ײతͳઆ໌ΛؚΊͨཧղ͠΍͍͢ߏ੒ͱ ͠·ͨ͠ɻ RAG ͷ֩৺తͳ֓೦Ͱ͋ΔϕΫτϧྨࣅ౓ͷஈ֊తղઆ͔Β࢝ΊɺBM25 ͱϕΫ τϧݕࡧΛൺֱͨ͠ϋΠϒϦουݕࡧͷཧ࿦ɺධՁख๏Λ঺հ͠·͢ɻ ࢿྉ࡞੒ʹ͍ͭͯɿ • ͜ͷࢿྉ͸ओʹ Claude Code ͕จষΛ࡞੒͠ɺൃදऀ͕ਫ਼ࠪɾߍӾɾՃචɾम ਖ਼͠·ͨ͠ɻޡΓ΍ݹ͍৘ใؚ͕·Ε͍ͯΔՄೳੑ͕͋Γ·͢ͷͰɺ࠷৽ͷ৘ ใ͸ެࣜυΩϡϝϯτ΍࿦จΛࢀর͍ͯͩ͘͠͞ɻ ର৅ಡऀɿ • RAG γεςϜͷجૅ͔Βֶͼ͍ͨํ • ධՁख๏Λମܥతʹཧղ͍ͨ͠ํ • ࣮૷Λ௨ͯ͡ཧղΛਂΊ͍ͨํ
4. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 4 2 جૅ஌ࣝฤ 2.1 RAG γεςϜͱ͸Կ͔ 2.1.1 ैདྷͷ LLM ͷݶք େن໛ݴޠϞσϧʢLarge Language Model, LLMʣͱ͸ɿ LLM ͸๲େͳςΩετ σʔλͰ܇࿅͞Εͨਓ޻஌ೳϞσϧͰɺࣗવͳจষͷཧղɾੜ੒Λߦ͏͜ͱ͕Ͱ͖· ͢ɻChatGPT ͷ GPT γϦʔζ΍ ClaudeɺGoogle Gemini ͳͲ͕୅දతͳྫͰ͢ɻ͜ ΕΒͷϞσϧ͸ɺΠϯλʔωοτ্ͷจॻɺॻ੶ɺ࿦จͳͲ͔Βֶश͠ɺਓؒͷΑ͏ ͳࣗવͳର࿩΍จষੜ੒ΛՄೳʹ͍ͯ͠·͢ɻ ͔͠͠ɺLLM ʹ͸ҎԼͷࠜຊతͳݶք͕͋Γ·͢ɿ • ஌ࣝΧοτΦϑ໰୊ɿ܇࿅σʔλͷ࣌఺·Ͱͷ৘ใ͔͠஌Βͳ͍ • Hallucinationʢݬ֮ʣ ɿࣄ࣮ʹج͔ͮͳ͍ɺ΋ͬͱ΋Β͍͠ӕΛੜ੒͢Δ • υϝΠϯݻ༗஌ࣝͷෆ଍ɿاۀ಺෦ͷจॻ΍ઐ໳෼໺ͷ࠷৽৘ใΛ஌Βͳ͍ ஌ࣝΧοτΦϑͷ۩ମྫɿ • GPT-4o(2025 ೥ 1 ݄ΞοϓσʔτҎ߱) ͸ 2024 ೥ 6 ݄ʢͨͩ͠ɺWeb ݕࡧػೳ ʹΑΓ࠷৽৘ใʹΞΫηεՄೳʣ ྫ͑͹ɺ ʮ2024 ೥ͷ࠷৽ͷ AI ن੍๏ʹ͍ͭͯڭ͑ͯʯͱ࣭໰ͯ͠΋ɺ͜ΕΒͷϞσ ϧ͸܇࿅࣌఺Ҏ߱ͷ৘ใΛ࣋ͨͳ͍ͨΊɺݹ͍৘ใΛ౴͑Δ͔ɺଘࡏ͠ͳ͍๏཯Λ࡞ Γग़ͯ͠͠·͏Մೳੑ͕͋Γ·͢ɻ Hallucination ͷ۩ମྫɿ • ଘࡏ͠ͳ͍ٕज़༻ޠɿ ʮQuantum-RAGʯ΍ʮNeuroVector EmbeddingʯͳͲɺ࣮ ࡏ͠ͳ͍ٕज़֓೦Λ૑࡞ 2.1.2 RAG ͕ղܾ͢Δ໰୊ RAGʢRetrieval-Augmented Generationʣ͸ɺ ʮݕࡧ֦ͯ͠ுͨ͠ੜ੒ʯͱ͍͏ҙຯͰɺ ҎԼͷ࢓૊ΈͰ্هͷ໰୊Λղܾ͠·͢ [1]ɿ 1. ֎෦৘ใͷࢀরɿ֎෦σʔλϕʔε͔Βؔ࿈จॻΛݕࡧ͠ɺLLM ͷ஌ࣝΛิ׬ 2. ࣄ࣮ʹجͮ͘ճ౴ɿݕࡧͨ͠จॻͷ಺༰Λࠜڌͱͯ͠ճ౴ੜ੒ʢHallucination ͷ཈੍ʣ 3. ΧελϜ஌ࣝͷ׆༻ɿاۀݻ༗ͷจॻ΍ϚχϡΞϧΛࢀরՄೳ ಛʹɺRAG γεςϜ͸ݕࡧ͞Ε࣮ͨࡍͷจॻΛࠜڌͱͯ͠ճ౴Λੜ੒͢ΔͨΊɺ LLM ୯ମͰͷճ౴ͱൺֱͯ͠ Hallucination ͕େ෯ʹ཈੍͞Ε·͢ɻ͜Ε͸ɺϞσϧ ͕ʮ஌Βͳ͍͜ͱΛ஌͍ͬͯΔ;ΓΛ͢ΔʯͷͰ͸ͳ͘ɺ ʮݕࡧͨ͠৘ใʹج͍ͮͯ ճ౴͢Δʯͱ͍͏੍໿͕ಇͨ͘ΊͰ͢ɻ
5. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 5 2.1.3 RAG ͷશମΞʔΩςΫνϟ RAG γεςϜ͸ҎԼͷ 3 ͭͷओཁίϯϙʔωϯτ͔Βߏ੒͞Ε·͢ɿ 1. จॻσʔλϕʔεɿݕࡧର৅ͱͳΔจॻ܈ʢPDFɺWeb ϖʔδɺࣾ಺จॻͳͲʣ 2. ݕࡧγεςϜɿϢʔβʔͷ࣭໰ʹؔ࿈͢ΔจॻΛݟ͚ͭΔγεςϜ 3. ੜ੒γεςϜɿݕࡧ݁ՌΛࢀরͯ͠ճ౴Λੜ੒͢Δ LLM ॲཧͷྲྀΕɿ࣭໰ʮ࠷৽ͷ AI ن੍ʹ͍ͭͯʯ→ ؔ࿈จॻΛݕࡧ → ݕࡧ݁ՌΛ LLM ʹ౉͢ → ݕࡧ݁Ռʹج͍ͮͯճ౴ੜ੒ 2.2 ϕΫτϧͱྨࣅ౓ͷجૅʢஈ֊తઆ໌ʣ RAG ͷݕࡧγεςϜΛཧղ͢ΔͨΊʹɺ·ͣϕΫτϧͱྨࣅ౓ʹ͍ͭͯஈ֊తʹֶ ͼ·͠ΐ͏ [2]ɻ 2.2.1 Level 0ɿ࠲ඪฏ໘ͰͷϕΫτϧʢߴߍ਺ֶϨϕϧʣ ϕΫτϧͱ͸ɿେ͖͞ͱ޲͖Λ࣋ͭྔͰ͢ɻ࠲ඪฏ໘Ͱ͸໼ҹͰදݱͰ͖·͢ɻྫɿ ఺ (3, 4) ΁ͷϕΫτϧ ⃗ v = (3, 4) ϕΫτϧͷ௕͞ɿ |⃗ v| = √ 32 + 42 = √ 9 + 16 = 5 (1) 2 ͭͷϕΫτϧͷؔ܎ɿ • ⃗ a = (3, 4) ͱ ⃗ b = (6, 8) ͸ಉ͡ํ޲Λ޲͍͍ͯΔ: ⃗ b = 2⃗ a • ⃗ a = (3, 4) ͱ ⃗ c = (4, −3) ͸௚ަ͍ͯ͠Δʢਨ௚ʣ 2.2.2 Level 1ɿ಺ੵͱίαΠϯྨࣅ౓ʢཧ޻ܥ 1 ೥Ϩϕϧʣ ಺ੵͷఆٛɿ 2 ͭͷϕΫτϧ ⃗ a = (a1 , a2 ) ⃗ b = (b1 , b2 ) ͷ಺ੵ͸࣍ͷΑ͏ʹఆٛ͞Ε·͢ɻ ⃗ a ·⃗ b = a1 b1 + a2 b2 (2)
6. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 6 زԿֶతҙຯɿ ⃗ a ·⃗ b = |⃗ a||⃗ b| cos θ (3) ͜͜Ͱ θ ͸ 2 ͭͷϕΫτϧͷͳ֯͢Ͱ͢ɻ ίαΠϯྨࣅ౓ɿ cos θ = ⃗ a ·⃗ b |⃗ a||⃗ b| (4) ߦྻͷੵͷجૅʢAttention ܭࢉͷ४උʣ ɿ ޙͷ Attention ܭࢉͰඞཁʹͳΔߦྻͷੵʹ͍ͭͯɺ͜͜Ͱ؆୯ʹઆ໌͠·͢ɻ • ߦྻͱϕΫτϧͷੵɿ A ͕ m × n ߦྻɺ⃗ x ͕ n ࣍ݩϕΫτϧͷͱ͖ (A⃗ x)i = n ∑ j=1 Aij xj (5) • ߦྻͱߦྻͷੵɿ A ͕ m × n ߦྻɺB ͕ n × p ߦྻͷͱ͖ɺੵ AB ͸ m × p ߦྻ (AB)ij = n ∑ k=1 Aik Bkj (6) • ۩ମྫʢ2×2 ͷ৔߹ʣ ɿ ( a b c d ) ( e f g h ) = ( ae + bg af + bh ce + dg cf + dh ) (7) • సஔߦྻɿ AT ͸ A ͷߦͱྻΛೖΕସ͑ͨߦྻ A = ( 1 2 3 4 ) ⇒ AT = ( 1 3 2 4 ) (8) ۩ମྫͰܭࢉͯ͠Έ·͠ΐ͏ɿ ⃗ a = (3, 4) ⃗ b = (6, 8) ͷ৔߹ɿ ⃗ a ·⃗ b = 3 × 6 + 4 × 8 = 18 + 32 = 50 (9) |⃗ a| = √ 32 + 42 = 5 (10) |⃗ b| = √ 62 + 82 = 10 (11) cos θ = 50 5 × 10 = 1 (12) cos θ = 1 ͳͷͰɺ θ = 0° ͭ·Γಉ͡ํ޲Λ޲͍͍ͯ·͢ɻ
7. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 7 2.2.3 Level 2ɿߴ࣍ݩۭؒͰͷ௚ײతཧղ ͳͥߴ࣍ݩ͕ඞཁ͔ɿ୯ޠ΍จॻΛ 2 ࣍ݩͰදݱ͢Δʹ͸৘ใ͕଍Γ·ͤΜɻݱ࣮త ʹ͸ 100ʙ1500 ࣍ݩͷϕΫτϧΛ࢖༻͠·͢ɻ ߴ࣍ݩͰͷྨࣅ౓ɿ • 2 ࣍ݩɿ ⃗ a = (a1 , a2 ) • ߴ࣍ݩɿ ⃗ a = (a1 , a2 , a3 , . . . , an ) ܭࢉํ๏͸ಉ͡Ͱ͢ɿ ⃗ a ·⃗ b = a1 b1 + a2 b2 + · · · + an bn = n ∑ i=1 ai bi (13) ௚ײతཧղɿ • ίαΠϯྨࣅ౓͕ 1 ʹ͍ۙ → ࣅͨํ޲ → ҙຯ͕ࣅ͍ͯΔ • ίαΠϯྨࣅ౓͕ 0 ʹ͍ۙ → ௚ަ → ؔ܎͕ͳ͍ • ίαΠϯྨࣅ౓͕-1 ʹ͍ۙ → ൓ରํ޲ → ରཱతͳҙຯ 2.3 Embedding ͷຊ࣭ 2.3.1 ʮ୯ޠ → ਺஋ϕΫτϧʯͷҙຯ ͳͥϕΫτϧͰදݱ͢Δͷ͔ɿίϯϐϡʔλ͸਺஋͔͠ཧղͰ͖·ͤΜɻ ʮݘʯ ʮೣʯ ʮࣗ ಈंʯͱ͍ͬͨ୯ޠΛɺίϯϐϡʔλ͕ʮࣅ͍ͯΔ/ࣅ͍ͯͳ͍ʯΛ൑அͰ͖Δܗʹ ม׵͢Δඞཁ͕͋Γ·͢ɻ naive ͳํ๏ͷ໰୊ɿ • ࣙॻॱɿ ʮݘʯ→ 1, ʮೣʯ→ 2, ʮࣗಈंʯ→ 3 • ໰୊ɿ ʮݘʯͱʮೣʯͷྨࣅੑ͕ʮݘʯͱʮࣗಈंʯͷྨࣅੑΑΓܭࢉͰ͖ͳ͍ Embedding ͷղܾࡦɿ୯ޠΛߴ࣍ݩϕΫτϧͰදݱ͠ɺྨࣅͨ͠ҙຯͷ୯ޠΛ ͍ۙҐஔʹ഑ஔɿ ʮݘʯ → (0.2, −0.1, 0.8, 0.3, . . .) (14) ʮೣʯ → (0.3, −0.2, 0.7, 0.4, . . .) (15) ʮࣗಈंʯ → (−0.5, 0.8, −0.1, −0.2, . . .) (16) ͜ͷ৔߹ɺ ʮݘʯͱʮೣʯͷίαΠϯྨࣅ౓͸ߴ͘ɺ ʮݘʯͱʮࣗಈंʯͷྨࣅ౓͸௿ ͘ͳΓ·͢ɻ
8. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 8 2.3.2 Word2Vec→BERT→ ݱ୅ख๏ͷਐԽ Word2Vecʢ2013 ೥ʣ ɿ[3] •ʮपғͷ୯ޠ͔Βத৺ͷ୯ޠΛ༧ଌʯ·ͨ͸ʮத৺ͷ୯ޠ͔ΒपғΛ༧ଌʯ • ྫɿ ʮೣ͕ڕΛ৯΂Δʯ→ ʮ͕ɺΛɺ৯΂Δʯ͔ΒʮೣɺڕʯΛ༧ଌ ݶքɿจ຺Λߟྀ͠ͳ͍ʢ ʮ͖͘ʯ͸ฉ͔͘ޮ͔͘Θ͔Βͳ͍ʣ BERTʢ2018 ೥ʣ ɿ[4] • Transformer ٕज़Λ࢖༻͠ɺจ຺Λߟྀͨ͠ Embedding • ྫɿ ʮԻָΛ͖͘ʯͱʮༀ͕͖͘ʯͰҟͳΔϕΫτϧ • ಛ௃ɿ૒ํ޲తͳจ຺ཧղʢલޙͷ୯ޠΛಉ࣌ʹߟྀʣ ݱ୅ख๏ɿ • 2024 ೥ʙ ɿOpenAI text-embedding-3-small/large ͸ɺ MRL ʢMatryoshka Rep- resentation Learningʣٕज़ʹΑΓ࣍ݩ਺Λॊೈʹௐ੔ՄೳͰɺ଎౓ͱਫ਼౓ ͷτϨʔυΦϑΛ࠷దԽͰ͖Δ • จॻશମͷҙຯΛଊ͑Δೳྗ޲্ • RAG ʹಛԽͨ͠࠷దԽʢݕࡧੑೳ޲্ʣ 2.3.3 ͳͥҙຯతྨࣅੑ͕ܭࢉͰ͖Δͷ͔ ෼෍Ծઆɿ ʮࣅͨจ຺Ͱ࢖ΘΕΔ୯ޠ͸ࣅͨҙຯΛ࣋ͭʯ ۩ମྫɿ •ʮݘ͸๸͑Δʯ ʮೣ͸໐͘ʯ→ ಈ෺ͷߦಈύλʔϯ •ʮϦϯΰ͸੺͍ʯ ʮόφφ͸ԫ৭͍ʯ→ Ռ෺ͷ৭ύλʔϯ •ʮ౦ژ͸େ౎ࢢͩʯ ʮϩϯυϯ͸େ౎ࢢͩʯ→ ౎ࢢͷಛ௃ύλʔϯ ͜ΕΒͷύλʔϯΛେྔͷςΩετ͔Βֶश͢Δ͜ͱͰɺ୯ޠͷҙຯؔ܎ΛϕΫτϧ ۭؒʹຒΊࠐΉ͜ͱ͕Ͱ͖·͢ɻ ਺ֶత௚ײɿߴ࣍ݩۭؒͰ͸ɺҙຯతʹࣅͨ֓೦ಉ͕࢜ʮΫϥελʔʯΛܗ੒͠ ·͢ɻ Ϋϥελʔͱ͸ɿΫϥελʔͱ͸ɺσʔλϙΠϯτ͕ີू͍ͯ͠ΔྖҬͷ͜ͱͰ ͢ɻ͜͜Ͱݴ͏σʔλϙΠϯτͱ͸ɺϕΫτϧ্ۭؒͷ 1 ͭ 1 ͭͷ఺ͷ͜ͱͰɺ͜ͷ จ຺Ͱ͸ʮ֤୯ޠͷϕΫτϧʯΛࢦ͠·͢ɻྫ͑͹ɺ ʮೣʯͱ͍͏୯ޠ͕ 300 ࣍ݩͷϕ Ϋτϧ (0.2, −0.1, 0.8, ...) Ͱදݱ͞Ε͍ͯΔͱ͖ɺ͜ͷϕΫτϧ͕ 300 ࣍ݩ্ۭؒͷ 1 ͭͷσʔλϙΠϯτʢ఺ʣͱͳΓ·͢ɻ ॏཁɿͳͥࣅͨҙຯͷ୯ޠ͕ۙ͘ʹ഑ஔ͞ΕΔͷ͔ʁ͜Ε͸୯ͳΔۮવͰ͸ͳ͘ɺ ֶशͷաఔͰҙਤతʹͦ͏ͳΔΑ͏ʹ܇࿅͞ΕΔ͔ΒͰ͢ɻWord2Vec ΍ BERT ͳͲ ͷϞσϧ͸ɺ ʮࣅͨจ຺Ͱ࢖ΘΕΔ୯ޠ͸ࣅͨҙຯΛ࣋ͭʯͱ͍͏Ծઆʢ෼෍Ծઆʣʹ جֶ͍ͮͯश͠·͢ɻ ֶशͷ࢓૊Έʢ؆ུԽʣ ɿ
9. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 9 1. େྔͷςΩετ͔Βʮೣ͸ϛϧΫΛҿΉʯ ʮݘ͸ਫΛҿΉʯͷΑ͏ͳจΛऩू 2.ʮೣʯͱʮݘʯ͕ࣅͨจ຺ʢ ʮʙΛҿΉʯ ʣͰ࢖ΘΕΔ͜ͱΛݕग़ 3. ܇࿅Λ௨ͯ͡ʮೣʯͱʮݘʯͷϕΫτϧΛۙ͘ʹ഑ஔ͢ΔΑ͏ʹௐ੔1 4. ݁Ռͱͯ͠ɺࣅͨҙຯͷ୯ޠ͕ࣗવͱΫϥελʔΛܗ੒ ͦͷ݁Ռɺߴ࣍ݩͷϕΫτϧۭؒͰ͸ɺࣅͨҙຯΛ࣋ͭ୯ޠͷϕΫτϧ͕ޓ͍ʹ ۙ͘ʹ഑ஔ͞Εɺ·ͱ·ΓʢΫϥελʔʣΛܗ੒͠·͢ɻྫ͑͹ɿ • ಈ෺Ϋϥελʔɿ ʮݘʯ ʮೣʯ ʮ͏͗͞ʯ ʮϥΠΦϯʯͳͲͷϕΫτϧ͕ۙ͘ʹू ·Δ • Ռ෺Ϋϥελʔɿ ʮΓΜ͝ʯ ʮΈ͔Μʯ ʮόφφʯ ʮͿͲ͏ʯͳͲͷϕΫτϧ͕ۙ ͘ʹू·Δ • ৐Γ෺Ϋϥελʔɿ ʮंʯ ʮόεʯ ʮిंʯ ʮඈߦػʯͳͲͷϕΫτϧ͕ۙ͘ʹू ·Δ 2.4 Transformer ͷجૅʢRAG ཧղʹඞཁͳ෦෼ʣ 2.4.1 Attention ػߏͷ௚ײతઆ໌ ਓؒͷ஫ҙػߏͱͷྨൺɿ ʮೣ͕ڕΛ৯΂Δʯͱ͍͏จΛཧղ͢Δ࣌ɺਓؒ͸ɿ •ʮೣʯΛཧղ͢Δ࣌ɺ ʮ৯΂Δʯ΍ʮڕʯʹ஫ҙΛ޲͚Δ •ʮ৯΂ΔʯΛཧղ͢Δ࣌ɺ ʮ୭͕ʯ ʢೣʣ ʮԿΛʯ ʢڕʣʹ஫ҙΛ޲͚Δ Attention ͷ਺ֶతఆٛʢ؆ུ൛ʣ ɿ ֤୯ޠ͕ଞͷ͢΂ͯͷ୯ޠʹରͯ͠ʮͲΕ͘Β͍஫ҙΛ޲͚Δ͔ʯΛܭࢉ [5]ɿ Attention(Q, K, V ) = softmax ( QKT √ dk ) V (17) ه߸ͷఆٛɿ • Q ∈ Rn×dk ɿQuery ߦྻʢ ʮԿʹ͍ͭͯ஌Γ͍͔ͨʯΛද͢ߦྻʣ • K ∈ Rm×dk ɿKey ߦྻʢ ʮԿ͕ར༻Մೳ͔ʯΛද͢ߦྻʣ • V ∈ Rm×dv ɿValue ߦྻʢ ʮ࣮ࡍͷ৘ใʯΛද͢ߦྻʣ • dk ɿKey/Query ϕΫτϧͷ࣍ݩ਺ʢεΧϥʔ஋ʣ • KT ∈ Rdk×mɿK ͷసஔߦྻ • nɿQuery ͷ਺ʢ௨ৗ͸୯ޠ਺ʣ ɺmɿKey/Value ͷ਺ Attention ؔ਺ͷ஋Ҭɿ Attention(Q, K, V ) : Rn×dk × Rm×dk × Rm×dv → Rn×dv (18) தؒܭࢉͷܗঢ়ɿ 1ৄࡉͳ਺ཧతഎܠʹ͍ͭͯ͸ຊࢿྉͰ͸ׂѪ͠·͢ɻ
10. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 10 • QKT ∈ Rn×mɿ֤ Query ͱ Key ͷؔ࿈౓είΞߦྻ • softmax ( QKT √ dk ) ∈ Rn×mɿਖ਼نԽ͞Εͨ஫ҙॏΈߦྻʢ֤ߦͷ࿨ =1ʣ • ࠷ऴग़ྗɿRn×dv ߦྻʢ֤ Query ʹର͢Δจ຺ߟྀࡁΈදݱʣ ͜͜Ͱɺsoftmax ؔ਺͸ҎԼͷΑ͏ʹఆٛ͞Ε·͢ɻ ೖྗϕΫτϧ x = (x1 , x2 , ..., xn )T ∈ Rn ʹରͯ͠ɿ softmax(x)i = exi ∑ n j=1 exj (19) softmax ͷه߸ͷఆٛɿ • x ∈ RnɿೖྗϕΫτϧʢn ࣍ݩ࣮਺ϕΫτϧʣ • softmax(x)i ɿग़ྗϕΫτϧͷ i ൪໨ͷཁૉ • xi ɿೖྗϕΫτϧ x ͷ i ൪໨ͷཁૉ • eɿωΠϐΞ਺ʢࣗવର਺ͷఈɺ໿ 2.718ʣ • exi ɿࢦ਺ؔ਺ʢe ͷ xi ৐ʣ softmax ؔ਺͸ɺ೚ҙͷ࣮਺஋ͷϕΫτϧΛ֬཰෼෍ʢ߹ܭ͕ 1 ʹͳΔඇෛͷ ஋ʣʹม׵͠·͢ɻ͜ΕʹΑΓɺ֤୯ޠ΁ͷʮ஫ҙͷॏΈʯ͕֬཰ͱͯ͠දݱ ͞Ε·͢ɻ ۩ମతͳܭࢉྫɿ ೖྗϕΫτϧ x = (2.0, 1.0, 0.1)T ʹରͯ͠ softmax Λܭࢉɿ ex1 = e2.0 ≈ 7.39 (20) ex2 = e1.0 ≈ 2.72 (21) ex3 = e0.1 ≈ 1.11 (22) ߹ܭ = 7.39 + 2.72 + 1.11 = 11.22 (23) ͕ͨͬͯ͠ɿ softmax(x)1 = 7.39 11.22 ≈ 0.66 (24) softmax(x)2 = 2.72 11.22 ≈ 0.24 (25) softmax(x)3 = 1.11 11.22 ≈ 0.10 (26) ग़ྗϕΫτϧ (0.66, 0.24, 0.10)T ͸֬཰෼෍ͱͳΓɺ߹ܭ͸ 1.00 ʹͳΓ·͢ɻ࠷ ΋େ͖ͳ஋ʢ2.0ʣʹରԠ͢Δཁૉ͕࠷΋ߴ͍֬཰ʢ0.66ʣΛ࣋ͪ·͢ɻ softmax ؔ਺ͷิ଍આ໌ɿ 1. ؔ਺໊ͷ༝དྷɿ softmax ͷ໊લ͸ʮsoft maximumʯʹ༝དྷ͍ͯ͠·͢ɿ • hard maxɿ࠷େ஋ͷΈΛ 1ɺଞΛ 0 ʹ͢Δ → [3.0, 1.0, 2.0] → [1, 0, 0] • soft maxɿ࠷େ஋ʹߴ͍֬཰Λ༩͑Δ͕ɺଞ΋ 0 ʹ͠ͳ͍ → [3.0, 1.0, 2.0] → [0.67, 0.09, 0.24]
11. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 11 •ʮsoftʯ͸࿈ଓతͰඍ෼ՄೳͳʮॊΒ͔͍ʯܾఆΛҙຯ͢Δ 2. Թ౓ύϥϝʔλɿ࣮ࡍʹ͸ softmax ʹ͸Թ౓ύϥϝʔλ T ΛಋೖͰ͖·͢ɿ softmax(x)i = exi/T ∑ n j=1 exj/T (27) Թ౓ͷޮՌɿ • T → 0ʢ௿Թʣ ɿhard max ʹۙͮ͘ʢ࠷େ஋ͷΈ͕ 1 ʹۙ͘ͳΔʣ • T = 1ɿඪ४తͳ softmaxʢ௨ৗͷࣜʣ • T → ∞ʢߴԹʣ ɿuniform ෼෍ʹۙͮ͘ʢ͢΂ͯ 1/n ʹۙ͘ͳΔʣ 3. ͳͥࢦ਺ؔ਺Λ࢖͏ͷ͔ɿ • ਖ਼ͷ஋อূɿ ex > 0ʢͲΜͳ࣮਺ x Ͱ΋ʣ→ ֬཰ͱͯ͠ద੾ • ࠩͷ֦େɿೖྗͷখ͞ͳࠩΛग़ྗͰେ͖֦͘େ → ࠷େ஋ΛΑΓ໌֬ʹڧ ௐ • ඍ෼ͷ؆ܿੑɿ (ex)′ = ex → ػցֶशͷޯ഑ܭࢉ͕ޮ཰త • ౷ܭతࣗવੑɿ࠷େΤϯτϩϐʔݪཧ͔Βࣗવʹಋग़͞ΕΔ Attention ͷ 2 ͭͷ໾ׂɿSelf-Attention ͱ Cross-Attention Attention ʹ͸େ͖͘෼͚ͯ 2 ͭͷ࢖ΘΕํ͕͋Γ·͢ɿ 1. Self-Attentionʢࣗݾ஫ҙػߏʣ • ໨తɿ 1 ͭͷೖྗʢྫɿ1 ͭͷจɺ1 ͭͷจॻʣͷ಺෦Ͱจ຺Λཧղ • Q, K, V ͷؔ܎ɿ͢΂ͯಉ͡ೖྗ͔Β࡞ΒΕΔ • RAG Ͱͷ࢖ΘΕํɿ – ຒΊࠐΈੜ੒࣌ɿจॻΛϕΫτϧԽ͢Δࡍʹจॻશମͷจ຺Λଊ͑Δ – ճ౴ੜ੒࣌ɿLLM ͕ੜ੒தͷ୯ޠͱաڈͷ୯ޠͷؔ܎ੑΛ൑அ 2. Cross-Attentionʢަࠩ஫ҙػߏʣ • ໨తɿ 2 ͭͷҟͳΔೖྗͷؒͰɺยํ͕΋͏ยํͷͲͷ෦෼ʹؔ࿈͍ͯ͠ Δ͔Λܭࢉ • Q, K, V ͷؔ܎ɿ Q ͕͋Δೖྗ͔ΒɺK ͱ V ͕ผͷೖྗ͔Β࡞ΒΕΔ • RAG Ͱͷ࢖ΘΕํɿ ʮ࣭໰ʯͱʮݕࡧ͞Εͨจॻ܈ʯΛؔ࿈෇͚ͯճ౴ੜ ੒ʢRAG ͷ֩৺෦෼ʣ Self-Attention ͷ۩ମతܭࢉྫɿ ʮೣ͕ڕΛ৯΂Δʯͱ͍͏จͰ Self-Attention ܭࢉɿ • 1. ຒΊࠐΈੜ੒࣌ɿจॻ΍ΫΤϦΛϕΫτϧԽ͢Δࡍɺ BERT ΍ Sentence- BERT ͳͲͷຒΊࠐΈϞσϧ಺෦Ͱ Attention ͕ಈ࡞͠ɺ୯ޠؒͷจ຺ؔ ܎Λଊ͑ͨີͳϕΫτϧදݱΛੜ੒
12. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 12 • 2. ճ౴ੜ੒࣌ɿ LLM ͕ݕࡧ͞Εͨจॻͱ࣭໰Λ౷߹ͯ͠ճ౴Λੜ੒͢Δ ࡍɺTransformer ΞʔΩςΫνϟ಺෦Ͱ Attention ͕ಈ࡞͠ɺ࣭໰ʹؔ࿈͢ Δจॻͷ෦෼ʹ஫ҙΛ޲͚ͳ͕ΒҰ؏ͨ͠ճ౴Λߏங • 3. จ຺ཧղɿݕࡧ͞Εͨෳ਺ͷจॻยʢνϟϯΫʣؒͷؔ܎ੑΛཧղ͠ɺ ໃ६͢Δ৘ใͷௐ੔΍ิ׬తͳ৘ใͷ౷߹Λߦ͏ ҎԼͷܭࢉྫ͸ɺ͜ΕΒͷॲཧͷجૅͱͳΔ Attention ϝΧχζϜͷಈ࡞ݪཧ Λ͍ࣔͯ͠·͢ɿ εςοϓ 1ɿߦྻͷઃఆ ೖྗσʔλͱ࣍ݩΛఆٛ͠ɺQɺKɺV ͷ 3 ͭͷߦྻΛॳظԽ͠·͢ɻ ୯ޠɿ[ʮೣʯ, ʮ͕ʯ, ʮڕʯ, ʮΛʯ, ʮ৯΂Δʯ]ɺdk = dv = 2ɺn = m = 5 ۩ମతͳҙຯɿ • QʢQueryʣ ɿ֤୯ޠ͕ʮԿΛ୳͍ͯ͠Δ͔ʯΛදݱ –ʮೣʯ: [1.0, 0.5] → ओޠͱͯ͠ಈ࡞ΛٻΊΔ –ʮڕʯ: [0.8, 0.9] → ໨తޠͱͯؔ͠܎ΛٻΊΔ –ʮ৯΂Δʯ: [2.0, 1.5] → ಈࢺͱͯ͠ओޠɾ໨తޠͱͷؔ܎Λڧ͘ٻΊΔ • KʢKeyʣ ɿ֤୯ޠ͕ʮͲΜͳ৘ใΛఏڙͰ͖Δ͔ʯΛදݱ –ʮೣʯ: [1.2, 0.6] → ಈ෺ɾओޠͷ৘ใΛఏڙ –ʮڕʯ: [0.9, 1.1] → ৯΂෺ɾ໨తޠͷ৘ใΛఏڙ –ʮ৯΂Δʯ: [1.8, 1.2] → ಈ࡞ɾؔ܎ͷ৘ใΛఏڙ • VʢValueʣ ɿ֤୯ޠͷʮ࣮ࡍͷҙຯ಺༰ʯΛදݱ –ʮೣʯ: [0.5, 0.3] → ಈ෺ͱͯ͠ͷಛ௃ –ʮڕʯ: [0.7, 0.8] → ৯΂෺ͱͯ͠ͷಛ௃ –ʮ৯΂Δʯ: [1.0, 0.9] → ಈ࡞ͷҙຯ಺༰ Q =       1.0 0.5 0.2 0.1 0.8 0.9 0.1 0.2 2.0 1.5       ∈ R5×2 (28) K =       1.2 0.6 0.1 0.3 0.9 1.1 0.2 0.4 1.8 1.2       ∈ R5×2 (29) V =       0.5 0.3 0.1 0.1 0.7 0.8 0.2 0.1 1.0 0.9       ∈ R5×2 (30)
13. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 13 εςοϓ 2ɿQKT ͷܭࢉ ֤ΫΤϦʢQʣͱ֤ΩʔʢKʣͷྨࣅ౓ΛଌΔͨΊɺߦྻੵ QKT Λܭࢉ͠·͢ɻ QKT = Q × KT =       1.0 0.5 0.2 0.1 0.8 0.9 0.1 0.2 2.0 1.5       × ( 1.2 0.1 0.9 0.2 1.8 0.6 0.3 1.1 0.4 1.2 ) (31) =       1.5 0.25 1.45 0.4 2.4 0.27 0.05 0.29 0.08 0.48 1.5 0.35 1.71 0.52 2.52 0.24 0.07 0.31 0.1 0.42 3.3 0.65 3.45 1.0 5.4       ∈ R5×5 (32) εςοϓ 3ɿεέʔϦϯά Ωʔͷ࣍ݩ਺ͷฏํࠜͰআࢉͯ͠ɺείΞͷ෼ࢄΛௐ੔͠ɺޯ഑ফࣦΛ๷͗· ͢ɻ QKT √ dk = QKT √ 2 ɿ QKT √ 2 =       1.06 0.18 1.03 0.28 1.70 0.19 0.04 0.21 0.06 0.34 1.06 0.25 1.21 0.37 1.78 0.17 0.05 0.22 0.07 0.30 2.34 0.46 2.45 0.71 3.83       (33) εςοϓ 4ɿsoftmaxʢߦ͝ͱʹద༻ʣ ֤ߦΛ֬཰෼෍ʹม׵͠ɺͲͷҐஔʹ஫ҙΛ޲͚Δ͔ͷॏΈΛܾఆ͠·͢ɻ ֤ߦʹ softmax Λద༻ͨ͠஫ҙॏΈߦྻɿ Attention Weight = softmax ( QKT √ dk ) (34) =       0.22 0.09 0.21 0.10 0.38 0.21 0.17 0.22 0.18 0.22 0.19 0.09 0.22 0.11 0.39 0.20 0.17 0.21 0.18 0.24 0.20 0.03 0.23 0.04 0.50       ∈ R5×5 (35) εςοϓ 5ɿ࠷ऴग़ྗߦྻ ஫ҙॏΈΛ࢖ͬͯόϦϡʔʢVʣΛॏΈ෇͖ฏۉ͠ɺจ຺ߟྀࡁΈͷදݱΛੜ੒ ͠·͢ɻ
14. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 14 Output = Attention Weight × V (36) =       0.22 0.09 0.21 0.10 0.38 0.21 0.17 0.22 0.18 0.22 0.19 0.09 0.22 0.11 0.39 0.20 0.17 0.21 0.18 0.24 0.20 0.03 0.23 0.04 0.50       ×       0.5 0.3 0.1 0.1 0.7 0.8 0.2 0.1 1.0 0.9       (37) =       0.68 0.62 0.45 0.43 0.70 0.64 0.47 0.45 0.74 0.68       ∈ R5×2 (38) ݁Ռͷղऍɿ • ֤ߦ͸ରԠ͢Δ୯ޠͷ৽͍͠දݱϕΫτϧʢจ຺ߟྀࡁΈʣ • 5 ߦ໨ʮ৯΂Δʯ: [0.74, 0.68] → ࣗ෼ࣗ਎ (0.50)ɺ ʮڕʯ(0.23)ɺ ʮೣʯ(0.20) ͷ৘ใΛॏΈ෇͖౷߹ • 1 ߦ໨ʮೣʯ: [0.68, 0.62] → ʮ৯΂Δʯ(0.38) ͷจ຺Λڧ͘൓ө • ग़ྗߦྻશମ͕ R5×2 Ͱɺ֤୯ޠͷจ຺ߟྀࡁΈදݱ Cross-Attention ͷ؆қܭࢉྫɿ RAG γεςϜʹ͓͚Δ Cross-Attention ܭࢉʢ࣭໰ͱݕࡧจॻؒͷؔ܎ੑʣ ɿ ઃఆɿ • Ϣʔβʔͷ࣭໰ɿ ʮRAG ͷϝϦοτ͸ԿͰ͔͢ʁ ʯ • ݕࡧ͞Εͨจॻ܈ɿ – จॻ Aɿ ʮRAG ͸ LLM ͷϋϧγωʔγϣϯʢݬ֮ʣΛ཈੍͢ΔޮՌ ͕͋Γ·͢ɻ ʯ – จॻ Bɿ ʮϕΫτϧݕࡧ͸ɺςΩετͷҙຯతͳྨࣅੑʹج͍ͮͯ৘ใ Λ୳͠·͢ɻ ʯ – จॻ Cɿ ʮ֎෦σʔλϕʔεΛࢀর͢ΔͨΊɺRAG ͸࠷৽ͷ৘ใʹ΋ ରԠͰ͖·͢ɻ ʯ Cross-Attention ʹ͓͚Δ Q, K, V ͷ໾ׂɿ • QʢQueryʣ ɿ࣭໰ऀ – ݩςΩετɿ ʮRAG ͷϝϦοτ͸ԿͰ͔͢ʁ ʯ – ҙຯɿ ʮࢲ͸ʰRAG ͷϝϦοτʱʹ͍ͭͯ஌Γ͍ͨʯͱ͍͏ҙਤΛ࣋ ͭϕΫτϧ – ໾ׂɿ֤จॻʹରͯ͠ʮ͜ͷ৘ใ͸ࢲͷ࣭໰ʹؔ܎͋Γ·͔͢ʁ ʯͱ ໰͍߹Θͤ
15. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 15 • KʢKeyʣ ɿࡧҾɺݟग़͠ – ݩςΩετɿݕࡧ͞Εͨจॻ܈ʮจॻ A, B, Cʯ – ҙຯɿ֤จॻ͕ʮͲͷΑ͏ͳ৘ใΛ͍࣋ͬͯΔ͔ʯΛࣔ͢ࡧҾͷΑ͏ ͳϕΫτϧ – ྫɿ ʮจॻ A ͸ϋϧγωʔγϣϯ཈੍ʹ͍ͭͯʯ ʮจॻ B ͸ϕΫτϧ ݕࡧʹ͍ͭͯʯ • VʢValueʣ ɿ৘ใຊମ – ݩςΩετɿݕࡧ͞Εͨจॻ܈ʮจॻ A, B, Cʯ – ҙຯɿ֤จॻͷʮ৘ใͷ۩ମతͳ಺༰ͦͷ΋ͷʯΛද͢ϕΫτϧ – ໾ׂɿ࠷ऴతʹ౷߹͞ΕΔ࣮ࡍͷ৘ใ಺༰ Cross-Attention ܭࢉͷྲྀΕɿ εςοϓ 1ɿQ ͱ K ͷর߹ʢؔ࿈౓ܭࢉʣ ࣭໰ QʢRAG ͷϝϦοτ͸ʁ ʣ͕ɺ֤จॻͷ KʢࡧҾʣͱͷؔ࿈౓Λܭࢉɿ ؔ࿈౓είΞ = QKT (39) = ( ࣭໰ϕΫτϧ ) ×    จॻ A ࡧҾT จॻ B ࡧҾT จॻ C ࡧҾT    (40) = ( 2.1 0.3 1.8 ) (41) ݁Ռͷղऍɿ • Q ͱ Kʢจॻ A: ϋϧγωʔγϣϯ཈੍ʣ→ ؔ࿈౓ 2.1ʢߴʣ • Q ͱ Kʢจॻ B: ϕΫτϧݕࡧʣ→ ؔ࿈౓ 0.3ʢ௿ʣ • Q ͱ Kʢจॻ C: ࠷৽৘ใʣ→ ؔ࿈౓ 1.8ʢߴʣ εςοϓ 2ɿ஫ҙͷॏΈ෇͚ʢsoftmax ద༻ʣ ܭࢉ͞Εͨؔ࿈౓είΞΛ softmax ؔ਺Ͱ֬཰ʹม׵ɿ Attention Weight = softmax(QKT ) (42) = softmax ( 2.1 0.3 1.8 ) (43) = ( 0.6 0.1 0.3 ) (44) ݁Ռͷղऍɿ • จॻ A ͷॏΈ → 0.6ʢ࠷΋ॏཁʣ • จॻ B ͷॏΈ → 0.1ʢ΄΅ແؔ܎ʣ • จॻ C ͷॏΈ → 0.3ʢ΍΍ॏཁʣ εςοϓ 3ɿV ͷ౷߹ʢॏΈ෇͖ฏۉʣ ͜ͷॏΈΛ࢖ͬͯɺ֤จॻͷ Vʢ৘ใຊମʣΛ౷߹ɿ ࠷ऴ৘ใ = Attention Weight × V (45) = 0.6 × จॻ A ͷ৘ใ + 0.1 × จॻ B ͷ৘ใ + 0.3 × จॻ C ͷ৘ใ (46)
16. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 16 Cross-Attention ͷ݁Ռɿ LLM ͸ʮϋϧγωʔγϣϯΛ཈੍͢Δ͜ͱʯ ʢจॻ AɺॏΈ 0.6ʣͱʮ࠷৽৘ใ ʹରԠͰ͖Δ͜ͱʯ ʢจॻ CɺॏΈ 0.3ʣͱ͍͏৘ใΛॏ఺తʹ࢖ͬͯɺ ʮRAG ͷ ϝϦοτ͸ɺϋϧγωʔγϣϯΛ཈੍͠ɺ࠷৽৘ใʹ΋ରԠͰ͖Δ఺Ͱ͢ʯͱ ͍ͬͨճ౴Λੜ੒͠·͢ɻ ͜ͷྫͰ͸ɺจॻ B ͷʮϕΫτϧݕࡧʯʹؔ͢Δ৘ใ͸࣭໰ʹ௚઀ؔ܎͠ͳ͍ ͨΊɺॏΈ͕ 0.1 ͱ௿͘཈͑ΒΕɺճ౴ੜ੒ʹ΄ͱΜͲӨڹ͠·ͤΜɻ Attention ͷ਺ࣜͷղऍɿ • QʢQueryʣ ɿ ʮԿʹ͍ͭͯ஌Γ͍͔ͨʯ→ ஫ҙΛ޲͚Δଆͷ୯ޠʢྫɿ ʮ৯ ΂Δʯ ʣ • K ʢKeyʣ ɿ ʮԿ͕ར༻Մೳ͔ʯ → ஫ҙΛ޲͚ΒΕΔଆͷ୯ޠ ʢྫɿ ʮೣʯ ʮڕʯ ʣ • V ʢValueʣ ɿ ʮ࣮ࡍͷ৘ใʯ→ ஫ҙΛ޲͚ͨ݁ՌಘΒΕΔ৘ใ ͳͥ͜ͷ਺͕ࣜ௚ײతཧղΛ൓ө͢Δͷ͔ɿ 1. QKT ɿ֤ Queryʢ஌Γ͍ͨ͜ͱʣͱ֤ Keyʢར༻Մೳͳ৘ใʣͷؔ࿈౓Λ ܭࢉ • ಺ੵ͕େ͖͍ = ࣅ͍ͯΔ = ؔ࿈ੑ͕ߴ͍ • ྫɿ ʮ৯΂ΔʯQuery ͱʮڕʯKey ͷ಺ੵ͕େ͖͍ = ҙຯతʹؔ࿈ 2. QKT √ dk ɿؔ࿈౓Λద੾ʹεέʔϦϯάʢ਺஋ͷ҆ఆԽʣ • √ dk ͰׂΔཧ༝ɿ࣍ݩ͕େ͖͘ͳΔͱ಺ੵ΋େ͖͘ͳΓ͗͢ΔͨΊ • εέʔϦϯάʹΑΓ softmax ͷޯ഑ফࣦΛ๷͙ 3. softmax(·)ɿؔ࿈౓Λ֬཰෼෍ʹม׵ →ʮͲͷ৘ใʹͲΕ͘Β͍஫ҙΛ޲ ͚Δ͔ʯͷॏΈ • શͯͷॏΈͷ߹ܭ = 1ʢ׬શͳ஫ҙͷ഑෼ʣ • ࠷΋ؔ࿈͢Δ୯ޠʹߴ͍ॏΈɺແؔ܎ͳ୯ޠʹ͸௿͍ॏΈ 4. ࠷ऴతʹ V ͱֻ͚ࢉɿॏΈʹԠ࣮ͯ͡ࡍͷ৘ใΛॏΈ෇͖ฏۉͰऔಘ • ॏཁͳ୯ޠͷ৘ใʢValueʣΛଟ͘औΓࠐΉ • ॏཁͰͳ͍୯ޠͷ৘ใ͸গ͚ͩ͠औΓࠐΉ • ݁Ռɿจ຺Λߟྀͨ͠৽͍͠୯ޠදݱ ਺ࣜͷ֤෦෼ͷ௚ײతҙຯɿ • QKT ɿ ʮ୭͕୭ʹ஫ҙΛ޲͚Δ΂͖͔ʯͷ਌࿨ੑߦྻ • softmaxɿ ʮ஫ҙͷڧ͞ʯΛ֬཰ͱͯ͠දݱ • V ɿ ʮ࣮ࡍʹऔΓग़͍ͨ͠৘ใʯ • શମɿ ʮؔ࿈͢Δ৘ใΛɺͦͷॏཁ౓ʹԠͯ͡૊Έ߹ΘͤΔʯ ͭ·Γɺ ʮ৯΂Δʯͱ͍͏ Query ʹରͯ͠ɺ ʮೣʯ ʮڕʯͱ͍͏ Key ͱͷؔ࿈౓ Λܭࢉ͠ɺͦͷॏΈͰʮೣʯ ʮڕʯͷ৘ใʢValueʣΛ૊Έ߹ΘͤΔ͜ͱͰɺจ຺ Λߟྀͨ͠දݱΛಘ͍ͯ·͢ɻ͜ΕʹΑΓɺಉ͡ʮ৯΂ΔʯͰ΋ʮೣ͕ڕΛ৯ ΂Δʯͱʮਓ͕ύϯΛ৯΂ΔʯͰ͸ҟͳΔදݱʹͳΓ·͢ɻ
17. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 17 2.4.2 ͳͥจ຺ΛཧղͰ͖Δͷ͔ ैདྷͷ໰୊ɿ ʮ͖͘ʯͱ͍͏୯ޠΛݟͯ΋ɺԻΛʮฉ͘ʯͳͷ͔ɺༀ͕ʮޮ͘ʯͳͷ͔Θ͔Β ͳ͍ɻ Transformer ͷղܾ๏ɿ จதͷ͢΂ͯͷ୯ޠͱͷؔ܎ੑΛಉ࣌ʹܭࢉɿ •ʮԻָΛ͖͘ʯ→ ʮԻָʯͱͷؔ܎͕ڧ͍ → ฉ͘ʢԻΛࣖͰײ͡Δʣ •ʮༀ͕͖͘ʯ→ ʮༀʯͱͷؔ܎͕ڧ͍ → ޮ͘ʢޮՌ͕͋Δʣ •ʮ࿩Λ͖͘ʯ→ ʮ࿩ʯͱͷؔ܎͕ڧ͍ → ฉ͘ʢࣖͰड͚औΔʣ RAG ͰͷԠ༻ɿ ॏཁͳલఏɿ Transformer ͸ݱ୅ͷ LLMʢChatGPTɺClaude ౳ʣͷج൫ٕज़ Ͱ͋ΓɺRAG ಛ༗ͷ΋ͷͰ͸͋Γ·ͤΜɻRAG γεςϜͰ͸ɺ͜ͷҰൠతͳ Transformer ٕज़ΛҎԼͷ 2 ͭͷ৔໘Ͱ׆༻͠·͢ɿ 1. ݕࡧஈ֊ʢRetrievalʣͰͷ׆༻ɿ • ࣭໰จͷཧղɿʮ࠷৽ͷ AI ن੍ʹ͍ͭͯʯ→ Transformer ͕ʮ࠷ ৽ʯ ʮAIʯ ʮن੍ʯͷจ຺తؔ܎Λཧղ • จॻͷཧղɿݕࡧର৅จॻ΋ Transformer ϕʔεͷϞσϧͰຒΊࠐΈ ϕΫτϧԽ • ؔ࿈ੑ൑அɿ࣭໰ͱจॻͷҙຯతྨࣅ౓Λɺจ຺Λߟྀͨ͠ϕΫτϧ Ͱܭࢉ 2. ੜ੒ஈ֊ʢGenerationʣͰͷ׆༻ɿ • ݕࡧͰಘΒΕͨจॻͱݩͷ࣭໰Λ૊Έ߹ΘͤͯɺLLMʢTransformer ϕ ʔεʣ͕ճ౴Λੜ੒ • ͜ͷࡍɺݕࡧ݁Ռͷจ຺Λਖ਼͘͠ཧղ͠ɺ࣭໰ʹద੾ʹ౴͑ΔͨΊʹ Transformer ͷจ຺ཧղೳྗ͕ෆՄܽ ͭ·ΓɺRAG ͸ Transformer ͷจ຺ཧղೳྗΛલఏͱͯ͠ɺͦΕΛݕࡧͱੜ੒ ͷ྆ํͰ׆༻͢ΔγεςϜͰ͢ɻ 3 ݕࡧٕज़ৄઆ 3.1 ϕΫτϧݕࡧͷ਺ֶతجૅͱ࣮૷ 3.1.1 ϕΫτϧݕࡧͷ׬શཧղ جຊతͳߟ͑ํɿϕΫτϧݕࡧ͸ ʮҙຯͷࣅ͍ͯΔจॻ͕ɺ ϕΫτϧۭؒͰ΋͍ۙҐஔ ʹ഑ஔ͞ΕΔΑ͏ʹຒΊࠐΈϞσϧֶ͕श͞Ε͍ͯΔʯ ͱ͍͏લఏʹج͍͍ͮͯ·͢ɻ ॏཁͳલఏɿϕΫτϧۭؒʢn ࣍ݩϢʔΫϦουۭؒ Rnʣࣗମʹ͸ҙຯతͳߏ଄ ͸͋Γ·ͤΜɻຒΊࠐΈϞσϧʢBERTɺSentence-BERT ౳ʣͷֶशʹΑΓɺҙຯత ʹྨࣅͨ͠จॻ͕ۙ͘ʹ഑ஔ͞ΕΔΑ͏ͳϕΫτϧԽ͕࣮ݱ͞Ε͍ͯ·͢ɻ ॲཧͷྲྀΕɿ
18. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 18 1. ࣄલॲཧɿશจॻΛ Embedding ϞσϧͰϕΫτϧԽͯ͠อଘ 2. ݕࡧ࣌ɿΫΤϦ΋ϕΫτϧԽ͠ɺอଘ͞ΕͨϕΫτϧͱͷྨࣅ౓Λܭࢉ 3. ϥϯΩϯάɿྨࣅ౓ͷߴ͍ॱʹจॻΛιʔτ ίαΠϯྨࣅ౓ͷৄࡉɿ similarity(q, d) = q · d |q||d| = ∑ n i=1 qi di √∑ n i=1 q2 i √∑ n i=1 d2 i (47) ۩ମతͳܭࢉྫɿ࣭໰ʮݘͷࣂ͍ํʯͱυΩϡϝϯτʮೣͷੈ࿩ํ๏ʯΛൺֱɿ Query = [0.8, 0.1, −0.2, 0.5, 0.3] (48) Document = [0.7, 0.2, −0.1, 0.4, 0.1] (49) ಺ੵ = 0.8 × 0.7 + 0.1 × 0.2 + (−0.2) × (−0.1) + 0.5 × 0.4 + 0.3 × 0.1 (50) = 0.56 + 0.02 + 0.02 + 0.20 + 0.03 = 0.83 (51) |q| = √ 0.82 + 0.12 + 0.22 + 0.52 + 0.32 = √ 1.03 ≈ 1.01 (52) |d| = √ 0.72 + 0.22 + 0.12 + 0.42 + 0.12 = √ 0.71 ≈ 0.84 (53) ྨࣅ౓ = 0.83 1.01 × 0.84 ≈ 0.98 (54) ߴ͍ྨࣅ౓ʢ0.98ʣ͕ग़͍ͯΔͷͰɺҙຯతʹؔ࿈͍ͯ͠Δ͜ͱ͕Θ͔Γ·͢ɻ 3.1.2 ϕΫτϧݕࡧͷ௕ॴͱ୹ॴ ௕ॴɿ • ҙຯతཧղɿ ʮंʯͱʮࣗಈंʯ ʮΫϧϚʯΛಉ֓͡೦ͱͯ͠ೝࣝ • ଟݴޠରԠɿ຋༁ͳ͠ͰҟͳΔݴޠؒͰͷݕࡧ͕Մೳ – ཧ༝ɿଟݴޠରԠͷຒΊࠐΈϞσϧʢྫɿmultilingual-BERTɺLaBSEʣ͸ɺ ҟͳΔݴޠͰಉ͡ҙຯΛ࣋ͭจষΛಉ͡ϕΫτϧۭؒͷ͍ۙҐஔʹ഑ஔ͢ ΔΑ͏ֶश͞Ε͍ͯΔ – ྫɿ ʮػցֶशʯ ʢ೔ຊޠʣͱʮmachine learningʯ ʢӳޠʣ͕ࣅͨϕΫτϧΛ ࣋ͭͨΊɺ೔ຊޠΫΤϦͰӳޠจॻΛݕࡧՄೳ – ࣮ݱํ๏ɿֶश࣌ʹಉ͡ҙຯͷଟݴޠจষϖΞΛ࢖༻͠ɺݴޠʹґଘ͠ͳ ͍ҙຯදݱΛ֫ಘ • ᐆດݕࡧɿ׬શҰகͰͳͯ͘΋ؔ࿈จॻΛൃݟ ୹ॴɿ • ܭࢉίετɿߴ࣍ݩϕΫτϧͷྨࣅ౓ܭࢉ͸ॏ͍ • Ωʔϫʔυݕࡧͷऑ͞ɿ ʮܕ൪ XYZ-123ʯͷΑ͏ͳݻ༗໊ࢺʹऑ͍ • ϒϥοΫϘοΫεɿͳͥͦͷจॻ͕બ͹Ε͔ͨͷઆ໌͕ࠔ೉
19. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 19 3.2 BM25 ͷৄࡉղઆͱ࣮૷ 3.2.1 BM25 ͷ਺ֶతఆٛͱ௚ײతཧղ BM25ʢBest Matching 25ʣ͸ɺ৘ใݕࡧ෼໺Ͱ࠷΋੒ޭͨ͠ख๏ͷҰͭͰ͢ [6, 7, 8]ɻ ໊લͷ༝དྷɿ • BMɿ ʮBest MatchingʯͷུͰɺ࠷దͳจॻϚονϯάΛߦ͏ͱ͍͏ҙຯ • 25ɿҰൠతʹɺ1994 ೥ͷ TREC-3 ձٞͰ 25 ൪໨ͷ࣮ݧͱͯ͠ఏҊ͞ΕͨͨΊ ͱݴΘΕ͍ͯ·͢ ྺ࢙తഎܠͱ࣮ݧతௐ੔ɿ BM25 ͸ཧ࿦తج൫ͱ࣮ݧతௐ੔ͷ྆ํΛ࣋ͬͯ ͍·͢ɻRobertson & Sparck Jones ͕ 1970 ೥୅ʹ֬཰త৘ใݕࡧϞσϧʢBinary Independence ModelʣΛఏҊ͠ɺ͜Ε͕ BM25 ͷཧ࿦తج൫ͱͳΓ·ͨ͠ɻ͔͠͠ɺ k1 ΍ b ͳͲͷύϥϝʔλ͸ɺTREC ͳͲͷେن໛ͳ৘ใݕࡧධՁίϯϖςΟγϣϯ Ͱ༷ʑͳσʔληοτΛ༻͍ͯ࠷దԽ͞Ε·ͨ͠ɻಛʹ k1=1.2, b=0.75 ͱ͍͏஋͸ɺ ଟ͘ͷ࣮ݧͰྑ޷ͳੑೳΛࣔͨͨ͠Ίඪ४஋ͱͳ͍ͬͯ·͢ɻͭ·ΓɺBM25 ͷࣜ͸ ʮ਺ֶతʹ׬ᘳʹಋग़͞ΕͨʯΘ͚Ͱ͸ͳ͘ɺ ʮཧ࿦తʹଥ౰Ͱɺ͔࣮ͭݧతʹ࠷΋ޮ Ռతͩͬͨʯύϥϝʔλͷ૊Έ߹ΘͤͳͷͰ͢ɻ LLM ͱͷؔ܎ɿ BM25 ͸ 1970 ೥୅͔Β։ൃ͞Εͨ఻౷తͳ৘ใݕࡧख๏Ͱɺ େن໛ݴޠϞσϧʢLLMʣΑΓ΋͸Δ͔ʹݹ͍ٕज़Ͱ͢ɻGoogle ݕࡧΤϯδϯ΍ Elasticsearch ͳͲͰ௕೥࢖ΘΕ͖ͯ·ͨ͠ɻRAG γεςϜͰ͸ɺLLM ͱ૊Έ߹Θͤ Δ͜ͱͰʮΩʔϫʔυݕࡧͷਫ਼౓ʯͱʮҙຯཧղʯͷ྆ํΛ׆༻Ͱ͖ΔͨΊॏཁͳ໾ ׂΛՌ͍ͨͯ͠·͢ɻ ௚ײతཧղɿBM25 ͕ߟྀ͢Δ 3 ͭͷཁૉ BM25 ͸ɺจॻͷؔ࿈ੑΛ൑அ͢ΔࡍʹҎԼͷ 3 ͭͷॏཁͳཁૉΛߟྀ͠·͢ɿ 1. ୯ޠͷॏཁ౓ʢIDFɿInverse Document Frequencyʣ ɿ ʮݘʯ͸ଟ͘ͷจॻʹ ग़ݱ͢Δ͕ɺ ʮࣲݘʯ͸গͳ͍ →ʮࣲݘʯͷํ͕ݕࡧʹ͓͍ͯॏཁ 2. ୯ޠͷग़ݱස౓ʢTFɿTerm Frequencyʣ ɿ͋ΔจॻͰʮػցֶशʯ͕ 10 ճग़ ݱ͍ͯ͠Ε͹ɺ1 ճ͔͠ग़ݱ͠ͳ͍จॻΑΓؔ࿈ੑ͕ߴ͍ʢͨͩ͠๞࿨͢Δʣ 3. จॻͷ௕͞ͷਖ਼نԽɿ௕͍จॻ͸୯ޠ͕ଟ͘ग़ݱ͠΍͍ͨ͢Ίɺ୹͍จॻͱެ ฏʹൺֱ͢Δʹ͸จॻ௕ʹΑΔௐ੔͕ඞཁɻྫ͑͹ɺ1000 ޠͷจॻͰʮAIʯ͕ 5 ճग़ݱ͢Δ৔߹ͱɺ100 ޠͷจॻͰʮAIʯ͕ 2 ճग़ݱ͢Δ৔߹ɺޙऀͷํ͕ີ ౓͕ߴؔ͘࿈ੑ͕ߴ͍ͱߟ͑ΒΕΔɻ ͳͥϕΫτϧݕࡧͱҟͳΔͷ͔ʁ • ϕΫτϧݕࡧɿจͷҙຯతͳྨࣅੑΛଌΔʢ ʮݘʯͱʮϖοτʯ͸ࣅ͍ͯΔʣ • BM25ɿ୯ޠͷҰகͱ౷ܭతͳॏཁੑΛଌΔʢ ʮݘʯͱʮϖοτʯ͸ผ෺ͱͯ͠ ѻ͏ʣ ͭ·ΓɺBM25 ͸ʮݕࡧΫΤϦͷ୯ޠ͕ͲΕ͚ͩॏཁͰɺͲΕ͚ͩจॻʹؚ·Εͯ ͍Δ͔ʯΛݡ͘ܭࢉ͢Δख๏Ͱ͢ɻ ׬શͳ਺ࣜɿ BM25(q, d) = n ∑ i=1 IDF(qi ) · f(qi , d) · (k1 + 1) f(qi , d) + k1 · (1 − b + b · |d| avgdl ) (55)
20. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 20 ֤߲ͷҙຯɿ • qi ɿΫΤϦதͷ i ൪໨ͷ୯ޠ • f(qi , d)ɿ୯ޠ qi ͕จॻ d தʹग़ݱ͢Δճ਺ʢTF: Term Frequencyʣ • |d|ɿจॻ d ͷ௕͞ʢ୯ޠ਺ʣ • avgdlɿίʔύεશମͷฏۉจॻ௕ • k1 , bɿௐ੔ύϥϝʔλʢ௨ৗ k1 = 1.5, b = 0.75ʣ IDFʢInverse Document Frequencyʣͷܭࢉɿ IDF(qi ) = ln ( N − df(qi ) + 0.5 df(qi ) + 0.5 ) (56) ͜͜Ͱɿ • Nɿίʔύεதͷ૯จॻ਺ • df(qi )ɿ୯ޠ qi ΛؚΉจॻͷ਺ 3.2.2 BM25 ͷ۩ମతܭࢉྫ ઃఆɿ • ΫΤϦɿ ʮػցֶशΞϧΰϦζϜʯ • จॻɿ ʮػցֶशͷ৽͍͠ΞϧΰϦζϜΛ։ൃͨ͠ɻ͜ͷΞϧΰϦζϜ͸...ʯ ʢ20 ୯ޠʣ • ίʔύεɿ1000 จॻɺฏۉจॻ௕ 15 ୯ޠ •ʮػցֶशʯ ɿ100 จॻʹग़ݱɺ ʮΞϧΰϦζϜʯ ɿ200 จॻʹग़ݱ IDF ͷܭࢉɿ IDF(ػցֶश) = ln ( 1000 − 100 + 0.5 100 + 0.5 ) = ln ( 900.5 100.5 ) ≈ 2.19 (57) IDF(ΞϧΰϦζϜ) = ln ( 1000 − 200 + 0.5 200 + 0.5 ) = ln ( 800.5 200.5 ) ≈ 1.38 (58) TF ੒෼ͷܭࢉʢ ʮػցֶशʯ͕ 1 ճɺ ʮΞϧΰϦζϜʯ͕ 2 ճग़ݱʣ ɿ TF(ػցֶश) = 1 · (1.5 + 1) 1 + 1.5 · (1 − 0.75 + 0.75 · 20 15 ) (59) = 2.5 1 + 1.5 · (0.25 + 1) = 2.5 2.875 ≈ 0.87 (60) TF(ΞϧΰϦζϜ) = 2 · (1.5 + 1) 2 + 1.5 · (0.25 + 1) = 5 3.875 ≈ 1.29 (61) ࠷ऴείΞɿ BM25 = 2.19 × 0.87 + 1.38 × 1.29 ≈ 1.91 + 1.78 = 3.69 (62)
21. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 21 3.2.3 BM25 ͷ௕ॴͱ୹ॴ ௕ॴɿ • ߴ଎ɿసஔΠϯσοΫεʹΑΔޮ཰తͳݕࡧ • Ωʔϫʔυݕࡧʹڧ͍ɿ׬શҰக΍ݻ༗໊ࢺʹ༗ޮ • ղऍՄೳɿͲͷ୯ޠ͕είΞʹد༩͔͕ͨ͠໌֬ • ࣮૷͕༰қɿෳࡶͳϞσϧෆཁ ୹ॴɿ • ޠኮͷҰகͷΈɿಉٛޠ΍ྨࣅ֓೦ΛೝࣝͰ͖ͳ͍ • ޠॱແࢹɿ ʮݘ͕ೣΛ௥͍͔͚Δʯͱʮೣ͕ݘΛ௥͍͔͚ΔʯΛ۠ผͰ͖ͳ͍ • จ຺ཧղෆՄɿ ʮ͸͠ʯ͕ڮɾസɾ୺ͷͲΕ͔Λ൑அͰ͖ͳ͍ 3.3 ϋΠϒϦουݕࡧ 3.3.1 ͳͥϋΠϒϦουݕࡧ͕ඞཁ͔ ୯ମͰ͸ݶք͕͋ΔϕΫτϧݕࡧͱ BM25 Λ૊Έ߹ΘͤΔ͜ͱͰɺ྆ํͷ௕ॴΛ׆ ͔ͤ·͢ɻ ૬ิతͳಛੑɿ ϕΫτϧݕࡧͱ BM25 ͸ɺͦΕͧΕ͕ҟͳΔڧΈΛ͓࣋ͬͯΓɺޓ͍ͷऑ఺Λิ ׬Ͱ͖·͢ɿ • ϕΫτϧݕࡧͷڧΈɿҙຯతͳྨࣅੑΛཧղʢ ʮݘʯͱʮϖοτʯͷؔ࿈ੑΛೝ ࣝʣ • BM25 ͷڧΈɿਖ਼֬ͳΩʔϫʔυϚονϯάʢ঎඼໊΍ݻ༗໊ࢺʹڧ͍ʣ • ૊Έ߹ΘͤͷޮՌɿ ʮMacBook Pro M3ʯͷΑ͏ͳݻ༗໊ࢺͱɺ ʮϊʔτύιί ϯʯͷΑ͏ͳҰൠ֓೦ͷ྆ํʹରԠ ݕࡧλΠϓ ϕΫτϧݕࡧ BM25 ϋΠϒϦουݕࡧ ҙຯతྨࣅੑ ⃝ △ ⃝ ΩʔϫʔυҰக △ ⃝ ⃝ ಉٛޠॲཧ ⃝ × ⃝ ݻ༗໊ࢺݕࡧ △ ⃝ ⃝ ܭࢉ଎౓ △ ⃝ △ ղऍੑ × ⃝ △ Table 1: ֤ݕࡧख๏ͷಛੑൺֱ ิ଍ɿ • ܭࢉ଎౓ɿϋΠϒϦουݕࡧ͸྆ํͷख๏Λ࣮ߦ͢ΔͨΊɺ୯ମͷख๏ΑΓ஗ ͘ͳΓ·͢ • ղऍੑɿ2 ͭͷҟͳΔख๏ͷείΞΛ݁߹͢ΔͨΊɺͳͥͦͷจॻ͕બ͹Εͨ ͔ͷઆ໌͕ෳࡶʹͳΓ·͢
22. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 22 3.3.2 ϋΠϒϦουݕࡧͷ࣮૷ํࣜ ॏཁɿܭࢉίετʹ͍ͭͯ ϋΠϒϦουݕࡧͰ͸ɺॏΈ܎਺ʹؔΘΒͣ྆ํͷख๏Λ׬શʹ࣮ߦ͢Δඞཁ͕ ͋Γ·͢ɿ • ͨͱ͑ϕΫτϧݕࡧ 10%ɺBM25 90% Ͱ΋ɺϕΫτϧݕࡧ͸શจॻʹରͯ͠ 100% ࣮ߦ͞Ε·͢ • ࠷ऴείΞͰॏΈ͕௿ͯ͘΋ɺܭࢉΛলུ͢Δ͜ͱ͸Ͱ͖·ͤΜ • ݁Ռͱͯ͠ɺඞͣ୯ମͷख๏ΑΓܭࢉίετ͕ߴ͘ͳΓ·͢ 1. είΞ݁߹ํࣜɿ Final Score = α · Vector Score + (1 − α) · BM25 Score (63) α ͸ॏΈ܎਺ʢ௨ৗ 0.5ʙ0.7ʣ 2. ϥϯΫ݁߹ํࣜʢReciprocal Rank Fusionʣ ɿ RRF Score = ∑ r∈R 1 k + r (64) ه߸ͷઆ໌ɿ • ϥϯΫʢrʣ ɿݕࡧ݁Ռʹ͓͚ΔॱҐɻ࠷΋ؔ࿈ੑ͕ߴ͍จॻ͕ϥϯΫ 1ɺ͕࣍ϥ ϯΫ 2ɺ... ͱͳΔ • Rɿ͋Δจॻ͕ෳ਺ͷݕࡧख๏ͰಘΒΕͨϥϯΫͷू߹ • kɿఆ਺ʢ௨ৗ 60ʣ ɻখ͍͞ϥϯΫʢ্ҐʣͷॏཁੑΛௐ੔͢Δύϥϝʔλ ۩ମྫɿ͋Δจॻ͕ϕΫτϧݕࡧͰ 3 ҐʢϥϯΫ 3ʣ ɺBM25 Ͱ 5 ҐʢϥϯΫ 5ʣ ͩͬͨ৔߹ɿ RRF Score = 1 60 + 3 + 1 60 + 5 = 1 63 + 1 65 ≈ 0.0317 (65) ิ଍ɿϕΫτϧݕࡧͱ BM25 ͷ 2 ख๏ϋΠϒϦουͷ৔߹ɺR ͷཁૉ਺͸࠷େ 2 ݸʹͳΓ·͢ɻ྆ํͷख๏Ͱώοτͨ͠จॻ͸ R = {rvector, rBM25}ɺยํͰͷΈώο τͨ͠จॻ͸ R = {rvector} ·ͨ͸ R = {rBM25} ͱͳΓ·͢ɻ 3. ஈ֊తߜΓࠐΈํࣜɿ 1. BM25 Ͱେ·͔ʹީิΛߜΓࠐΈʢߴ଎ʣ 2. ީิʹରͯ͠ϕΫτϧݕࡧͰਫ਼ີϥϯΩϯά ஫ҙɿ͜ͷํࣜͰ΋ BM25 ͷܭࢉίετ͕௥Ճ͞ΕΔͨΊɺϕΫτϧݕࡧ୯ମΑ Γߴ଎ʹͳΔ͜ͱ͸͋Γ·ͤΜɻͨͩ͠ɺϕΫτϧݕࡧͷର৅จॻ਺ΛݮΒͤΔͨ Ίɺશจॻʹରͯ྆͠ํΛ࣮ߦ͢ΔΑΓ͸ޮ཰తͰ͢ɻ
23. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 23 ࣮૷ํࣜͷ࢖͍෼͚ͱ࣮༻্ͷ܏޲ɿ ํࣜ ར఺ͱಛ௃ ద༻৔໘ 1. είΞ݁߹ • ࣮૷͕࠷΋γϯϓϧ • ύϥϝʔλ α Ͱॊೈʹௐ ੔Մೳ • LangChain/LlamaIndex ͳͲͷओཁϑϨʔϜϫʔ ΫͰඪ४తͳબ୒ࢶͷҰ ͭ • Ұൠతͳ RAG γεςϜ • ElasticsearchɺOpenSearch ͳͲͷݕࡧΤϯδϯ • LangChainɺ LlamaIndex ͳ ͲͷϑϨʔϜϫʔΫ 2. RRF • είΞͷਖ਼نԽ͕ෆཁ • ϥϯΩϯάͷΈΛߟྀ͢ ΔͨΊؤ݈ • 3 ͭҎ্ͷख๏ͷ݁߹ʹ΋ ֦ு༰қ • ෳ਺ͷݕࡧΤϯδϯΛ౷ ߹͢Δ৔߹ • είΞͷई౓͕େ͖͘ҟ ͳΔ৔߹ • ϝλαʔνΤϯδϯ 3. ஈ֊తߜΓࠐΈ • େن໛σʔληοτͰޮ ཰త • ϝϞϦ࢖༻ྔΛ࡟ݮ • ஗ԆධՁ͕Մೳ • จॻ਺͕ඇৗʹଟ͍৔߹ ʢ100 ສ݅Ҏ্ʣ • ϦΞϧλΠϜੑ͕ॏཁͳ ৔߹ • Ϧιʔε͕ݶΒΕͨ؀ڥ Table 2: ϋΠϒϦου࣮૷ํࣜͷൺֱ
24. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 24 3.4 Embedding ʹΑΔυΩϡϝϯτબ୒ͷྲྀΕ νϟϯΫ෼ׂʢChunkingʣͱ͸ɿେ͖ͳจॻΛݕࡧ͠΍͍͢খ͞ͳ୯Ґʹ෼ׂ͢Δ ॲཧͰ͢ɻ௕େͳจॻΛͦͷ··ຒΊࠐΈϕΫτϧԽ͢Δͱɺ༷ʑͳ࿩୊͕ࠞࡏͯ͠ ͠·͍ɺಛఆͷ࣭໰ʹର͢Δݕࡧਫ਼౓͕௿Լ͠·͢ɻͦͷͨΊɺจॻΛద੾ͳαΠζ ʢྫɿ500-1000 จࣈɺ100-300 τʔΫϯʣʹ෼ׂ͠ɺ֤νϟϯΫΛಠཱͨ͠ݕࡧ୯Ґͱ ͯ͠ѻ͍·͢ɻ 3.4.1 લॲཧεςοϓ 1. จॻͷνϟϯΫ෼ׂ 2. Embedding ϞσϧʹΑΔϕΫτϧԽ 3. ϕΫτϧσʔλϕʔε΁ͷ֨ೲ 3.4.2 ݕࡧ࣌ͷॲཧϑϩʔ 1. ΫΤϦͷϕΫτϧԽɿೖྗ͞Ε࣭ͨ໰จΛຒΊࠐΈϞσϧͰϕΫτϧԽ 2. ྨࣅจॻͷݕࡧɿϕΫτϧσʔλϕʔε͔Βྨࣅ౓ͷߴ͍จॻΛݕࡧ 3. ݁Ռͷฦ٫ɿ্Ґ k ݅ͷจॻΛฦ٫ 3.4.3 νϟϯΫ෼ׂͱݕࡧઓུ 1. ݻఆ௕෼ׂɿ • จࣈ਺·ͨ͸୯ޠ਺Ͱۉ౳ʹ෼ׂ • γϯϓϧ͕ͩจ຺్͕੾ΕΔՄೳੑ 2. ηϚϯςΟοΫ෼ׂɿ • ஈམ΍จষͷҙຯత·ͱ·ΓͰ෼ׂ • จ຺Λอ࣋͢Δ͕ॲཧ͕ෳࡶ 3. Φʔόʔϥοϓ෼ׂɿ • νϟϯΫؒͰҰ෦ॏෳΛ࣋ͨͤΔ • ৘ใͷܽམΛ๷͙͕৑௕ੑ͕૿Ճ 4. ߴ౓ͳݕࡧઓུɿ • ϝλσʔλͷ׆༻ɿνϟϯΫ෼ׂ࣌ʹϑΝΠϧ໊ɺϖʔδ൪߸ɺষλΠτϧɺ࠷ ऴߋ৽೔ͳͲͷϝλσʔλΛ෇༩͠ɺݕࡧ࣌ʹϑΟϧλϦϯά΍ LLM ΁ͷί ϯςΩετิڧʹ׆༻ • Parent Document Retriever (Small-to-Big)ɿݕࡧ༻ʹ࠷దԽ͞Εͨখ͞ͳ νϟϯΫʢsmall chunkʣͰݕࡧ͠ɺLLM ʹ͸ݕࡧ͞ΕͨνϟϯΫΛؚΉΑΓ޿ ൣғͷ਌νϟϯΫʢbig chunkʣΛ౉͢͜ͱͰɺݕࡧਫ਼౓ͱจ຺ཧղͷόϥϯε ΛऔΔख๏
25. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 25 ޮՌతͳνϟϯΫ෼ׂͷཁ఺ɿ • ֊૚త෼ׂɿஈམ → จ → จࣈͷ༏ઌॱҐͰ෼ׂ • ద੾ͳαΠζɿ௨ৗ 1000ʙ2000 จࣈఔ౓ • Φʔόʔϥοϓɿจ຺ͷ࿈ଓੑΛอͭͨΊ 200 จࣈఔ౓ͷॏෳ • ϝλσʔλ෇༩ɿνϟϯΫͷҐஔ৘ใͳͲΛه࿥ 4 ධՁख๏ 4.1 ධՁͷॏཁੑͱ՝୊ 4.1.1 ͳͥ RAG γεςϜͷධՁ͕ࠔ೉͔ RAG γεςϜͷධՁ͕ࠔ೉ͳཧ༝: 1. ଟஈ֊ͷॲཧɿݕࡧͱੜ੒ͷ྆ํΛධՁ͢Δඞཁ 2. ओ؍ੑɿճ౴ͷʮྑ͞ʯ͸ओ؍తͰఆྔԽ͕ࠔ೉ 3. ίϯςΩετґଘɿಉ࣭͡໰Ͱ΋จ຺ʹΑΓద੾ͳճ౴͕มΘΔ 4. ஌ࣝͷਖ਼֬ੑɿࣄ࣮ͷਖ਼֬ੑΛࣗಈ൑ఆ͢Δͷ͕ࠔ೉ 4.1.2 ධՁͷ؍఺ 1. ݕࡧੑೳͷධՁɿ • ؔ࿈จॻΛਖ਼͘͠औಘͰ͖͍ͯΔ͔ • ݕࡧ݁ՌͷϥϯΩϯά͸ద੾͔ • ඞཁͳ৘ใ͕ܽམ͍ͯ͠ͳ͍͔ 2. ੜ੒ੑೳͷධՁɿ • ݕࡧ͞Εͨจॻͷ಺༰Λਖ਼֬ʹ൓ө͍ͯ͠Δ͔ • ࣗવͰಡΈ΍͍͢จষʹͳ͍ͬͯΔ͔ • ࣭໰ʹద੾ʹ౴͍͑ͯΔ͔ 3. ૯߹ੑೳͷධՁɿ • ΤϯυϢʔβʔͷຬ଍౓ • γεςϜશମͷ༗༻ੑ • ճ౴ͷ৴པੑ
26. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 26 4.2 ݕࡧੑೳͷධՁࢦඪ 4.2.1 ఻౷తͳ IR ࢦඪ IRʢInformation Retrievalɿ৘ใݕࡧʣͱ͸ɿେྔͷ৘ใͷத͔ΒɺϢʔβʔͷ৘ ใཁٻʹؔ࿈͢Δ৘ใΛޮ཰తʹݟ͚ͭग़͢͜ͱΛ໨తͱͨ͠෼໺Ͱ͢ɻWeb αʔ νΤϯδϯɺਤॻؗݕࡧγεςϜɺ࿦จݕࡧγεςϜͳͲ͕ IR γεςϜͷ୅දྫͰ ͢ɻҎԼͷࢦඪ͸ɺ৘ใݕࡧγεςϜͷੑೳΛ٬؍తʹධՁ͢ΔͨΊʹ௕೥࢖ΘΕͯ ͖·ͨ͠ɻ 1. Precisionʢద߹཰ʣ ɿ Precision = औಘͨؔ͠࿈จॻ਺ औಘͨ͠จॻ਺ (66) ۩ମྫɿ • 10 ݅औಘͯ͠ɺͦͷ͏ͪ 7 ͕݅ؔ࿈ → Precision = 0.7 •ʮແବͳݕࡧ݁Ռ͕Ͳͷఔ౓ؚ·ΕΔ͔ʯΛଌΔࢦඪ 2. Recallʢ࠶ݱ཰ʣ ɿ Recall = औಘͨؔ͠࿈จॻ਺ શؔ࿈จॻ਺ (67) ۩ମྫɿ • ຊདྷ 20 ݅ͷؔ࿈จॻ͕͋Γɺ7 ݅औಘ → Recall = 0.35 •ʮඞཁͳ৘ใΛͲͷఔ౓࿙Β͞ͳ͍͔ʯΛଌΔࢦඪ 3. F1 είΞɿ F1 = 2 × Precision × Recall Precision + Recall (68) ద߹཰ͱ࠶ݱ཰ͷௐ࿨ฏۉͰɺόϥϯεͷऔΕͨධՁࢦඪɻ ௐ࿨ฏۉͱ͸ɿௐ࿨ฏۉ͸ɺ୯७ฏۉʢࢉज़ฏۉʣͱ͸ҟͳΓɺখ͍͞஋ͷӨڹΛ ڧ͘ड͚Δฏۉͷܭࢉํ๏Ͱ͢ɻ୯७ฏۉͰ͸ (P + R)/2 Ͱ͕͢ɺௐ࿨ฏۉͰ͸ 2PR P+R ͱͳΓ·͢ɻ͜ΕʹΑΓɺPrecision ͱ Recall ͷͲͪΒ͔Ұํ͕ۃ୺ʹ௿͍৔߹ɺF1 είΞ΋௿͘ͳΓ·͢ɻ ྫɿ • Precision=0.9, Recall=0.1 ͷ৔߹ • ୯७ฏۉɿ(0.9 + 0.1)/2 = 0.5 • ௐ࿨ฏۉʢF1ʣ ɿ2×0.9×0.1 0.9+0.1 = 0.18 ͜ͷΑ͏ʹɺௐ࿨ฏۉ͸྆ํͷࢦඪ͕ߴ͍৔߹ͷΈߴ͍஋Λࣔͨ͢ΊɺݕࡧγεςϜ ͷ૯߹ੑೳΛΑΓݫ͘͠ධՁͰ͖·͢ɻ 4. Mean Average Precision (MAP)ɿ MAP ͱ͸Կ͔ɿ Mean Average PrecisionʢMAPʣ͸ɺ৘ใݕࡧγεςϜͷ૯߹త ͳੑೳΛධՁ͢Δ࠷΋ॏཁͳࢦඪͷҰͭͰ͢ɻ୯ҰͷΫΤϦͰ͸ͳ͘ɺෳ਺ͷΫΤϦ ʹର͢ΔݕࡧੑೳͷฏۉΛଌఆ͠·͢ɻ MAP ͕ॏཁͳཧ༝ɿ
27. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 27 • ૯߹ධՁɿ 1 ͭͷΫΤϦ͚ͩͰͳ͘ɺγεςϜશମͷੑೳΛଌఆ • ϥϯΩϯάߟྀɿݕࡧ݁ՌͷॱҐ΋ධՁʹؚΉʢ্Ґʹ͋Δؔ࿈จॻ΄Ͳߴ͘ ධՁʣ • ඪ४ࢦඪɿ৘ใݕࡧͷֶज़ݚڀ΍঎༻γεςϜͰ޿͘࢖༻͞ΕΔ • ൺֱՄೳɿҟͳΔγεςϜؒͷੑೳൺֱ͕༰қ جຊతͳߟ͑ํɿ֤ΫΤϦʹ͍ͭͯʮؔ࿈จॻ͕ݟ͔ͭͬͨ࣌఺Ͱͷద߹཰ʯΛ ܭࢉ͠ɺͦΕΛશΫΤϦͰฏۉԽ͠·͢ɻͭ·Γɺ ʮؔ࿈จॻΛ্Ґʹ࣋ͬͯ͜ΒΕ ΔγεςϜ΄Ͳߴ͍είΞʯʹͳΓ·͢ɻ MAP = 1 |Q| |Q| ∑ q=1 AP(q) (69) ه߸ͷఆٛɿ • |Q|ɿධՁʹ࢖༻͢ΔΫΤϦͷ૯਺ • qɿ֤ΫΤϦʢq = 1, 2, ..., |Q|ʣ • AP(q)ɿΫΤϦ q ͷ Average Precision ͜͜Ͱ AP(q) ͸֤ΫΤϦͷ Average Precisionɿ AP(q) = ∑ n k=1 P(k) × rel(k) ؔ࿈จॻ਺ (70) AP(q) ͷه߸ͷఆٛɿ • nɿݕࡧ݁Ռͱͯ͠ฦ͞ΕΔจॻ਺ʢ௨ৗ 10 ݅·ͨ͸ 100 ݅ʣ • kɿݕࡧ݁ՌͷॱҐʢk = 1, 2, ..., nʣ • P(k)ɿॱҐ k Ͱͷద߹཰ʢ্Ґ k ݅தͷؔ࿈จॻ਺ / kʣ • rel(k)ɿॱҐ k ͷจॻ͕ؔ࿈จॻͳΒ 1ɺͦ͏Ͱͳ͚Ε͹ 0 • ؔ࿈จॻ਺ɿͦͷΫΤϦʹରͯ͠ਖ਼ղͱͯ͠ଘࡏ͢Δؔ࿈จॻͷ૯਺ MAP ͷܭࢉྫɿݕࡧ݁Ռ͕ [ؔ࿈, ඇؔ࿈, ؔ࿈, ؔ࿈, ඇؔ࿈] ͷ৔߹ɿ P(1) = 1/1 = 1.0 (ؔ࿈) (71) P(2) = 1/2 = 0.5 (ඇؔ࿈) (72) P(3) = 2/3 = 0.67 (ؔ࿈) (73) P(4) = 3/4 = 0.75 (ؔ࿈) (74) P(5) = 3/5 = 0.6 (ඇؔ࿈) (75) ؔ࿈จॻͰͷਫ਼౓ͷΈߟྀɿ AP = 1.0 + 0.67 + 0.75 3 = 0.81 (76)
28. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 28 4.2.2 ϥϯΩϯάධՁࢦඪ ݕࡧ݁ՌͷॱংΛߟྀͨ͠ධՁࢦඪ܈ɿ 1. Normalized Discounted Cumulative Gain (NDCG)ɿ্ҐͷจॻʹΑΓߴ ͍ॏΈΛ෇͚ͯධՁɻଟஈ֊ͷؔ࿈౓ʢ0-3 ͳͲʣʹରԠɻ 2. Mean Reciprocal Rank (MRR)ɿ࠷ॳͷؔ࿈จॻͷॱҐͷٯ਺ͷฏۉɻ ʮ౴ ͑Λ͙͢ʹݟ͚ͭΒΕΔ͔ʯΛଌΔɻ 3. Mean Average Precision (MAP)ɿ֤ؔ࿈จॻͰͷਫ਼౓ͷฏۉɻݕࡧ݁Ռશ ମͷ඼࣭Λ૯߹ධՁɻ ஫ҙɿ͜ΕΒͷࢦඪ͸ࠓճͷ࣮૷ྫͰ͸࢖༻͍ͯ͠ͳ͍͕ɺΑΓৄࡉͳݕࡧੑೳ ධՁ͕ඞཁͳ৔߹ʹ༗༻ɻ 4.3 ੜ੒඼࣭ͷධՁࢦඪ 4.3.1 ࢀরϕʔεධՁ 1. BLEUʢBilingual Evaluation Understudyʣ ɿػց຋༁༻ʹ։ൃɻn-gram ͷҰ க౓Λଌఆ [9]ɻࠓճͷ࣮૷Ͱ͸࢖༻ͤͣɻ 2. ROUGEʢRecall-Oriented Understudy for Gisting Evaluationʣ ɿ ཁ໿ͷධՁʹಛԽɻࢀরจͰͷ n-gram ࠶ݱ཰Λଌఆ [10]ɻ ROUGE-Nɿ ROUGE-N = ∑ S∈{ࢀরཁ໿} ∑ gram n ∈S Countmatch(gram n ) ∑ S∈{ࢀরཁ໿} ∑ gram n ∈S Count(gram n ) (77) ROUGE-LʢLongest Common Subsequenceʣ ɿ ROUGE-L = 2 × Rlcs × Plcs Rlcs + Plcs (78) 4.3.2 ҙຯతධՁࢦඪ 1. BERTScoreɿ BERT ຒΊࠐΈΛ࢖ͬͨҙຯతྨࣅ౓ධՁɻޠॱ΍ޠኮͷҧ͍ʹؤ݈ [11, 4]ɻ BERTScore = F1BERT = 2 × PBERT × RBERT PBERT + RBERT (79) ͜͜Ͱɿ PBERT = 1 |ˆ x| ∑ ˆ xi∈ˆ x max xj∈x ⃗ ˆi xT ⃗ xj (80) RBERT = 1 |x| ∑ xj∈x max ˆ xi∈ˆ x ⃗ xj T ⃗ ˆi x (81) ⃗ xj ͱ ⃗ ˆi x ͸ BERT ຒΊࠐΈϕΫτϧɻ 2. SentenceBERT cosine similarityɿ Similarity = ⃗ s1 · ⃗ s2 |∥⃗ s1 ∥| × |∥⃗ s2 ∥| (82) จશମͷҙຯతྨࣅ౓Λଌఆɻ
29. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 29 4.4 RAG ಛ༗ͷධՁࢦඪ RAG γεςϜʹಛԽͨ͠ධՁࢦඪͱͯ͠ɺFaithfulnessʢ஧࣮ੑʣ ɺAnswer Relevancy ʢ౴Ҋଥ౰ੑʣ ɺ Context Relevancy ʢίϯςΩετద߹ੑʣ ͳͲ͕͋Γɺ ͜ΕΒ͸ RAGAS ͳͲͷϑϨʔϜϫʔΫͰఏҊ͞Ε͍ͯ·͢ [12]ɻ RAGAS ʢRAG Assessmentʣ ͱ͸ɿ RAGAS ͸ ʮRAG Assessmentʯ ͷུͰɺ RAG γεςϜઐ༻ͷධՁϑϨʔϜϫʔΫͰ͢ɻैདྷͷ৘ใݕࡧධՁʢPrecision/Recall ౳ʣ Ͱ͸ଌఆͰ͖ͳ͍ʮੜ੒͞Εͨճ౴ͷ஧࣮ੑʯ ʮճ౴ͱ࣭໰ͷؔ࿈ੑʯ ʮݕࡧ͞Εͨί ϯςΩετͷద߹ੑʯΛධՁ͢ΔͨΊʹ։ൃ͞Ε·ͨ͠ɻ • ։ൃഎܠɿैདྷͷ IR ࢦඪͰ͸ɺ ʮݕࡧ͸ਖ਼͕֬ͩճ౴͕ؒҧ͍ͬͯΔʯ ʮճ౴͸ ࣗવ͕ͩࠜڌ͕ͳ͍ʯͱ͍ͬͨ RAG ಛ༗ͷ໰୊Λଊ͑ΒΕͳ͍ • ධՁํ๏ɿ LLM-as-a-Judge ͷख๏Λ࢖ͬͯࣗಈධՁΛߦ͏ • ར఺ɿਓखʹΑΔධՁ͕ࠔ೉ͳେن໛γεςϜͰ΋٬؍తධՁ͕Մೳ • ࣮૷ɿ Python ϥΠϒϥϦͱͯ͠ OSS Ͱఏڙ͞Ε͓ͯΓɺ࣮ࡍͷ RAG γες Ϝ։ൃͰ޿͘࢖༻͞Ε͍ͯΔ LLM-as-a-JudgeʢLLM ΛධՁऀͱͯ͠࢖͏Ξϓϩʔνʣ ɿ͜ΕΒͷධՁ͸ɺ͠ ͹͠͹ LLM-as-a-Judge ʹΑͬͯࣗಈԽ͞Ε·͢ɻྫ͑͹ Faithfulness ͸ɺ ʮੜ੒͞Ε ͨճ౴ͷ֤จΛɺผͷڧྗͳ LLMʢྫɿGPT-4oʣʹݟͤͯɺ ʰ͜ͷจ͸ɺ༩͑ΒΕͨ ࢀরจॻͷ಺༰͚ͩͰূ໌Ͱ͖·͔͢ʁYes/Noʱͱ໰͍͔͚ɺYes ͱ౴ׂ͑ͨ߹Λε ίΞԽ͢Δʯͱ͍͏ํ๏Ͱܭࢉ͞Ε·͢ɻ 4.4.1 ஧࣮ੑʢFaithfulnessʣ ੜ੒͞Εͨճ౴͕ݕࡧจॻͷ಺༰ʹ஧࣮͔ΛධՁɻ ܭࢉํ๏ɿ 1. ੜ੒ճ౴Λݪࢠతͳओுʢatomic claimsʣʹ෼ղ 2. ֤ओு͕ݕࡧจॻͰࢧ࣋͞Ε͍ͯΔ͔ΛνΣοΫ 3. ࢧ࣋͞ΕΔओுͷׂ߹Λܭࢉ Faithfulness = ࢧ࣋͞ΕΔओு਺ શओு਺ (83) 4.4.2 ౴Ҋଥ౰ੑʢAnswer Relevancyʣ ճ౴͕࣭໰ʹରͯ͠Ͳͷఔ౓ؔ࿈ੑ͕͋Δ͔ΛධՁɻ ܭࢉख๏ɿ 1. ੜ੒ճ౴͔Βෳ਺ͷٯ࣭໰Λੜ੒ 2. ٯ࣭໰ͱݩ࣭໰ͷྨࣅ౓Λܭࢉ 3. ྨࣅ౓ͷฏۉΛଥ౰ੑείΞͱ͢Δ
30. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 30 Answer Relevancy = 1 N N ∑ i=1 sim(q, q′ i ) (84) ͜͜Ͱɿ • qɿݩ࣭໰ʢϢʔβʔ͕ೖྗ࣭ͨ͠໰ʣ • q′ i ɿੜ੒͞Εͨ i ൪໨ͷٯ࣭໰ • Nɿੜ੒͢Δٯ࣭໰ͷ਺ʢ௨ৗ 3-5 ݸʣ • sim(q, q′ i )ɿ࣭໰ؒͷྨࣅ౓ؔ਺ sim ؔ਺ͷઆ໌ɿ sim(q, q′ i ) ͸ 2 ͭͷ࣭໰จͷҙຯతྨࣅ౓Λ 0-1 ͷ஋Ͱଌఆ͢ Δؔ਺Ͱ͢ɻҰൠతͳܭࢉํ๏ɿ • ೖྗɿݩ࣭໰ q ͱٯ࣭໰ q′ i ͷ 2 ͭͷςΩετ • ॲཧɿຒΊࠐΈϞσϧʢBERTɺSentence-BERT ౳ʣΛ࢖ͬͯ྆ํΛϕΫτϧԽ ͠ɺίαΠϯྨࣅ౓Λܭࢉ • ग़ྗɿ 0ʢશ͘ྨࣅ͍ͯ͠ͳ͍ʣ͔Β 1ʢ׬શʹಉ͡ҙຯʣ·Ͱͷྨࣅ౓είΞ ྨࣅ౓͕ߴ͍΄Ͳɺੜ੒͞Εͨճ౴͕ݩ࣭໰ʹద੾ʹ౴͍͑ͯΔ͜ͱΛࣔ͠·͢ɻ 4.4.3 ίϯςΩετద߹ੑʢContext Relevancyʣ ݕࡧ͞Εͨจॻ͕࣭໰ʹରͯ͠Ͳͷఔ౓ؔ࿈͍ͯ͠Δ͔ΛධՁɻ ධՁํ๏ɿ 1. ݕࡧ͞ΕͨίϯςΩετʢจॻ܈ʣͷ֤จʹ͍ͭͯɺLLM Λ࢖ͬͯʮ͜ͷจ͸ ࣭໰ʹ౴͑ΔͨΊʹඞཁͰ͔͢ʁ ʯΛ൑ఆ 2.ʮඞཁʯͱ൑ఆ͞Εͨจͷ਺Λʮؔ࿈จ਺ʯͱ͢Δ 3. ݕࡧ͞ΕͨίϯςΩετશମͷจ਺ͰׂΔ Context Relevancy = ؔ࿈จ਺ ૯ݕࡧจॻ਺ (85) ৄࡉઆ໌ɿ • ؔ࿈จ਺ɿݕࡧ͞Ε֤ͨจॻʢνϟϯΫʣͷதͰɺ࣭໰ʹ౴͑ΔͨΊʹ࣮ࡍʹ ඞཁͳจͷ਺ɻLLM-as-a-Judge ʹΑͬͯʮ͜ͷจ͸࣭໰ʰ￿￿ʱʹ౴͑ΔͨΊʹඞ ཁͰ͔͢ʁ ʯͱ͍͏ܗͰ֤จΛධՁ • ૯ݕࡧจॻ਺ɿ RAG γεςϜ͕ݕࡧͯ͠औಘͨ͠શจॻʢνϟϯΫʣͷ૯਺ ʢ௨ৗ 3-10 ݅ʣ • ద༻ର৅ɿ͜ͷࢦඪ͸ݕࡧγεςϜશମʹରͯ͠ద༻͞Εɺ֤จॻݸผͰ͸ͳ ͘ݕࡧ݁Ռηοτશମͷ඼࣭Λଌఆ • ໨తɿແؔ܎ͳจॻͷࠞೖΛݕग़͠ɺݕࡧਫ਼౓ͷվળ఺Λಛఆ
31. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 31 4.4.4 LLM-as-a-Judge ͷݶքͱ஫ҙ఺ LLM ධՁऀʹ΋ऑ఺͕͋Γ·͢ɻྫ͑͹ɺධՁϓϩϯϓτͷॻ͖ํʹΑͬͯධՁ݁ Ռ͕ͿΕΔҐஔόΠΞεʢϦετͷ࠷ॳ΍࠷ޙʹ͋Δ΋ͷΛߴ͘ධՁ͠΍͍͢ͳͲʣ ΍ɺධՁऀࣗ਎ͷ஌ࣝʹΑΔϋϧγωʔγϣϯʢධՁࣗମ͕ؒҧ͍ͬͯΔՄೳੑʣͳ ͲͰ͢ɻͦͷͨΊɺෳ਺ͷϓϩϯϓτͰࢼͨ͠Γɺෳ਺ͷ LLM ͰධՁͯ͠ଟ਺ܾΛ ͱͬͨΓ͢ΔͳͲͷ޻෉͕ॏཁʹͳΓ·͢ɻ ओཁͳ՝୊ͱରࡦɿ • ҐஔόΠΞεɿϦετ಺ͷ߲໨ͷॱংΛม͑ͯෳ਺ճධՁ • ϓϩϯϓτґଘੑɿҟͳΔදݱͷධՁϓϩϯϓτͰ݁ՌΛݕূ • ධՁऀͷϋϧγωʔγϣϯɿෳ਺ͷ LLMʢGPT-4ɺClaudeɺGemini ౳ʣͰධՁ ͯ͠ଟ਺ܾ • จ຺௕੍ݶɿ௕͍จॻ͸෼ׂͯ͠ධՁ͠ɺޙͰ౷߹ • ࠶ݱੑͷ໰୊ɿ temperature=0 ʹઃఆ͠ɺseed ஋Λݻఆ 4.4.5 ൃలతͳධՁ؍఺ ࣮༻తͳ RAG γεςϜͰ͸ɺ্هͷجຊతͳධՁࢦඪʹՃ͑ͯɺҎԼͷΑ͏ͳൃల తͳ؍఺΋ॏཁʹͳΓ·͢ɿ 1. Noise RobustnessʢϊΠζ଱ੑʣ ɿݕࡧ݁Ռʹແؔ܎ͳจॻʢϊΠζʣؚ͕· Ε͍ͯͨ৔߹Ͱ΋ɺͦΕʹ࿭Θ͞Εͣʹਖ਼͍͠ճ౴Λੜ੒Ͱ͖Δ͔ΛධՁ͠·͢ɻ • ධՁํ๏ɿҙਤతʹϊΠζจॻΛݕࡧ݁Ռʹࠞೖͤ͞ɺճ౴඼࣭ͷྼԽ౓Λଌ ఆ • ॏཁੑɿ࣮ࡍͷγεςϜͰ͸׬ᘳͳݕࡧ͸ࠔ೉Ͱɺແؔ܎ͳ৘ใ΁ͷରॲೳྗ ͕࣮༻ੑΛࠨӈ • ଌఆࢦඪɿϊΠζ༗ແͰͷճ౴඼࣭ൺֱɺϊΠζจॻ΁ͷݴٴ཰ 2. Negative Rejectionʢڋ൱ೳྗʣ ɿݕࡧͨ͠จॻͷதʹճ౴ͷࠜڌͱͳΔ৘ใ ͕ଘࡏ͠ͳ͍৔߹ʹɺ ʮ෼͔Γ·ͤΜʯ΍ʮ৘ใ͕͋Γ·ͤΜʯͱਖ਼௚ʹճ౴Ͱ͖Δ ͔ΛධՁ͠·͢ɻ • ධՁํ๏ɿճ౴ෆՄೳͳ࣭໰ʹରͯ͠ɺద੾ʹʮෆ໌ʯͱճ౴͢Δ͔Λ൑ఆ • ॏཁੑɿϋϧγωʔγϣϯ๷ࢭͱ৴པੑ֬อͷͨΊɺෆਖ਼֬ͳਪଌΑΓ੣࣮ͳ ڋ൱͕ॏཁ • ଌఆࢦඪɿڋ൱͢΂͖࣭໰Ͱͷਖ਼͍͠ڋ൱཰ɺڋ൱࣌ͷઆ໌ͷద੾ੑ ͜ΕΒͷൃలతධՁ͸ɺجຊతͳਫ਼౓ࢦඪ͚ͩͰ͸ଊ͑ΒΕͳ͍ʮ࣮༻γεςϜ ͱͯ͠ͷ඼࣭ʯΛଌఆ͢Δ্Ͱॏཁͳ؍఺ͱͳΓ·͢ɻ
32. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 32 5 ࣮ࡍʹಈ͔ͤΔίʔυྫ ຊষͰ͸ɺલষ·ͰֶΜͩ RAG γεςϜͷཧ࿦Λ࣮ફ͢ΔͨΊͷίʔυྫΛఏڙ ͠·͢ɻίʔυ͸ Google Colab Ͱ࣮ߦՄೳͰɺݕࡧख๏ͷҧ͍ʹΑΔਫ਼౓޲্Λ ࣮ࡍʹମݧ͢Δ͜ͱΛ໨తͱ͍ͯ͠·͢ɻͨͩ͠ɺൃදऀ͕࣮ߦͨ͠ͱ͜Ζɺτ ʔΫϯফඅ͕େ͖͍ͨΊɺແྉͷ API ΩʔͰ͸్தͰϨʔτϦϛοτʹ౸ୡ͠ ͯ͠·͍ɺग़ྗͷݕূ͕Ͱ͖·ͤΜͰͨ͠ɻͦͷͨΊɺਫ਼౓ධՁͷϩδοΫʹ໰୊ ͕͋ΔՄೳੑ͕͋Γ·͢ɻ ͋͘·Ͱ΋ֶशͷͨΊͷࢀߟίʔυͱͯ͠͝ར༻͍ͩ͘͞ɻ 5.1 ؀ڥߏங 5.1.1 Google Colab Ͱͷ࣮ߦखॱ ࣮ߦํ๏ɿ 1. Google Colab ʹΞΫηεʢhttps://colab.research.google.comʣ 2. ৽͍͠ϊʔτϒοΫΛ࡞੒ 3. ҎԼͷϥΠϒϥϦΠϯετʔϧίʔυΛ࣮ߦ 4. ֤ίʔυϒϩοΫΛॱ൪ʹ࣮ߦ ϥΠϒϥϦΠϯετʔϧͷઆ໌ɿҎԼͷίʔυ͸ɺRAG γεςϜͷ࣮૷ͱධՁʹ ඞཁͳ͢΂ͯͷϥΠϒϥϦΛΠϯετʔϧ͠·͢ɻ͜ͷίʔυΛ࠷ॳʹ࣮ߦͯͩ͘͠ ͍͞ɿ Listing 1: ඞཁͳϥΠϒϥϦͷΠϯετʔϧ 1 # ࣮ ૷ Ͱ ࢖ ༻ ͢ Δ ϥ Π ϒ ϥ Ϧ Λ1ߦ Ͱ Π ϯ ε τ ʔ ϧ 2 !pip install sentence -transformers rank-bm25 scikit -learn google - generativeai -q 5.2 แׅత RAG ධՁσϞɿݕࡧ͔Βੜ੒ɺධՁ·Ͱ ͜ͷίʔυͷ໨తͱҙਤɿຊηΫγϣϯͰ͸ɺRAG γεςϜͷ׬શͳධՁϑϩʔΛ ମݧͰ͖Δ౷߹σϞΛఏڙ͠·͢ɻϕΫτϧݕࡧͱϋΠϒϦουݕࡧͷൺֱ͔Βɺ࣮ ࡍͷ LLM ʹΑΔճ౴ੜ੒ɺͦͯ͠ LLM-as-a-Judge ʹΑΔߴ౓ͳධՁ·ͰɺҰ࿈ͷ ྲྀΕΛ࣮૷͠·͢ɻ ֶशͷྲྀΕɿ 1. ݕࡧख๏ͷൺֱʢϕΫτϧݕࡧ vs ϋΠϒϦουݕࡧʣ 2. Gemini API Λ࢖࣮ͬͨࡍͷճ౴ੜ੒ 3. LLM-as-a-Judge ʹΑΔ஧࣮ੑɾଥ౰ੑධՁ 4. ݕࡧख๏͕࠷ऴతͳճ౴඼࣭ʹ༩͑ΔӨڹͷ෼ੳ ظ଴͞ΕΔֶश੒Ռɿ • ݕࡧਫ਼౓͕ճ౴඼࣭ʹ༩͑Δ௚઀తӨڹͷཧղ
33. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 33 • Gemini Λ׆༻ͨ͠ LLM-as-a-Judge ධՁख๏ͷशಘ • RAG γεςϜશମͷධՁϑϩʔͷཧղ • ࣮ફతͳ RAG ධՁγεςϜͷߏஙೳྗ ࣮ߦ४උɿ 1. Gemini API ΩʔΛऔಘʢhttps://aistudio.google.com/app/apikeyʣ 2. Google Colab ͷʮൿີ৘ใʯʹʮGOOGLE_API_KEYʯͱͯ͠ઃఆ ࣮ߦํ๏ɿҎԼͷίʔυΛ Google Colab ͷ৽͍͠ηϧʹίϐʔ & ϖʔετͯ͠ ࣮ߦ͍ͯͩ͘͠͞ɿ Listing 2: แׅత RAG ධՁσϞʢݕࡧɾੜ੒ɾධՁ౷߹ʣ 1 # ඞ ཁ ͳ ϥ Π ϒ ϥ Ϧ ͷ Π ϯ ε τ ʔ ϧ ʢ Google Colab Ͱ ࣮ ߦʣ 2 !pip install sentence -transformers rank-bm25 scikit -learn google - generativeai -q 3 4 import numpy as np 5 import os 6 from typing import List, Dict, Any 7 from sentence_transformers import SentenceTransformer 8 from rank_bm25 import BM25Okapi 9 from sklearn.metrics.pairwise import cosine_similarity 10 import google.generativeai as genai 11 import json 12 from google.colab import userdata 13 14 # Gemini API Ω ʔ ͷ ઃ ఆ 15 genai.configure(api_key=userdata.get('GOOGLE_API_KEY')) 16 17 if not userdata.get('GOOGLE_API_KEY'): 18 print("஫ ҙ: Gemini API Ω ʔ ͕ ઃ ఆ ͞ Ε ͯ ͍ · ͤ Μ") 19 print("Google Colab ͷ ʮൿ ີ ৘ ใʯ Ͱ GOOGLE_API_KEY Λ ઃ ఆ ͠ ͯ ͘ ͩ ͞ ͍") 20 print("API Ω ʔ औ ಘ: https://aistudio.google.com/app/apikey") 21 22 # ==================== 23 # 1. RAG γ ε ς Ϝ ͷ ࣮ ૷ 24 # ==================== 25 26 class RAGSystem: 27 def __init__(self, documents: List[str]): 28 print("RAG γ ε ς Ϝ Λ ॳ ظ Խ த ...") 29 self.documents = documents 30 31 # ຒ Ί ࠐ Έ Ϟ σ ϧ ॳ ظ Խ 32 self.embedding_model = SentenceTransformer('all-MiniLM -L6-v2' ) 33 print("จ ॻ ͷ ຒ Ί ࠐ Έ Λ ܭ ࢉ த ...") 34 self.doc_embeddings = self.embedding_model.encode(documents)
34. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 34 35 36 # BM25 ॳ ظ Խ 37 tokenized_docs = [doc.split() for doc in documents] 38 self.bm25 = BM25Okapi(tokenized_docs) 39 print(f"ॳ ظ Խ ׬ ྃ: {len(documents)}݅ ͷ จ ॻ Λ ొ ࿥") 40 41 def _vector_search(self, query: str, top_k: int) -> List[Dict[str , Any]]: 42 query_embedding = self.embedding_model.encode([query]) 43 similarities = np.dot(self.doc_embeddings , query_embedding.T) .flatten() 44 top_indices = np.argsort(similarities)[::-1][:top_k] 45 46 return [{ 47 'document': self.documents[idx], 48 'score': float(similarities[idx]), 49 'method': 'vector', 50 'doc_id': idx 51 } for idx in top_indices] 52 53 def _hybrid_search(self, query: str, top_k: int, alpha: float = 0.6) -> List[Dict[str, Any]]: 54 # ϕ Ϋ τ ϧ ݕ ࡧ ε ί Ξ 55 query_embedding = self.embedding_model.encode([query]) 56 vector_scores = np.dot(self.doc_embeddings , query_embedding.T ).flatten() 57 58 # BM25 ε ί Ξ 59 query_tokens = query.split() 60 bm25_scores = self.bm25.get_scores(query_tokens) 61 62 # ਖ਼ ن Խ 63 vector_scores = self._normalize_scores(vector_scores) 64 bm25_scores = self._normalize_scores(bm25_scores) 65 66 # ϋ Π ϒ Ϧ ο υ ε ί Ξ 67 hybrid_scores = alpha * vector_scores + (1 - alpha) * bm25_scores 68 top_indices = np.argsort(hybrid_scores)[::-1][:top_k] 69 70 return [{ 71 'document': self.documents[idx], 72 'score': float(hybrid_scores[idx]), 73 'vector_score': float(vector_scores[idx]), 74 'bm25_score': float(bm25_scores[idx]), 75 'method': 'hybrid', 76 'doc_id': idx 77 } for idx in top_indices] 78 79 def _normalize_scores(self, scores: np.ndarray) -> np.ndarray: 80 min_score = scores.min()
35. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 35 81 max_score = scores.max() 82 if max_score - min_score == 0: 83 return np.zeros_like(scores) 84 return (scores - min_score) / (max_score - min_score) 85 86 def search(self, query: str, method: str = "vector", top_k: int = 3) -> List[Dict[str, Any]]: 87 if method == "vector": 88 return self._vector_search(query, top_k) 89 elif method == "hybrid": 90 return self._hybrid_search(query, top_k) 91 else: 92 raise ValueError("method must be 'vector' or 'hybrid'") 93 94 def generate_answer(self, query: str, retrieved_docs: List[Dict[ str, Any]]) -> str: 95 """Gemini Λ ࢖ ͬ ͯ ճ ౴ Λ ੜ ੒""" 96 if not userdata.get('GOOGLE_API_KEY'): 97 return "Gemini API Ω ʔ ͕ ઃ ఆ ͞ Ε ͯ ͍ ͳ ͍ ͨ Ίɺ ճ ౴ Λ ੜ ੒ Ͱ ͖ · ͤ Μɻ " 98 99 context = "\n".join([f"จ ॻ{i+1}: {doc['document ']}" for i, doc in enumerate(retrieved_docs)]) 100 101 try: 102 model = genai.GenerativeModel('gemini -2.5-flash') 103 prompt = f"""͋ ͳ ͨ ͸ ఏ ڙ ͞ Ε ͨ จ ॻ ͷ Έ Λ ج ʹɺ ਖ਼ ֬ Ͱ ؆ ܿ ͳ ճ ౴ Λ ੜ ੒ ͢ Δ ઐ ໳ త ͳ Ξ γ ε λ ϯ τ Ͱ ͢ɻ ఏ ڙ ͞ Ε ͨ จ ॻ ʹ ؚ · Ε ͯ ͍ ͳ ͍ ৘ ใ ͸ ઈ ର ʹ ؚ Ί ͳ ͍ Ͱ ͘ ͩ ͞ ͍ɻ 104 105 ࣭ ໰: {query} 106 107 ؔ ࿈ จ ॻ: 108 {context} 109 110 ্ ه ͷ จ ॻ ͷ Έ Λ ࢀ ߟ ʹ ͠ ͯɺ ࣭ ໰ ʹ ճ ౴ ͠ ͯ ͘ ͩ ͞ ͍ɻ """ 111 112 response = model.generate_content(prompt) 113 return response.text.strip() 114 except Exception as e: 115 return f"ճ ౴ ੜ ੒ Τ ϥ ʔ: {str(e)}" 116 117 # ==================== 118 # 2. ධ Ձ γ ε ς Ϝ ͷ ࣮ ૷ 119 # ==================== 120 121 class RAGEvaluator: 122 def __init__(self): 123 self.embedding_model = SentenceTransformer('all-MiniLM -L6-v2' ) 124 self.rouge_scorer = rouge_scorer.RougeScorer(['rouge1', '
36. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 36 rougeL'], use_stemmer=True) 125 126 def evaluate_retrieval(self, retrieved_docs: List[Dict], relevant_doc_ids: List[int]) -> Dict[str, float]: 127 retrieved_ids = [doc['doc_id'] for doc in retrieved_docs] 128 relevant_set = set(relevant_doc_ids) 129 retrieved_set = set(retrieved_ids) 130 131 intersection = retrieved_set.intersection(relevant_set) 132 precision = len(intersection) / len(retrieved_set) if retrieved_set else 0 133 recall = len(intersection) / len(relevant_set) if relevant_set else 0 134 f1 = 2 * precision * recall / (precision + recall) if ( precision + recall) else 0 135 136 return {'precision': precision , 'recall': recall , 'f1': f1} 137 138 def evaluate_semantic_similarity(self, generated: str, reference: str) -> float: 139 gen_emb = self.embedding_model.encode([generated]) 140 ref_emb = self.embedding_model.encode([reference]) 141 return float(cosine_similarity(gen_emb , ref_emb)[0][0]) 142 143 def evaluate_rouge(self, generated: str, reference: str) -> Dict[ str, float]: 144 scores = self.rouge_scorer.score(reference , generated) 145 return { 146 'rouge1_f': scores['rouge1'].fmeasure , 147 'rougeL_f': scores['rougeL'].fmeasure 148 } 149 150 # ==================== 151 # 3. ς ε τ σ ʔ λ ͷ ४ උ 152 # ==================== 153 154 # ࣮ ࡍ ͷ ٕ ज़ จ ॻ σ ʔ λ 155 documents = [ 156 "ػ ց ֶ श ͸ ਓ ޻ ஌ ೳ ͷ Ұ ෼ ໺ Ͱɺ ί ϯ ϐ ϡ ʔ λ ͕ σ ʔ λ ͔ Β ࣗ ಈ త ʹ ύ λ ʔ ϯ Λ ֶ श ͠ɺ ༧ ଌ ΍ ෼ ྨ Λ ߦ ͏ ٕ ज़ Ͱ ͢ɻ ڭ ࢣ ͋ Γ ֶ शɺ ڭ ࢣ ͳ ͠ ֶ शɺ ڧ Խ ֶ श ͷ3ͭ ͷ ओ ཁ ͳ ֶ श ํ ࣜ ͕ ͋ Γ · ͢ɻ ", 157 "ਂ ૚ ֶ श ʢσ Ο ʔ ϓ ϥ ʔ χ ϯ άʣ ͸ ػ ց ֶ श ͷ ख ๏ ͷ Ұ ͭ Ͱɺ ଟ ૚ ͷ χ ϡ ʔ ϥ ϧ ω ο τ ϫ ʔ Ϋ Λ ࢖ ༻ ͠ ͯ σ ʔ λ ͷ ෳ ࡶ ͳ ύ λ ʔ ϯ Λ ֶ श ͠ · ͢ɻ ը ૾ ೝ ࣝɺ ࣗ વ ݴ ޠ ॲ ཧɺ Ի ੠ ೝ ࣝ ͳ Ͳ ͷ ෼ ໺ Ͱ ֵ ໋ త ͳ ੒ Ռ Λ ্ ͛ ͯ ͍ · ͢ɻ ", 158 "ࣗ વ ݴ ޠ ॲ ཧ ʢ NLP ʣ ͸ɺ ਓ ؒ ͕ ࢖ ༻ ͢ Δ ࣗ વ ݴ ޠ Λ ί ϯ ϐ ϡ ʔ λ ʹ ཧ ղɾ ॲ ཧ ͞ ͤ Δ ٕ ज़ ෼ ໺ Ͱ ͢ɻ ς Ω ε τ ෼ ੳɺ ػ ց ຋ ༁ɺ ࣭ ໰ Ԡ ౴ γ ε ς Ϝɺ ν ϟ ο τ Ϙ ο τ ͳ Ͳ ͷ Ԡ ༻ ͕ ͋ Γ · ͢ɻ ", 159 "RAG ʢ Retrieval -Augmented Generation ʣ ͸ɺ ֎ ෦ ͷ ஌ ࣝ ϕ ʔ ε ͔ Β ؔ ࿈ ৘ ใ Λ ݕ ࡧ ͠ɺ ͦ ͷ ৘ ใ Λ ج ʹ ͠ ͯ େ ن ໛ ݴ ޠ Ϟ σ ϧ ͕ ճ ౴ Λ ੜ ੒ ͢ Δ ख ๏ Ͱ ͢ɻ ϋ ϧ γ ω ʔ γ ϣ ϯ ͷ ཈ ੍ ͱ ࠷ ৽ ৘ ใ ͷ ׆ ༻ ͕ Մ ೳ Ͱ
37. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 37 ͢ɻ ", 160 "ϕ Ϋ τ ϧ ݕ ࡧ ͸ɺ จ ॻ ΍ ୯ ޠ Λ ߴ ࣍ ݩ ϕ Ϋ τ ϧ ۭ ؒ ʹ Τ ϯ ϕ σ Ο ϯ ά ͠ɺ ϕ Ϋ τ ϧ ؒ ͷ ྨ ࣅ ౓ Λ ܭ ࢉ ͠ ͯ ؔ ࿈ ͢ Δ ৘ ใ Λ ݕ ࡧ ͢ Δ ख ๏ Ͱ ͢ɻ ҙ ຯ త ͳ ྨ ࣅ ੑ Λ ଊ ͑ Δ ͜ ͱ ͕ Ͱ ͖ · ͢ɻ ", 161 "Transformer Ξ ʔ Ω ς Ϋ ν ϟ ͸ɺ Ξ ς ϯ γ ϣ ϯ ػ ߏ Λ த ֩ ͱ ͠ ͨ ਂ ૚ ֶ श Ϟ σ ϧ Ͱɺ ࣗ વ ݴ ޠ ॲ ཧ ʹ ͓ ͍ ͯ େ ͖ ͳ ੒ ޭ Λ ऩ Ί ͯ ͍ · ͢ɻ GPT ɺ BERT ɺ T5 ͳ Ͳ ͷ ଟ ͘ ͷ େ ن ໛ ݴ ޠ Ϟ σ ϧ ͕ Transformer Λ ϕ ʔ ε ͱ ͠ ͯ ͍ · ͢ɻ ", 162 "ڭ ࢣ ͋ Γ ֶ श ͸ɺ ೖ ྗ σ ʔ λ ͱ ਖ਼ ղ ϥ ϕ ϧ ͷ ϖ Ξ Λ ࢖ ͬ ͯ Ϟ σ ϧ Λ ܇ ࿅ ͢ Δ ػ ց ֶ श ͷ ख ๏ Ͱ ͢ɻ ෼ ྨ ໰ ୊ ΍ ճ ؼ ໰ ୊ ͷ ղ ܾ ʹ ࢖ ༻ ͞ Εɺ ܾ ఆ ໦ɺ α ϙ ʔ τ ϕ Ϋ λ ʔ Ϛ γ ϯɺ ϥ ϯ μ Ϝ ϑ Υ Ϩ ε τ ͳ Ͳ ͷ Ξ ϧ ΰ Ϧ ζ Ϝ ͕ ͋ Γ · ͢ɻ ", 163 "Ξ ς ϯ γ ϣ ϯ ػ ߏ ͸ɺ ೖ ྗ ͷ ֤ ෦ ෼ ʹ ର ͠ ͯ ಈ త ʹ ॏ ཁ ౓ Λ ܭ ࢉ ͠ɺ ॏ ཁ ͳ ෦ ෼ ʹ Α Γ ଟ ͘ ͷ ஫ ҙ Λ ޲ ͚ Δ ࢓ ૊ Έ Ͱ ͢ɻ ػ ց ຋ ༁ ΍ จ ॻ ཁ ໿ ͳ Ͳ ͷ λ ε Ϋ Ͱ େ ෯ ͳ ੑ ೳ ޲ ্ Λ ΋ ͨ Β ͠ · ͠ ͨɻ ", 164 "χ ϡ ʔ ϥ ϧ ω ο τ ϫ ʔ Ϋ ͸ɺ ਓ ؒ ͷ ೴ ͷ ਆ ܦ ࡉ ๔ ʢχ ϡ ʔ ϩ ϯʣ ͷ ಇ ͖ Λ ໛ ฿ ͠ ͨ ػ ց ֶ श Ϟ σ ϧ Ͱ ͢ɻ ೖ ྗ ૚ɺ Ӆ Ε ૚ɺ ग़ ྗ ૚ ͔ Β ߏ ੒ ͞ Εɺ ॏ Έ ͱ ό Π Ξ ε Λ ௐ ੔ ͢ Δ ͜ ͱ Ͱ ༷ ʑ ͳ λ ε Ϋ Λ ֶ श Ͱ ͖ · ͢ɻ ", 165 "ڧ Խ ֶ श ͸ɺ Τ ʔ δ Σ ϯ τ ͕ ؀ ڥ ͱ ͷ ૬ ޓ ࡞ ༻ Λ ௨ ͡ ͯɺ ใ ु Λ ࠷ େ Խ ͢ Δ Α ͏ ͳ ߦ ಈ Λ ֶ श ͢ Δ ػ ց ֶ श ͷ ख ๏ Ͱ ͢ɻ ή ʔ ϜAI ɺ ϩ Ϙ ο τ ੍ ޚɺ ࣗ ಈ ӡ స ͳ Ͳ ͷ ෼ ໺ Ͱ Ԡ ༻ ͞ Ε ͯ ͍ · ͢ɻ " 166 ] 167 168 # ධ Ձ ༻ ς ε τ έ ʔ ε 169 test_cases = [ 170 { 171 "query": "ਂ ૚ ֶ श ͱ χ ϡ ʔ ϥ ϧ ω ο τ ϫ ʔ Ϋ ͷ ؔ ܎ ʹ ͭ ͍ ͯ ڭ ͑ ͯ", 172 "relevant_docs": [1, 8], # ਂ ૚ ֶ श ͱ χ ϡ ʔ ϥ ϧ ω ο τ ϫ ʔ Ϋ ͷ จ ॻ 173 "expected_answer": "ਂ ૚ ֶ श ͸ ଟ ૚ ͷ χ ϡ ʔ ϥ ϧ ω ο τ ϫ ʔ Ϋ Λ ࢖ ༻ ͢ Δ ػ ց ֶ श ख ๏" 174 }, 175 { 176 "query": "RAG γ ε ς Ϝ ͷ ࢓ ૊ Έ ͱ ϝ Ϧ ο τ", 177 "relevant_docs": [3], # RAG ͷ จ ॻ 178 "expected_answer": "RAG ͸ ֎ ෦ ஌ ࣝ ϕ ʔ ε ͔ Β ৘ ใ Λ ݕ ࡧ ͠ ͯ ճ ౴ Λ ੜ ੒ ͠ɺ ϋ ϧ γ ω ʔ γ ϣ ϯ ཈ ੍ ͱ ࠷ ৽ ৘ ใ ׆ ༻ ͕ Մ ೳ" 179 }, 180 { 181 "query": "Transformer Ξ ʔ Ω ς Ϋ ν ϟ ͱ Ξ ς ϯ γ ϣ ϯ ػ ߏ", 182 "relevant_docs": [5, 7], # Transformer ͱ Attention ͷ จ ॻ 183 "expected_answer": "Transformer ͸ Ξ ς ϯ γ ϣ ϯ ػ ߏ Λ த ֩ ͱ ͠ ͨ ਂ ૚ ֶ श Ϟ σ ϧ Ͱ ଟ ͘ ͷ େ ن ໛ ݴ ޠ Ϟ σ ϧ ͷ ϕ ʔ ε" 184 }, 185 { 186 "query": "ػ ց ֶ श ͷ ֶ श ํ ࣜ ͷ छ ྨ", 187 "relevant_docs": [0, 6, 9], # ػ ց ֶ शɺ ڭ ࢣ ͋ Γ ֶ शɺ ڧ Խ ֶ श ͷ จ ॻ 188 "expected_answer": "ػ ց ֶ श ʹ ͸ ڭ ࢣ ͋ Γ ֶ शɺ ڭ ࢣ ͳ ͠ ֶ शɺ
38. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 38 ڧ Խ ֶ श ͷ3ͭ ͷ ओ ཁ ͳ ֶ श ํ ࣜ ͕ ͋ Δ" 189 } 190 ] 191 192 # ==================== 193 # 4. ࣮ ݧ ࣮ ߦ ͱ ݁ Ռ ද ࣔ 194 # ==================== 195 196 def run_comparison_experiment(): 197 print("=" * 60) 198 print("ϕ Ϋ τ ϧ ݕ ࡧ vs ϋ Π ϒ Ϧ ο υ ݕ ࡧ ਫ਼ ౓ ൺ ֱ ࣮ ݧ") 199 print("=" * 60) 200 201 # γ ε ς Ϝ ॳ ظ Խ 202 rag_system = RAGSystem(documents) 203 evaluator = RAGEvaluator() 204 205 results = {"vector": [], "hybrid": []} 206 207 for i, test_case in enumerate(test_cases): 208 print(f"\nς ε τ έ ʔ ε {i+1}: {test_case['query ']}") 209 print("-" * 50) 210 211 for method in ["vector", "hybrid"]: 212 # ݕ ࡧ ࣮ ߦ 213 retrieved_docs = rag_system.search(test_case['query'], method=method , top_k=3) 214 215 # ݕ ࡧ ੑ ೳ ධ Ձ 216 retrieval_metrics = evaluator.evaluate_retrieval( retrieved_docs , test_case['relevant_docs']) 217 218 # ݕ ࡧ ͞ Ε ͨ จ ॻ ͔ Β ؆ қ ճ ౴ ੜ ੒ 219 context = " ".join([doc['document'][:100] + "..." for doc in retrieved_docs]) 220 221 # ҙ ຯ త ྨ ࣅ ౓ ධ Ձ 222 semantic_sim = evaluator.evaluate_semantic_similarity( context , test_case['expected_answer']) 223 224 result = { 225 'query': test_case['query'], 226 'method': method , 227 'retrieval_f1': retrieval_metrics['f1'], 228 'retrieval_precision': retrieval_metrics['precision' ], 229 'retrieval_recall': retrieval_metrics['recall'], 230 'semantic_similarity': semantic_sim , 231 'retrieved_docs': [doc['doc_id'] for doc in retrieved_docs] 232 }
39. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 39 233 results[method].append(result) 234 235 print(f"{method.upper()}ݕ ࡧ:") 236 print(f" ݕ ࡧF1: {retrieval_metrics['f1 ']:.3f}") 237 print(f" ݕ ࡧ ਫ਼ ౓: {retrieval_metrics['precision ']:.3f} ") 238 print(f" ݕ ࡧ ࠶ ݱ ཰: {retrieval_metrics['recall ']:.3f}" ) 239 print(f" ҙ ຯ త ྨ ࣅ ౓: {semantic_sim:.3f}") 240 print(f" ݕ ࡧ จ ॻID: {[doc['doc_id'] for doc in retrieved_docs]}") 241 242 # ૯ ߹ ݁ Ռ ͷ ܭ ࢉ ͱ ද ࣔ 243 print("\n" + "=" * 60) 244 print("૯ ߹ ݁ Ռ ൺ ֱ") 245 print("=" * 60) 246 247 for method in ["vector", "hybrid"]: 248 method_results = results[method] 249 avg_f1 = np.mean([r['retrieval_f1'] for r in method_results]) 250 avg_precision = np.mean([r['retrieval_precision'] for r in method_results]) 251 avg_recall = np.mean([r['retrieval_recall'] for r in method_results]) 252 avg_semantic = np.mean([r['semantic_similarity'] for r in method_results]) 253 254 print(f"\n{method.upper()}ݕ ࡧ ͷ ฏ ۉ ੑ ೳ:") 255 print(f" ฏ ۉ ݕ ࡧF1: {avg_f1:.3f}") 256 print(f" ฏ ۉ ݕ ࡧ ਫ਼ ౓: {avg_precision:.3f}") 257 print(f" ฏ ۉ ݕ ࡧ ࠶ ݱ ཰: {avg_recall:.3f}") 258 print(f" ฏ ۉ ҙ ຯ త ྨ ࣅ ౓: {avg_semantic:.3f}") 259 260 # վ ળ ౓ ͷ ܭ ࢉ 261 vector_avg_f1 = np.mean([r['retrieval_f1'] for r in results[' vector']]) 262 hybrid_avg_f1 = np.mean([r['retrieval_f1'] for r in results[' hybrid']]) 263 # LLM-as-a-Judge ධ Ձ γ ε ς Ϝ 264 class LLMJudge: 265 def __init__(self): 266 if not userdata.get('GOOGLE_API_KEY'): 267 print("஫ ҙ: Gemini API Ω ʔ ͕ ઃ ఆ ͞ Ε ͯ ͍ · ͤ Μɻ LLM-as-a -Judge ධ Ձ ͸ ར ༻ Ͱ ͖ · ͤ Μɻ ") 268 269 def evaluate_faithfulness(self, answer: str, context: str) -> Dict[str, Any]: 270 """஧ ࣮ ੑ ධ Ձɿ ճ ౴ ͕ จ ॻ ʹ ஧ ࣮ ͔ Λ Gemini Ͱ ൑ ఆ""" 271 if not userdata.get('GOOGLE_API_KEY'): 272 return {"score": 0.0, "reason": "API Ω ʔ ະ ઃ ఆ"} 273
40. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 40 274 prompt = f"""Ҏ Լ ͷ จ ॻ ͱ ճ ౴ Λ ධ Ձ ͠ ͯ ͘ ͩ ͞ ͍ɻ 275 276 ʲจ ॻʳ 277 {context} 278 279 ʲճ ౴ʳ 280 {answer} 281 282 ʲධ Ձ ج ४ʳ 283 ճ ౴ ͷ ಺ ༰ ͕ ఏ ڙ ͞ Ε ͨ จ ॻ ͷ Έ ʹ ج ͮ ͍ ͯ ͍ Δ ͔ Λ ൑ ఆ ͠ ͯ ͘ ͩ ͞ ͍ɻ 284 - จ ॻ ʹ ه ࡌ ͞ Ε ͯ ͍ ͳ ͍ ৘ ใ ͕ ؚ · Ε ͯ ͍ Δ ৔ ߹ ͸ ஧ ࣮ ੑ ͕ ௿ ͍ 285 - จ ॻ ͷ ಺ ༰ ͱ ໃ ६ ͢ Δ ৘ ใ ͕ ؚ · Ε ͯ ͍ Δ ৔ ߹ ͸ ஧ ࣮ ੑ ͕ ௿ ͍ 286 - จ ॻ ͷ ಺ ༰ ͷ Έ Ͱ ߏ ੒ ͞ Ε ͯ ͍ Δ ৔ ߹ ͸ ஧ ࣮ ੑ ͕ ߴ ͍ 287 288 1-5ͷ ఺ ਺ Ͱ ධ Ձ ͠ɺ ͦ ͷ ཧ ༝ ΋ આ ໌ ͠ ͯ ͘ ͩ ͞ ͍ɻ 289 ճ ౴ ͸ ඞ ͣ Ҏ Լ ͷ ܗ ࣜ Ͱ ฦ ͠ ͯ ͘ ͩ ͞ ͍ɿ 290 ఺ ਺: [1-5 ͷ ੔ ਺] 291 ཧ ༝: [ධ Ձ ཧ ༝ Λ ؆ ܿ ʹ]""" 292 293 try: 294 model = genai.GenerativeModel('gemini -2.5-flash') 295 response = model.generate_content(prompt) 296 content = response.text.strip() 297 298 # ε ί Ξ ந ग़ 299 lines = content.split('\n') 300 score = 3.0 # σ ϑ Υ ϧ τ 301 reason = "ධ Ձ Λ औ ಘ Ͱ ͖ · ͤ Μ Ͱ ͠ ͨ" 302 303 for line in lines: 304 if '఺ ਺:' in line or 'ε ί Ξ:' in line: 305 try: 306 score = float(line.split(':')[1].strip()) / 5.0 # 0-1ʹ ਖ਼ ن Խ 307 except: 308 pass 309 elif 'ཧ ༝:' in line: 310 reason = line.split(':', 1)[1].strip() 311 312 return {"score": score, "reason": reason} 313 except Exception as e: 314 return {"score": 0.0, "reason": f"Τ ϥ ʔ: {str(e)}"} 315 316 def evaluate_relevancy(self, question: str, answer: str) -> Dict[ str, Any]: 317 """ଥ ౰ ੑ ධ Ձɿ ճ ౴ ͕ ࣭ ໰ ʹ ద ੾ ʹ ౴ ͑ ͯ ͍ Δ ͔ Λ Gemini Ͱ ൑ ఆ"" " 318 if not userdata.get('GOOGLE_API_KEY'): 319 return {"score": 0.0, "reason": "API Ω ʔ ະ ઃ ఆ"} 320 321 prompt = f"""Ҏ Լ ͷ ࣭ ໰ ͱ ճ ౴ Λ ධ Ձ ͠ ͯ ͘ ͩ ͞ ͍ɻ
41. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 41 322 323 ʲ࣭ ໰ʳ 324 {question} 325 326 ʲճ ౴ʳ 327 {answer} 328 329 ʲධ Ձ ج ४ʳ 330 ճ ౴ ͕ ࣭ ໰ ʹ ର ͠ ͯ Ͳ ͷ ఔ ౓ ద ੾ ʹ ౴ ͑ ͯ ͍ Δ ͔ Λ ൑ ఆ ͠ ͯ ͘ ͩ ͞ ͍ɻ 331 - ࣭ ໰ ͷ ҙ ਤ Λ ਖ਼ ͠ ͘ ཧ ղ ͠ ͯ ͍ Δ 332 - ࣭ ໰ ʹ ର ͢ Δ ௚ ઀ త ͳ ճ ౴ Λ ؚ Μ Ͱ ͍ Δ 333 - ճ ౴ ͷ ಺ ༰ ͕ ࣭ ໰ ͱ ؔ ࿈ ͠ ͯ ͍ Δ 334 335 1-5ͷ ఺ ਺ Ͱ ධ Ձ ͠ɺ ͦ ͷ ཧ ༝ ΋ આ ໌ ͠ ͯ ͘ ͩ ͞ ͍ɻ 336 ճ ౴ ͸ ඞ ͣ Ҏ Լ ͷ ܗ ࣜ Ͱ ฦ ͠ ͯ ͘ ͩ ͞ ͍ɿ 337 ఺ ਺: [1-5 ͷ ੔ ਺] 338 ཧ ༝: [ධ Ձ ཧ ༝ Λ ؆ ܿ ʹ]""" 339 340 try: 341 model = genai.GenerativeModel('gemini -2.5-flash') 342 response = model.generate_content(prompt) 343 content = response.text.strip() 344 345 # ε ί Ξ ந ग़ 346 lines = content.split('\n') 347 score = 3.0 # σ ϑ Υ ϧ τ 348 reason = "ධ Ձ Λ औ ಘ Ͱ ͖ · ͤ Μ Ͱ ͠ ͨ" 349 350 for line in lines: 351 if '఺ ਺:' in line or 'ε ί Ξ:' in line: 352 try: 353 score = float(line.split(':')[1].strip()) / 5.0 # 0-1ʹ ਖ਼ ن Խ 354 except: 355 pass 356 elif 'ཧ ༝:' in line: 357 reason = line.split(':', 1)[1].strip() 358 359 return {"score": score, "reason": reason} 360 except Exception as e: 361 return {"score": 0.0, "reason": f"Τ ϥ ʔ: {str(e)}"} 362 363 # ౷ ߹ ධ Ձ γ ε ς Ϝ 364 class ComprehensiveRAGEvaluator: 365 def __init__(self): 366 self.embedding_model = SentenceTransformer('all-MiniLM -L6-v2' ) 367 self.llm_judge = LLMJudge() 368 369 def evaluate_retrieval(self, retrieved_docs: List[Dict], relevant_doc_ids: List[int]) -> Dict[str, float]:
42. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 42 370 """ݕ ࡧ ੑ ೳ ධ Ձ""" 371 retrieved_ids = [doc['doc_id'] for doc in retrieved_docs] 372 relevant_set = set(relevant_doc_ids) 373 retrieved_set = set(retrieved_ids) 374 375 intersection = retrieved_set.intersection(relevant_set) 376 precision = len(intersection) / len(retrieved_set) if retrieved_set else 0 377 recall = len(intersection) / len(relevant_set) if relevant_set else 0 378 f1 = 2 * precision * recall / (precision + recall) if ( precision + recall) else 0 379 380 return {'precision': precision , 'recall': recall , 'f1': f1} 381 382 # ==================== 383 # 4. แ ׅ త ࣮ ݧ ࣮ ߦ 384 # ==================== 385 386 def run_comprehensive_experiment(): 387 print("=" * 60) 388 print("แ ׅ త RAG ධ Ձ ࣮ ݧ: ݕ ࡧ→ੜ ੒→ධ Ձ") 389 print("=" * 60) 390 391 # γ ε ς Ϝ ॳ ظ Խ 392 rag_system = RAGSystem(documents) 393 evaluator = ComprehensiveRAGEvaluator() 394 395 results = {"vector": [], "hybrid": []} 396 397 for i, test_case in enumerate(test_cases): 398 print(f"\nς ε τ έ ʔ ε {i+1}: {test_case['query ']}") 399 print("-" * 60) 400 401 for method in ["vector", "hybrid"]: 402 print(f"\n{method.upper()}ݕ ࡧ ࣮ ߦ த ...") 403 404 # 1. ݕ ࡧ ࣮ ߦ 405 retrieved_docs = rag_system.search(test_case['query'], method=method , top_k=3) 406 407 # 2. ճ ౴ ੜ ੒ 408 generated_answer = rag_system.generate_answer(test_case[' query'], retrieved_docs) 409 410 # 3. ݕ ࡧ ੑ ೳ ධ Ձ 411 retrieval_metrics = evaluator.evaluate_retrieval( retrieved_docs , test_case['relevant_docs']) 412 413 # 4. LLM-as-a-Judge ධ Ձ 414 context = "\n".join([doc['document'] for doc in
43. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 43 retrieved_docs]) 415 faithfulness = evaluator.llm_judge.evaluate_faithfulness( generated_answer , context) 416 relevancy = evaluator.llm_judge.evaluate_relevancy( test_case['query'], generated_answer) 417 418 # ݁ Ռ อ ଘ 419 result = { 420 'query': test_case['query'], 421 'method': method , 422 'generated_answer': generated_answer , 423 'retrieval_f1': retrieval_metrics['f1'], 424 'retrieval_precision': retrieval_metrics['precision' ], 425 'retrieval_recall': retrieval_metrics['recall'], 426 'faithfulness_score': faithfulness['score'], 427 'faithfulness_reason': faithfulness['reason'], 428 'relevancy_score': relevancy['score'], 429 'relevancy_reason': relevancy['reason'], 430 'retrieved_docs': [doc['doc_id'] for doc in retrieved_docs] 431 } 432 results[method].append(result) 433 434 # ݁ Ռ ද ࣔ 435 print(f"ੜ ੒ ճ ౴: {generated_answer[:100]}...") 436 print(f"ݕ ࡧ F1: {retrieval_metrics['f1 ']:.3f}") 437 print(f"஧ ࣮ ੑ: {faithfulness['score ']:.3f} - { faithfulness['reason ']}") 438 print(f"ଥ ౰ ੑ: {relevancy['score ']:.3f} - {relevancy[' reason ']}") 439 print(f"ݕ ࡧ จ ॻ ID: {[doc['doc_id'] for doc in retrieved_docs]}") 440 441 # ૯ ߹ ݁ Ռ ෼ ੳ 442 print("\n" + "=" * 60) 443 print("૯ ߹ ݁ Ռ ൺ ֱ") 444 print("=" * 60) 445 446 for method in ["vector", "hybrid"]: 447 method_results = results[method] 448 avg_retrieval = np.mean([r['retrieval_f1'] for r in method_results]) 449 avg_faithfulness = np.mean([r['faithfulness_score'] for r in method_results]) 450 avg_relevancy = np.mean([r['relevancy_score'] for r in method_results]) 451 452 print(f"\n{method.upper()}ݕ ࡧ ͷ ฏ ۉ ੑ ೳ:") 453 print(f" ݕ ࡧF1: {avg_retrieval:.3f}") 454 print(f" ஧ ࣮ ੑ: {avg_faithfulness:.3f}")
44. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 44 455 print(f" ଥ ౰ ੑ: {avg_relevancy:.3f}") 456 print(f" ૯ ߹ ε ί Ξ: {(avg_retrieval + avg_faithfulness + avg_relevancy) / 3:.3f}") 457 458 # վ ળ ޮ Ռ ͷ ෼ ੳ 459 vector_total = np.mean([ 460 (r['retrieval_f1'] + r['faithfulness_score'] + r[' relevancy_score']) / 3 461 for r in results['vector'] 462 ]) 463 hybrid_total = np.mean([ 464 (r['retrieval_f1'] + r['faithfulness_score'] + r[' relevancy_score']) / 3 465 for r in results['hybrid'] 466 ]) 467 468 improvement = ((hybrid_total - vector_total) / vector_total) * 100 if vector_total > 0 else 0 469 470 print(f"\nϋ Π ϒ Ϧ ο υ ݕ ࡧ ʹ Α Δ վ ળ:") 471 print(f" ૯ ߹ ε ί Ξ վ ળ: {improvement:+.1f}%") 472 473 if improvement > 5: 474 print(" → ϋ Π ϒ Ϧ ο υ ݕ ࡧ ͕ େ ෯ ʹ ༏ लʂ ") 475 elif improvement > 0: 476 print(" → ϋ Π ϒ Ϧ ο υ ݕ ࡧ ͕ ΍ ΍ ༏ ल") 477 else: 478 print(" → ͜ ͷ σ ʔ λ η ο τ Ͱ ͸ ࠩ ͕ খ ͞ ͍") 479 480 return results 481 482 # ࣮ ݧ ࣮ ߦ 483 print("࣮ ݧ Λ ։ ࢝ ͠ · ͢ ...") 484 experiment_results = run_comprehensive_experiment() 485 486 print("\n" + "=" * 60) 487 print("࣮ ݧ ׬ ྃʂ ") 488 print("͜ ͷ ౷ ߹ σ Ϟ Ͱ ͸ Ҏ Լ Λ ࣮ ݱ ͠ · ͠ ͨɿ ") 489 print("• ϕ Ϋ τ ϧ ݕ ࡧ vs ϋ Π ϒ Ϧ ο υ ݕ ࡧ ͷ ൺ ֱ") 490 print("• Gemini API ʹ Α Δ ࣮ ࡍ ͷ ճ ౴ ੜ ੒") 491 print("• LLM-as-a-Judge ʹ Α Δ ٬ ؍ త ධ Ձ") 492 print("• ݕ ࡧ ख ๏ ͕ ճ ౴ ඼ ࣭ ʹ ༩ ͑ Δ Ө ڹ ͷ ෼ ੳ") 493 print("=" * 60) 5.3 ຊষͷ·ͱΊͱ࣍ͷεςοϓ ͜ͷষͰֶΜͩ͜ͱɿ • ϕΫτϧݕࡧͱϋΠϒϦουݕࡧͷ࣮૷ͱੑೳൺֱ • Gemini API Λ࢖࣮ͬͨࡍͷ RAG ճ౴ੜ੒
45. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 45 • LLM-as-a-Judge ʹΑΔ٬؍తͳ඼࣭ධՁ • ݕࡧख๏͕࠷ऴతͳճ౴඼࣭ʹ༩͑ΔӨڹͷఆྔԽ ࣮ફతͳԠ༻ɿࠓճͷ౷߹σϞ͸ҎԼͷ৔໘Ͱ׆༻Ͱ͖·͢ɿ • ࣗࣾσʔλͰͷݕࡧख๏ͷબ୒ • RAG γεςϜͷ඼࣭վળͷޮՌଌఆ • ৽͍͠ධՁࢦඪͷ௥Ճɾݕূ • νϡʔχϯάύϥϝʔλͷ࠷దԽ ࣍ͷεςοϓɿ • ΑΓେ͖ͳσʔληοτͰͷ࣮ݧ • ଞͷݕࡧख๏ʢElasticsearchɺPinecone ౳ʣͱͷൺֱ • νϟϯΩϯάઓུ΍ΤϯϕσΟϯάϞσϧͷ࠷దԽ • υϝΠϯݻ༗σʔλͰͷੑೳݕূ • ଎౓޲্ɾεέʔϥϏϦςΟͷ࠷దԽ – ϕΫτϧσʔλϕʔεͷಋೖʢChromaɺQdrantɺWeaviate ౳ʣ – ۙࣅ࠷ۙ๣୳ࡧʢANNʣͷ׆༻ʢHNSWɺIVF ౳ʣ – ΠϯσοΫεߏஙͷ࠷దԽͱΩϟογϡઓུ – ฒྻॲཧʹΑΔݕࡧɾੜ੒ͷߴ଎Խ – ϝϞϦޮ཰తͳνϟϯΩϯάͱόονॲཧ ิ଍ɿϕΫτϧσʔλϕʔεͷબ୒ࢦ਑ ઐ༻ϕΫτϧσʔλϕʔεʢChromaɺQdrantɺWeaviate ౳ʣvs PostgreSQL + pgvector ֦ுɿ ઐ༻ϕΫτϧ DB ͷ༏Ґੑɿ – ଎౓ɿϕΫτϧݕࡧʹಛԽͨ͠ઃܭʹΑΓɺେن໛σʔλʢ100 ສϕΫτ ϧҎ্ʣͰ਺ഒʙ਺ेഒߴ଎ – ઐ༻ΞϧΰϦζϜɿ HNSW ౳ͷߴ଎ۙࣅݕࡧΞϧΰϦζϜΛඪ४࣮૷ – ϝϞϦޮ཰ɿϕΫτϧσʔλͷޮ཰త഑ஔɺSQL Φʔόʔϔουͳ͠ – ฒྻॲཧɿϕΫτϧݕࡧઐ༻ͷฒྻΤϯδϯɺGPU ΞΫηϥϨʔγϣϯ ରԠ PostgreSQL + pgvector ͷར఺ɿ – γεςϜ౷߹ɿطଘͷ PostgreSQL ΠϯϑϥΛ׆༻Մೳ – ৴པੑɿ ACID ಛੑɺτϥϯβΫγϣϯɺόοΫΞοϓ౳ͷاۀϨϕϧػ ೳ – σʔλ݁߹ɿ SQL ͰଞͷϏδωεσʔλͱ༰қʹ݁߹Մೳ
46. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 46 – খن໛Ͱ͸े෼ɿ 100 ສϕΫτϧҎԼͰ͸ੑೳࠩ͸খ͍͞ બ୒ج४ɿ – ઐ༻ϕΫτϧ DBɿߴ଎ݕࡧɾେن໛σʔλɾϦΞϧλΠϜੑΛॏࢹ͢Δ ৔߹ – PostgreSQL + pgvectorɿطଘγεςϜ౷߹ɾσʔλ੔߹ੑɾதن໛σ ʔλͷ৔߹ 6 ࢀߟจݙɾิ଍ࢿྉ References [1] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Timo Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459–9474, 2020. [2] Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to information retrieval. Cambridge university press, 2008. [3] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [6] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333– 389, 2009. [7] Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern information retrieval: The concepts and technology behind search. Addison-Wesley, 2011. [8] Elastic N.V. Elasticsearch: The definitive guide. https://www.elastic.co/guide/, 2023. [9] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. [10] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. [11] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
47. RAG× ධՁͰ௫ΉʂԠ౴ਫ਼౓޲্Ϩγϐ 47 [12] Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023.