AzureのPrompt Flowを使ったRAG精度評価とつらみ

by id32h6kz

Slide 1

Slide 1 text

AzureͷPrompt FlowΛ࢖ͬͨ RAGਫ਼౓ධՁͱͭΒΈ KDDI ΞδϟΠϧ։ൃηϯλʔ גࣜձࣾ 2024.07.11 Findy Lunch LT

Slide 2

Slide 2 text

ࣗݾ঺հ • ͍ͰΈͭ • KDDI ΞδϟΠϧ։ൃηϯλʔ גࣜձࣾ • ۀ຿ɿੜ੒AIΛར༻ͨ͠ΞϓϦέʔγϣϯ։ൃ • ޷͖ͳAzureαʔϏεɿDocument Intelligence

Slide 3

Slide 3 text

͓͠ͳ͕͖ • ͸͡Ίʹ • ΞʔΩςΫνϟ঺հ • ࠓ೔఻͍͑ͨ͜ͱ • Prompt FlowͱRAGASͷ঺հ • ·ͱΊ

Slide 4

Slide 4 text

͸͡Ίʹ • ༷ʑͳυΩϡϝϯτΛRAGͱͯ͠ੜ੒AIʹճ౴ͤ͞ΔαʔϏε • ·ͩ։ൃதͷͨΊɺϦϦʔε͸͍ͯ͠·ͤΜ • ਫ਼౓޲্ͦͷ΋ͷ͸·ͩ·ͩ໛ࡧத

Slide 5

Slide 5 text

ࠓճLTͷϕʔεͱͳΔγεςϜͷ ؆୯ͳΞʔΩςΫνϟ঺հ ར༻ऀ υΩϡϝϯτ Azure App Service Document Intelligence AI Search gpt-4o ᶃ ᶄ ᶅ ᶆ

Slide 6

Slide 6 text

ຊ೔఻͍͑ͨ͜ͱ

Slide 7

Slide 7 text

RAGͷਫ਼౓޲্ͬͯେมʂ

Slide 8

Slide 8 text

Ҋ͕݅ελʔτͨ͠λΠϛϯάͰPOͷํʹ ͜͏ݴΘΕ·ͨ͠

Slide 9

Slide 9 text

ࠓ͸·ͩճ౴ਫ਼౓͕௿͍

Slide 10

Slide 10 text

ճ౴ਫ਼౓͸80%Λ໨ࢦ͔͢Β

Slide 11

Slide 11 text

ࣗ༝ʹೖྗ͢Δ͜ͱͷͰ͖Δɺνϟοτ ϘοτͰճ౴ਫ਼౓80%ͬͯͲ͏͍͏͜ͱʁ

Slide 12

Slide 12 text

ͦ΋ͦ΋ճ౴ਫ਼౓͕ߴ͍/௿͍ͱײ͡Δͷ͸ ײ֮ͳͷͰɺ࣮ࡍͲΕ͘Β͍ͳͷ͔ΛՄࢹ Խ͍ͨ͠

Slide 13

Slide 13 text

ͦ͜Ͱ·ͣਫ਼౓ධՁʹ͍ͭͯͲ͏͢Δ΂͖ ͔ݕ౼͠·ͨ͠

Slide 14

Slide 14 text

ਫ਼౓ධՁͷඞཁੑ • ճ౴ਫ਼౓ͷՄࢹԽ • ࠓޙͷվળͷࢦ਑ • վળͷޮՌݕূ

Slide 15

Slide 15 text

ਫ਼౓ධՁͷओͳख๏ • खಈධՁ(ਓ͕ؒߦ͏ධՁ) • γεςϜʹରͯ͠QΛ౤͛ɺAͷྑ͠ѱ͠Λਓ͕ؒ൑அ͢Δ • ࣗಈධՁ(LLMͳͲ͕ߦ͏ධՁ) • γεςϜʹରͯ͠QΛ౤͛ɺAͷྑ͠ѱ͠ΛLLM͕൑அ͢Δ

Slide 16

Slide 16 text

Slide 17

Slide 17 text

ਫ਼౓ධՁʹ࢖͑ͦ͏ͳαʔϏε • Prompt Flow • 🦜⚒LangSmith • LangFuse

Slide 18

Slide 18 text

ਫ਼౓ධՁʹ࢖͑ͦ͏ͳαʔϏε • Prompt Flow • 🦜⚒LangSmith • LangFuse

Slide 19

Slide 19 text

Prompt Flowͷ֓ཁ • LLMΛ׆༻ͨ͠ΞϓϦέʔγϣϯͷ։ൃޮ཰ԽΛਤΔͨΊͷαʔ Ϗε • ࣭໰͔Βճ౴·ͰͷҰ࿈ͷFlowΛઃఆ࣮͠ߦ͢Δ͜ͱ͕Ͱ͖Δ • ֤Flow͸LLMΛ࢖ͬͨςΩετੜ੒΍ϓϩϯϓτ࡞੒ɺPython ͷ࣮ߦ͕Մೳͱͳ͓ͬͯΓɺ͜ΕΒΛ૊Έ߹Θͤͯશମͷϑϩʔ Λ࡞੒͢Δ

Slide 20

Slide 20 text

Prompt FlowΛબΜͩཧ༝ • AzureΛར༻͍ͯͨͨ͠Ίɺಋೖϋʔυϧ͕௿͍ • RAGͷධՁ͚ͩͰͳ͘։ൃͷࣄલݕূʹ΋ར༻͢Δ͜ͱ͕Ͱ͖ Δ(promptΛमਖ਼͓ͯ͠ࢼ͠౳)

Slide 21

Slide 21 text

RAGASͱ͸ • PythonͰ࡞ΒΕͨRAGධՁ༻ϑϨʔϜϫʔΫɺPrompt Flowʹ૊ΈࠐΉ͜ͱ͕Մೳ • 9ͭͷϝτϦΫεͰճ౴ΛධՁ͢Δ͜ͱ͕Մೳ • https://docs.ragas.io/en/stable/concepts/metrics/index.html

Slide 22

Slide 22 text

࣮ࡍʹPrompt FlowͱRAGASΛ࢖ͬͯධ ՁΛͲͷΑ͏ʹߦ͏͔

Slide 23

Slide 23 text

Prompt FlowͱRAGASͷ঺հ ඪ४ϑϩʔ(ΞϓϦέʔγϣϯΛ࠶ݱͨ͠ϑϩʔ)

Slide 24

Slide 24 text

Prompt FlowͱRAGASͷ঺հ ධՁϑϩʔ(RAGASݺͿϑϩʔ)

Slide 25

Slide 25 text

Prompt FlowͱRAGASͷ঺հ RAGASͷධՁ݁Ռ

Slide 26

Slide 26 text

Prompt FlowͱRAGASͷ঺հ RAGASͷධՁ݁Ռ ճ౴ͷਖ਼֬͞ ίϯςΩετʹର͢Δճ౴ͷ஧࣮౓ ਖ਼౴ͱίϯςΩετͷؔ࿈౓ ࣭໰ٴͼਖ਼౴ʹର͢ΔίϯςΩετͷਫ਼౓

Slide 27

Slide 27 text

Prompt FlowͱRAGASΛ ࢖ͬͯྑ͔ͬͨ͜ͱ • ϒϥ΢β͔Β৮ΕΔͨΊɺΤϯδχΞ͚ͩͰͳ͘PO΋৮ΕΔ • ճ౴ਫ਼౓޲্ͷͨΊͷࢪࡦ͕ຊ౰ʹޮՌ͕͋ͬͨͷ͔Λ֬ೝ͢ Δ͜ͱ͕Ͱ͖Δ

Slide 28

Slide 28 text

Prompt Flow࠷ߴʂ

Slide 29

Slide 29 text

PO΋৮ͬͯ͘Εͯɺධ൑΋͍͍ʂ

Slide 30

Slide 30 text

Ͱ΋ʂ

Slide 31

Slide 31 text

Prompt Flowͷ͕ͭ͜͜Β͍ʂ • Prompt Flowͷιʔεߋ৽͕େม • ࣮ࡍͷΞϓϦέʔγϣϯͱPrompt Flowͷίʔυ͸ผ෺ • ैྔ՝ۚͳͷͰɺ౎౓ىಈ͍͕ͨ͠VMͷىಈʹඇৗʹ͕࣌ؒ ͔͔Δ(5෼Ҏ্)

Slide 32

Slide 32 text

ͦΜͳதɺଟ͘ͷ՝୊͸

Slide 33

Slide 33 text

Visual Studio Codeͷ֦ுػೳ΍Prompt Flow CLI toolΛ࢖͏ͱղফͰ͖Δ

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

ͦͷଞͷ՝୊ • RAGASͰධՁ͢ΔͨΊͷQAσʔλͷ࡞੒͕೉͍͠ • ݱঢ়QAͷࣗಈੜ੒͸͋·Γਫ਼౓͕ग़ͳ͔ͬͨͨΊਓͷखͰ࡞ ੒͍ͯ͠Δɻ • େྔʹ࡞Δʹ͸͕͔͔࣌ؒΔͨΊɺܧଓతʹ௥Ճ͢Δӡ༻͕ ྑͦ͞͏

Slide 36

Slide 36 text

কདྷతʹ໨ࢦ͍ͨ͜͠ͱ • Prompt FlowͱGithub Actionsͷ࿈ܞ • LangFuseΛར༻ͨ͠feedbackऔಘ • ධՁ༻ͷQAσʔλΛ࡞੒͢Δ޻਺ͷ࡟ݮ

Slide 37

Slide 37 text

·ͱΊ • RAGͷਫ਼౓޲্ʹ͸ධՁ͕ෆՄܽ • खಈͰͷධՁ͸ݱ࣮తͰ͸ͳ͘ɺͱͯ΋େม • RAG͸ճ౴ਫ਼౓ΛධՁ͢Δ͜ͱͰɺରࡦΛߟ͑Δ͜ͱ͕Ͱ͖Δ • ఆظతʹධՁΛ͢Δ͜ͱͰɺߦͳ͍ͬͯΔࢪࡦͷޮՌΛଌఆ͢Δ͜ͱ͕Ͱ͖ Δ • QAϦετͷ࡞੒ʹ͕͔͔࣌ؒΔͷͰɺલ΋ͬͯ࣌ؒΛઃ͚࡞੒͢Δͷ͕ྑ͍

Slide 38

Slide 38 text

͋Γ͕ͱ͏͍͟͝·ͨ͠