Foundation Models でオンデバイスRAGを試みる

by p0dee

Slide 1

Slide 1 text

Foundation Models Ͱ ΦϯσόΠεRAGΛࢼΈΔ extension DC 2025 Day1@DeNA, 2025/10/01 Takeshi Tanaka @p0dee

Slide 2

Slide 2 text

Tanaka Takeshi X: p0dee iOS/Android Engineer झຯ͸δϟζͱےτϨͰ͢ɻ

Slide 3

Slide 3 text

📖 ೔ه

Slide 4

Slide 4 text

📖 --.

Slide 5

Slide 5 text

📖 --. 😕

Slide 6

Slide 6 text

📖 Foundation Models (On-device LLM)

Slide 7

Slide 7 text

🤷 📖

Slide 8

Slide 8 text

RAG LLM͕֎෦ιʔε͔Β৘ใऔಘ͠ճ౴Λੜ੒͢Δ Retrieval-Augmented Generation

Slide 9

Slide 9 text

TOOL CALLING ࣭໰ ճ౴ 👤 📖 ݕࡧΫΤϦ ֘౰৘ใ

Slide 10

Slide 10 text

TOOL CALLING 👤 📖 ࣭໰ ճ౴ ɾ'.͸࣭໰Λਖ਼͘͠ղऍ͠ɺద੾ͳݕࡧΫΤϦΛੜ੒Ͱ͖Δͷ͔ ɾ'.͸ಘΒΕͨ೔هσʔλΛਖ਼͘͠ղऍ͠ղ౴Ͱ͖Δͷ͔ ݕࡧΫΤϦ ֘౰৘ใ

Slide 11

Slide 11 text

౴͑ TOOL CALLING 👤 ࣭໰ ճ౴ Instruction: “Ϣʔβʔ͸ࣗ෼ͷ೔هʹ͍࣭ͭͯ໰Λ౤͔͚͛·͢ɻ ͋ͳͨ͸ؔ࿈͢ΔΩʔϫʔυΛ࢖༻ͯ͠೔هͷσʔλΛݕࡧ͠ɺ ͦͷ࣭໰ʹճ౴͠·͢ɻ” ݻఆͷμϛʔσʔλ ݕࡧΫΤϦ ֘౰৘ใ 2025೥07݄18೔ ࢄ൅Λ໎͍ͬͯΔ೔ه 2025೥6݄9೔ ࢄ൅ʹߦͬͨ೔ه 2025݄4݄08೔ ࢄ൅ʹߦͬͨ͜ͱΛ ༑ୡʹؾ͕͍ͭͯ΋Β͑ͨ (ຊจলུ)

Slide 12

Slide 12 text

TOOL CALLING 2025೥07݄18೔ ࢄ൅Λ໎͍ͬͯΔ೔ه 2025೥6݄9೔ ࢄ൅ʹߦͬͨ೔ه 2025݄4݄08೔ ࢄ൅ʹߦͬͨ͜ͱΛ ༑ୡʹؾ͕͍ͭͯ΋Β͑ͨ (ຊจলུ) ݻఆͷμϛʔσʔλ “ࢄ൅” 👍'.͸࣭໰Λਖ਼͘͠ղऍ͠ɺద੾ͳݕࡧΫΤϦΛੜ੒Ͱ͖ͨ 👍'.͸ಘΒΕͨ೔هσʔλΛਖ਼͘͠ղऍ͠ղ౴Ͱ͖ͨ “൅Λ੾ͬͨ”

Slide 13

Slide 13 text

TOOL CALLING 👤 📖 ࣭໰ ճ౴ ݕࡧΫΤϦ ֘౰৘ใ 👍

Slide 14

Slide 14 text

TOOL CALLING 👤 📖 ࣭໰ ճ౴ ɾ'.ʹ೔هΛɺΦϯσόΠεͰɺͲͷΑ͏ʹݕࡧͤ͞Δ͔ ɾݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖Δ͔ʢ෦෼ҰகʻηϚϯςΟοΫʣ

Slide 15

Slide 15 text

Spotlight (On-device searching)

Slide 16

Slide 16 text

https://developer.apple.com/jp/videos/play/wwdc2024/10131/

Slide 17

Slide 17 text

👤 📖 ࣭໰ ճ౴ 👍೔هΛΦϯσόΠεͰݕࡧͰ͖ͨ ❌ݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖ͳ͔ͬͨʢηϚϯςΟοΫݕࡧͷਫ਼౓ෆ଍ʣ

Slide 18

Slide 18 text

https://x.com/ShunTakeishi/status/1958363232068128879

Slide 19

Slide 19 text

ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ ࡞ͬͯΈΔʂ

Slide 20

Slide 20 text

4UFQจॻ ೔هΤϯτϦ ͝ͱʹϕΫτϧԽʢ/-$POUFYUVBM&NCFEEJOHʣ 4UFQ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ "DDFMFSBUFPS.-5FOTPS 4UFQྨࣅ౓είΞͰιʔτ͠ɺ্ҐΛώοτͱ͢Δ

Slide 21

Slide 21 text

􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿

Slide 22

Slide 22 text

͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺ ՆͰ΋ఈʹྫྷͨ͞Λ΋ͭ੨͍ͦΒɺ ͏͍ͭ͘͠৿Ͱ০ΒΕͨϞϦʔΦࢢɺ ߫֎ͷ͗Β͗Βͻ͔Δ૲ͷ೾ɻ /-$POUFYUVBM&NCFEEJOH 􀈿

Slide 23

Slide 23 text

͋ͷ / Πʔ / ϋ / τʔ / ϰΥ / ͷ / ͖͢ / ͱ / ͓ͬ / ͨ / ෩ / ɺ / Ն / Ͱ΋ / ఈ / ʹ / ྫྷ / ͨ / ͞ / Λ / ΋ͭ / ੨͍ / ͦΒ / ɺ / ͏ / ͭ / ͘ / ͍͠ / ৿ / Ͱ / ০Β / Ε / ͨ / Ϟ / Ϧʔ / Φ / ࢢ / ɺ / ߫֎ / ͷ / ͗ / Β / ͗ / Β / ͻ͔Δ / ૲ / ͷ / ೾ / ɻ

Slide 24

Slide 24 text

͋ͷɹɹ[-0.231, 0.027, 0.210 ... -0.128, 0.172, -0.151] Πʔɹɹ[ 0.035, -0.122, 0.064 ... -0.127, -0.109, -0.100] ϋɹɹɹ[-0.052, 0.195, 0.281 ... 0.003, 0.064, -0.048] τʔɹɹ[ 0.240, 0.309, 0.173 ... 0.095, 0.054, -0.172] … ͻ͔Δɹ[ 0.054, 0.104, -0.062 ... 0.048, 0.160, -0.001] ૲ɹɹɹ[ 0.055, -0.059, 0.142 ... -0.024, -0.082, -0.045] ͷɹɹɹ[-0.160, -0.062, 0.464 ... -0.056, 0.215, -0.091] ೾ɹɹɹ[ 0.041, 0.101, 0.223 ... 0.092, 0.027, 0.069]

Slide 25

Slide 25 text

Slide 26

Slide 26 text

􀈿 ɹɹɹ[ 0.022, -0.000, 0.034 ... 0.007, 0.019, -0.007] 􀈿 ɹɹɹ[ 0.081, -0.033, -0.040 ... 0.056, -0.015, 0.017] 􀈿 ɹɹɹ[ 0.015, -0.003, 0.040 ... 0.054, -0.005, -0.004] … 􀈿 ɹɹɹ[ 0.057, 0.012, -0.050 ... 0.082, -0.002, 0.057] 􀈿 ɹɹɹ[ 0.018, 0.021, -0.023 ... -0.070, -0.017, 0.006] 􀈿 ɹɹɹ[-0.022, 0.005, -0.008 ... -0.020, 0.021, -0.044] 4UFQจॻ ೔هΤϯτϦ ͝ͱʹϕΫτϧԽʢ/-$POUFYUVBM&NCFEEJOHʣ

Slide 27

Slide 27 text

􀈿 ɹɹɹ[ 0.022, -0.000, 0.034 ... 0.007, 0.019, -0.007] 􀈿 ɹɹɹ[ 0.081, -0.033, -0.040 ... 0.056, -0.015, 0.017] 􀈿 ɹɹɹ[ 0.015, -0.003, 0.040 ... 0.054, -0.005, -0.004] … 􀈿 ɹɹɹ[ 0.057, 0.012, -0.050 ... 0.082, -0.002, 0.057] 􀈿 ɹɹɹ[ 0.018, 0.021, -0.023 ... -0.070, -0.017, 0.006] 􀈿 ɹɹɹ[-0.022, 0.005, -0.008 ... -0.020, 0.021, -0.044] ʁɹɹɹ[-0.062, -0.036, -0.046 ... -0.009, -0.020, 0.011] 4UFQ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ “ඒ͍͠৿Ͱ০ΒΕͨࢢ͸ʁ”

Slide 28

Slide 28 text

􀈿 0.022 -0.000 ... 0.019 -0.007 􀈿 0.081 -0.033 ... -0.015 0.017 􀈿 0.015 -0.003 ... -0.005 -0.004 ɹɹɹɹɹɹɹɹɹɹɹ…ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ 􀈿 0.057 0.012 ... -0.002 0.057 􀈿 0.018 0.021 ... -0.017 0.006 􀈿 -0.022 0.005 ... 0.021 -0.044 4UFQ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ -0.062 -0.036 -0.046 ... -0.009 -0.020 0.011 -0.012 0.802 0.029 ... 0.510 -0.001 -0.910 = ֤จॻΛࣔ͢ߦϕΫτϧɾ࣭໰ΫΤϦΛࣔ͢ྻϕΫτϧͷ಺ੵ͕ ίαΠϯྨࣅ౓ d ʹͳΔʢʹ͍ۙ΄Ͳྨࣅɺʹ͍ۙ΄Ͳ૬ҧʣ

Slide 29

Slide 29 text

􀈿 0.022 -0.000 ... 0.019 -0.007 􀈿 0.081 -0.033 ... -0.015 0.017 􀈿 0.015 -0.003 ... -0.005 -0.004 ɹɹɹɹɹɹɹɹɹɹɹ…ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ 􀈿 0.057 0.012 ... -0.002 0.057 􀈿 0.018 0.021 ... -0.017 0.006 􀈿 -0.022 0.005 ... 0.021 -0.044 4UFQྨࣅ౓είΞͰιʔτ͠ɺ্ҐΛώοτͱ͢Δ -0.062 -0.036 -0.046 ... -0.009 -0.020 0.011 -0.012 0.802 0.029 ... 0.510 -0.001 -0.910 = ֤จॻΛࣔ͢ߦϕΫτϧɾ࣭໰ΫΤϦΛࣔ͢ྻϕΫτϧͷ಺ੵ͕ ίαΠϯྨࣅ౓ d ʹͳΔʢʹ͍ۙ΄Ͳྨࣅɺʹ͍ۙ΄Ͳ૬ҧʣ

Slide 30

Slide 30 text

“࠷ޙʹ൅੾ͬͨͷ͍ͭʁ” “મ౬ͰԿҿΜͩʁ” “٤஡ళͰձͬͨͷ୭ʁ” “όΠτͷ஌ਓʹ͋ͬͨͷͲ͜ʁ” ˞ݸͷ୹จ "*ੜ੒ ͔Βݕࡧ #1 #1 #1 #3

Slide 31

Slide 31 text

import Accelerate import CoreML internal import NaturalLanguage let embedding: NLContextualEmbedding = … // 1ςΩετΛϕΫτϧԽʢmean pooling + L2ਖ਼نԽʣ func tensorize(text: String, asColumnVector: Bool = false) throws -> MLTensor? { let result = try embedding.embeddingResult(for: text, language: language) let dim = embedding.dimension var sum = [Float](repeating: 0, count: dim) var count = 0 result.enumerateTokenVectors(in: text.startIndex.. Float ΛҰׅม׵ var vecF = [Float](repeating: 0, count: dim) vDSP_vdpsp(vecD, 1, &vecF, 1, vDSP_Length(dim)) // sum += vecF vDSP_vadd(sum, 1, vecF, 1, &sum, 1, vDSP_Length(dim)) count += 1 return true } guard count > 0 else { return nil } // mean poolingʢsum /= countʣ var invN = 1.0 / Float(count) vDSP_vsmul(sum, 1, &invN, &sum, 1, vDSP_Length(dim)) // L2 ਖ਼نԽ let normVector = l2Normalize(sum) let shape = asColumnVector ? [normVector.count, 1] : [1, normVector.count] return MLTensor(shape: shape, scalars: normVector) } // L2 ਖ਼نԽ private func l2Normalize(_ vec: [Float]) -> [Float] { var norm: Float = 0 vDSP_svesq(vec, 1, &norm, vDSP_Length(vec.count)) norm = sqrtf(norm) + 1e-12 vDSP_vsdiv(vec, 1, &norm, &vec, 1, vDSP_Length(vec.count)) return vec } iOSʹ૊Έࠐ·ΕͨBERTͰςΩετຒΊࠐΈɾϕΫτϧݕࡧΛΦϯσόΠε࣮ߦ͢Δ #Mac – Qiita ௥هʢ2025/10/05ʣɿͪ͜Βͷهࣄ͕ΑΓࢀߟʹͳΔͱࢥΘΕ·͢ʂ https://qiita.com/5enxia/items/eea83cfd277584e00748

Slide 32

Slide 32 text

var contents: [UUID : String] = [:] var tensors: [UUID : MLTensor] = [:] func embed(items: [String]) throws { var ret: [UUID : String] = [:] do { try items.forEach { item in let tensor = try encode(text: item) let uuid = UUID() tensors[uuid] = tensor ret[uuid] = item } } catch { // error handling } contents = ret } func search(query: String, topCount: Int) async -> [String] { // υΩϡϝϯτΛूੵͨ͠ D*M ߦྻ let flatteneds = tensors.values.map { $0.flattened() } let docsTensor = MLTensor(stacking: flatteneds) // ΫΤϦϕΫτϧ guard let queryTensor = try? encode(text: query, asColumnVector: true) else { return [] } let product = docsTensor.matmul(queryTensor) let calcScores = await product.shapedArray(of: Float.self).scalars let arr = Array(zip(tensors.map(\.key), calcScores)) // ྨࣅ౓Ͱιʔτ্͠ҐN݅Λநग़ let sorted = arr.sorted { $0.1 > $1.1 }.prefix(topCount) // ্ҐUUID͝ͱʹରԠ͢ΔυΩϡϝϯτΛ contents ͔Β୳ࡧ return ... }

Slide 33

Slide 33 text

ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ 👍೔هΛΦϯσόΠεͰݕࡧͰ͖ͨ 👍ݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖ͨ

Slide 34

Slide 34 text

ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ 👍 👍

Slide 35

Slide 35 text

εΫγϣ

Slide 36

Slide 36 text

⁉ εΫγϣ

Slide 37

Slide 37 text

εΫγϣ πʔϧݺͼग़ͯ͠ͳ͍ ؔ܎ͷͳ͍ݕࡧΩʔϫʔυͰݕࡧ ଘࡏ͠ͳ͍೔هΤϯτϦΛ፻଄

Slide 38

Slide 38 text

2025೥8݄

Slide 39

Slide 39 text

ηϚϯςΟοΫݕࡧͷ࣮૷ΞϓϩʔνΛ঺հ ɾNLContextualEmbedding + MLTensor FM͸ϞσϧΞοϓσʔτʹ൐͏඼࣭ͷมԽʹ஫ҙ ɾϨεϙϯε඼࣭ͷ҆ఆੑ ɾπʔϧݺͼग़͠ͷ࠶ݱੑ ໋ྩදݱ΍ɺखॱΛ໌ࣔͨ͠ϓϩϯϓτɺfew-shot ͳͲ͋ΒΏΔςΫχοΫΛࢼ͕ͨ͠ޮՌ͕ݟΒΕͳ͍

Slide 40

Slide 40 text

FMΛࣂ͍׳Β͢ํ๏ ڭ͍͑ͯͩ͘͞ 🙇

Slide 41

Slide 41 text

Foundation Models Ͱ ΦϯσόΠεRAGΛࢼΈΔ extension DC 2025 Day1@DeNA, 2025/10/01 Takeshi Tanaka @p0dee