Foundation Models で オンデバイスRAGを試みる

extension DC 2025 Day1@DeNA (2025/10/01)　で話した内容の、スキップしたスライドも含めたノーカット版です。

紹介ツイート：https://x.com/ShunTakeishi/status/1958363232068128879
登壇者の試行錯誤ブログ：https://p0dee.com/blog/tag/foundation-models/

2025/10/05：ソースコードを修正
本プレゼンで紹介している実装については、以下のQiita記事がより参考になると思います。
https://qiita.com/5enxia/items/eea83cfd277584e00748

Transcript

1. Foundation Models Ͱ ΦϯσόΠεRAGΛࢼΈΔ extension DC 2025 Day1@DeNA, 2025/10/01 Takeshi Tanaka @p0dee
2. Tanaka Takeshi X: p0dee iOS/Android Engineer झຯ͸δϟζͱےτϨͰ͢ɻ
3. 📖 ೔ه
4. 📖 --.
5. 📖 --. 😕
6. 📖 Foundation Models (On-device LLM)
7. 🤷 📖
8. RAG LLM͕֎෦ιʔε͔Β৘ใऔಘ͠ճ౴Λੜ੒͢Δ Retrieval-Augmented Generation
9. TOOL CALLING ࣭໰ ճ౴ 👤 📖 ݕࡧΫΤϦ ֘౰৘ใ
10. TOOL CALLING 👤 📖 ࣭໰ ճ౴ ɾ'.͸࣭໰Λਖ਼͘͠ղऍ͠ɺద੾ͳݕࡧΫΤϦΛੜ੒Ͱ͖Δͷ͔
    ɾ'.͸ಘΒΕͨ೔هσʔλΛਖ਼͘͠ղऍ͠ղ౴Ͱ͖Δͷ͔ ݕࡧΫΤϦ ֘౰৘ใ
11. ౴͑ TOOL CALLING 👤 ࣭໰ ճ౴ Instruction: “Ϣʔβʔ͸ࣗ෼ͷ೔هʹ͍࣭ͭͯ໰Λ౤͔͚͛·͢ɻ ͋ͳͨ͸ؔ࿈͢ΔΩʔϫʔυΛ࢖༻ͯ͠೔هͷσʔλΛݕࡧ͠ɺ ͦͷ࣭໰ʹճ౴͠·͢ɻ” ݻఆͷμϛʔσʔλ ݕࡧΫΤϦ ֘౰৘ใ 2025೥07݄18೔ ࢄ൅Λ໎͍ͬͯΔ೔ه 2025೥6݄9೔ ࢄ൅ʹߦͬͨ೔ه 2025݄4݄08೔ ࢄ൅ʹߦͬͨ͜ͱΛ ༑ୡʹؾ͕͍ͭͯ΋Β͑ͨ (ຊจলུ)
12. TOOL CALLING 2025೥07݄18೔ ࢄ൅Λ໎͍ͬͯΔ೔ه 2025೥6݄9೔ ࢄ൅ʹߦͬͨ೔ه 2025݄4݄08೔ ࢄ൅ʹߦͬͨ͜ͱΛ ༑ୡʹؾ͕͍ͭͯ΋Β͑ͨ (ຊจলུ) ݻఆͷμϛʔσʔλ “ࢄ൅” 👍
    '.͸࣭໰Λਖ਼͘͠ղऍ͠ɺద੾ͳݕࡧΫΤϦΛੜ੒Ͱ͖ͨ
    👍
    '.͸ಘΒΕͨ೔هσʔλΛਖ਼͘͠ղऍ͠ղ౴Ͱ͖ͨ “൅Λ੾ͬͨ”
13. TOOL CALLING 👤 📖 ࣭໰ ճ౴ ݕࡧΫΤϦ ֘౰৘ใ 👍
14. TOOL CALLING 👤 📖 ࣭໰ ճ౴ ɾ'.ʹ೔هΛɺΦϯσόΠεͰɺͲͷΑ͏ʹݕࡧͤ͞Δ͔
    ɾݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖Δ͔ʢ෦෼ҰகʻηϚϯςΟοΫʣ
15. Spotlight (On-device searching)
16. https://developer.apple.com/jp/videos/play/wwdc2024/10131/
17. 👤 📖 ࣭໰ ճ౴ 👍
    ೔هΛΦϯσόΠεͰݕࡧͰ͖ͨ
    ❌
    ݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖ͳ͔ͬͨʢηϚϯςΟοΫݕࡧͷਫ਼౓ෆ଍ʣ
18. https://x.com/ShunTakeishi/status/1958363232068128879
19. ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ ࡞ͬͯΈΔʂ
20. 4UFQ
    
    จॻ ೔هΤϯτϦ ͝ͱʹϕΫτϧԽʢ/-$POUFYUVBM&NCFEEJOHʣ
    4UFQ
    
    ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ
    
    
    
    
    
    
    
    
    
    
    
    
    
    "DDFMFSBUF
    PS
    .-5FOTPS
    4UFQ
    
    ྨࣅ౓είΞͰιʔτ͠ɺ্ҐΛώοτͱ͢Δ
21. 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿 􀈿
22. ͋ͷΠʔϋτʔϰΥͷ͖͢ͱ͓ͬͨ෩ɺ ՆͰ΋ఈʹྫྷͨ͞Λ΋ͭ੨͍ͦΒɺ ͏͍ͭ͘͠৿Ͱ০ΒΕͨϞϦʔΦࢢɺ ߫֎ͷ͗Β͗Βͻ͔Δ૲ͷ೾ɻ /-$POUFYUVBM&NCFEEJOH 􀈿
23. ͋ͷ / Πʔ / ϋ / τʔ / ϰΥ / ͷ / ͖͢ / ͱ / ͓ͬ / ͨ / ෩ / ɺ / Ն / Ͱ΋ / ఈ / ʹ / ྫྷ / ͨ / ͞ / Λ / ΋ͭ / ੨͍ / ͦΒ / ɺ / ͏ / ͭ / ͘ / ͍͠ / ৿ / Ͱ / ০Β / Ε / ͨ / Ϟ / Ϧʔ / Φ / ࢢ / ɺ / ߫֎ / ͷ / ͗ / Β / ͗ / Β / ͻ͔Δ / ૲ / ͷ / ೾ / ɻ
24. ͋ͷɹɹ[-0.231, 0.027, 0.210 ... -0.128, 0.172, -0.151] Πʔɹɹ[ 0.035, -0.122, 0.064 ... -0.127, -0.109, -0.100] ϋɹɹɹ[-0.052, 0.195, 0.281 ... 0.003, 0.064, -0.048] τʔɹɹ[ 0.240, 0.309, 0.173 ... 0.095, 0.054, -0.172] … ͻ͔Δɹ[ 0.054, 0.104, -0.062 ... 0.048, 0.160, -0.001] ૲ɹɹɹ[ 0.055, -0.059, 0.142 ... -0.024, -0.082, -0.045] ͷɹɹɹ[-0.160, -0.062, 0.464 ... -0.056, 0.215, -0.091] ೾ɹɹɹ[ 0.041, 0.101, 0.223 ... 0.092, 0.027, 0.069]
25. ͋ͷɹɹ[-0.231, 0.027, 0.210 ... -0.128, 0.172, -0.151] Πʔɹɹ[ 0.035, -0.122, 0.064 ... -0.127, -0.109, -0.100] ϋɹɹɹ[-0.052, 0.195, 0.281 ... 0.003, 0.064, -0.048] τʔɹɹ[ 0.240, 0.309, 0.173 ... 0.095, 0.054, -0.172] … ͻ͔Δɹ[ 0.054, 0.104, -0.062 ... 0.048, 0.160, -0.001] ૲ɹɹɹ[ 0.055, -0.059, 0.142 ... -0.024, -0.082, -0.045] ͷɹɹɹ[-0.160, -0.062, 0.464 ... -0.056, 0.215, -0.091] ೾ɹɹɹ[ 0.041, 0.101, 0.223 ... 0.092, 0.027, 0.069] ɹɹɹɹ[ 0.022, -0.000, 0.034 ... 0.007, 0.019, -0.007] ϓʔϦϯάʢྫ: Mean poolingʣ ਖ਼نԽ (L2 normalization)
26. 􀈿 ɹɹɹ[ 0.022, -0.000, 0.034 ... 0.007, 0.019, -0.007] 􀈿 ɹɹɹ[ 0.081, -0.033, -0.040 ... 0.056, -0.015, 0.017] 􀈿 ɹɹɹ[ 0.015, -0.003, 0.040 ... 0.054, -0.005, -0.004] … 􀈿 ɹɹɹ[ 0.057, 0.012, -0.050 ... 0.082, -0.002, 0.057] 􀈿 ɹɹɹ[ 0.018, 0.021, -0.023 ... -0.070, -0.017, 0.006] 􀈿 ɹɹɹ[-0.022, 0.005, -0.008 ... -0.020, 0.021, -0.044] 4UFQ
    
    จॻ ೔هΤϯτϦ ͝ͱʹϕΫτϧԽʢ/-$POUFYUVBM&NCFEEJOHʣ
    
27. 􀈿 ɹɹɹ[ 0.022, -0.000, 0.034 ... 0.007, 0.019, -0.007] 􀈿 ɹɹɹ[ 0.081, -0.033, -0.040 ... 0.056, -0.015, 0.017] 􀈿 ɹɹɹ[ 0.015, -0.003, 0.040 ... 0.054, -0.005, -0.004] … 􀈿 ɹɹɹ[ 0.057, 0.012, -0.050 ... 0.082, -0.002, 0.057] 􀈿 ɹɹɹ[ 0.018, 0.021, -0.023 ... -0.070, -0.017, 0.006] 􀈿 ɹɹɹ[-0.022, 0.005, -0.008 ... -0.020, 0.021, -0.044] ʁɹɹɹ[-0.062, -0.036, -0.046 ... -0.009, -0.020, 0.011] 4UFQ
    
    ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ
    “ඒ͍͠৿Ͱ০ΒΕͨࢢ͸ʁ”
28. 􀈿 0.022 -0.000 ... 0.019 -0.007 􀈿 0.081 -0.033 ... -0.015 0.017 􀈿 0.015 -0.003 ... -0.005 -0.004 ɹɹɹɹɹɹɹɹɹɹɹ…ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ 􀈿 0.057 0.012 ... -0.002 0.057 􀈿 0.018 0.021 ... -0.017 0.006 􀈿 -0.022 0.005 ... 0.021 -0.044 4UFQ
    
    ࣭໰ΫΤϦΛϕΫτϧԽ͠ɺ֤จॻϕΫτϧͱͷྨࣅ౓Λܭࢉ
    -0.062 -0.036 -0.046 ... -0.009 -0.020 0.011 -0.012 0.802 0.029 ... 0.510 -0.001 -0.910 = ֤จॻΛࣔ͢ߦϕΫτϧɾ࣭໰ΫΤϦΛࣔ͢ྻϕΫτϧͷ಺ੵ͕
    ίαΠϯྨࣅ౓
    d
    ʹͳΔʢʹ͍ۙ΄Ͳྨࣅɺʹ͍ۙ΄Ͳ૬ҧʣ
29. 􀈿 0.022 -0.000 ... 0.019 -0.007 􀈿 0.081 -0.033 ... -0.015 0.017 􀈿 0.015 -0.003 ... -0.005 -0.004 ɹɹɹɹɹɹɹɹɹɹɹ…ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ 􀈿 0.057 0.012 ... -0.002 0.057 􀈿 0.018 0.021 ... -0.017 0.006 􀈿 -0.022 0.005 ... 0.021 -0.044 4UFQ
    
    ྨࣅ౓είΞͰιʔτ͠ɺ্ҐΛώοτͱ͢Δ -0.062 -0.036 -0.046 ... -0.009 -0.020 0.011 -0.012 0.802 0.029 ... 0.510 -0.001 -0.910 = ֤จॻΛࣔ͢ߦϕΫτϧɾ࣭໰ΫΤϦΛࣔ͢ྻϕΫτϧͷ಺ੵ͕
    ίαΠϯྨࣅ౓
    d
    ʹͳΔʢʹ͍ۙ΄Ͳྨࣅɺʹ͍ۙ΄Ͳ૬ҧʣ
30. “࠷ޙʹ൅੾ͬͨͷ͍ͭʁ” “મ౬ͰԿҿΜͩʁ” “٤஡ళͰձͬͨͷ୭ʁ” “όΠτͷ஌ਓʹ͋ͬͨͷͲ͜ʁ” ˞ݸͷ୹จ "*ੜ੒ ͔Βݕࡧ #1 #1 #1 #3
31. import Accelerate import CoreML internal import NaturalLanguage let embedding: NLContextualEmbedding = … // 1ςΩετΛϕΫτϧԽʢmean pooling + L2ਖ਼نԽʣ func tensorize(text: String, asColumnVector: Bool = false) throws -> MLTensor? { let result = try embedding.embeddingResult(for: text, language: language) let dim = embedding.dimension var sum = [Float](repeating: 0, count: dim) var count = 0 result.enumerateTokenVectors(in: text.startIndex..<text.endIndex) { vecD, range in // Double -> Float ΛҰׅม׵ var vecF = [Float](repeating: 0, count: dim) vDSP_vdpsp(vecD, 1, &vecF, 1, vDSP_Length(dim)) // sum += vecF vDSP_vadd(sum, 1, vecF, 1, &sum, 1, vDSP_Length(dim)) count += 1 return true } guard count > 0 else { return nil } // mean poolingʢsum /= countʣ var invN = 1.0 / Float(count) vDSP_vsmul(sum, 1, &invN, &sum, 1, vDSP_Length(dim)) // L2 ਖ਼نԽ let normVector = l2Normalize(sum) let shape = asColumnVector ? [normVector.count, 1] : [1, normVector.count] return MLTensor(shape: shape, scalars: normVector) } // L2 ਖ਼نԽ private func l2Normalize(_ vec: [Float]) -> [Float] { var norm: Float = 0 vDSP_svesq(vec, 1, &norm, vDSP_Length(vec.count)) norm = sqrtf(norm) + 1e-12 vDSP_vsdiv(vec, 1, &norm, &vec, 1, vDSP_Length(vec.count)) return vec } iOSʹ૊Έࠐ·ΕͨBERTͰςΩετຒΊࠐΈɾϕΫτϧݕࡧΛΦϯσόΠε࣮ߦ͢Δ #Mac – Qiita ௥هʢ2025/10/05ʣɿͪ͜Βͷهࣄ͕ΑΓࢀߟʹͳΔͱࢥΘΕ·͢ʂ https://qiita.com/5enxia/items/eea83cfd277584e00748
32. var contents: [UUID : String] = [:] var tensors: [UUID : MLTensor] = [:] func embed(items: [String]) throws { var ret: [UUID : String] = [:] do { try items.forEach { item in let tensor = try encode(text: item) let uuid = UUID() tensors[uuid] = tensor ret[uuid] = item } } catch { // error handling } contents = ret } func search(query: String, topCount: Int) async -> [String] { // υΩϡϝϯτΛूੵͨ͠ D*M ߦྻ let flatteneds = tensors.values.map { $0.flattened() } let docsTensor = MLTensor(stacking: flatteneds) // ΫΤϦϕΫτϧ guard let queryTensor = try? encode(text: query, asColumnVector: true) else { return [] } let product = docsTensor.matmul(queryTensor) let calcScores = await product.shapedArray(of: Float.self).scalars let arr = Array(zip(tensors.map(\.key), calcScores)) // ྨࣅ౓Ͱιʔτ্͠ҐN݅Λநग़ let sorted = arr.sorted { $0.1 > $1.1 }.prefix(topCount) // ্ҐUUID͝ͱʹରԠ͢ΔυΩϡϝϯτΛ contents ͔Β୳ࡧ return ... }
33. ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ 👍
    ೔هΛΦϯσόΠεͰݕࡧͰ͖ͨ
    👍
    ݕࡧΫΤϦͷ༳ΕΛٵऩͰ͖ͨ
34. ࣗલݕࡧ Τϯδϯ 👤 📖 ࣭໰ ճ౴ 👍 👍
35. εΫγϣ
36. ⁉ εΫγϣ
37. εΫγϣ πʔϧݺͼग़ͯ͠ͳ͍ ؔ܎ͷͳ͍ݕࡧΩʔϫʔυͰݕࡧ ଘࡏ͠ͳ͍೔هΤϯτϦΛ፻଄
38. 2025೥8݄
39. ηϚϯςΟοΫݕࡧͷ࣮૷ΞϓϩʔνΛ঺հ ɾNLContextualEmbedding + MLTensor FM͸ϞσϧΞοϓσʔτʹ൐͏඼࣭ͷมԽʹ஫ҙ ɾϨεϙϯε඼࣭ͷ҆ఆੑ ɾπʔϧݺͼग़͠ͷ࠶ݱੑ ໋ྩදݱ΍ɺखॱΛ໌ࣔͨ͠ϓϩϯϓτɺfew-shot ͳͲ͋ΒΏΔςΫχοΫΛࢼ͕ͨ͠ޮՌ͕ݟΒΕͳ͍
40. FMΛࣂ͍׳Β͢ํ๏ ڭ͍͑ͯͩ͘͞ 🙇
41. Foundation Models Ͱ ΦϯσόΠεRAGΛࢼΈΔ extension DC 2025 Day1@DeNA, 2025/10/01 Takeshi Tanaka @p0dee