Core ML版Stable DiffusionをiOSで快適に動かす

「Mobile勉強会 Wantedly × チームラボ #11」での発表資料です。
https://wantedly.connpass.com/event/297601/

記事： Core ML版Stable DiffusionをiOSで快適に動かす

Transcript

1. Core ML൛ Stable Diffusion Λ iOS Ͱշదʹಈ͔͢ అ मҰ @shu223
2. ࣗݾ঺հ • అ मҰ • @shu223 (GitHub, Qiita, Zenn, note, , YouTube, Podcast, etc...) • ॻ੶ʢ঎ۀग़൛4࡭ɺݸਓग़൛ଟ਺ @BOOTHʣ:
3. • 1೥ؒݚڀ։ൃ෦ͷϚωʔδϟʔʹνϟϨϯδ • 12೥ؒiOS͚ͩ΍͖ͬͯͨΤϯδχΞͷΩϟϦΞνΣϯδ • 2023೥6݄͔ΒΤϯδχΞʹ෮ؼ • Ϛωʔδϟʔ͔ΒΤϯδχΞʹ໭Γ·͢ • ࠷ۙͷ͓࢓ࣄ • λον໊ࢗަ׵
4. ຊ೔ͷ͓୊ʹ͍ͭͯ • ❌ ʮStable Diffusionʯͱ͍͏ը૾ੜ੒AIͷղઆ • ⭕ ڊେͳϞσϧΛiOSͰಈ͔͢ࡍͷߟ͑ํ΍צॴ
5. Stable Diffusionͱ͸ • ը૾ੜ੒AI • ೖྗςΩετʹԠͯ͡ը૾Λࣗಈ ੜ੒͢Δtext-to-imageϞσϧ • 2022.8 Φʔϓϯιʔεͱͯ͠ެ։
6. Core ML Stable Diffusion ͱ͸ Core ML ϑΥʔϚοτʹม׵͞Εͨ Stable Diffusion ͷϞσϧ • ैདྷͷϞσϧΛAppleϋʔυ΢ΣΞͰಈ͔͢ => CPUͷΈར ༻ • Core MLϞσϧ => CPU, GPU, Neural EngineΛར༻ → Appleͷϋʔυ΢ΣΞΛ࠷େݶ׆͔ͤΔͷ͕Core MLϞσϧ 1 1 ৄࡉͳ࠷దԽͷղઆɿ Deploying Transformers on the Apple Neural Engine - Apple Machine Learning Research
7. Core ML Stable Diffusion ͷϦϙδτϦ https://github.com/apple/ml-stable-diffusion • Appleۘ੡ • Ϟσϧม׵ίʔυɺmacOS/iOSͰಈ͔ͨ͢ΊͷSwiftίʔυ౳ ΛؚΉ • ެࣜϒϩάهࣄɿ Stable Diffusion with Core ML on Apple Silicon - Apple Machine Learning Research
8. ॳొ৔࣌Ҏ߱ͷਐԽ 2022೥12݄ͷॳެ։࣌Ҏ߱͋·Γ࿩୊ʹͳ͍ͬͯͳ͍͕ɺணʑ ͱਐԽ͍ͯ͠Δ • ControlNetαϙʔτ • SDXLαϙʔτ • 6ϏοτྔࢠԽʢؔ࿈هࣄɿ 1, 2, 3ʣ • Multilingual text encoder αϙʔτ • ؔ࿈هࣄɿ [iOS 17] ଟݴޠBERTຒΊࠐΈϞσϧͷαϙʔ
9. Core ML Stable Diffusion ͷԿ͕خ͍͠ͷ͔ʁ • खݩͷMacͰɺແྉ͔ͭΦϑϥΠϯͰ࢖͍·͘ΕΔ • ࣗલͷmacOS/iOSΞϓϦʹ૊ΈࠐΊΔ • For me: ༷ʑͳϊ΢ϋ΢͕٧·ͬͨCore MLͷެࣜαϯϓϧ
10. CLI͔Βಈ͔͢ SwiftͰॻ͔ΕͨCLIαϯϓϧ StableDiffusionSampleΛར༻ swift run StableDiffusionSample \ "a photo of an astronaut riding a horse on mars" \ --resource-path </path/to/models> --seed 93 \ --output-path </path/to/output/image> • --resource-path • ίϯύΠϧࡁΈCore MLϞσϧϑΝΠϧ .mlmodelc ͕ ೖͬͨσΟϨΫτϦͷύεΛࢦఆ
11. ࣗ࡞ΞϓϦʹ૊ΈࠐΉ 1/2 Swift Package ͱͯ͠ϓϩδΣΫτʹ௥Ճ͢Δ • → Package Product "StableDiffusion" Λλʔήοτʹ௥Ճ
12. ࣗ࡞ΞϓϦʹ૊ΈࠐΉ 2/2 import StableDiffusion ... let pipeline = try StableDiffusionPipeline(resourcesAt: resourceURL) pipeline.loadResources() let image = try pipeline.generateImages(prompt: prompt, seed: seed).first
13. Hugging FaceʹΑΔαϯϓϧΞϓ Ϧ • GitHub • App Store
14. Diffusers Λ iPhone 15 Pro Ͱಈ͔ͯ͠ΈΔ
15. Ϟσϧͷμ΢ϯϩʔυ ʢॳճͷΈඞཁʣ
16. Ϟσϧͷϩʔυ ʢىಈຖʹඞཁʣ
17. ⚠ ϩʔσΟϯά͕ͱͯͭ΋ͳ͘௕͍
18. ىಈ͢Δ౓ʹ 5෼ 2 Pipeline loaded in 293.2635090351105 2 Ϟσϧͷμ΢ϯϩʔυ͕׬ྃͨ͠ঢ়ଶ͔Βͷىಈ
19. ىಈঢ়ଶ
20. ⚠ ը૾ੜ੒࣌ʹඞͣΫϥογϡ
21. ࢖༻ϝϞϦྔͷεύΠΫ ʢiPhone 15 Proར༻ʣ
22. iOSͰʮ·ͱ΋ʹʯಈ͔͢
23. iOSͰ·ͱ΋ʹಈ͔͢ • ࢖༻ϝϞϦྔΛখ͘͢͞Δ • ىಈ࣌ؒΛ୹͘͢Δ • ը૾ੜ੒࣌ؒΛ୹͘͢Δ
24. iOSͰ·ͱ΋ʹಈ͔͢ • ࢖༻ϝϞϦྔΛখ͘͢͞Δ • ىಈ࣌ؒΛ୹͘͢Δ • ը૾ੜ੒࣌ؒΛ୹͘͢Δ
25. ࢖༻ϝϞϦྔΛখ͘͢͞Δํ๏ • reduceMemory ΦϓγϣϯΛ༗ޮʹ͢Δ • αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ
26. ࢖༻ϝϞϦྔΛখ͘͢͞Δํ๏ • ⭕ reduceMemory ΦϓγϣϯΛ༗ޮʹ͢Δ • αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ
27. reduceMemory Λ༗ޮʹ͢Δ ModelInfo.swift ͷ reduceMemory ϓϩύςΟΛܾఆ͢Δ ॲཧΛॻ͖׵͑Δ 3 var reduceMemory: Bool { // Enable on iOS devices, except when using quantization if runningOnMac { return false } if isXL { return !deviceHas8GBOrMore } return !(quantized && deviceHas6GBOrMore) } 3 ݱߦ࣮૷Ͱ͸iPhone 15 Proͷ৔߹͸ඞͣແޮԽ͞Εͯ͠·͏
28. reduceMemory Λ༗ޮʹ͢ΔͱͲ͏ͳΔ͔ʁ • ඞཁͳͱ͖ʹϞσϧΛϩʔυ͠ɺ͙͢ʹΞϯϩʔυ͢Δ • ը૾ੜ੒ॲཧͷΦʔόʔϔουΛ૿΍͢͜ͱʹͳΔ • υΩϡϝϯτͰ͸ղઆ͞Ε͍ͯͳ͍͕ɺιʔείʔυΛݟΔ ͱىಈ࣌ʹ prewarmResources ͱ͍͏ॲཧΛߦ͍ͬͯΔ public func loadResources() throws { if reduceMemory { try prewarmResources() } ...
29. prewarmResources loadResources ޙ͙͢ʹ unloadResources Λߦ͏ func prewarmResources() throws { try loadResources() unloadResources() } Both .mlpackage and .mlmodelc models are compiled upon first load. ʢதུʣ .mlmodelc files do cache this compiled asset and non-first load times are reduced to just a few seconds.
30. σϞ
31. ը૾ੜ੒࣌ʹΫϥογϡ͠ͳ͘ͳͬͨ
32. ը૾ੜ੒ͷॲཧ࣌ؒ Got images: [Optional(<CGImage 0x12810d510> (IP) <<CGColorSpace 0x283e04780> (kCGColorSpaceDeviceRGB)> width = 768, height = 768, bpc = 8, bpp = 24, row bytes = 2304 kCGImageAlphaNone | 0 (default byte order) | kCGImagePixelFormatPacked is mask? No, has masking color? No, has soft mask? No, has matte? No, should interpolate? Yes)] in 83.99049496650696 1ຕ͋ͨΓ 83.99 ඵ ىಈ࣌ؒ΋ґવͱͯ͠ 5෼ ͔͔͍ͬͯΔ
33. iOSͰ·ͱ΋ʹಈ͔͢ • ࢖༻ϝϞϦྔΛখ͘͢͞Δ • ىಈ࣌ؒΛ୹͘͢Δ • ը૾ੜ੒࣌ؒΛ୹͘͢Δ
34. ىಈ࣌ؒΛ୹͘͢Δํ๏ ىಈ࣌ؒͷ΄ͱΜͲ͕ɺϞσϧͷϩʔυ 4 ʢ࣮ଶͱͯ͠͸ MLModel ͷॳظԽʣʹ͕͔͔͍࣌ؒͬͯΔ → αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ 4 ϥΠϒϥϦ಺ͷ loadResources ϝιου
35. ը૾ੜ੒࣌ؒΛ୹͘͢Δํ๏ αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ The Neural Engine is capable of accelerating models with low-bit palettization: 1, 2, 4, 6 or 8 bits. ʢχϡʔϥϧɾΤϯδϯ͸ɺ௿ ϏοτͷύϨλΠζͰϞσϧΛߴ଎ԽͰ͖Δʣ compressed weights are faster to fetch from memory ... ʢѹॖ͞ Εͨ΢ΣΠτ͸ϝϞϦ͔ΒͷϑΣον͕ߴ଎ʹͳΓɺʣ 5 5 apple/ml-stable-diffusion: Stable Diffusion with Core ML on Apple Silicon ͷREADMEΑΓ
36. ʢ࠶ܝʣ࢖༻ϝϞϦྔΛখ͘͢͞Δํ๏ • reduceMemory ΦϓγϣϯΛ༗ޮʹ͢Δ • ⭕ αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ
37. ͱʹ͔͘Ϟσϧͷѹॖ͕ॏཁ
38. ݱࡏͷ࢖༻ϞσϧΛௐ΂Δ 1/3 let loader = PipelineLoader(model: iosModel()) func iosModel() -> ModelInfo { guard deviceSupportsQuantization else { return ModelInfo.v21Base } if deviceHas6GBOrMore { return ModelInfo.xlmbpChunked } return ModelInfo.v21Palettized } → iPhone 15 ProͰ͸ ModelInfo.xlmbpChunked Λར༻ 6 6 deviceSupportsQuantization ͕ true ͔ͭ deviceHas6GBOrMore ͕ true
39. ݱࡏͷ࢖༻ϞσϧΛௐ΂Δ 2/3 static let xlmbpChunked = ModelInfo( modelId: "apple/coreml-stable-diffusion-xl-base-ios", modelVersion: "SDXL base (768, iOS) [4 bit]", supportsEncoder: false, quantized: true, isXL: true )
40. ݱࡏͷ࢖༻ϞσϧΛௐ΂Δ 3/3 apple/coreml-stable-diffusion-xl-base-ios • iOS޲͚ʹ 4.04 ࠞ߹ϏοτྔࢠԽʢmixed-bit palettizationʣͨ͠΋ͷ • ɾɾɾͰ͸͋Δ͕ɺ΋ͱ΋ͱ͕SDXLϞσϧͳͷͰΫιͰ͔ ͍ • Unet͚ͩͰ 1.3 GBҎ্
41. ϞσϧΛѹॖ͢Δ or Ϟσϧͷѹॖ൛Λ୳͢
42. Hugging Face Ͱެ։͞Ε͍ͯΔ Core ML Stable Diffusion Ϟσϧ → 2023೥10݄ݱࡏͰ16छྨ
43. ! ͲΕΛબ΂͹͍͍ͷ͔
44. ͍ܰSDϞσϧΛબͿϙΠϯτ • SDXLͰ͸ͳ͍΋ͷɾɾɾXL͸࠷৽ͷߴը࣭൛Ͱ͋ΓɺσΧ ͍ • palletized ͱ͍͍ͭͯΔ΋ͷɾɾɾѹॖ͞Ε͍ͯΔ͜ͱΛࣔ ͢ • 16bit -> 6-bit • SDόʔδϣϯͷҧ͍ʹΑΔαΠζͷҧ͍͸͋·Γͳ͍ → "apple/coreml-stable-diffusion-2-1-base-palettized"
45. σϞ
46. ֤छύϑΥʔϚϯε • ىಈ࣌ؒ 7ɿ100ඵલޙʢ300%ߴ଎Խʣ • ࢖༻ϝϞϦྔͷϐʔΫ஋ 8ɿ໿850MB • ը૾ੜ੒ॲཧ࣌ؒɿ10ඵҎԼʢ800%ߴ଎Խʣ 8 αΠζͷখ͍͞Ϟσϧͷ࢖༻ʹ൐͍ɺreduceMemory ͸࠶ͼແޮԽͨ͠ 7 ΞϓϦͷىಈ࣌ؒશମͰ͸ͳ͘ɺύΠϓϥΠϯͷϩʔυ࣌ؒΛܭଌ
47. ͞ΒͳΔىಈ࣌ؒͷ୹ॖ ϞσϧͷϩʔυʢMLModelͷॳظԽʣ͕90%Ҏ্Λ઎ΊΔ time to load unet: 57.41793203353882 time to load textEncoder: 30.462964057922363 time to load decoder: 3.1042929887771606 time to load encoder: 2.146523952484131 Pipeline resources loaded in 106.28662097454071
48. ͞ΒͳΔϞσϧѹॖ Mixed-Bit Palettization ެࣜϦϙδτϦREADME ͷ "Advanced Weight Compression (Lower than 6-bits)" ʹৄࡉ͋Γ
49. more • ControlNet • Multilingual text encoder • ؔ࿈هࣄɿ [iOS 17] ଟݴޠBERTຒΊࠐΈϞσϧͷαϙʔ τ
50. ·ͱΊ Stable DiffusionͷΑ͏ͳΫισΧCore MLϞσϧΛiOSͰಈ͔͢ • ΩϟογϡػߏΛར༻ͯ͠౎౓ϩʔυ & Ξϯϩʔυ • → ϝϞϦ࢖༻ྔΛ཈͑Δ • ϞσϧαΠζΛখ͘͢͞Δ • → ىಈ࣌ؒɾϝϞϦ࢖༻ྔɾը૾ੜ੒࣌ؒ͢΂ͯվળ • Hugging Face୳͢/coremltoolsͰࣗલѹॖ/Mixed-Bit Palettization
51. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ