[Swift Heroes '24] Delightful on-device AI experiences

Currently, numerous APIs allow developers to run Stable Diffusion models on the server and integrate AI into their apps by making network requests.

While these APIs are great, they can become very costly and don't allow for building offline-first applications. In this session, I will discuss a feature I have been building for my app QReate that uses Stable Diffusion to generate art-like QR codes completely on-device using CoreML.

Transcript

1. Delightful on-device AI experiences @POLPIELLADEV @POLPIELLADEV@IOSDEV.SPACE 🦸 SWIFT HEROES 2024
2. App maker and content creator based in Barcelona Hi! I’m Pol
4. Get early access to Helm!
9. So… I had an idea
11. 🔐 PRIVACY FRIENDLY 🎨 HIGHLY CUSTOMIZABLE 🛫 OFFLINE FIRST
13. I need this NOW!
14. The concepts
16. Stable Di ff usion
18. A neon drawing of an astronaut fl oating in space, digital art by Mór Than, unsplash contest winner, space art, sci- fi , retrowave, synthwave
20. https://inpainter.vercel.app/
21. Stable Di ff usion
22. Stable Di ff usion + ControlNet
23. A high quality photo of a sur fi ng dog
24. ControlNet Input
25. ControlNet Input https://github.com/apple/ml-stable-di ff usion?tab=readme-ov- fi le#-using-controlnet
26. A drawing of Darth Vader from Star Wars
27. ControlNet Input
28. ControlNet Input
29. Finding the right solution
32. Houston, we have TWO problems
33. 🔐 PRIVACY FRIENDLY 🎨 HIGHLY CUSTOMIZABLE 🛫 OFFLINE FIRST #1
34. 🛫 OFFLINE FIRST #1
35. #2
36. Change the pricing model? #2 🤑
37. On-device Stable Di ff usion
39. ⚙ PyTorch to CoreML Converter 📦 Swi ft Package to load models
44. Did somebody say ControlNet?
48. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion
49. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion python3 -m venv venv source venv/bin/activate pip install -e .
51. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion python3 -m venv venv source venv/bin/activate pip install -e . python -m python_coreml_stable_diffusion.torch2coreml \ -- bundle-resources-for-swift-cli \ -- attention-implementation SPLIT_EINSUM_V2 \ -- convert-unet \ -- convert-text-encoder \ -- convert-vae-decoder \ -- convert-vae-encoder \ -- model-version runwayml/stable-diffusion-v1-5 \ -- unet-support-controlnet \ -- convert-controlnet DionTimmer/controlnet_qrcode-control_v1p_sd15 \ -o generated
52. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion python3 -m venv venv source venv/bin/activate pip install -e . python -m python_coreml_stable_diffusion.torch2coreml \ -- bundle-resources-for-swift-cli \ -- attention-implementation SPLIT_EINSUM_V2 \ -- convert-unet \ -- convert-text-encoder \ -- convert-vae-decoder \ -- convert-vae-encoder \ -- model-version runwayml/stable-diffusion-v1-5 \ -- unet-support-controlnet \ -- convert-controlnet DionTimmer/controlnet_qrcode-control_v1p_sd15 \ -o generated
53. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion python3 -m venv venv source venv/bin/activate pip install -e . python -m python_coreml_stable_diffusion.torch2coreml \ -- bundle-resources-for-swift-cli \ -- attention-implementation SPLIT_EINSUM_V2 \ -- convert-unet \ -- convert-text-encoder \ -- convert-vae-decoder \ -- convert-vae-encoder \ -- model-version runwayml/stable-diffusion-v1-5 \ -- unet-support-controlnet \ -- convert-controlnet DionTimmer/controlnet_qrcode-control_v1p_sd15 \ -o generated
54. git clone https: / / github.com/apple/ml-stable-diffusion.git cd ml-stable-diffusion python3 -m venv venv source venv/bin/activate pip install -e . python -m python_coreml_stable_diffusion.torch2coreml \ -- bundle-resources-for-swift-cli \ -- attention-implementation SPLIT_EINSUM_V2 \ -- convert-unet \ -- convert-text-encoder \ -- convert-vae-decoder \ -- convert-vae-encoder \ -- model-version runwayml/stable-diffusion-v1-5 \ -- unet-support-controlnet \ -- convert-controlnet DionTimmer/controlnet_qrcode-control_v1p_sd15 \ -o generated
56. Loading Stable Di ff usion models
57. // swift-tools-version: 5.9 import PackageDescription let package = Package( name: "StableDiffusionControlNet", platforms: [.macOS(.v14)], dependencies: [ ], targets: [ ] )
58. // swift-tools-version: 5.9 import PackageDescription let package = Package( name: "StableDiffusionControlNet", platforms: [.macOS(.v14)], dependencies: [ ], targets: [ ] ) .package(url: "https: // github.com/apple/ml-stable-diffusion.git", exact: "1.1.0")
59. // swift-tools-version: 5.9 import PackageDescription let package = Package( name: "StableDiffusionControlNet", platforms: [.macOS(.v14)], dependencies: [ .package(url: "https: // github.com/apple/ml-stable-diffusion.git", exact: "1.1.0") ], targets: [ ] ) .executableTarget( name: "StableDiffusionControlNet", dependencies: [.product(name: "StableDiffusion", package: "ml-stable-diffusion")], resources: [.process("Resources")] ),
60. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
61. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
62. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
63. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
64. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
65. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: true) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
66. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: false) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
67. import Foundation import StableDiffusion import CoreML func generate(prompt: String, startingImageURL: URL, numberOfImages: Int) async throws -> [CGImage] { guard let resourcesURL = Bundle.module.url(forResource: "Resources", withExtension: nil) ?. path() else { return [] } let url = URL(fileURLWithPath: resourcesURL) let configuration = MLModelConfiguration() configuration.computeUnits = .all let pipeline = try StableDiffusionPipeline(resourcesAt: url, controlNet: ["DiontimmerControlnetQrcodeControlV1PSd15"], configuration: configuration, disableSafety: false, reduceMemory: false) try pipeline.loadResources() let startingNSImage = NSImage(contentsOf: startingImageURL) ?. resized(to: .init(width: 512, height: 512)) guard let startingImage = startingNSImage ?. cgImage(forProposedRect: nil, context: nil, hints: nil) else { return [] } var pipelineConfig = StableDiffusionPipeline.Configuration(prompt: prompt) pipelineConfig.negativePrompt = "ugly, disfigured, low quality, blurry, nsfw" pipelineConfig.controlNetInputs = [startingImage] pipelineConfig.startingImage = startingImage pipelineConfig.useDenoisedIntermediates = true pipelineConfig.strength = 0.9 pipelineConfig.seed = UInt32.random(in: (0 .. < UInt32.max)) pipelineConfig.guidanceScale = 7.5 pipelineConfig.stepCount = 50 pipelineConfig.originalSize = 512 pipelineConfig.targetSize = 512 pipelineConfig.imageCount = numberOfImages return try pipeline.generateImages(configuration: pipelineConfig, progressHandler: { _ in true }) .compactMap { $0 } }
68. let prompt = """ Style-NebMagic, award winning photo, A Dark-Eyed Junco, sitting Great Basin National Park, intricate, nature background, wildlife photography, hyper realistic, Style-LostTemple, deep shadow, high contrast, dark, sunrise, morning, full moon """
69. let prompt = """ Style-NebMagic, award winning photo, A Dark-Eyed Junco, sitting Great Basin National Park, intricate, nature background, wildlife photography, hyper realistic, Style-LostTemple, deep shadow, high contrast, dark, sunrise, morning, full moon """ let url = URL(filePath: "/my-qr-code.png")
70. let prompt = """ Style-NebMagic, award winning photo, A Dark-Eyed Junco, sitting Great Basin National Park, intricate, nature background, wildlife photography, hyper realistic, Style-LostTemple, deep shadow, high contrast, dark, sunrise, morning, full moon """ let url = URL(filePath: "/my-qr-code.png") let image = try await generate( prompt: prompt, startingImageURL: url, numberOfImages: 1 )
72. https://github.com/huggingface/swi ft -coreml-di ff users
73. One more thing…
76. Host your models remotely And load them on demand
78. let modelURL = URL(string: "https: // huggingface.co/:user/:model/resolve/main/:file.zip?download=true")! let (location, downloadFileResponse) = try await URLSession.shared.download(from: modelURL) guard let httpResponse = downloadFileResponse as? HTTPURLResponse, httpResponse.statusCode == 200 else { exit(1) } try FileManager.default .moveItem( at: location, to: URL.desktopDirectory.appending(component: "model.zip") ) Model from HuggingFace
79. Thank you for listening! @POLPIELLADEV @POLPIELLADEV@IOSDEV.SPACE 🦸 SWIFT HEROES 2024