Building Hybrid, Agentic Apps with Apple’s Foundation Models Framework

Apple’s Foundation Model framework is the unified API for on-device and cloud-hosted large language models. With it, developers can build intelligent features for their apps that are type-safe, flexible, and leverage existing capabilities of their apps. Learn how to bring intelligent features to your iOS apps using LanguageModelSession and cloud-hosted models like Claude, PCC, and Gemini.

Transcript

1. Building hybrid, agentic apps with Apple’s Foundation Models framework Peter Friese · Staff Developer Relations Engineer, Google
3. peterfriese.dev/newsletter @peterfriese
5. Architectural design patterns @peterfriese ! s e p i c e r y r a Culin
7. Foundation Models @peterfriese
8. On-device AI - benefits 1/ Available offline 2/ Zero inference cost 3/ 100% private on device @peterfriese
9. 3 lines of Swift import FoundationModels let model = SystemLanguageModel.default let session = LanguageModelSession(model: model) let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) print(response.content) @peterfriese n o i t a r e n e g Async/await
11. Assembling prompts with strings var prompt = "Identify the ingredients in this photo.\n" prompt += "Suggest a recipe matching my profile.\n" if dietaryProfile.isVegetarian { prompt += "The dish MUST be strictly vegetarian.\n" n a e t a l o p r e t n i 't n a c u o Y } ! g n i r if !dietaryProfile.allergies.isEmpty { t S a o image int prompt += "Exclude: \(dietaryProfile.allergies.joined(separator: ", ")).\n" } prompt += "How do we interpolate countertopPhoto.cgImage here? let response = try await session.respond(to: prompt) @peterfriese "
12. Recipe 1a: Multimodal @PromptBuilder let response = try await session.respond { "Identify the ingredients in this photo." "Suggest a recipe matching my profile." if let cgImage = countertopPhoto.cgImage { Attachment(cgImage) } dietaryProfile kitchenEquipment } @peterfriese e d i s n i t h g i r s t c u r t s n i a m Do ! t p m o r p e h t
13. Recipe 1b: Conform to PromptRepresentable struct DietaryProfile: PromptRepresentable { var isVegetarian: Bool var allergies: [String] var maxPrepMinutes: Int @PromptBuilder var promptRepresentation: Prompt { "Max prep time: \(maxPrepMinutes) minutes." if isVegetarian { "The dish MUST be strictly vegetarian." } if !allergies.isEmpty { "Exclude allergens: \(allergies.joined(separator: ", "))." } } } @peterfriese w o l f l o r t n o c Declarative
14. Declarative prompt assembly @peterfriese
15. Mise en place, not string soup @peterfriese
18. Strings are great for markdown... 's l a u t x e T h t i Rendered w ! t x e T d e r u t c Stru o d w o h g n i r Still just a St & e m i T p e r p we store e s a b a t a d a n i s t n e i d e r g in ? g n i s r a p t u o with @peterfriese
19. Recipe 2: Type-safe output with @Generable @Generable struct GeneratedRecipe: Equatable { @Guide(description: "An appetizing dish title") var title: String @Guide(description: "Prep minutes", .range(5!!...120)) var prepMinutes: Int @Guide(description: "Ingredients", .count(3!!...10)) var ingredients: [GeneratedIngredient] var instructions: [String] } @peterfriese d e c r o f n e l e d o m e h t s e d i u G ! g n i d o c e d n e during tok
20. Streaming PartiallyGenerated into SwiftUI @Observable final class InspireViewModel { var partial: GeneratedRecipe.PartiallyGenerated? func generate(from prompt: Prompt) async throws { let stream = session.streamResponse( to: prompt, generating: GeneratedRecipe.self ) for try await snapshot in stream { self.partial = snapshot.content } } } @peterfriese ! r a f o s d e v i e c e r g n i h t y r e Ev
21. Inside the @Generable macro @peterfriese
24. An agent is an LLM with tools @peterfriese
25. The native tool calling loop @peterfriese
26. Recipe 3: Native tool calling (EventKit) struct AddToGroceryListTool: FoundationModels.Tool { let name = "addToGroceryList" let description = "Add items to iOS Reminders." @Generable struct Arguments { @Guide(description: "Items with quantities") var items: [String] } func call(arguments: Arguments) async throws !-> String { ! s r e d n i m e R S let store = EKEventStore() O i o t y l t c e r i Writes d try await store.requestFullAccessToReminders() for item in arguments.items { let reminder = EKReminder(eventStore: store) reminder.title = item reminder.calendar = store.defaultCalendarForNewReminders() try store.save(reminder, commit: false) } try store.commit() return "Added \(arguments.items.count) items to Reminders." } }
27. Injecting tools into LanguageModelSession let session = LanguageModelSession( model: SystemLanguageModel.default, tools: [ CheckGroceryListTool(), AddToGroceryListTool(), NutritionLookupTool(), CookbookLookupTool() ] ) let reply = try await session.respond( to: "Add missing Miso Salmon ingredients to Reminders." ) @peterfriese
29. So… can we do everything on-device? @peterfriese
30. On-device AI - limitations 1/ Device availability 2/ 8,192-token context window 3/ Latency & thermal limits 4/ Multi-step frontier reasoning @peterfriese
31. Latency & context window benchmark Input document Tokens On-device (iOS 27) Gemini 3.6 Flash Gemini 3.5 Flash Lite Crazy Ones 131 2.20s 3.83s 1.57s Google IPO letter 621 3.77s 5.59s 1.52s Stanford speech 2,759 6.95s 6.24s 1.61s House of Usher 3,950 10.89s 9.52s 1.72s App Store guidelines 60,655 Exceeds 8k 10.88s 1.88s @peterfriese
33. One protocol, every model @peterfriese
34. On-device system model import FoundationModels l e d o m e c n e g i l l e t n I e l p p A 3B on-device let model = SystemLanguageModel.default let session = LanguageModelSession(model: model) let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) @peterfriese
35. Swapping in Private Cloud Compute import FoundationModels let model = SystemLanguageModel.privateCloudCompute let session = LanguageModelSession(model: model) let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) @peterfriese ! l l a c ) : o t ( d n o p s e r & n o i s Same ses
36. Swapping in Anthropic Claude import ClaudeForFoundationModels import FoundationModels let model = ClaudeLanguageModel(name: "claude-sonnet-4-6") let session = LanguageModelSession(model: model) let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) @peterfriese l a i c i f f o 's c i p o Anthr r e t p a d a l e d o LanguageM
37. Swapping in Google Gemini import FirebaseAILogic import FoundationModels let model = FirebaseAI.firebaseAI() .geminiLanguageModel(name: "gemini-3.6-flash") let session = LanguageModelSession(model: model) e h t d n i h e b i n i m e G n e k o t 1M ! I P A n o i s s same se let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) @peterfriese
38. OpenAI & local Ollama servers import FoundationModels import FoundationModelsUtilities let model = ChatCompletionsLanguageModel( baseURL: URL(string: "http:!//localhost:11434/v1")!, modelName: "llama3.3:70b" ) let session = LanguageModelSession(model: model) let response = try await session.respond( to: "Suggest a quick weeknight cherry tomato pasta." ) @peterfriese ! M L L v r o , o i d u t S M L , a m Local Olla
39. System One models
40. System One models 1/ System 1 vs. System 2 thinking Fast, instinctive classification (< 20ms) vs. deliberative LLM reasoning 2/ Sub-20ms on the Apple Neural Engine Purpose-built classifiers (Laya & Jev) that use a fraction of LLM energy 3/ Constrained decision primitives Deterministic enums (choice), yes/no checks (noul), and 1–5 rubrics (score) 4/ Conforms to LanguageModel Open-source SystemOneFoundationModels plugs straight into LanguageModelSession @peterfriese
41. Calling a System One decision model import SystemOneFoundationModels @Generable struct ChefTriageDecision { var intent: CulinaryIntent var isAllergenSafe: Bool @Guide(.range(1!!...5)) var complexityScore: Int } !// choice !-> enum !// noul !-> Yes/No !// score !-> rubric let laya = try LayaOnDeviceLanguageModel() let fastSession = LanguageModelSession(model: laya) let decision = try await fastSession.respond( to: "Can I swap peanut oil for sesame oil?", generating: ChefTriageDecision.self ).content @peterfriese ! e n i g n E l a r u e N e l p p A e h < 20ms on t
42. Running the pass: DynamicProfile @peterfriese
43. /ˈprǝʊfaɪl/ 1/ An outline of something, especially a person's face, as seen from one side 2/ A description of somebody/something that gives useful information @peterfriese
44. DynamicProfile 1/ Declarative station configuration Bundles Instructions, Tools, .model(...), and sampling modifiers in one value 2/ SwiftUI-style var body composition Switches profiles dynamically with standard if / else and switch control flow 3/ Hot-swappable mid-session Mutate session.activeProfile on the fly without losing conversation history @peterfriese
45. Defining a station profile let sousChefProfile = LanguageModelSession.Profile { Instructions(""" You are a concise kitchen sous-chef. Focus on quick unit conversions and substitutions. """) Tools { CheckGroceryListTool() AddToGroceryListTool() } } .model(SystemLanguageModel.default) .temperature(0.2) let session = LanguageModelSession(profile: sousChefProfile) @peterfriese ! s r e i f i d o m e l y t s I U t f i w S
47. Recipe 4: Availability & capability fallback struct AvailabilityFallbackProfile: LanguageModelSession.DynamicProfile { var body: some LanguageModelSession.DynamicProfile { let system = SystemLanguageModel.default if system.isAvailable !&& system.capabilities.contains(.vision) { Profile { Instructions("Identify the ingredients concisely.") } ! h c a e r % 0 0 1 .model(system) r o f k c a b l l a f Cloud } else { Profile { Instructions("Identify the ingredients and suggest pairings.") } .model(FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-3.6-flash")) } } } @peterfriese
50. Take 1: The naive do / catch fallback func importRecipe(from text: String) async throws !-> GeneratedRecipe { do { let local = LanguageModelSession(model: SystemLanguageModel.default) return try await local.respond(to: text, generating: GeneratedRecipe.self).content } catch { let cloud = LanguageModelSession(model: geminiFlashModel) return try await cloud.respond(to: text, generating: GeneratedRecipe.self).content } } y r e t t a b & s 1 1 Wastes ! s t r a t s n e v e i n i m e G e r befo @peterfriese
51. Why naive do / catch fallback hurts @peterfriese
52. Recipe 5a: Pre-flight token budgeting struct HybridRecipeProfile: LanguageModelSession.DynamicProfile { let fitsOnDevice: Bool init(for text: String) async { ! e z i S t x e t n o c 2 9 1 let system = SystemLanguageModel.default , 8 t s n i a g a k c e h c t c a x E let tokens = (try? await system.tokenCount(for: text)) !?? 0 let outputHeadroom = 1_000 self.fitsOnDevice = system.isAvailable !&& (tokens + outputHeadroom) !<= system.contextSize } } @peterfriese
53. Recipe 5b: Model-specific instructions extension HybridRecipeProfile { var body: some LanguageModelSession.DynamicProfile { if fitsOnDevice { Profile { Instructions("Extract the recipe concisely.") } .model(SystemLanguageModel.default) .temperature(0.2) } else { Profile { Instructions("Extract recipe & infer prep times.") } .model(geminiFlashModel) .temperature(0.4) } } } @peterfriese ! i n i m e G r o f t p m o r p r e h c Ri
56. Three-tier agentic architecture @peterfriese
57. Calling the System One gatekeeper let laya = try LayaOnDeviceLanguageModel() let fastSession = LanguageModelSession(model: laya) let decision = try await fastSession.respond( to: userMessage, generating: ChefTriageDecision.self ).content let profile = TriagedChefProfile(decision: decision) let session = LanguageModelSession(profile: profile) let reply = try await session.respond(to: userMessage) @peterfriese e r o f e b n o i s s e Routes the s ! M L L y n a g wakin
58. Recipe 6: System One gatekeeper profile struct TriagedChefProfile: LanguageModelSession.DynamicProfile { let decision: ChefTriageDecision var body: some LanguageModelSession.DynamicProfile { if !decision.isAllergenSafe { Profile { Instructions("Warn about allergen conflict.") } .model(SystemLanguageModel.default) } else if decision.complexityScore !<= 2 { Profile { Instructions("Answer concisely.") } .model(SystemLanguageModel.default) } else { Profile { Instructions("Plan full menus & macros.") } .model(geminiFlashModel) } } } @peterfriese ! i n i m e G o t s e t a l a c s e x e l p Com
61. Recipe 7: Switching profiles mid-session !// 1. Start in fast On-Device Pantry Station let session = LanguageModelSession( profile: pantryCheckProfile ) let r1 = try await session.respond( to: "Do I have miso paste on my list?" ) !// 2. Escalate mid-session (transcript preserved!) session.activeProfile = executiveChefProfile .reasoningLevel(.high) let plan = try await session.respond( to: "Build a 3-day meal plan using that miso." ) @peterfriese
64. c i g o L I A e s a b e r i F e h t k Chec models page! @peterfriese
65. Recipe 8: OTA tuning with Remote Config let remoteConfig = RemoteConfig.remoteConfig() try await remoteConfig.fetchAndActivate() let modelName = remoteConfig["chef_model"].stringValue !// !!... let model = FirebaseAI.firebaseAI() .geminiLanguageModel(name: modelName) @peterfriese p p A n a t u o h t i w s l e d o m p Swa ! e t a d p u e r o St
66. Don’t leave the kitchen doors unlocked @peterfriese
67. Lock the kitchen doors with App Check #if targetEnvironment(simulator) let providerFactory = AppCheckDebugProviderFactory() #else let providerFactory = AppAttestProviderFactory() #endif AppCheck.setAppCheckProviderFactory(providerFactory) FirebaseApp.configure() let ai = FirebaseAI.firebaseAI( backend: .googleAI(), useLimitedUseAppCheckTokens: true ) @peterfriese ! s n e k o t t s e u q e r r e p d e t c Replay-prote
69. Apple’s declarative DSL journey
70. Apple’s declarative DSL journey @PromptBuilder & @Generable Hy bri d ag en t ic AI nM od els Fo u nd ati o Da ta Sw if t nte nts Ap pI @Model & @Query Sw if t UI @ViewBuilder @AppShortcutsBuilder @DynamicProfileBuilder 2019 2022 2023 2025 2026 @peterfriese
71. Recap 1/ Prompt & plate declaratively Compose multimodal inputs with @PromptBuilder and stream @Generable structs 2/ Give your agent hands Connect models to EventKit, databases & APIs via FoundationModels.Tool 3/ One protocol, every model Plug on-device, PCC, Claude, Gemini, OpenAI & System One into LanguageModelSession 4/ Orchestrate with DynamicProfile Route by availability, token budget, System One triage, or workflow phase @peterfriese
72. e v i t c a r e t n Try the i ! e s r u o c C c o D s l a e M y l d n e i Fr @peterfriese
73. Thanks! @peterfriese & e d o c e l p m a Slides, s s l e d o M n o i t a d n u o F e n O m e t Sys
74. Q&A @peterfriese