Accessing Gemini through Apple’s Foundation Model framework

At WWDC 2026, Apple opened the Foundation Models framework to third-party model adapters. Starting with iOS 27, macOS 27, iPadOS 27, watchOS 27, and visionOS 27, model providers can implement the public LanguageModel protocol and plug in.

In this talk, I show how to use this integration in your iOS apps.

Transcript

1. @peterfriese.dev Created by Mamank from Noun Project https: / / peterfriese.dev peterfriese Peter Friese, Staff Developer Relations Engineer, Google Accessing Gemini through Apple’s Foundation Model framework
2. Peter Friese The Google office in Amsterdam Fun fact: same glasses as Paul Hudson @peterfriese.dev
3. powered by
4. let session = LanguageModelSession() let response = try await session.respond(to: """ Summarize this text: Here’s to the crazy ones. The m i sfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote the m, disagree wi th the m, glorify or vilify the m. About the only thing you can’t do is ignore the m. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do. """ ) print(response.content)
5. On-device AI - benefits 1/ Available offline 2/ No cost 3/ Privacy
6. Context window size: 4096 tokens
8. Gemini Developer API Ve rt ex AI Gemini API Gemini models Firebase client SDKs 􀟜
9. let response = try await session.respond(to: """ Summarize this text: An Owner’s Manual for Google’s Shareholders Google is not a conventional company. We do not intend to become one. Throughout Google’s evolution as a privately held company, we have managed Google differently. We have also emphasized an a tm osphere of creativity and challenge, which has helped us provide unbiased, accurate and free access to information for those who rely on us around the world. Now the t im e has come for the company to move to public ownership. This change wi ll bring i m portant benefits for our employees, for our present and future shareholders, for our customers, and most of all for Google users. . """ ) print(response.content) let session = LanguageModelSession()
10. let response = try await session.respond(to: """ Summarize this text: An Owner’s Manual for Google’s Shareholders Google is not a conventional company. We do not intend to become one. Throughout Google’s evolution as a privately held company, we have managed Google differently. We have also emphasized an a tm osphere of creativity and challenge, which has helped us provide unbiased, accurate and free access to information for those who rely on us around the world. Now the t im e has come for the company to move to public ownership. This change wi ll bring i m portant benefits for our employees, for our present and future shareholders, for our customers, and most of all for Google users. . """ ) print(response.content) let session = LanguageModelSession()
11. let response = try await session.respond(to: """ Summarize this text: An Owner’s Manual for Google’s Shareholders Google is not a conventional company. We do not intend to become one. Throughout Google’s evolution as a privately held company, we have managed Google differently. We have also emphasized an a tm osphere of creativity and challenge, which has helped us provide unbiased, accurate and free access to information for those who rely on us around the world. Now the t im e has come for the company to move to public ownership. This change wi ll bring i m portant benefits for our employees, for our present and future shareholders, for our customers, and most of all for Google users. . """ ) print(response.content) let session = LanguageModelSession() let model = FirebaseAI.firebaseAI().ge mi niLanguageModel(name: "ge mi ni-3.5-flash")
12. let response = try await session.respond(to: """ Summarize this text: An Owner’s Manual for Google’s Shareholders Google is not a conventional company. We do not intend to become one. Throughout Google’s evolution as a privately held company, we have managed Google differently. We have also emphasized an a tm osphere of creativity and challenge, which has helped us provide unbiased, accurate and free access to information for those who rely on us around the world. Now the t im e has come for the company to move to public ownership. This change wi ll bring i m portant benefits for our employees, for our present and future shareholders, for our customers, and most of all for Google users. . """ ) print(response.content) let model = FirebaseAI.firebaseAI().ge mi niLanguageModel(name: "ge mi ni-3.5-flash") let session = LanguageModelSession(model: model)
13. Grounding with Google Maps
14. extension FindPointsOfInteres t T ool { func suggestions(category: Category) - > [String] { s w i tch category { case .restaurant : ["Restaurant 1", "Restaurant 2", "Restaurant 3"] case .campground : ["Campground 1", "Campground 2", "Campground 3"] case .hotel : ["Hotel 1", "Hotel 2", "Hotel 3"] case .cafe : ["Cafe 1", "Cafe 2", "Cafe 3"] case . m useum : [ " M useum 1", "M useum 2", "M useum 3"] case . m arina : [ " M arina 1", "M arina 2", "M arina 3"] case .nationa l M onument : [ "T he National Rock 1", "T he National Rock 2", “The National Rock 3"] } } }
15. let model = FirebaseAI .firebaseAI(backend: .vertexAI(location: "global")) .ge m i niLanguageModel( name: "ge mi ni-3.1-flash-lite" )
16. let model = FirebaseAI .firebaseAI(backend: .vertexAI(location: "global")) .ge m i niLanguageModel( name: "ge mi ni-3.1-flash-lite" ) let session = LanguageModelSession( model: model )
17. let model = FirebaseAI .firebaseAI(backend: .vertexAI(location: "global")) .ge m i niLanguageModel( name: "ge mi ni-3.1-flash-lite" , serverTools: [.googleMaps()] ) let session = LanguageModelSession( model: model )
18. let model = FirebaseAI .firebaseAI(backend: .vertexAI(location: "global")) .ge m i niLanguageModel( name: "ge mi ni-3.1-flash-lite" , serverTools: [.googleMaps()] ) let session = LanguageModelSession( model: model , instructions: Instructions { "Use Google Maps to find real places." } )
19. let model = FirebaseAI .firebaseAI(backend: .vertexAI(location: "global")) .ge m i niLanguageModel( name: "ge mi ni-3.1-flash-lite" , serverTools: [.googleMaps()] ) let session = LanguageModelSession( model: model , instructions: Instructions { "Use Google Maps to find real places." } ) let response = try await session.respond(to: """ Find 3 real \(arguments.pointOfInterest.rawValue) near \(landmark.name) at coordinates (\(landmark.latitude), \(landmark.longitude)) based on this query: \(arguments.naturalLanguageQuery). """ ) print(response.content)
20. Grounding with Google Search
21. let model = FirebaseAI.firebaseAI().ge mi niLanguageModel( name: "ge m i ni-3.1-flash-lite", serverTools: [.googleSearch()] )
22. let model = FirebaseAI.firebaseAI().ge mi niLanguageModel( name: "ge m i ni-3.1-flash-lite", serverTools: [.googleSearch()] ) let session = LanguageModelSession( model: model, instructions: """ Provide a detailed description, general information, opening hours, and upco mi ng events for the requested place. Use Google Search to find the most up-to-date information. """ )
23. let model = FirebaseAI.firebaseAI().ge mi niLanguageModel( name: "ge m i ni-3.1-flash-lite", serverTools: [.googleSearch()] ) let session = LanguageModelSession( model: model, instructions: """ Provide a detailed description, general information, opening hours, and upco mi ng events for the requested place. Use Google Search to find the most up-to-date information. """ ) let stream = session.streamResponse( generating: LandmarkDetailResponse.self, includeSchemaInPrompt: true, options: GenerationOptions(samplingMode: .greedy), prompt: "\(self.activity T i tle) in \(self.landmarkName)." )
24. @Generable struct LandmarkDetailResponse: Codable { @Guide(description: "A detailed summary of what you can do at the place.") let description: String @Guide(description: "A list of strings w i th details like address, phone.") let generalInfo: [String]? @Guide(description: "A list of opening hours.") let openingT im es: [OpeningHour]? @Guide(description: "A list of strings containing events currently on.”) let upco mi ngEvents: [String]? }
25. for entry in session.transcript { if case let .response(responseEntry) = entry { if let metadata = responseEntry . m etadata["groundingMetadata"] as? GroundingMetadata { self.groundingMetadata = metadata } } }
26. Multi-modal prompts
27. let model = FirebaseAI .firebaseAI() .ge m i niLanguageModel(name: "ge mi ni-3.1-flash-lite") let session = LanguageModelSession( model: model, instructions: "You are a visual landmark identifier." )
28. let model = FirebaseAI .firebaseAI() .ge m i niLanguageModel(name: "ge mi ni-3.1-flash-lite") let session = LanguageModelSession( model: model, instructions: "You are a visual landmark identifier." ) let response = try await session.respond(generating: LandmarkIdentification.self) { Attachment(cg Im age) "Identify the landmark in this im age and provide a short description." }
29. let model = FirebaseAI .firebaseAI() .ge m i niLanguageModel(name: "ge mi ni-3.1-flash-lite") let session = LanguageModelSession( model: model, instructions: "You are a visual landmark identifier." ) let response = try await session.respond(generating: LandmarkIdentification.self) { Attachment(cg Im age) "Identify the landmark in this im age and provide a short description." } guard let cg Im age = im age.cg I m age else { throw NSError(domain: "VisualLandmarkIdentifier", code: 3, userInfo: [NSLocalizedDescriptionKey: "Failed to get CG Im age from UI Im age"]) }
30. let model = FirebaseAI .firebaseAI() .ge m i niLanguageModel(name: "ge mi ni-3.1-flash-lite") let session = LanguageModelSession( model: model, instructions: "You are a visual landmark identifier." ) let response = try await session.respond(generating: LandmarkIdentification.self) { Attachment(cg Im age) "Identify the landmark in this im age and provide a short description." } guard let cg Im age = im age.cg I m age else { throw NSError(domain: "VisualLandmarkIdentifier", code: 3, userInfo: [NSLocalizedDescriptionKey: "Failed to get CG Im age from UI Im age"]) }
31. Getting ready for production
32. Gemini models Illegitimate request Legitimate request App Check w/ replay protection X
34. One more thing…
35. One more thing…
36. Thanks!
37. Thanks! @peterfriese.dev Created by Mamank from Noun Project https: / / peterfriese.dev peterfriese peterfriese Slides for this talk
38. Build an AI-powered meal- planner with Firebase
40. Q&A @peterfriese.dev Created by Mamank from Noun Project https: / / peterfriese.dev peterfriese peterfriese