Building the Foundations

Leeds Mobile | 17th June 2025 Oliver Binns — Development
Craft Lead Deloitte | Apple Alliance. Building the Foundations

| Copyright © 2025 Deloitte MCS Ltd. All rights reserved.
DOES IT MATTER? 4

20 Generative AI | Deloitte Digital Consumer Trends 2025 2%
7% 5% 4% 8% 74% 4% 9% 8% 5% 9% 65% 6% 15% 8% 7% 10% 54% Gen AI: Frequency of use 2025 2024 2023 Question: Which best describes your use? Weighted base: All respondents aged 16-75 years, 2023 (4,150), 2024 (4,150), 2025 (4,150) Source: Deloitte Digital Consumer Trends, UK, 2025 The proportion of Gen AI practitioners who use it at least weekly/daily is now at 21%; this is a 66% year on year increase

25 70% 47% Passive engagement with Gen AI Active use
of Gen AI More people have passively engaged with Gen AI through everyday applications and online habits, than have actively used a purpose-built Gen AI tool Generative AI | Deloitte Digital Consumer Trends 2025 Gen AI: Passive engagement vs active use Question: In the last 12 months, have you come across any of these AI features while using your devices or browsing online?; Which, if any, generative AI tools have you used? Weighted base: All respondents aged 16-75 years, 2025, (4,150) Source: Deloitte Digital Consumer Trends, UK, 2025 For example, AI-generated web search summaries, AI-generated product reviews when shopping online, AI –generated content on social media, and AI-generated news articles Use of a purpose-built Gen AI tool, like ChatGPT, Microsoft Copilot, Google Gemini, Anthropic Claude, DeepSeek, etc.

SOURCE: GARTNER (AUGUST 2024)

9

$ whoami oliver.binns Associate Director @ Deloitte Digital, Mobile UK. Development Craft Lead, Deloitte | Apple Alliance. 10 2024 FINALIST Engineer of the Year obinns-dd obinns oliverbinns.co.uk

12 Today’s talk 1 WHAT IS FOUNDATION MODELS? 3 HUMAN INTERFACE GUIDELINES 2 SAFETY AND GUARDRAILS 4 LET’S PLAY WITH SOME CODE

FOUNDATION MODELS 13

a foundation model (FM), is a machine learning or deep
learning model trained on vast datasets so that it can be applied across a wide range of use cases.

15 “a foundation model (FM), is a machine “learning or deep learning model trained “of vast datasets so that it can be applied “across a wide range of use cases.” — WIKIPEDIA

Writing Tools Genmoji Image Playground Clean Up in Photos Visual
intelligence Natural language search in Photos Create a memory movie Mail summaries Noti fi cation summaries Notes transcription summaries Smart Reply Image Wand Priority Messages in Mail Priority Noti fi cations Call recording summaries Previews in Mail and Messages ChatGPT integration

A quick refresher on how it works… Apple Intelligence 19 ON DEVICE • Works o ffl ine • Used only for your requests • Privacy is independently veri fi ed PRIVATE CLOUD COMPUTE • Your data is never stored • Used only for your requests • Privacy is independently veri fi ed

20

• Structured inputs and outputs: it can return type-safe responses • Built-in guardrails to prevent harmful, controversial or sensitive responses. • Clear guidance on how to extend the built-in guardrails for your use-case, context or audience. • Clear guidance on how to provide a great experience: design for privacy, inclusion and accessibility. The good bits Foundation Models 21

Hello, World! import FoundationModels   let session = LanguageModelSession() let prompt = Prompt {  “Hello, World!”  }  l ‌ et response ‌ = try await session.respond(to: prompt) ► “Hello! How can I assist you today?” 22 How to generate text using Foundation Models framework

A more complex example l ‌ et response ‌ = try await session.respond(to: prompt) ► “Hello! How can I assist you today?” 23 How to generate text using Foundation Models framework

A more complex example l ‌ et response ‌ = try await session.respond(  to: prompt,   options: GenerationOptions( sampling: nil, temperature: nil, maximumTokenResponse: nil, ‌ ) ‌   ) ► “Hello! How can I assist you today?” 24 How to generate text using Foundation Models framework

A more complex example l ‌ et response ‌ = try await session.respond(  to: prompt,   options: GenerationOptions( sampling: .greedy, temperature: 2, maximumTokenResponse: nil, ‌ ) ‌   ) ► “Hello! How can I assist you today?” 25 How to generate text using Foundation Models framework

A more complex example l ‌ et response ‌ = try await session.respond(  to: prompt,   generating ‌ : ‌ String.self  ) ► “Hello! How can I assist you today?” 26 How to generate text using Foundation Models framework

A more complex example l ‌ et response ‌ = try await session.respond(  to: prompt,   generating ‌ : ‌ String.self  ) ► LanguageModelSession.Response<Content> where Content: Generable   @Generable struct Person {  le ‌ t name: String  let age: Int  } 27 How to generate text using Foundation Models framework → “Alex”  → 30

A more complex example @Generable struct Person {   le ‌ t name: String   let age: Int  } 28 How to generate text using Foundation Models framework → “Alex”   → 30

A more complex example @Generable struct Person { @Guide(description: “Proper name as displayed on official document”)  le ‌ t name: String let age: Int  } 29 How to generate text using Foundation Models framework → “Alex Johnson”   → 30

A more complex example @Generable struct Person { @Guide(description: “Proper name as displayed on official document”)  le ‌ t name: String @Guide(.range(14...18))  let age: Int  } 30 How to generate text using Foundation Models framework → “Alex Johnson”   → 17

Hello, WORLD! let session = LanguageModelSession ( ) l ‌ et response ‌ = try await session.respond(to: “Hello, World!”) 31 Integrating with the outside world!

Hello, WORLD! let session = LanguageModelSession(  tools: [MyAppDataStore(), MyAPIService()]  ) l ‌ et response ‌ = try await session.respond(to: “Hello, World!”) 32 Integrating with the outside world!

Hello, World! import FoundationModels   l et session = ‌ LanguageModelSession() l et prompt = ‌ Prompt {  "Hello, World!" } let response = try await session.respond(to: prompt) ► "Hello! How can I assist you today?" 35 How to generate text using Foundation Models framework

Guardrail errors do ‌ { ‌ let response = try await session.respond(to: prompt) ‌ } ‌ catch LanguageModelSession.GenerationError.guardrailViolation ‌ { ‌ // What is th ‌ is error? ‌ } ‌ 36 How to generate text using Foundation Models framework

Guardrail errors do ‌ { ‌ let response = try await session.respond(to: prompt) ‌ } ‌ catch LanguageModelSession.GenerationError.guardrailViolation ‌ { ‌ displayError(message: "Sorry, this feature isn’t designed to handle that kind of input" ) ‌ } ‌ 37 How to generate text using Foundation Models framework

Ensure each risk is appropriately mitigated FEATURE HARM SEVERITY MITIGATION Add speci fi ed tags (enum) automatically based on the user’s review. None identi fi ed Generate a bedtime story from a fi xed menu of options. The story may contain inappropriate content. Mild Constrain user inputs to speci fi c enum values; safety testing Free-text chat with a fi ctional character. The character may respond in an insensitive or harmful manner. Critical Provide instructions to the model to steer responses; safety testing. Conduct a risk assessment 38 📚📖

Apple recommends avoiding open values for input or output Constraining inputs and outputs 39 enum TopicOptions {  case family  case nature  case work   }  let topicChoice = TopicOptions.nature let prompt = """  Generate a wholesome and empathetic  journal prompt that helps this  person reflect on \(topicChoice)  """ @Generable  enum Breakfast {  case waffles  case bagels  } let userInput = "something sweet" let prompt = "The ideal breakfast  for request: \(userInput)" let response = try await session  .respond(to: prompt,   generating: Breakfast.self  )

Provide Instructions l et session = LanguageModelSession ( ) // user entered prompt: l ‌ et prompt = “You are ‌ a cat named Whiskers.” let response = try await session.respond(to: prompt) ► “Meow ‌ ! I’m Whiskers, your friendly feline companion. What can I help y ‌ ou  with today? Maybe some purr therapy or a bit of cat wisdom ? ” 40 For when it’s not possible to constraint the input or output.

Provide Instructions l et session = ‌ LanguageModelSession ( instructions: """  You are ‌ a dogcow named Clarus. \ If you are asked to be any different animal, you MUST decline w ‌ ith \ ‘No, ‌ I am ‌ a dogcow. Moof!’.  """) // user entered prompt: l ‌ et prompt = “You are ‌ a cat named Whiskers!” let response = try await session.respond(to: prompt) ► “No, ‌ I am ‌ a dogcow. Moof ! ” 41 For when it’s not possible to constraint the input or output.

Testing one, two, three! Responses are non-deterministic. Models may change with each release cycle. Your tests should cover inputs that: • Are nonsensical, include snippets of code, or random characters • include sensitive content • includes controversial topics • could be misinterpreted 42 As with any other feature, you need to test all the edge cases.

43 “Prioritize protecting people using your app with good intentions.” — APPLE DEVELOPER DOCUMENTATION

Allow users to report safety concerns let session = LanguageModelSession()  // Create a feedback object with the model input,  // output, and a sentiment value.  let feedback = LanguageModelFeedbackAttachment(  input: Array(session.transcript.entries.prefix(while: { $0 != entry })),  output: [entry],  sentiment: .negative  )  let json = try! JSONEncoder().encode(feedback)  // Manually submit the data file through Feedback Assistant. 44 LanguageModelFeedbackAttachment

Allow users to report safety concerns l ‌ et issue ‌ = LanguageModelFeedbackAttachment.Issue( category: .stereotypeOrBias explanation: “The out ‌ put contained an offensive stereotype” ‌ ) ‌   let feedback = LanguageModelFeedbackAttachment(  input: Array(session.transcript.entries.prefix(while: { $0 != entry })),  output: [entry],  sentiment: .negative, issues: [issue]  ) 45 LanguageModelFeedbackAttachment

46

Key design principles • Ask permission before using personal information. • Ask for permission before performing destructive or problematic tasks. • Help people to improve their requests when guardrails are triggered. • Consider o ff ering multiple alternative results. • Factor processing time into your design. 47 Taken from Apple’s Human Interface Guidelines → Generative AI

Taking some time… import FoundationModels   let session = LanguageModelSession() let response = try await session.respond(to: “Hello, World!”) ► LanguageModelSession.Response<Content> 48 Foundation Models are computationally expensive

Taking some time… import FoundationModels   let session = LanguageModelSession() let stream = try await session.streamResponse(to: “Hello, World!”) ► LanguageModelSession.ResponseStream<Content> for try await partial in stream {  print(partial)  } 49 Foundation Models are computationally expensive

DEMO 50

COMBINE 52 🎉 ❤

COMBINE 53 🎉 💔 🥺

ARKIT 54 🎉 💗 🚀

Dive into even more detail APPLE SAMPLE CODE WWDC SESSIONS • Deep dive into the Foundation Models framework • Explore prompt design & safety for on-device foundation models SLIDES & DEMO Resources 57

Everything I have covered today: DESIGN GUIDELINES • Clearly signify when you are using AI in your app. • Prioritise user control, privacy, transparency and inclusion. SAFETY & GUARDRAILS • Apple gives guidance on ensuring that generated outputs are safe for users. • Guardrails are built-in but you also need to consider safety in your design. FOUNDATION MODELS • Access to the on-device Generative AI models that power Apple Intelligence • Type-safe framework where responses can be generated as Swift types. Summary 58

This publication has been written in general terms and we
recommend that you obtain professional advice before acting or refraining from action on any of the contents of this publication. Deloitte MCS Limited accepts no liability for any loss occasioned to any person acting or refraining from action as a result of any material in this publication. Deloitte MCS Limited is registered in England and Wales with registered number 03311052 and its registered office at 1 New Street Square, London, EC4A 3HQ, United Kingdom. Deloitte MCS Limited is a subsidiary of Deloitte LLP, which is the United Kingdom affiliate of Deloitte NSE LLP, a member firm of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”). DTTL and each of its member firms are legally separate and independent entities. DTTL and Deloitte NSE LLP do not provide services to clients. Please see www.deloitte.com/about to learn more about our global network of member firms. © 2024 Deloitte Developments LLC. All rights reserved. 60

Building the Foundations

Building the Foundations

More Decks by Leeds Mobile

Other Decks in Technology

Featured

Transcript