Model Mondays S2E08: On-Device and Local AI

On-device inference is critical if you want to run AI models locally, on your own hardware, e.g., for edge computing needs. Join as as we talk to Maanav Dalal about Foundry Local – a solution built on ONNX Runtime (for use in CPUs, NPUs & GPUs) & taking you from prototype to production.

On our customer stories segment we're also joined by Marilyn Morgan Westner (Co-founder and Chief Experience Officer) and Alex Westner, (Co-founder and CEO) at Xander Glasses. Learn how they bring sight to sound with AI-driven captioning on their wearable assistive device.

Register:
Livestreamed: Aug 04 / https://developer.microsoft.com/en-us/reactor/events/26127/
Discord AMA: Aug 08 / https://discord.gg/azureaifoundry?event=1382863345777901670

Catch-up After:
Livestream Recording: https://www.youtube.com/watch?v=ILBDDCJ0d9g
AMA Transcript: https://github.com/orgs/azure-ai-foundry/discussions/108
AMA Schedule: https://aka.ms/model-mondays/forum

About Model Mondays:
GitHub: https://aka.ms/model-mondays/
Livestream: https://aka.ms/model-mondays/rsvp
Discord: https://aka.ms/model-mondays/discord

Transcript

1. Maanav Dalal Product Manger, Core AI / Microsoft MODEL MONDAYS S2:E08 Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:15 PM ET Tools Spotlight On: On-Device & Local AI
2. Co-founder & Chief Experience Officer, Xander MODEL MONDAYS S2:E08 Marilyn Morgan Westner Co-founder & CEO, Xander Alex Westner Customer Stories: Xander Industry Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:15 PM ET
3. Sharmila Chockalingam Senior PMM, Azure AI Weekly Highlights WHAT’S NEW IN AZURE AI FOUNDRY THIS WEEK?
4. ❶ · Try · RFT Observability RFT Observability Provides real-time, in-depth visibility into your RFT job by automatically kicking off evaluation (“auto-evals”) that shows the detailed finetuning progress at each checkpoint.. Read:| https://techcommunity.microsoft.com/category/ai/blog/azure-ai-services-blog
5. ❷ · Try · Github Spark Github Spark Build and ship full-stack intelligent apps using natural language with access to the full power of the GitHub platform—no setup, no configuration, and no headaches. Read: https://github.blog/changelog/2025-07-23-github-spark-in-public-preview-for-copilot-pro-subscribers/
6. ❸ · Try · DAViD Models DAVID These are models that require only a fraction of the cost of training and inference when compared with foundational models of similar accuracy SynthHuman Dataset: https://github.com/microsoft/DAViD Read Paper: https://arxiv.org/abs/2507.15365
7. ❹ · Try · Agent Experience Optimization Agent Experience Optimization Make your content easily discoverable by AI agents recent best practices, including how Microsoft’s NLWeb project fits in, and other emerging standards Explore Series: https://aka.ms/the-future-of-ai View: https://aka.ms/model-mondays/forum
8. ❺ · Watch · MCP for Beginners MCP For Beginners core concepts, hands-on development, advanced implementation, and real-world case studies. Watch: https://aka.ms/mcp-videos View: https://aka.ms/mcp-for-beginners
9. Nitya Narasimhan Senior AI Advocate, Microsoft Weekly Spotlight HOSTED BY
10. Maanav Dalal Product Manger, Core AI / Microsoft On-Device & Local AI MODEL MONDAYS S2:E08 Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:00 PM ET Tools
11. ❶ · Read · Documentation Foundry Local enables efficient, secure, and scalable AI model inference directly on your devices! Understand the Foundry Local Architecture Docs: https://learn.microsoft.com/azure/ai-foundry/foundry-local Join: https://aka.ms/model-mondays/discord Key features include Local inference (on-device) with data privacy, Model customization with cost efficiency (local hardware), Seamless integrations with CLI, SDK and REST API
12. ❷ · Watch · Build Session Foundry Local – Building cutting-edge on-device AI experiences Build on-device AI with Azure AI Foundry, ONNX Runtime & Rich Tooling Watch: https://build.microsoft.com/sessions/BRK146 View: https://aka.ms/model-mondays/forum Build intelligent on-device AI with Azure AI Foundry. With seamless integration and top performance across CPU, GPU, and NPU, as well as the seamless integration of Azure AI Foundry Model catalog and inferencing APIs for great developer experience cross local and cloud. Experience improved performance, enhanced security, and cost-efficient operations that minimize errors while enabling real-time processing both locally and in the cloud.
13. ❸ · Explore · Repository What link is the QR Code pointing to Takeaway message with more context for what you learn from that link Repo: https://github.com/microsoft/Foundry-Local View: https://aka.ms/model-mondays/forum
14. ❹ · Try · See it in practice Get Hands-on with a practical fine-tuning lab using Foundry Local Highlights use of local models & hardware for on-device inference Hands-on Lab: https://github.com/microsoft/Build25-LAB329 View: https://aka.ms/model-mondays/forum
15. ❺ · Explore · Olive Toolkit Want to optimize AI models for targeted hardware? Try the Olive (ONNX Live) toolkit! Helps build optimal ONNX models e.g., for use in Foundry Local Explore: https://github.com/microsoft/Olive View: https://aka.ms/model-mondays/forum
16. Co-founder and Chief Experience Officer, Xander MODEL MONDAYS S2:E08 Marilyn Morgan Westner Co-founder and CEO, Xander Alex Westner Customer Stories: Xander Industry With: Nitya Narasimhan Visit: https://www.xanderglasses.com/
17. ❶ · Learn · Sight For Sound Xander Captioning Glasses let you “see” what people are saying (sight for sound) Glasses listen and transcribe speech into text in real time powered by Azure AI Action: https://aka.ms/model-mondays/customer-stories View: https://www.xanderglasses.com
18. Monalisa Whalin Principal PM / Core AI @Microsoft Models for AI Agents MODEL MONDAYS S2:E09 Models Hosted By: April Gittens AUG 11, 2025 · 1:30-2:00 PM ET
19. Maanav Dalal Product Manger, Core AI / Microsoft Foundry Friday AMA AZURE AI FOUNDRY DISCORD ON-DEVICE AND LOCAL AI AUG 08, 2025 · 1:30-2:00 PM ET Tools
20. Fri 1:30pm ET Join The Discord https://aka.ms/model-mondays/Discord Links To Resources Summary of AMA https://aka.ms/model-mondays/Forum #ModelMondays WATCH LIVE ON MONDAY FIND RECAPS ON FORUM JOIN AMA ON FRIDAY Mon 1:30pm ET Register Here https://aka.ms/model-mondays/RSVP