Efficient model customization with Microsoft Foundry - for Agentic AI

Customize AI models and optimize performance for agentic AI scenarios with targeted fine-tuning in Microsoft Foundry. Learn to adapt agent tone with SFT, reduce cost & latency with Distillation, and enhance developer experience with synthetic data generation, custom evaluation, and developer tier.

Delivered at AI Tour NYC 2026
https://aitour.microsoft.com/flow/microsoft/nyc26/sessioncatalog/page/sessioncatalog/session/1760572081812001YGbG

Transcript

1. Efficient model customization with Microsoft Foundry Nitya Narasimhan, PhD Senior AI Advocate Core AI DevRel @Microsoft
2. What we will cover today Unlock business value with fine-tuning Model customization is critical when you need to improve the quality, accuracy and operating costs for your agentic AI apps Fine-tuning in Microsoft Foundry Why is model customization the right choice here? Why is Microsoft Foundry the best platform to use? Demo: Basic Fine-Tuning Understand supervised fine-tuning Fine-tuning for Agentic Applications Why do models struggle with multi-turn tool calling? How does fine-tuning improve agent performance? Summary Model Customization – unlocks business value for enterprise Microsoft Foundry - streamlines fine-tuning for AI developers Demo: Custom Grader Write evaluators to test target criteria Demo: Agentic Fine-Tuning SFT + Distillation to improve tool calling Demo: Synthetic Data Get high-quality datasets with less effort
3. Zava Retail is building “Cora” – an AI Agent App Bruno Zhao Zava Retail Customer I want to buy the right products for my DIY project. Who can I talk to? Make it helpful Customize Cora’s tone & response to be polite and conversational Robin Counts Retail Store Manager I want to build customer loyalty. Can we ensure its responses are valid? Make it accurate Improve Cora’s use of tool selection, parameter propagation, policy adherence Kian Lambert App Dev Manager I want to save operating costs! Can I teach a cheaper model this task? Make it cost-effective Distill Cora’s task-specific behaviors into smaller models without losing accuracy
4. Unlock business value with model optimization
5. I’m painting my living room wall. What paint should I buy? Tone and style Query Extraction Example responses Personalization Intent mapping Inventory retrieval User input Context engineering Desired Output Model adaptation Prompt Engineering Retrieval Augmented Generation (RAG) I recommend our Eggshell Paint Would you like to know more about color choices Fine Tuning The Model Customization Journey For Cora Teach AI models new behaviors Improve accuracy by using tools & knowledge better Reduce costs by distilling behavior to smaller models
6. What does fine-tuning mean? LLM Fine-tuned LLM Fine-tuning refers to customizing a pre-trained LLM with additional training on a specific task or new dataset for enhanced performance, new skills, or improved accuracy
7. Why should we fine-tune? Domain-specific optimization Task-specific optimization Reduced token consumption Efficient resource utilization Smaller models, faster response Shorter prompts, improve response Improve quality Reduce cost Reduce latency Example: We have a domain-specific focus (retail) and task-specific focus (question-answering). Let’s think about how fine-tuning can help optimization.
8. How do you fine-tune a foundation model? Generic LLM/SLM Language capability Reasoning capability World knowledge Instruction-following Coding Chat Etc. Instruction tuning (SFT) Instruction-response data, task-specific data, other annotated data Alignment tuning (RLHF, DPO) Human preference data, reward model data, preference score data Fine-tuning Fine-tuned LLM/SLM Adapted with your data, for your use case Teach it to copy behaviors Teach it to reason better
9. Supervised fine-tuning Learns to copy behaviors Ex: Content generation task Reinforcement fine-tuning Learns to reason better Ex: Complex logic tasks Model distillation Transfers learning to smaller model Optimize for COST Vision fine-tuning Preference fine-tuning Hybrid fine-tuning Improve image understanding Ex: ClassificationTask Prioritize good vs. bad Ex: Tone adaptation Improve context selection in RAG Optimize for PRECISION Why should we fine-tune on Microsoft Foundry?
10. Part 1 – Basic Fine Tuning Make it helpful Customize Cora’s tone & response to be polite & conversational
11. Demo 1: Supervised Fine Tuning – Basic workflow Decide vision and scope Choose base model Choose FT technique Dataset Fine-tuning Evaluation Deploy and monitor Regularly benchmark and iterate!
12. Microsoft Foundry We’re making every aspect of fine-tuning better No Data? No problem! Synthesize training data from documents and code Public Preview Train for Less! 50% discount with Dev Training tier – and higher quota Public Preview Open-Source Models Ministral, OSS-20B, Llama 3.3 70B, and Qwen3 32B – in the same UX Public Preview Agentic RFT Fine tune for tool use in chain of thought for o4-mini & GPT-5 Private Preview AI.Azure.com
13. Synthetic data generation Pre-defined recipes like Q&A and tool calling (for agents) Upload PDFs, docs, or code Our multi-agent framework creates high quality training data Public preview
14. Demo 2: Synthetic Data & Graders – Improve DX Objective Teach a best-in-class model to call the right tools to solve complex business problems Model: GPT 4.1-nano Technique: Supervised fine-tuning Data: Synthetic data generation Train: Foundry UI & Python SDK Evaluate: Foundry evals Deploy: DevTier Regularly benchmark and iterate!
15. Part 2 – Agentic Fine Tuning Make it accurate Improve Cora’s use of tool selection, parameter propagation & policy adherence Make it cost-effective Distill Cora’s behaviors into a smaller model without losing accuracy
16. Demo: SFT + Distillation – Improve Agent Tool Calling Objective Teach a best-in-class model to call the right tools to solve complex business problems Model: GPT 4.1-nano Technique: Supervised fine-tuning Data: Synthetic data generation Train: Foundry UI & Python SDK Evaluate: Foundry evals Deploy: DevTier Regularly benchmark and iterate!
17. Developer Training Tier Training can be expensive— especially for RFT models! DevTier training offers a 50% discount Jobs execute on pre-emptible capacity—think of it like spot VMs for training Public preview
18. Demo: Deploy & Test on DevTier – Improve DX +Cost Objective Teach a best-in-class model to call the right tools to solve complex business problems Model: GPT 4.1-nano Technique: Supervised fine-tuning Data: Synthetic data generation Train: Foundry UI & Python SDK Evaluate: Foundry evals Deploy: DevTier Regularly benchmark and iterate!
19. Microsoft Foundry We’re making every aspect of fine-tuning better No Data? No problem! Synthesize training data from documents and code Public Preview Train for Less! 50% discount with Dev Training tier – and higher quota Public Preview Open-Source Models Ministral, OSS-20B, Llama 3.3 70B, and Qwen3 32B – in the same UX Public Preview Agentic RFT Fine tune for tool use in chain of thought for o4-mini & GPT-5 Private Preview AI.Azure.com
20. Trainer 4. Trainer updates model weights to produce the best CoT Let me guess x = apple I need to subtract 5 x = -5 I need to subtract 1 x = 1 2. Model generates multiple samples 0 1 0.5 3. Grader assign samples a score between 0-1. E.g., 0.5 if output is a number, 0.5 if output is correct Grader What is x? x+5=0 1. Prompt sent to model Model Reinforcement Fine-Tuning (RFT) teaches the model to produce outputs that score highly on a learned reward metric. RFT can elicit more structured, goal-directed reasoning behaviors that are not directly obtainable through imitation alone Reinforcement Fine Tuning: Improve Model Reasoning
21. In Preview: Agentic RFT with GPT-5 Objective Teach a best-in-class model to call the right tools to solve complex business problems Model: GPT-5 Technique: Reinforcement fine-tuning Data: 10 manually curated examples Train: Foundry UI & Python SDK Evaluate: Foundry evals Deploy: DevTier Regularly benchmark and iterate! Sign up at aka.ms/agentic-rft-preview
22. Summary & Next Steps
23. Recap: Model customization unlocks business value I’m painting my living room wall User input System prompt Few shot examples Longer prompts Add my data Grounded responses Imprecise selection Prompt engineering + RAG Simplified workflows Better dev experience Synthetic Data & Graders Good choice! I recommend our Eggshell Paint. Would you like to know more about color choices? LLM output Shorter prompts Better response quality Better tool calling Supervised fine-tuning SFT + Distillation Cheaper models with comparable accuracy
24. Recap: Microsoft Foundry makes fine-tuning seamless Model choice The best models from the best providers Choose serverless or managed compute Reliability 99.9% availability for Azure OpenAI models Latency guarantees with PTU-M Foundry platform Everything you need in one place: models, training, evaluation, deployments, and metrics Scalability Start with low cost DevTier to experiment Scale up with PTU-M for production workloads