From Catalog To Cloud: An AI Engineer's Journey

Keynote for the 2024 Community Days Event

Abstract:
So you want to build a generative AI application - a custom copilot - for your enterprise. Who should do this work? ML Engineers build models and Software Engineers ship application products - but who bridges the gap beween Model and Product?

Say hello to the rise of the AI Engineer. In this talk we'll look at the AI Engineer journey from Catalog to Code, as we talk about the key skills that this role requires for building applications from first prompt to final product.

Transcript

1. From Catalog To Cloud: An AI Engineer’s Journey Nitya Narasimhan, PhD Senior AI Advocate, Microsoft Follow Me For AI Updates At: https://linkedin.com/in/nityan 2024 Community Days Keynote
2. Let me give you the Big Picture of AI Engineering! ž Motivation – The Rise of the AI Engineer ž Catalog – Understand Model Choice ž Code – Understand Development Workflow ž Cloud – From Prototype To Production ž Copilot – Learn By Deconstructing Samples
3. The Rise Of The AI Engineer ML Engineers know how to build language models Software Engineers know how to build production apps There is a knowledge and skills gap between these two roles
4. Does this app use generative AI? (Natural Language, Task Types) Want to build a custom copilot? Does this app have private data? (How do I ground responses?) What models do I need to use? (How do I find or select them?) What architecture should I use? (How do I orchestrate app flow?) What platform should I use? (Go from prompt to production)
5. I build foundation AI Models I ship AI Products “AI Engineering” bridges the skills gap between machine learning (models) and software engineering (products) Amanda Silver - CVP, Developer Division @Microsoft - Here's a brief overview of some insights that our amazing DevDiv User Research team has put together. • Model Selection • Prompt Engineering • Fine Tuning • Retrieval Augmented Generation • AI-Assisted Evaluation • End-to-End Development
6. The AI Engineer’s Journey – From Catalog To Cloud Improve response quality. How do I design prompts? Model Selection Prompt Engineering RAG Architecture Fine Tuning Model Evaluation Azure AI Platform Models drive your AI apps. How do you choose one? Ground response in my data. How do I orchestrate this? Retrain base model with my data for cost & capability. Define quality & safety metrics, AI-assisted flows Streamline development with unified tools & flows CATALOG CLOUD
7. Let’s Talk About Model Selection How do I make the right choice?
8. Oct 2024 | The Future Of AI Is Model Choice - Tech Community The Paradox of Choice – leads to decision fatigue The Developer Challenge – tradeoff quality and cost The Filter Process – reduce catalog to manageable subset
9. Oct 2024 | The Future Of AI Is Model Choice - Tech Community Pick a model that can do the task you want – or support fine-tuning to do it Then see if the precision works for you – or if you need specialization Now you need to make a decision on cost vs. quality The Azure AI Model Catalog can help! Demo: Azure AI Model Catalog
10. Let’s Talk About Development How can I improve the responses from my model choice in a way that lets me tradeoff cost and complexity as needed
11. Let’s Talk About Prompt Engineering User Prompt = the user question asked Model Prompt = the enhanced question send to the model Prompt Engineering is about adding context and instructions to user prompts to improve the quality of final response
12. May 2024 | Generative AI For Beginners - Chapter 4 System Context: define personas and behavior Primary Content: add related knowledge to ground responses Model Configuration: change models or their parameters for quality Prompt Template: have reusable prompts with placeholders for data – for popular tasks or application domains Demo: GitHub Model Marketplace
13. Let’s Talk About RAG Architecture
14. Sep 2024 | RAG With Azure AI Studio | Slides & Video Embedding Model: change text question into vectorized query Retrieval Service: find matching results in a relevant search index Semantic Ranking: put results in order that can fit query context best Prompt Templating: put retrieved knowledge in a prompt template to augment user question
15. Sep 2024 | RAG With Azure AI Studio | Slides & Video Embedding Model: change text question into vectorized query Retrieval Service: find matching results in a relevant search index Semantic Ranking: put results in order that can fit query context best Prompt Templating: put retrieved knowledge in a prompt template to augment user question
16. Let’s Talk About Fine Tuning Base Model = pre- trained on large data Fine-Tuned Variant = trades off quality or cost for complexity Fine Tuning is about retraining that model with your own data set to enhance quality or capabilities
17. May 2024 | Generative AI For Beginners - Chapter 28 Demo: Fine-Tuning With Open AI Concepts: what is fine- tuning & why use it? Process: how do we get started & what are the main challenges faced? Dataset Creation: how do we find data for this (own vs. synthetic) Fine-Tuning Workflow: run job, host variant, compare performance Find & Publish Variants: in community hubs
18. Let’s Talk About Evaluation How do I make the right choice?
19. Application Lifecycle (GenAIOps) Operationalizing Building/ augmenting Ideating/ exploring Managing BUSINESS NEED
20. Yes Yes 1. Ideating/exploring 2. Building/augmenting 3. Operationalizing
21. Let’s Talk About AI-Assisted Evaluation
22. Let’s Talk About Evaluator Metrics & Custom Evaluators for Quality Coherence: does the response make sense for the given question Fluency: is the response grammatically correct and well-written? Relevance: did given response answer the question asked? Groundedness: did the response use the data context we provided? Custom: define metric and its scoring criteria
23. Let’s Talk About Developer Tools How do I streamline my e2e workflow?
24. Let’s Talk About The Azure AI Platform DEMO: AI Apps Gallery Model catalog: go from discovery to evaluation to deployment in mins. Hub & Project: manage AI resources & monitor deployed applications with unified platform Developer Tools: from provisioning to deploy, with CLI, SDK support. Trustworthy AI: built-in safety systems, content filters & evaluators. AI Templates: sample apps for key scenarios
25. TRY IT: Contoso Chat Workshop Nov 2024 | Build a Custom Retail Copilot Code-First With Azure AI Star Repo and watch for updates during Microsoft Ignite Try The Workshop and register for an in-venue session at MSIgnite or on Microsoft AI Tour.
26. ž Microsoft Ignite – LAB 401 · THR 502 ž Microsoft AI Tour – WRK 550 · Track 5 ž Contoso Chat – Self-Guided Workshop ž #30DaysOfIA – Azure AI Week (5 Posts) ž Model Choice – The Future of AI Blog ž Generative AI For Beginners – Free Course Follow Me For AI Updates At: https://linkedin.com/in/nityan Want a copy of theses slides, resource links or high-resolution versions of the sketchnotes from this talk? Follow me on LinkedIn for a blog post! Thank You!
27. From Catalog To Cloud: An AI Engineer’s Journey Nitya Narasimhan, PhD Senior AI Advocate, Microsoft Follow Me For AI Updates At: https://linkedin.com/in/nityan THANK YOU FOR LISTENING!!