Azure Zurich User Group AI & LLM Soirée ChatGPT over your own data (and beyond)

Marco Gerber Senior Cloud Engineer | Microsoft Azure MVP scopewyse GmbH marco.gerber@scopewyse.com linkedin.com/in/marcogerber @rolebasedpotato marcogerber.ch About me | Tech Azure Architecture, Security, AI, Automation About me | Private Winterthur, Alpine sport, Photography, Travel

The AI journey November 30, 2022 Launch of ChatGPT January, 2023 OpenAI and Microsoft further extend their partnership January, 2023 Microsoft announces Azure OpenAI Service now GA November, 2021 Microsoft introduces Azure OpenAI Service (Preview) The rise of Large Language Models (LLM) The rise of Large Multimodal Models (LMM) and Small Language Models (SLM) Build and integration of AI application Today Microsoft and OpenAI exclusive partnership July 22, 2019 Data partnerships OpenAI examples: - Apple - Reddit - Stack Overflow - Shutterstock - BuzzFeed - GitHub - Vox Media - Financial Times OpenAI Partnerships List (foundationinc.co)

Initial thoughts We cannot use the public ChatGPT due to privacy and compliance reasons. Let‘s deploy Azure OpenAI and use it as internal ChatGPT. Wouldn‘t it be great if we could integrate our own documents and create a knowledge base bot? Well, let‘s use „On Your Data“ in Azure OpenAI and integrate it as a Copilot inside Teams. Prompt-in-prompt-out and asking questions is great, but we need AI to do stuff for us. Let‘s bring in an orchestration layer and chain services together.

▪ Large Language Model (LLM) ▪ GPT-4 (OpenAI) ▪ Llama 3 (Meta) ▪ Gemini (Google DeepMind) ▪ Large Multimodal Model ▪ GPT-4 Turbo with Vision (OpenAI | Images, Video, Text) ▪ GPT-4o (OpenAI | Audio, Images, Video, Text) ▪ Sora (OpenAI | Video generation) ▪ Small Language Model (SLM) ▪ Phi-3 Family ▪ Orca 2 ▪ They can: Put one sequence of character (tokens) after the other (i.e. text generation) ▪ They can not: Run functions, call API‘s, retrieve documents, etc. Therefore, ChatGPT is not a model, it‘s an AI application! LLM / LMM / SLM

▪ Platform/Gallery to share and collaborate on AI content: ▪ Models ▪ Datasets ▪ Spaces ▪ Forum ▪ Etc. Hugging Face

Data enrichment techniques Fine-tuning • Retraining a base model (like GPT-4) with a custom dataset • Adapt or encance to a specific behaviour • Static, not suitable for changing data • Costly to train and maintain Retrieval-Augemented Generation (RAG) Embeddings • Format: Vectors • Good for similarity search rgb(255,180,0) rgb(250,145,0) • Use your own data • Static and changing data • APIs, Databases, Storage (PDF, Word, Markdown, etc.) and many more • Trust through citations GPT-4 model training • Costs of more than $100 Millionen • Trained on ~25‘000 Nvidia A100 Tensor Core GPUs simultaneously (cluster provided by Microsoft) • Continuous training of 90 – 100 days

Google search: 0.0003 kWh ChatGPT query: 0.0029 kWh Not so fun fact: Energy consumption X10 (search only, no running cost, no training, no employees, etc.) Source: AI’s Power Demand: Calculating ChatGPT’s electricity consumption for handling over 156.4 billion user queries every year - BestBrokers.com

Azure OpenAI On Your Data (RAG) How it works Tell me something about our vacation policy. Generated answer, grounded using data from the AI Search index Index documents from storage account Query AI Search Azure OpenAI GPT-4 Azure AI Search Storage Account hosting company documents

Azure OpenAI On Your Data (RAG)

I want more Tell me something about our vacation policy. Generated answer, grounded using data from the AI Search index Index documents from storage account Query AI Search Azure OpenAI GPT-4 Azure AI Search Storage Account hosting company documents User: How many holidays do I have per year? ... Application: Do you want create a vacation request? ... Application: Please provide a start and end date. ... HR API Create vacation request Inform superiors Prompt orchestration

▪ Middleware to blend code/logic and AI ▪ Use AI models to complete tasks, i.e. evaluate what the user wants ▪ Integrate business logic ▪ Trigger actions ▪ ... What is prompt orchestration Prompt Flow Semantic Kernel LangChain

▪ Open-source SDK developed by Microsoft ▪ Python, C# and Java ▪ Goal: 1. Understand what the user wants and what to do (Planner) 2. Easy integration of AI services (Services) 3. Define functions, which are triggered by the planner (Plugins) Semantic Kernel 1. Classify the damage into one of these categories: CAR_INSURANCE, HOUSE_INSURANCE, PET_INSURANCE, OTHER. Do not add any other descriptions. 2. What kind of car damage is it? (PARKING_DAMAGE, THIEF, BROKEN GLASS, OTHER) 3. What is the car make? 4. What is the car model? 5. What is the license number? 6. What is the estimated damage cost (<1000 CHF for minor damage, > 5000 CHF for medium, 10000 CHF for severe. - Replace with MINOR, MEDIUM or SEVERE) 7. Explain the situation in the picture Use planner to find out which plugin to run Run InsurancePolicyPlugin.GetPolicies plugin Generate and ground user feedback Open insurance application and upload photo of damage Azure OpenAI GPT-4 Turbo w/ Vision Analyse image using prompt engineering Look up insurance coverage Your car is covered, do you want to submit a claim? Submit a claim in the insurance management tool I have a car damage Wep app running Semantic Kernel Call API

Demo Prompt orchestration using Semantic Kernel

▪ Think different ▪ Service availability ▪ Responsible and trusted AI ▪ Transparency of training data ▪ Data sources ▪ Biased: Race, gender, age, ethnicity, etc. ▪ Regulation ▪ Security, Networking, Monitoring, Logging ▪ Python basics are beneficial ▪ Be open and eager to learn. Have fun! Key takeaways

Questions?