Slide 1

Slide 1 text

Azure Zurich User Group AI & LLM Soirée ChatGPT over your own data (and beyond)

Slide 2

Slide 2 text

Marco Gerber Senior Cloud Engineer | Microsoft Azure MVP scopewyse GmbH [email protected] linkedin.com/in/marcogerber @rolebasedpotato marcogerber.ch About me | Tech Azure Architecture, Security, AI, Automation About me | Private Winterthur, Alpine sport, Photography, Travel

Slide 3

Slide 3 text

The AI journey November 30, 2022 Launch of ChatGPT January, 2023 OpenAI and Microsoft further extend their partnership January, 2023 Microsoft announces Azure OpenAI Service now GA November, 2021 Microsoft introduces Azure OpenAI Service (Preview) The rise of Large Language Models (LLM) The rise of Large Multimodal Models (LMM) and Small Language Models (SLM) Build and integration of AI application Today Microsoft and OpenAI exclusive partnership July 22, 2019 Data partnerships OpenAI examples: - Apple - Reddit - Stack Overflow - Shutterstock - BuzzFeed - GitHub - Vox Media - Financial Times OpenAI Partnerships List (foundationinc.co)

Slide 4

Slide 4 text

Initial thoughts We cannot use the public ChatGPT due to privacy and compliance reasons. Let‘s deploy Azure OpenAI and use it as internal ChatGPT. Wouldn‘t it be great if we could integrate our own documents and create a knowledge base bot? Well, let‘s use „On Your Data“ in Azure OpenAI and integrate it as a Copilot inside Teams. Prompt-in-prompt-out and asking questions is great, but we need AI to do stuff for us. Let‘s bring in an orchestration layer and chain services together.

Slide 5

Slide 5 text

▪ Large Language Model (LLM) ▪ GPT-4 (OpenAI) ▪ Llama 3 (Meta) ▪ Gemini (Google DeepMind) ▪ Large Multimodal Model ▪ GPT-4 Turbo with Vision (OpenAI | Images, Video, Text) ▪ GPT-4o (OpenAI | Audio, Images, Video, Text) ▪ Sora (OpenAI | Video generation) ▪ Small Language Model (SLM) ▪ Phi-3 Family ▪ Orca 2 ▪ They can: Put one sequence of character (tokens) after the other (i.e. text generation) ▪ They can not: Run functions, call API‘s, retrieve documents, etc. Therefore, ChatGPT is not a model, it‘s an AI application! LLM / LMM / SLM

Slide 6

Slide 6 text

▪ Platform/Gallery to share and collaborate on AI content: ▪ Models ▪ Datasets ▪ Spaces ▪ Forum ▪ Etc. Hugging Face

Slide 7

Slide 7 text

Data enrichment techniques Fine-tuning • Retraining a base model (like GPT-4) with a custom dataset • Adapt or encance to a specific behaviour • Static, not suitable for changing data • Costly to train and maintain Retrieval-Augemented Generation (RAG) Embeddings • Format: Vectors • Good for similarity search rgb(255,180,0) rgb(250,145,0) • Use your own data • Static and changing data • APIs, Databases, Storage (PDF, Word, Markdown, etc.) and many more • Trust through citations GPT-4 model training • Costs of more than $100 Millionen • Trained on ~25‘000 Nvidia A100 Tensor Core GPUs simultaneously (cluster provided by Microsoft) • Continuous training of 90 – 100 days

Slide 8

Slide 8 text

Google search: 0.0003 kWh ChatGPT query: 0.0029 kWh Not so fun fact: Energy consumption X10 (search only, no running cost, no training, no employees, etc.) Source: AI’s Power Demand: Calculating ChatGPT’s electricity consumption for handling over 156.4 billion user queries every year - BestBrokers.com

Slide 9

Slide 9 text

Azure OpenAI On Your Data (RAG) How it works Tell me something about our vacation policy. Generated answer, grounded using data from the AI Search index Index documents from storage account Query AI Search Azure OpenAI GPT-4 Azure AI Search Storage Account hosting company documents

Slide 10

Slide 10 text

Azure OpenAI On Your Data (RAG)

Slide 11

Slide 11 text

I want more Tell me something about our vacation policy. Generated answer, grounded using data from the AI Search index Index documents from storage account Query AI Search Azure OpenAI GPT-4 Azure AI Search Storage Account hosting company documents User: How many holidays do I have per year? ... Application: Do you want create a vacation request? ... Application: Please provide a start and end date. ... HR API Create vacation request Inform superiors Prompt orchestration

Slide 12

Slide 12 text

▪ Middleware to blend code/logic and AI ▪ Use AI models to complete tasks, i.e. evaluate what the user wants ▪ Integrate business logic ▪ Trigger actions ▪ ... What is prompt orchestration Prompt Flow Semantic Kernel LangChain

Slide 13

Slide 13 text

▪ Open-source SDK developed by Microsoft ▪ Python, C# and Java ▪ Goal: 1. Understand what the user wants and what to do (Planner) 2. Easy integration of AI services (Services) 3. Define functions, which are triggered by the planner (Plugins) Semantic Kernel 1. Classify the damage into one of these categories: CAR_INSURANCE, HOUSE_INSURANCE, PET_INSURANCE, OTHER. Do not add any other descriptions. 2. What kind of car damage is it? (PARKING_DAMAGE, THIEF, BROKEN GLASS, OTHER) 3. What is the car make? 4. What is the car model? 5. What is the license number? 6. What is the estimated damage cost (<1000 CHF for minor damage, > 5000 CHF for medium, 10000 CHF for severe. - Replace with MINOR, MEDIUM or SEVERE) 7. Explain the situation in the picture Use planner to find out which plugin to run Run InsurancePolicyPlugin.GetPolicies plugin Generate and ground user feedback Open insurance application and upload photo of damage Azure OpenAI GPT-4 Turbo w/ Vision Analyse image using prompt engineering Look up insurance coverage Your car is covered, do you want to submit a claim? Submit a claim in the insurance management tool I have a car damage Wep app running Semantic Kernel Call API

Slide 14

Slide 14 text

Demo Prompt orchestration using Semantic Kernel

Slide 15

Slide 15 text

▪ Think different ▪ Service availability ▪ Responsible and trusted AI ▪ Transparency of training data ▪ Data sources ▪ Biased: Race, gender, age, ethnicity, etc. ▪ Regulation ▪ Security, Networking, Monitoring, Logging ▪ Python basics are beneficial ▪ Be open and eager to learn. Have fun! Key takeaways

Slide 16

Slide 16 text

Questions?