SDD 2024 - Large Language Models, data & APIs: integrating generative AI power – with Python & .NET

Human language as a universal interface for software solutions – sounds exciting! In this session, Christian offers an introduction to the integration of generative AI and Large Language Models (LLMs) into your applications. Using Python and .NET APIs, he shows how to exploit the potential of LLMs for various use cases.
At the core of the integration are the attendee’s data, API, and document. We’ll cover architecture patterns like In-Context Learning, Retrieval-Augmented Generation (RAG), Reasoning & Acting (ReAct), and Agents. These techniques are crucial for the development of modern, AI-driven business applications.
For more complex requirements, you’ll see how to employ LangChain for Python and Semantic Kernel for .NET. These frameworks open up enhanced opportunities for text understanding and generation, especially concerning prolonged conversations.
Christian will also provide an overview of Cloud-based solutions, including Azure Open AI Service and Fermyon Serverless AI, focusing on the option to provide and utilize own LLMs based on GPT-4 or Llama2.
Come and join to experience a pragmatic approach to integrating generative AI into business applications!

Transcript

1. Large Language Models, data & APIs: Integrating Generative AI power – with Python & .NET Christian Weyer Co-Founder & CTO @christianweyer
2. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Christian Weyer Co-Founder & CTO @ Thinktecture AG 2
3. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Our journey today 3 AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems
4. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET AI all-the-things? 4
5. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET A.I. 5
6. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers
7. § LLMs generate text based on input § LLMs can understand text – this changes a lot § Without having to train them on domains or use cases § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Large Language Models (LLMs) 7 Text… – really, just text?
8. Talk to company knowledge Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 8
9. § LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Large Language Models demystified 9
10. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET LLMs in your Solutions 10
11. § LLMs are always part of end-to-end architectures § HTTP/Web/REST APIs § Databases § Client apps (Web, desktop, mobile) § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything! Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET End-to-end architectures with LLMs 11
12. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Using LLMs: It’s just HTTP APIs Inference, FTW. 12
13. GPT-4 API access OpenAI Playground Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 13
14. § Barebones SDKs § Abstraction over HTTP APIs § E.g. Open AI SDK, Mistral SDK § Available for any programming language § Also available from other providers Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Building LLM-based end-to-end applications 14
15. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET The best tool for any developer to talk to LLMs! 15 🙈
16. § OSS framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources § Alternatives like LlamaIndex, Haystack, etc. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET LangChain - building LLM-based applications 16
17. § Microsoft’s open-source framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Chain is collection of Plugins § Planners are similar to Agents in LangChain § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Semantic Kernel 17
18. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Talk to your Data 18
19. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 19 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB
20. RAG: Chat with web site documents Semantic Kernel, GPT-3.5-Turbo Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 20
21. RAG: Learning about company’s policies via Slack LangChain, Weaviate – Mixtral-8x7B on Groq Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 21
22. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Talk to your Systems 22
23. § LLM can be instructed to, e.g. § Do sentiment analysis § Extract information from text § Extract structured information § Schema description: JSON, TypeScript types, etc. § Implement via tools like Kor, TypeChat, or Open AI Function / Tool Calling Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Extracting meaning in text for structured data 23
24. Extracting structured data from text: Flight search OpenAI Function Calling, LangChain, FastAPI, JS - GPT-4-Turbo Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 24
25. Extracting structured data from voice: Form filling JSON extraction, OpenAI JS SDK, Angular Forms - Mixtral-8x7B on Groq Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 25
26. § Integrate LLM-external systems to aid LLMs § Tool / function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Extending LLM capabilities 26 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'
27. Extending LLM capabilities OpenAI Tool Calling with Semantic Kernel Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 27
28. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Democratizing Generative AI 28
29. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 29 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source
30. § Open-source community drives innovation in Generative AI § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Open-source LLMs thrive 30
31. Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain, StreamLit Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 31
32. Offline in browser: Running open-source LLMs with WebAssembly & WebGPU web-llm Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 32
33. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET End-to-End 33
34. 34 Angular PWA OpenAI Speech-to-Text TT Panorama Gateway OpenAI GPT-4 OpenAI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio TT Panorama Query Panorama API Availability Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET
35. Ask for experts availability in my company systems Angular, Speech-to-text, internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-Turbo Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 35
36. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Exciting Times… 36
37. § LLMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SLMs: specialized, fine-tuned for domains § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Current state 37
38. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET 2024: The year of SLMs & CPU inference 38
39. Thank you! Christian Weyer https://thinktecture.com/christian-weyer 39