Slide 1

Slide 1 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Christian Weyer @christianweyer CTO, Technology Catalyst

Slide 2

Slide 2 text

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Christian Weyer Co-Founder & CTO @ Thinktecture AG 2

Slide 3

Slide 3 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Our journey 3 AI all-the- things? Integrating LLMs Selected Scenarios End-to-End Exciting Times… Democratizing Generative AI

Slide 4

Slide 4 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET AI all-the-things? 4

Slide 5

Slide 5 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET AI all-the-things? 5 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers

Slide 6

Slide 6 text

§ LLMs generate text based on input § LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Large Language Models (LLMs) 6 Text… – really, just text?

Slide 7

Slide 7 text

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Large Language Models demystified 7

Slide 8

Slide 8 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Integrating LLMs 8

Slide 9

Slide 9 text

§ LLMs are always part of end-to-end architectures § HTTP/Web/REST APIs § Databases § Client apps (Web, desktop, mobile) § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything! Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET End-to-end architectures with LLMs 9

Slide 10

Slide 10 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Using LLMs: It’s just APIs ! Inference, FTW. 10

Slide 11

Slide 11 text

GPT-4 API access via OpenAI Playground Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 11

Slide 12

Slide 12 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET The best tool for .NET developers to talk to LLMs! 12 🙈

Slide 13

Slide 13 text

§ OSS framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET LangChain - building LLM-based applications 13

Slide 14

Slide 14 text

§ Microsoft’s OSS framework to integrate LLMs into applications § .NET, Python, and Java versions § .NET 1.0 RTM in December 2023 § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Planners are orchestrating LLMS interactions § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Semantic Kernel - building LLM-based applications 14

Slide 15

Slide 15 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Selected Scenarios 15

Slide 16

Slide 16 text

Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Typical LLM scenarios: 16

Slide 17

Slide 17 text

Extracting meaning in text § LLM can be instructed to, e.g. § Do sentiment analysis § Extract information from text § Extracting structured information § JSON, TypeScript types, etc. § Via tools like Kor, TypeChat, or Open AI Functions Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Typical LLM scenarios: 17

Slide 18

Slide 18 text

Extracting structured data (LangChain + Kor) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 18

Slide 19

Slide 19 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Answering Questions on Data - Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 19 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA

Slide 20

Slide 20 text

Learning about my company’s policies via Slack (LangChain) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 20

Slide 21

Slide 21 text

Chat with web site documents (Semantic Kernel) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 21

Slide 22

Slide 22 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Democratizing Generative AI 22

Slide 23

Slide 23 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Other providers Antrophic Cohere HuggingFace … Open-source Edge IoT Server Desktop Mobile Web Open-source 23

Slide 24

Slide 24 text

§ Open-source community drives innovation in Generative AI § HuggingFace is central place for it § Literally, every week a new and “better” LLM shows up 🤓 § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Open-source LLMs thrive 24

Slide 25

Slide 25 text

Local RAG with Mistral OSS LLM (llama.cpp & LM Studio) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 25

Slide 26

Slide 26 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET End-to-End 26

Slide 27

Slide 27 text

Ask for expert availability in my company systems (Speech-to-text, my own API, GPT-4, Text-to-speech) Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET DEMO 27

Slide 28

Slide 28 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Exciting Times 28

Slide 29

Slide 29 text

§ LLMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Current state 29

Slide 30

Slide 30 text

Potential for LLM-powered human-machine workflows via universal interface agents Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Outlook 30

Slide 31

Slide 31 text

Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Outlook 31

Slide 32

Slide 32 text

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 32 Selected demos: Extract structured information: https://github.com/thinktecture-labs/llm-extract-structured-information-langchain-kor Local RAG with PDFs: https://github.com/thinktecture-labs/rag-chat-with-pdf-local-llm

Slide 33

Slide 33 text

§ Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ § LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf Large Language Models, Data & APIs Integrating Generative AI Power into your solutions - with Python & .NET Links 33