LLMs, Data & APIs: Integrating AI power into your applications (with Semantic Kernel & LangChain)

Let's look beyond the ChatGPT hype! In this session, Christian delves into the world of Large Language Models (LLMs), data, and APIs - focusing on how to integrate AI capabilities seamlessly into your .NET-based applications. We will explore practical use cases that demonstrate the potential of LLMs (such as GPT) and discuss how to incorporate AI techniques into existing applications and architectures. Attendees will gain insights into Microsoft's Semantic Kernel tooling for .NET developers, as well as a glimpse into alternative solutions like LangChain. Lastly, we will address the importance of not only leveraging cloud-based and closed-source systems (like OpenAI) but also considering on-premises and open-source options to accommodate diverse organizational needs. Come in and find out!

Transcript

1. LLMs, Data & APIs: Integrating AI power into your applications (with Semantic Kernel & LangChain) Christian Weyer @christianweyer CTO, Technology Catalyst
2. § What & why? § LLM integration aspects § Some use cases § Agents & LLMs § LLM landscape § Current state & Outlook LLMs, Data & APIs Integrating AI power into your applications Talking points 2
3. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com LLMs, Data & APIs Integrating AI power into your applications Christian Weyer Co-Founder & CTO @ Thinktecture AG 3
4. LLMs, Data & APIs Integrating AI power into your applications Intro 4
5. Speech OpenAI Whisper tortoise-tts … LLMs, Data & APIs Integrating AI power into your applications Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 5
6. Speech OpenAI Whisper tortoise-tts … LLMs, Data & APIs Integrating AI power into your applications Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 6
7. § Large Language Models (LLMs) generate text based on input § Plus, LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language as a first-class citizen in software architecture 🤯 * LLMs are not perfect – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with LLMs, Data & APIs Integrating AI power into your applications Text… – really, just text? 7 Yeah: Large Language Models
8. “Hm… what … how… erm… who could know this…? 🤔” Learning about my company’s policies in Slack LLMs, Data & APIs Integrating AI power into your applications DEMO 8 Demos are using OpenAI GPT-3.5-turbo or GPT-4
9. § LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through LLMs, Data & APIs Integrating AI power into your applications Large Language Models demystified 9
10. LLMs, Data & APIs Integrating AI power into your applications Integration 10
11. LLMs, Data & APIs Integrating AI power into your applications Using LLMs: It’s just APIs ! 11
12. LLMs, Data & APIs Integrating AI power into your applications The best tool for .NET developers to talk to LLMs! 12 🙈
13. § Most popular open-source framework for developing applications powered by LLMs § Several releases per week § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of actions in code § Abstractions for § Loading text from a wide range of sources § Vector stores § Prompts & LLMs (local and remote) § Tools and Agents LLMs, Data & APIs Integrating AI power into your applications LangChain 13
14. § Microsoft’s open-source framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities into a single unit of functionality § Semantic functions for prompting § Native functions to run local code § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading text § .NET version has no support to run a model locally (yet?) § Can only call (Azure) OpenAI API and Hugging Face Inference API LLMs, Data & APIs Integrating AI power into your applications Semantic Kernel 14
15. LLMs, Data & APIs Integrating AI power into your applications Use cases 15
16. Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 16
17. Understanding text § LLMs understand a given text § LLM can be instructed to e.g., extract information out of the text § Tools like Kor or Open AI Functions can be used to extract information in a structured way § JSON, Python classes, etc. LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 17
18. Extracting structured data (LangChain) LLMs, Data & APIs Integrating AI power into your applications DEMO 18
19. Answering Questions / Chat with documents § Mixed usage of text embedding and a large language model § Retrieval-augmented generation (RAG) § Index one or more documents § Use text embedding to index documents § Store embeddings in a vector database § User asks question § Create embedding for the question § Search vector database to find relevant text § Use LLM with relevant input text & question to create an answer LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 19
20. LLMs, Data & APIs Integrating AI power into your applications Answering Questions / Chat with documents - Architecture Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 20 Vector DB Embedding model Embedding model 💡 Indexing QA
21. Chat with web site documents (Semantic Kernel) LLMs, Data & APIs Integrating AI power into your applications DEMO 21
22. LLMs, Data & APIs Integrating AI power into your applications Agents 22
23. § Involve an LLM making decisions (via ReAct – Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools or functions to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) LLMs, Data & APIs Integrating AI power into your applications Agents: Multi-steps workflows with LLM interaction 23
24. LLMs, Data & APIs Integrating AI power into your applications ReAct – with function calling 24 LLM My code Query Some API Some database Prompt Function Definitions Final answer Answer ❓ ❓ ❗ 💡
25. Stock market agent with OpenAI function calling (LangChain) LLMs, Data & APIs Integrating AI power into your applications DEMO 25
26. LLMs, Data & APIs Integrating AI power into your applications Landscape 26
27. § Open AI § Azure Open AI Service § Google Model Garden on Vertex AI § Amazon Sagemaker § Hugging Face § Self-hosting (does not really scale) § Device / browser LLMs, Data & APIs Integrating AI power into your applications LLM providers 27
28. LLMs, Data & APIs Integrating AI power into your applications Wrap-Up 28
29. § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § Bias of the models § Which data was the model trained on? § Finding the right prompt can be tedious § Prompting differs between LLMs § Prompt injection as a threat § Will improve over time § Context size is currently limited § Will likely diminish over time § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data LLMs, Data & APIs Integrating AI power into your applications Current state 29
30. Huge potential for LLM-driven human-machine workflows via a UI (universal interface) LLMs, Data & APIs Integrating AI power into your applications Outlook 30
31. Thank you! Christian Weyer https://thinktecture.com/christian-weyer 31
32. § Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ § LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 LLMs, Data & APIs Integrating AI power into your applications Links 32