LLMs, Data & APIs: Integrating AI power into your applications (with Semantic Kernel & LangChain) Christian Weyer @christianweyer CTO, Technology Catalyst

§ What & why? § LLM integration aspects § Some use cases § Agents & LLMs § LLM landscape § Current state & Outlook LLMs, Data & APIs Integrating AI power into your applications Talking points 2

§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com LLMs, Data & APIs Integrating AI power into your applications Christian Weyer Co-Founder & CTO @ Thinktecture AG 3

LLMs, Data & APIs Integrating AI power into your applications Intro 4

Speech OpenAI Whisper tortoise-tts … LLMs, Data & APIs Integrating AI power into your applications Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 5

Speech OpenAI Whisper tortoise-tts … LLMs, Data & APIs Integrating AI power into your applications Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 6

§ Large Language Models (LLMs) generate text based on input § Plus, LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language as a first-class citizen in software architecture 🤯 * LLMs are not perfect – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with LLMs, Data & APIs Integrating AI power into your applications Text… – really, just text? 7 Yeah: Large Language Models

“Hm… what … how… erm… who could know this…? 🤔” Learning about my company’s policies in Slack LLMs, Data & APIs Integrating AI power into your applications DEMO 8 Demos are using OpenAI GPT-3.5-turbo or GPT-4

§ LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through LLMs, Data & APIs Integrating AI power into your applications Large Language Models demystified 9

LLMs, Data & APIs Integrating AI power into your applications Integration 10

LLMs, Data & APIs Integrating AI power into your applications Using LLMs: It’s just APIs ! 11

LLMs, Data & APIs Integrating AI power into your applications The best tool for .NET developers to talk to LLMs! 12 🙈

§ Most popular open-source framework for developing applications powered by LLMs § Several releases per week § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of actions in code § Abstractions for § Loading text from a wide range of sources § Vector stores § Prompts & LLMs (local and remote) § Tools and Agents LLMs, Data & APIs Integrating AI power into your applications LangChain 13

§ Microsoft’s open-source framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities into a single unit of functionality § Semantic functions for prompting § Native functions to run local code § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading text § .NET version has no support to run a model locally (yet?) § Can only call (Azure) OpenAI API and Hugging Face Inference API LLMs, Data & APIs Integrating AI power into your applications Semantic Kernel 14

LLMs, Data & APIs Integrating AI power into your applications Use cases 15

Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 16

Understanding text § LLMs understand a given text § LLM can be instructed to e.g., extract information out of the text § Tools like Kor or Open AI Functions can be used to extract information in a structured way § JSON, Python classes, etc. LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 17

Extracting structured data (LangChain) LLMs, Data & APIs Integrating AI power into your applications DEMO 18

Answering Questions / Chat with documents § Mixed usage of text embedding and a large language model § Retrieval-augmented generation (RAG) § Index one or more documents § Use text embedding to index documents § Store embeddings in a vector database § User asks question § Create embedding for the question § Search vector database to find relevant text § Use LLM with relevant input text & question to create an answer LLMs, Data & APIs Integrating AI power into your applications Typical LLM use cases 19

LLMs, Data & APIs Integrating AI power into your applications Answering Questions / Chat with documents - Architecture Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 20 Vector DB Embedding model Embedding model 💡 Indexing QA

Chat with web site documents (Semantic Kernel) LLMs, Data & APIs Integrating AI power into your applications DEMO 21

LLMs, Data & APIs Integrating AI power into your applications Agents 22

§ Involve an LLM making decisions (via ReAct – Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools or functions to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) LLMs, Data & APIs Integrating AI power into your applications Agents: Multi-steps workflows with LLM interaction 23

LLMs, Data & APIs Integrating AI power into your applications ReAct – with function calling 24 LLM My code Query Some API Some database Prompt Function Definitions Final answer Answer ❓ ❓ ❗ 💡

Stock market agent with OpenAI function calling (LangChain) LLMs, Data & APIs Integrating AI power into your applications DEMO 25

LLMs, Data & APIs Integrating AI power into your applications Landscape 26

§ Open AI § Azure Open AI Service § Google Model Garden on Vertex AI § Amazon Sagemaker § Hugging Face § Self-hosting (does not really scale) § Device / browser LLMs, Data & APIs Integrating AI power into your applications LLM providers 27

LLMs, Data & APIs Integrating AI power into your applications Wrap-Up 28

§ Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § Bias of the models § Which data was the model trained on? § Finding the right prompt can be tedious § Prompting differs between LLMs § Prompt injection as a threat § Will improve over time § Context size is currently limited § Will likely diminish over time § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data LLMs, Data & APIs Integrating AI power into your applications Current state 29

Huge potential for LLM-driven human-machine workflows via a UI (universal interface) LLMs, Data & APIs Integrating AI power into your applications Outlook 30

Thank you! Christian Weyer https://thinktecture.com/christian-weyer 31

§ Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ § LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 LLMs, Data & APIs Integrating AI power into your applications Links 32