Large Language Models, Daten & APIs: Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co.

Jajaja, der ChatGPT-Hype... Jenseits dessen taucht Christian in die Welt der Large Language Models (LLMs), Daten und APIs ein und konzentriert sich darauf, wie Sie AI-Funktionalität sinnvoll in Ihre Anwendungen integrieren können. Wir werden pragmatische Anwendungsfälle untersuchen, die das Potenzial von LLMs (wie GPT) demonstrieren - und erörtern, wie AI-Techniken in bestehende Architekturen einbezogen werden können. Die Teilnehmer erhalten Einblicke in LangChain als Python-Toolkit und die Möglichkeiten der Integration von LLMs. Zudem werden wir uns damit befassen, nicht nur Cloud-basierte und Closed-Source-Systeme (wie OpenAI) zu nutzen, sondern auch On-Premises- und Open-Source-Optionen in Betracht zu ziehen, um unterschiedlichen organisatorischen Anforderungen gerecht zu werden. Come in and find out!

Transcript

1. Large Language Models, Daten & APIs: Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Christian Weyer @christianweyer CTO, Technology Catalyst
2. § What & why? § LLM integration aspects § Selected use cases § Agents & LLMs § LLM landscape § Current state & Outlook Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Talking points 2
3. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Christian Weyer Co-Founder & CTO @ Thinktecture AG 3
4. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Intro 4
5. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. What’s in a name? 5
6. § LLMs generate text based on input § LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 * LLMs are not “perfect” – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Text… – really, just text? 6 Yeah: Large Language Models (LLMs)
7. § LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Large Language Models demystified 7
8. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Integration 8
9. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Using LLMs: It’s just APIs! Inference, FTW. 9
10. GPT-4 API access via OpenAI Playground Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 10 Demos are using OpenAI GPT-3.5-turbo or GPT-4(-Turbo)
11. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. The best tool for Java developers to talk to LLMs! 11 🙈
12. § Open-source framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. LangChain 12
13. § No official Java version of LangChain available § LangChain4J is community-driven § Works with Java 8 or higher § Supports Spring Boot 2 and 3 Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. LangChain4J 13
14. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Use cases 14
15. Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Typical LLM use cases 15
16. Understanding text § LLM can be instructed to e.g. § do sentiment analysis § extract information from text § Tools (like Kor, Open AI Functions – or TypeChat) for extracting information in a structured way reliably § JSON, Python classes, TypeScript types etc. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Typical LLM use cases 16
17. Extracting structured data (LangChain + Kor) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 17
18. Answering Questions / Chat with documents § Mixed usage of text embedding and a large language model § Retrieval-augmented generation (RAG) § Index one or more documents § Use text embedding to index documents § Store embeddings in a vector database § User asks question § Create embedding for the question § Search vector database to find relevant text § Use LLM with relevant input text & question to create an answer Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Typical LLM use cases 18
19. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Answering Questions / Chat with documents - RAG Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 19 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA
20. Chat with my company’s policies (LangChain) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 20
21. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Agents 21
22. § Involve an LLM making decisions (via ReAct – Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Agents: Multi-steps workflows with LLM interaction 22
23. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. ReAct – with function calling 23 LLM My code Query Some API Some database Prompt Function definitions Final answer Answer ❓ ❓ ❗ 💡
24. Stock market agent with OpenAI function calling (Python OpenAI SDK) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 24
25. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Landscape 25
26. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. LLMs everywhere 26 OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Other providers Antrophic Cohere HuggingFace … Open-source Edge IoT Server Desktop Mobile Web Open-source
27. § Open-source community drives innovation in Generative AI § HuggingFace is central place for it § Literally, every week a new and “better” LLM shows up 🤓 § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Open-source LLMs thrive 27
28. Mistral OSS LLM locally on Apple M1 Max (llama.cpp & LM Studio) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 28
29. Running OSS LLM offline in the browser - with Wasm & WebGPU (web-llm) Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. DEMO 29
30. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Wrap-Up 30
31. § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Current state 31
32. Huge potential for LLM-driven human-machine workflows via universal interface agents Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Outlook 32
33. Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Outlook 33
34. § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai- services/openai/reference § Hugging Face Inference Endpoints § https://huggingface.co/docs/inference- endpoints/api_reference § LangChain § https://www.langchain.com/ § LangChain4J § https://github.com/langchain4j/langchain4j § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Llama.cpp § https://github.com/ggerganov/llama.cpp § Web-llm § https://webllm.mlc.ai/ § The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) § https://arxiv.org/pdf/2309.17421.pdf § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for- large-language-model- applications/assets/PDF/OWASP-Top-10-for- LLMs-2023-slides-v1_0_1.pdf Large Language Models, Daten & APIs Integration von Generative-AI-Power in Ihre Anwendungen mit LangChain & Co. Links 34