Generative AI: Integration von Large Language Models in eigene Software - APIs, Daten & Code

Jajaja, der ChatGPT-Wahnsinn. Einfach Text in ein Chat-Fenster tippen und staunen ... Jo, nice. Aber was ist das wirklich Interessante daran für uns Softwarearchitekten und -entwickler? Christian zeichnet in dieser Night Session jenseits des aktuellen Hypes ein Bild, in dem Large Language Models (LLMs) und die menschliche Sprache ein wichtiger und zentraler Bestandteil zukünftiger Softwarelösungen sein kann. Die Mächtigkeit semantischer Schnittstellen via LLMs, gepaart mit klassischen Daten-Architekturen, APIs und Workflows, eröffnet interessante Use Cases für die Mensch-Maschine- respektive Mensch-Code-Interaktion. Die Session bringt die neuen Ideen von generativer AI & Large Language Models in Einklang mit existierenden Architekturansätzen, damit wir als Developer das mögliche Potenzial für künftige Lösungen und Produkte besser einschätzen können.

Transcript

1. Generative AI: Integration von Large Language Models in eigene Software - APIs, Daten & Code Christian Weyer @christianweyer CTO, Technology Catalyst
2. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com Integration von Large Language Models in eigene Software APIs, Daten & Code Christian Weyer Co-Founder & CTO @ Thinktecture AG 2
3. Integration von Large Language Models in eigene Software APIs, Daten & Code Our journey 3 (Gen) AI all- the-things? Integrating LLMs Selected Scenarios Democratizing Generative AI Exciting Times… Agents
4. Integration von Large Language Models in eigene Software APIs, Daten & Code AI all-the-things? 4
5. Integration von Large Language Models in eigene Software APIs, Daten & Code AI all-the-things? 5 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers
6. § LLMs generate text based on input § LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Integration von Large Language Models in eigene Software APIs, Daten & Code Large Language Models (LLMs) 6 Text… – really, just text?
7. § LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Integration von Large Language Models in eigene Software APIs, Daten & Code Large Language Models demystified 7
8. Integration von Large Language Models in eigene Software APIs, Daten & Code Integrating LLMs 8
9. § LLMs are always part of end-to-end architectures § HTTP/Web/REST APIs § Databases § Client apps (Web, desktop, mobile) § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything! Integration von Large Language Models in eigene Software APIs, Daten & Code End-to-end architectures with LLMs 9
10. Integration von Large Language Models in eigene Software APIs, Daten & Code Using LLMs: It’s just APIs ! Inference, FTW. 10
11. GPT-4 API access via OpenAI Playground Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 11
12. Integration von Large Language Models in eigene Software APIs, Daten & Code The best tool for developers to talk to LLMs! 12
13. § OSS framework for developing applications powered by LLMs § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources Integration von Large Language Models in eigene Software APIs, Daten & Code E.g.: LangChain - building LLM-based applications 13
14. Integration von Large Language Models in eigene Software APIs, Daten & Code Selected Scenarios 14
15. Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. Integration von Large Language Models in eigene Software APIs, Daten & Code Typical LLM scenarios: 15
16. Extracting meaning in text § LLM can be instructed to, e.g. § Do sentiment analysis § Extract information from text § Extracting structured information § JSON, Python classes, TypeScript types etc. § Via tools like JSON Schema or Open AI Functions Integration von Large Language Models in eigene Software APIs, Daten & Code Typical LLM scenarios: 16
17. Extracting structured data (LangChain + Kor) Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 17
18. Integration von Large Language Models in eigene Software APIs, Daten & Code Answering Questions on Data - Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 18 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA
19. Learning about my company’s policies via Slack (LangChain) Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 19
20. Integration von Large Language Models in eigene Software APIs, Daten & Code Towards Autonomy 20
21. § Involve an LLM in making decisions (via ReAct – Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) Integration von Large Language Models in eigene Software APIs, Daten & Code Agents: Multi-step workflows 21
22. Integration von Large Language Models in eigene Software APIs, Daten & Code Agents & ReAct – with OpenAI function calling LLM My code Query Some API Some database Prompt Function definitions Final answer Answer ❓ ❓ ❗ 💡 22 Function Function
23. Ask for experts availability in my company systems (Whisper, node.js OpenAI SDK – Function Calling, TTS) Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 23
24. Integration von Large Language Models in eigene Software APIs, Daten & Code 24 Angular PWA (&node.js Proxy) OpenAI Speech-to-Text TT “Panorama” (company-internal) OpenAI GPT-4 OpenAI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data Query Panorama API Generate response with experts availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio
25. Integration von Large Language Models in eigene Software APIs, Daten & Code Democratizing Generative AI 25
26. Integration von Large Language Models in eigene Software APIs, Daten & Code LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Other providers Antrophic Cohere HuggingFace … Open-source Edge IoT Server Desktop Mobile Web Open-source 26
27. § Open-source community drives innovation in Generative AI § HuggingFace is central place for it § Literally, every week a new and “better” LLM shows up 🤓 § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Integration von Large Language Models in eigene Software APIs, Daten & Code Open-source LLMs thrive 27
28. Running Mistral OSS LLM locally (llama.cpp & LM Studio) Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 28
29. Integration von Large Language Models in eigene Software APIs, Daten & Code Exciting Times 29
30. § LLMs enable new scenarios & use cases to incorporate human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Integration von Large Language Models in eigene Software APIs, Daten & Code Current state 30
31. Potential for LLM-powered human-machine workflows via universal interface agents Integration von Large Language Models in eigene Software APIs, Daten & Code Outlook 31
32. Integration von Large Language Models in eigene Software APIs, Daten & Code Outlook 32
33. Thank you! Christian Weyer https://thinktecture.com/christian-weyer 33
34. § Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ § LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf Integration von Large Language Models in eigene Software APIs, Daten & Code Links 34