Semantic AI: A Language Model & an Embedding Model walk into a bar... Christian Weyer | Co-Founder & CTO | Thinktecture AG | [email protected] 

Semantic AI A Language Model & an Embedding Model walk into a bar... Our journey Models for our software Lightweight RAG Semantic Routing Observability Structured Output / Tool Calling 2 

Semantic AI A Language Model & an Embedding Model walk into a bar... MODELS FOR OUR SOFTWARE 3 

Language Models understand and generate semantically rich human language, transforming it into text or structured data for both humans and machines. ⚠ Non-deterministic: same input can lead to different outputs. Embedding Models capture semantic meaning by encoding human language into numerical vector representations, facilitating understanding, comparison, and retrieval for both humans and machines. ✅ Deterministic: same input always results in the same embedding. Semantic AI A Language Model & an Embedding Model walk into a bar... 🫱 🫲 Semantic AI Generative AI 4 

§ Language & embedding models part of end-to-end architectures § Embedding models can be run locally § Optimized for CPU § Language models still hard to run locally § High GPU power § High VRAM § High memory bandwidth Semantic AI A Language Model & an Embedding Model walk into a bar... API-based AI model integrations 5 

Semantic AI A Language Model & an Embedding Model walk into a bar... Classical applications & UIs API-based data Document-based data 6 

Semantic AI A Language Model & an Embedding Model walk into a bar... Language-enabled “UIs” – Talk-to-TT 7 

Semantic AI A Language Model & an Embedding Model walk into a bar... C4 system context diagram § Various tech stacks § Docker-based distributed system 8 

Semantic AI A Language Model & an Embedding Model walk into a bar... PATTERN LIGHTWEIGHT RAG 9 

Semantic AI A Language Model & an Embedding Model walk into a bar... Talking to documents (Retrieval-augmented generation) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Results Question Answ er w / sources LLM Embedding Model Embedding Model 💡 Indexing / Embedding Question Answering .md, .docx, .pdf etc. “Lorem ipsum…?” 💡 Vector DB 10 

§ Frameworks § LangChain § FastEmbed § Lightweight & efficient for generating text embeddings § Embedding model § jinaai/jina-embeddings-v2-base-de (local) – 768 dims § Vector store § PostgreSql (pgvector) vector store § LLM/SLM § Llama 3.3 70B on Cerebras (very fast) Semantic AI A Language Model & an Embedding Model walk into a bar... Technical implementation – Lightweight RAG 11 

Semantic AI A Language Model & an Embedding Model walk into a bar... PATTERN STRUCTURED OUTPUT 12 

§ Tools integration is being standardized with MCP Semantic AI A Language Model & an Embedding Model walk into a bar... Talking to APIs (Function / Tool calling) 13 “When is CW available for a two-days workshop?” System Prompt (+ employee data) + Schema (for structured output) Web API Availability business logic 

§ Frameworks § Pydantic § Instructor § Methodology § Schema with JSON Mode (not Function Calling) § SLM/LLM § Llama 3.3 70B on Cerebras (very fast) Semantic AI A Language Model & an Embedding Model walk into a bar... Technical implementation – Structured Output 14 

Semantic AI A Language Model & an Embedding Model walk into a bar... PATTERN SEMANTIC ROUTING 15 

Semantic AI A Language Model & an Embedding Model walk into a bar... Semantics-based decisions for user interactions Guarding (e.g. prompt injection) Routing (selecting correct target) “Lorem ipsum…?” Target RAG Target API Call Target … something else … Fine-tuned NLP Model Embedding Model 16 

Guarding § Frameworks § llm-guard § HuggingFace Transformers § NLP model § deepset/ deberta-v3-base-injection (local) Routing § Frameworks § semantic-routing § FastEmbed § Embedding model § intfloat/ multilingual-e5-large (local) – 1024 dims § Vector store § PostgreSql (pgvector) Semantic AI A Language Model & an Embedding Model walk into a bar... Technical implementation – Semantic Guarding & Routing 17 

Semantic AI A Language Model & an Embedding Model walk into a bar... PATTERN / SOLUTION OBSERVABILITY 18 

Semantic AI A Language Model & an Embedding Model walk into a bar... Things can get… overwhelming 19 

§ Methodology § Open Telemetry (OTel) § Frameworks § OTel Python packages § LogFire SDK § Tools § LogFire, LangFuse § Any OTel-enabled system Semantic AI A Language Model & an Embedding Model walk into a bar... Technical implementation - Observability 20 

Semantic AI A Language Model & an Embedding Model walk into a bar... Typical Semantic AI patterns & solutions – in end-to-end software engineering Lightweight RAG Structured Output Semantic Guarding & Routing Insightful Observability 21 

Semantic AI A Language Model & an Embedding Model walk into a bar... 22 AI solutions are ≅10% AI and 100% software engineering. 22 

Thank you! Christian Weyer [email protected] https://thinktecture.com/christian-weyer 23