Techorama NL 2025: Semantic AI: A Language & an Embedding Model walk into a bar...

Generative AI has reshaped how we build intelligent systems, but true AI reliability goes beyond text generation. Semantic AI shifts the focus from mere language generation to in-depth understanding and structured decision-making, where language models and embedding models work together to power robust applications.
SLMs & LLMs process and generate natural language, enabling structured outputs for APIs and interactive applications. Embedding models, on the other hand, capture semantic relationships within data, enhancing query filtering, task routing, and retrieval accuracy. Together, they create AI systems that are more context-aware, scalable, and precise.
Christian explores how structured outputs from LMs and embedding-driven semantics can work together to improve an AI system's performance. Through a real-world example, he demonstrates how query validation, task routing, and API integration create robust AI-powered interactions.

Transcript

1. Semantic AI: A Language & an Embedding Model walk into a bar... Christian Weyer | Co-Founder & CTO | Thinktecture AG | christian.weyer@thinktecture.com
2. Semantic AI A Language & an Embedding Model Walk Into a Bar... Our journey Models for our software Lightweight RAG Semantic Routing Observability Structured Output / Tool Calling 2
3. Semantic AI A Language & an Embedding Model Walk Into a Bar... MODELS FOR OUR SOFTWARE 3
4. Language Models understand and generate semantically rich human language, transforming it into text or structured data for both humans and machines. ⚠ Non-deterministic: same input can lead to different outputs. Embedding Models capture semantic meaning by encoding human language into numerical vector representations, facilitating understanding, comparison, and retrieval for both humans and machines. ✅ Deterministic: same input always results in the same embedding. Semantic AI A Language & an Embedding Model Walk Into a Bar... 🫱 🫲 Semantic AI Generative AI 4
5. § Language & embedding models part of end-to-end architectures § Embedding models can be run locally § Optimized for CPU § Language models still hard to run locally § High GPU power § High VRAM § High memory bandwidth Semantic AI A Language & an Embedding Model Walk Into a Bar... API-based AI model integrations 5
6. Semantic AI A Language & an Embedding Model Walk Into a Bar... Classical applications & UIs API-based data Document-based data 6
7. Semantic AI A Language & an Embedding Model Walk Into a Bar... Language-enabled “UIs” – Talk-to-TT 7
8. Semantic AI A Language & an Embedding Model Walk Into a Bar... C4 system context diagram § Various tech stacks § Docker-based distributed system 8
9. Semantic AI A Language & an Embedding Model Walk Into a Bar... PATTERN LIGHTWEIGHT RAG
10. Semantic AI A Language & an Embedding Model Walk Into a Bar... Talking to documents (Retrieval-augmented generation) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Results Question Answ er w / sources LLM Embedding Model Embedding Model 💡 Indexing / Embedding Question Answering .md, .docx, .pdf etc. “Lorem ipsum…?” 💡 Vector DB 10
11. § Frameworks § LangChain § FastEmbed § Lightweight & efficient for generating text embeddings § Embedding model § jinaai/jina-embeddings-v2-base-de (local) – 768 dims § Vector store § PostgreSql (pgvector) vector store § LLM/SLM § Llama 3.3 70B on Cerebras (very fast) Semantic AI A Language & an Embedding Model Walk Into a Bar... Technical implementation – Lightweight RAG 11
12. Semantic AI A Language & an Embedding Model Walk Into a Bar... PATTERN STRUCTURED OUTPUT
13. § Tools integration (and more) is being standardized with MCP Semantic AI A Language & an Embedding Model Walk Into a Bar... Talking to APIs (Function / Tool calling) “When is CW available for a two-days workshop?” System Prompt (+ employee data) + Schema (for structured output) Web API Availability business logic 13
14. § Frameworks § Pydantic § Instructor § Methodology § Schema with JSON Mode (not Function Calling) § SLM/LLM § Llama 3.3 70B on Cerebras (very fast) Semantic AI A Language & an Embedding Model Walk Into a Bar... Technical implementation – Structured Output 14
15. Semantic AI A Language & an Embedding Model Walk Into a Bar... PATTERN SEMANTIC ROUTING
16. Semantic AI A Language & an Embedding Model Walk Into a Bar... Semantics-based decisions for user interactions Guarding (e.g. prompt injection) Routing (selecting correct target) “Lorem ipsum…?” Target RAG Target API Call Target … something else … Fine-tuned NLP Model Embedding Model 16
17. Guarding § Frameworks § llm-guard § HuggingFace Transformers § NLP model § deepset/ deberta-v3-base-injection (local) Routing § Frameworks § semantic-routing § FastEmbed § Embedding model § intfloat/ multilingual-e5-large (local) – 1024 dims § Vector store § PostgreSql (pgvector) Semantic AI A Language & an Embedding Model Walk Into a Bar... Technical implementation – Semantic Guarding & Routing 17
18. Semantic AI A Language & an Embedding Model Walk Into a Bar... PATTERN / SOLUTION OBSERVABILITY
19. Semantic AI A Language & an Embedding Model Walk Into a Bar... Things can get… overwhelming 19
20. § Methodology § Open Telemetry (OTel) § Frameworks § OTel Python packages § LogFire SDK § Tools § LogFire, LangFuse § Any OTel-enabled system Semantic AI A Language & an Embedding Model Walk Into a Bar... Technical implementation - Observability 20
21. Semantic AI A Language & an Embedding Model Walk Into a Bar... Typical Semantic AI patterns & solutions – in end-to-end software engineering Lightweight RAG Structured Output Semantic Guarding & Routing Insightful Observability 21
22. Semantic AI A Language & an Embedding Model Walk Into a Bar... AI-based solutions are ≅10% AI and 100% software engineering.
23. Thank you! Christian Weyer christian.weyer@thinktecture.com https://thinktecture.com/christian-weyer