MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Marco Frodl [email protected] Principal Consultant for Generative AI @marcofrodl 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions About Me Marco Frodl Principal Consultant for Generative AI Thinktecture AG X: @marcofrodl E-Mail: [email protected] LinkedIn: https://www.linkedin.com/in/marcofrodl/ https://www.thinktecture.com/thinktects/marco-frodl/ 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🚀 https://www.aurelio.ai/semantic-router Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow, unreliable LLM generations to make tool-use or safety decisions, we use the magic of semantic vector space — routing our requests using semantic meaning. 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🚀 https://www.aurelio.ai/semantic-router Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow, unreliable LLM generations to make tool-use or safety decisions, we use the magic of semantic vector space — routing our requests using semantic meaning. It’s perfect for: input guarding, topic routing, tool-use decisions. 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🚀 in Numbers In my RAG example, Semantic Router using remote services is 3.4 times faster than an LLM and it is 30 times less expensive. Semantic Router with local model is 7.7 times faster and 60 times less expensive vs LLM. 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Really? More Safety? Better Speed? Less Budget? 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Refresher: What is RAG? “Retrieval-Augmented Generation (RAG) extends the capabilities of LLMs to an organization's internal knowledge, all without the need to retrain the model. 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Refresher: What is RAG? https://aws.amazon.com/what-is/retrieval-augmented-generation/ “Retrieval-Augmented Generation (RAG) extends the capabilities of LLMs to an organization's internal knowledge, all without the need to retrain the model. It references an authoritative knowledge base outside of its training data sources before generating a response” 

Ask me anything MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Simple RAG Question Prepare Search Search Results Question Answer LLM Vector DB Embedding Model Question as Vector Workflow Terms - Retriever - Chain Elements Embedding- Model Vector- DB Python LLM LangChain 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Demo: Simple RAG 

Our sample content MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Simple RAG in a nutshell 

Which retriever do you want? MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Multiple Retriever 

Best source determination before the search MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Advanced RAG Question Retriever Selection 0-N Search Results Question Answer LLM Embedding Model Vector DB A Question as Vector Vector DB B LLM Prepare Search or 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Demo: Dynamic Retriever Selection with LLM 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Embedding Model 

Best source determination before the search MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Advanced RAG Question Retriever Selection 0-N Search Results Question Answer LLM Embedding Model Vector DB A Question as Vector Vector DB B LLM Prepare Search or 

Best source determination before the search MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Advanced RAG w/ Semantic Router Question Retriever Selection 0-N Search Results Question Answer Embedding Model Vector DB A Question as Vector Vector DB B LLM Prepare Search or Embedding Model 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Demo: Semantic Router with RAG 

LLM as Router MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🐌 

Semantic Router with remote embedding model MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🚀 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Demo: Semantic Router running locally 

Semantic Router with local embedding model MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Turbo 🚀 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Speed & Budget in Numbers SR Remote is 3.4 times faster than LLM (0,62s vs 0,18s) SR Local is 7.75 times faster than LLM (0,62s vs 0,08s) SR Remote is 30 times cheaper than LLM ($0,60 vs $0,02) SR Local is 60 times cheaper than LLM ($0,60 vs $0,01) 

Tool-Call needed? Tool Calling Workflow with LLM MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Tool-use Decisions Answer Execute Tool-Call Prepare Tool-Call More Tools? Regular LLM-Call Question LLM LLM LLM 

Tool-Call needed? Tool Calling Workflow with LLM + Semantic Router MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Tool-use Decisions Answer Execute Tool-Call Prepare Tool-Call More Tools? Regular LLM-Call Embedding Model LLM LLM Question 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Demo: Semantic Router tool-use decisions 

MLCon 2025 Beyond LLMs: Using Embedding Models for Input Guarding, Semantic Routing, and Tool Decisions Yes, please! More Safety! Better Speed! Less Budget! 

Thank you! Any questions? Marco Frodl @marcofrodl Principal Consultant for Generative AI