Engineering AI Agents that Don’t Break

Event link: https://luma.com/ftrqjbaa

AI agents are having a moment, but most of them are little more than fragile prototypes that break under pressure. Together, we’ll explore why agentic systems fail in practice, and how to fix that with real engineering principles. In this talk, you’ll learn how to build agents that are modular, observable, and ready for production. If you’re tired of LLM demos that don’t deliver, this talk is your blueprint for building agents that work.

Transcript

1. Engineering AI Agents that Donʼt Break Bilge Yücel, Developer Relations Engineer @ deepset
2. Bilge Yucel Hello 👋 • Developer Relations Engineer at deepset • Istanbul → Berlin in/bilge-yucel @bilgeyucl
3. Our Context Company Solving Custom AI challenges since 2018. HQ in Berlin and NYC. Backed by: Leading open source framework & commercial platforms for custom enterprise-grade AI Products Used by 70 Thought leaders
4. Where are you on your agent journey?
5. What is an AI agent? An AI agent is an LLM-based system that autonomously pursues a goal by interacting with its environment using tools. Human LLM Call Environment Action Feedback Stop
6. Why Agents Fail? 🤔
7. “Agent involves a lot more work than expected and 90% of it is pure engineering. Nothing really to do with LLM, but it is how to blend LLM into an agentic workflow that makes senseˮ - a Haystack community member
8. Failure Reason 1 Too Much Trust, Too Little Design ❌ No fallback plans ❌ Brittle prompts ❌ Too many tools ❌ No observability ❌ No guardrails
9. Failure Reason 2 Wrong Use Case • You have complex, multi-step problems requiring diverse actions • Tasks involve multiple tools / sources • Goals are clear but the optimal path to achieve them isn't predetermined • When interactions can follow predictable patterns (e.g. Q&A • Tasks can be decomposed into clear steps • Stability, robustness and efficiency are prioritized over automation potential Agents for Reasoning, Pipelines for Defined Flow
10. AI Workflows & Agents Designing for the right level of agentic behavior is a strategic choice Operational Risk Autonomy AI Agent Deterministic AI Workflow Variables, outputs, com plexity • You can decide how much “powerˮ to give • Balance of Performance vs. Speed vs. Costs • 🔐 Modularity
11. • Open-source AI orchestration framework by deepset • Provides the tools that Python developers need to build real world, agentic AI systems with visibility and control • Building blocks: Components & Pipelines
12. Haystack Agents User Request Agent LLM (e.g. OpenAI, Anthropic, Gemini) System Prompt Python Functions External APIs Haystack Components MCP Servers Generated Answer
13. Github Issue Resolver Agent Build a GitHub Issue Resolver Agent
14. Practical Tips For Building Agents • Find the optimal level of autonomy. Have the control over your system. • Minimize tools. Each new tool adds failure risk. • Split agents. Donʼt make one agent do everything. Use planners, coders, fixers if needed. • Build ‘fatʼ tools. Encapsulate logic, retries, and fallback paths into the tool logic. • Build guardrails. Validate inputs, actions and outputs. • Log everything. Add tracing to every step, tool, and output. • Human-in-the-loop. Let people oversee and guide agents.
15. Unbreakable AI Agents with Haystack Build a Tool-Calling Agent AI Guardrails: Content Moderation and Safety with Open Language Models Creating a Multi-Agent System with Haystack Trace and Evaluate RAG with Arize Phoenix DevOps Support Agent with Human in the Loop Build a GitHub Issue Resolver Agent Build a GitHub PR Creator Agent Building AI Agents with Haystack
16. Thank you bilge.yucel@deepset.ai Contact us www.deepset.ai www.haystack.deepset.ai