Go Beyond Basic RAG with Agentic Behavior

RAG revolutionized AI by merging search and generation, and agentic behavior takes this search to the next level by enabling LLMs to make decisions and call tools. This talk covers agentic behavior’s key features: tool integration and reasoning, along with a live demo.

Demo colab notebook: https://colab.research.google.com/drive/1c4Q1oBpmLWINXFavaWphsGEnu6zH2Zho?usp=sharing

Transcript

1. Go Beyond Basic RAG with Agentic Behavior
2. Bilge Yücel Hello 👋 • Developer Relations Engineer at deepset • Computer Science → Artificial Intelligence • Berlin, Germany • Bbuzz ‘24, Bbuzz ‘25, Bbuzz ‘26… bilgeyucel in/bilge-yucel @bilgeyucl
3. Agenda 01 Retrieval Augmented Generation 02 Agentic Behavior 03 Agentic RAG Implementation 04 Demo
4. Retrieval Augmented Generation
5. Retrieval Augmented Generation RAG
6. Retrieval Augmented Generation RAG • Use LLMs generative capabilities, not their knowledge • LLM is “augmentedˮ with a retrieval step • Ground the generative modelʼs output in real-world data, so answers stay factual and relevant Use Cases: Customer Support FAQs, Enterprise Knowledge Bases Search Engines & QA
7. Benefits of RAG • Private knowledge • Up-to-date information • Reduced hallucination • Cost-effective • Transparency
8. Go Beyond Standard RAG • Complex user queries often need multiple retrievals or deeper reasoning ◦ “Compare product A and product B using customer feedback.ˮ • Sometimes, initial retrieval doesnʼt provide enough context ◦ Go to alternative resources • Need for dynamic and iterative search strategies to enhance accuracy ◦ Query writing, other retrieval strategies (hybrid, HyDE…) • and more…
9. Agentic Behavior
10. What is an AI agent? An AI agent is a system that autonomously pursues a goal by interacting with its environment and using tools. Human LLM Call Environment Action Feedback Stop
11. What is an AI agent? Overview of a LLM-powered autonomous agent system Source: https://lilianweng.github.io/posts/20230623-agent/
12. What is Agentic Behavior? giving the model the ability to choose what to do next
13. What is Agentic Behavior? • AI workflow with agentic components ◦ Decision making LLM ◦ A reasoning step + self-reflection ◦ Tools ◦ Memory • Planning + Action • Prompting strategies: CoT → ReAct • Tool-calling giving the model the ability to choose what to do next
14. Chain of Thought Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - https://arxiv.org/pdf/2201.11903
15. Chain of Thought Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - https://arxiv.org/pdf/2201.11903
16. REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS - https://arxiv.org/pdf/2210.03629 Reason + Act = ReAct
17. ReAct Prompt
18. Tool Calling Tool calling refers to the ability of a language model to dynamically invoke external functions and APIs as part of its reasoning process. LLM Answer Generation Tool Calling: Hybrid Retrieval Tool Calling: Weather API LLM Response Tool Calling: Web Search Tool Calling: Flight Radar API
19. Tool/Function Calling • Passing prompt + functions/tools (name, description, parameters) • User query → function name + JSON object for arguments • Use arguments to call the function
20. Tool/Function Calling Source: https://platform.openai.com/docs/guides/function-calling?api-mode=chat
21. Tool/Function Calling Source: https://platform.openai.com/docs/guides/function-calling?api-mode=chat
22. Agentic RAG Implementation
23. Agentic RAG • Deterministic → non-deterministic • LLM as the brain and decides on the next action • Access to tools and can go to alternative resources: Web, another database • Update the retrieval: query rewriting, query decomposition, multi-step retrieval
24. • Open-source AI orchestration framework by deepset • deepset AI Platform is built on Haystack • Provides the tools that Python developers need to build real world, agentic AI systems • Building blocks: Components & Pipelines
25. Haystack Ecosystem 160 Components 70 Integrations 21k+ Stars 70 Tutorials & Recipes
26. Components
27. Pipelines → Assemble components into workflows
28. Agentic RAG
29. Agent Component User Request Agent LLM (e.g. OpenAI, Anthropic, Gemini) System Prompt Tool ComponentTool MCPTool ToolSet Generated Answer
30. Agent Observability
31. Demo
32. Scan the QR Code to receive slidedeck and demo notebook shared during the presentation! Thank You! bilgeyucel in/bilge-yucel @bilgeyucl