Building advanced RAGs with Elasticsearch \ OpenSearch

RAG (Retrieval Augmented Generation) is built in 3 steps - understanding the query, executing a retrieval step and and generating a response. Each step has it's own unique characteristics and challenges. In this session we'll go through the common pitfalls (and solutions!) for each, and demonstrate how you can easily build an efficient RAG using Elasticsearch or OpenSearch.

Presented at Build Stuff Conference December 2025

Transcript

1. bigdataboutique.com Advanced RAG with Elasticsearch & OpenSearch
2. About Me Liza Katz Software Engineer, Team Lead, Solution Architect Past: IronSrc (Unity), Elastic
3. A bit of history
4. LLM Application Architecture • Patterns of textual interactions with LLMs ◦ Chatbot: Answers Questions ◦ Workflow: Series of steps / tasks ◦ RAG: Retrieval Augmented Generation ◦ Agent: Tools to get data and perform actions
5. LLM Application Architecture • Patterns of textual interactions with LLMs ◦ Chatbot: Answers Questions ◦ Workflow: Series of steps / tasks ◦ RAG: Retrieval Augmented Generation ◦ Agent: Tools to get data and perform actions
6. LLM Application Architecture • Patterns of textual interactions with LLMs ◦ Chatbot: Answers Questions ◦ Workflow: Series of steps / tasks ◦ RAG: Retrieval Augmented Generation ◦ Agent: Tools to get data and perform actions
7. LLM Application Architecture • Patterns of textual interactions with LLMs ◦ Chatbot: Answers Questions ◦ Workflow: Series of steps / tasks ◦ RAG: Retrieval Augmented Generation ◦ Agent: Tools to get data and perform actions
8. LLM Application Architecture • Patterns of textual interactions with LLMs ◦ Chatbot: Answers Questions ◦ Workflow: Series of steps / tasks ◦ RAG: Retrieval Augmented Generation ◦ Agent: Tools to get data and perform actions
9. When to RAG? • LLM Knowledge Cutoff • Better grounding • Talk to company knowledge • Talk to numeric data
10. RAG is dead, long live RAG ☠
11. • Pre Processing • Retrieve • Generate So what is RAG?
12. Choosing a vector DB • Cost • Performance • Scalability • Operations • Open Source The best vector DB might be the one you already have
13. I want to return the shoes I ordered 👟 🏠 🏻 ♂ 💻 • Personal information • Orders • Company Policy • Similar Tickets “Bob”
14. • What demos show you • What happens for real Advanced RAG Techniques
15. Semantic Search “I want to return the shoes I ordered” Customers may return most items within 30 days of delivery. Return Policy If your product arrives damaged or incorrect, you may request…
16. • Clean up • Chunking • Formatting • Model selection • Question extraction (FAQ) Embedding Strategy
17. • Query pre processing Keyword Extraction
18. Keyword Extraction “I want to return the shoes I ordered” If you wish to Return an Item You Ordered, please.. Returning Shoes Purchased Online Order Status and Order Changes can be viewed… Return Shoes Order LLM
19. • Filter by label, category, location, source, etc. • Scope to user, product, role Semantic Search + Metadata
20. Semantic Search + Metadata “I want to return the shoes I ordered” Customers may return most items within 30 days of delivery. Customers must notify support of returns within 15 days (Europe) If your product arrives damaged or incorrect, you may request… Label: returns Location: Europe, Global LLM
21. • Both of both worlds Hybrid Search + RRF
22. Hybrid Search + RRF “I want to return the shoes I ordered” Customers may return most items within 30 days of delivery. Returning Shoes Purchased Online Order Status and Order Changes can be viewed… If your product arrives damaged or incorrect, you may request… Customers must notify support of returns within 10 days (Europe)
23. • Use an AI model to reorder results • Based on relevancy to the question Rerank
24. Rerank “I want to return the shoes I ordered” Customers must notify support of returns within 15 days (Europe) Returning Shoes Purchased Online Order Status and Order Changes can be viewed… If your product arrives damaged or incorrect, you may request… Customers may return most items within 30 days of delivery.
25. Time Decay • Use function_score + knn to prioritize recent documents
26. Time Decay “I want to return the shoes I ordered” Returning Shoes Purchased Online (2025-11-01) Customers must notify support of returns in 10 days (2023-01-01) Order Status and Order Changes can be viewed… If your product arrives damaged or incorrect, you may request… Customers may return most items within 30 days of delivery.
27. A word on evaluation • Human Evaluation • Production Feedback • LLM as a judge • Search Evaluation
28. Summary • LLM applications are brand new • RAG is not dead • The best vector DB might be the one you already have • Get your embedding + chunking strategy right • Hybrid search, reranking, and time decay • EVALUATION is everything
29. bigdataboutique.com info@bigdataboutique.com Contact Liza Katz (lizak) pulse.support