June 2024 - Navigating RAG Optimization with an Evaluation Driven Compass

Retrieval Augmented Generation (RAG) has become a cornerstone for integrating domain-specific content and addressing hallucinations in AI applications. As the adoption of RAG solutions intensifies across industries, a pressing challenge emerges: understanding and identifying where within the complex RAG framework changes and improvements can be made. This talk delves into the methodology of extracting crucial indicators from your RAG pipeline, empowering informed decision-making during experimentation. Join us as we navigate an end-to-end process for RAG experimentation and evaluation, offering insights into optimizing performance and addressing hurdles along the RAG implementation journey.

Recorded live in San Francisco at the AI Engineer World's Fair.
Video here: https://www.youtube.com/watch?v=DId2KP8Ykz4

Transcript

1. Navigating RAG Optimization with an Evaluation-Driven Compass | Atita Arora Solution Architect, Qdrant Deanna Emery Founding AI Researcher, Quotient AI
2. | Key discussion points : • Core challenges of RAG • Improvement Techniques • Evaluation based RAG Optimization • Results and Conclusions
3. | Flavours of RAG
4. Enterprise-Ready, Massive-Scale Vector Search Technology for the Next AI Generation Performance Centric Quick and easy to start Resource optimization focussed All embeddings supported OOTB Fully Open Source project Scalability Oriented Most Loved open-source vector search database > 10 K+ Adopters Worldwide > 5 M+ Downloads > 18 K+ Github Stars
5. | Challenges of RAG
6. | RAG Improvement techniques Data Cleaning ℹ Leverage Metadata Advanced data extraction ✂ Data Chunking Embedding Model Retrieval Window 🔍 Indexing Algorithms 🗂 Multi Vector Indexing Document Reranking LLM Prompt Engineering Prompt Agents
7. | Evaluation is Paramount !! Why should you evaluate ? • Establish Trust (Reputation and Confidence) • Correlate outcomes wrt your use case • Validation that your application avoids common pitfalls • Criterion to make a go / no-go decisions • Roadmap for improvements • Compliance and Ethics
8. Domain-specific eval is essential for high-quality RAG apps RAG quality is inherently use-case-dependent. It depends on the database and its contents. With Quotient we can evaluate and iterate on LLM products using domain-specific, data-backed experimentation.
9. |
10. Demo Walkthrough Find our notebook here:
11. | Evaluation Metrics • We want to minimize hallucinations in the LLM outputs. • For our evaluation, we will be considering the following metrics, with a focus on Faithfulness: ◦ Context Relevance ◦ Chunk Relevance ◦ Faithfulness ◦ ROUGE-L ◦ BERTScore ◦ BERT Sentence Similarity
22. | To sum up • Domain understanding helps address challenges like document order, terminology, and chain of thought. • Avoid over-optimizing your first run; there's no substitute for evaluation-based improvements. • Your data is a crucial determinant of the complexity of RAG. • Regularly update your evaluation dataset to keep it aligned with the latest challenges in your LLM application. • Evaluate with a combination of carefully chosen metrics to effectively diagnose issues. • Ensure the scalability of your evaluation process to accommodate future expansions and refinements. • LLMs, combined with human evaluations, are among the most effective methods for assessing LLM-based applications. Find our notebook here:
23. | References • https://www.youtube.com/watch?v=3MEMPZR1aZA • https://qdrant.tech/articles/rapid-rag-optimization-with-qdrant-and-quoti ent/ • https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building- with-llms-part-i/ • https://superlinked.com/vectorhub/evaluating-retrieval-augmented-gen eration-a-framework-for-assessment
24. Thank you !! A free forever 1GB cluster included for trying out. No credit card required. Feel free to reach us at : info@qdrant.com Request access to Quotient to get started! Learn more at: https://www.quotientai.co/