RAG vs Fine-Tuning: Choosing the Right Approach for Your GenAI App

A practical breakdown of when to use Retrieval-Augmented Generation vs. fine-tuning when building GenAI applications — covering the RAG pipeline (chunking, embedding, retrieval, reranking, generation), hybrid search, and a decision framework for choosing between the two approaches.
Originally presented at DevFest Cape Town 2025.

Transcript

1. Cape Town 2025 RAG vs Fine-Tuning: Choosing the Right Approach for Your GenAI Apps Abed Matini Senior Backend Developer, Ogilvy
2. Agenda • What is Fine-Tuning? What is RAG? • Search in RAG: Keyword, Semantic, Hybrid • Live Demo • RAG vs. Fine-Tuning • Advanced RAG Techniques
3. Why vanilla LLMs fall short • Knowledge cutoff; no access to private data • Hallucinations and lack of citations • Compliance, auditability, and governance needs • Rapidly changing information • Cost and maintenance in real apps
4. LLMs are already powerful What LLMs don’t know ✓ Summarizing text × Private databases and internal docs ✓ Generating and rewriting content × Hard-to-access or siloed information ✓ Code completion and transformation × Real-time updates and fresh content ✓ Reasoning (with limits) × Organization-specific terminology & style
5. What is Fine-Tuning? Fine-tuning involves further training a pre-trained LLM on a taskspecific dataset, fine-tuning helps it specialize in a particular area. • Requires labeled data and training infrastructure
6. Fine-Tuning Key Benefits • Further trains model weights for specific tasks or styles • Improves consistency, structured outputs, domain vocabulary • Improved accuracy, enhanced specificity, • Reduced hallucinations & bias.
7. What is RAG? Retrieval-Augmented Generation (RAG) is a framework that enhances LLMs by connecting them to external knowledge sources.
8. RAG Pipeline Overview Response + Citations Prompt Retriever Augmented Prompt Knowledge Base LLM
9. RAG Key Benefits • Access to up-to-date information • Reduce hallucinations • Ability to use private or domain-specific data • Transparent source attribution (Grounded) • Cost-effective (no retraining needed)
10. RAG Components LLM Knowledge Base Generates answers from the augmented prompt The content pool: docs, sites, databases, APIs. Retriever Searches the knowledge base to fetch the most relevant context for a query.
11. RAG Workflow Chunking Embedding Storing • Ingest and chunk content (overlap/semantic) • Embed chunks (choose embedding model) • Store embeddings (vector DB) • Retrieve top_k relevant chunks • Rerank or compress context • Generate answer with citations Retrieval Reranking Generation
14. RAG Workflow Chunking Embedding Storing Retrieval Reranking Generation Break the source content into smaller, manageable sections for easier processing.
15. RAG Workflow Chunking Embedding Storing Retrieval Reranking Generation Convert each section into a numerical representation that captures its meaning.
16. RAG Workflow Chunking Embedding Storing Retrieval Reranking Generation Save these embeddings in a vector database for efficient semantic search.
17. RAG Workflow Chunking Embedding Storing Retrieval Reranking Find and return the most relevant sections based on a query. Knowledge base Generation
18. RAG Workflow Chunking Embedding Storing Retrieval Reranking Reorder the retrieved sections to prioritize the most relevant ones. Generation
19. RAG Workflow Chunking Embedding Storing Retrieval Use the selected sections to produce the final answer. Reranking Generation
20. Search Methods Keyword Search Semantic Search Hybrid Search ✓ Precise matching ✓ Understands meaning ✓ Combines both ✓ Uses algorithms like ✓ Uses vector ✓ Metadata Filtering BM25, TF-IDF ✓ Fast × embeddings ✓ Finds conceptually Struggles with synonyms related content × May miss exact terms ✓ Balances precision and recall ✓ More comprehensive results ✓ Best approach for most RAG systems
21. Semantic Search Similar Sentences Pet
22. Hybrid Search 1. Result A 1. Result A 2. Result B 2. Result B 3. Result C Keyword Search 50 result each! 3. Result C Metadata Filter 1. Result C 1. Result C Retriever 2. Result B 2. Result B Semantic Search 3. Result X Metadata Filter 3. Result X Top_k results
23. Search Methods Examples • Query: “renewal policy for enterprise contracts” • Keyword finds “renewal” sections • Semantic finds “contract extension” and “rollover” • Hybrid returns both; filters to source=“policy-docs”, region=“ZA”
25. Cape Town DEMO
26. Fine-Tuning vs RAG: Comparison Advantages Disadvantages - Training takes time Fine-Tuning - Matches org-style with unsupervised training - Requires expertise - Can align with compliance & proprietary data - Not available for all models - No source references - Higher risk of hallucination - Fast setup, no fine-tuning needed - Can use new documents in minutes RAG - Managed solutions available - Provides source references - Lower hallucination risk - Not great at summarizing whole documents
27. Decision Framework
28. Cape Town Start with Rag for custom document Q&A. Use Fine-tuning if you need tasks like summarization or ogspecific style.
29. Advanced Rag Techniques Metadata Filtering Hypothetical Question Generation Multi-Source Retrieval Reranking Prompt Optimization Self-Evaluation
30. Key Takeaways • RAG grounds LLMs with real, fresh knowledge and citations • Fine-tuning improves style/structure and task performance • Hybrid approach often wins in production • ADK makes RAG implementations faster and maintainable • Evaluate, monitor, and iterate continuously
31. Thank You https://www.linkedin.com/in/matini https://google.github.io/adk-doc