RAG for Small LLM

LLMs are pretty amazing at understanding natural language. When you combine them with RAG (Retrieval-Augmented Generation), it’s like a match made in heaven. You can use this combo for all sorts of things like answering questions on a knowledge base, interactive chatbots, smart coding assistants, and more.

​But if you’re worried about privacy, using cloud-based LLMs like GPT or Gemini can be a problem.

​​What if you could get an astonishingly good RAG using a small LLM?

Transcript

1. Advanced RAG for Small LLM
3. llama.cpp Phi 2 Pico Jarvis
4. github.com/ariya/pico-jarvis
6. Zero-Shot Prompt
7. Zero-Shot Prompt
9. You are a helpful assistant. Answer verbosely. user assistant system
10. You are a helpful assistant. Answer concisely.
12. Chain of Thought
14. You are a research assistant with access to Google search. Given the question from the user, your task is to use Google to search for the answer. Think step by step.
16. JSON Mode
17. JSON Mode
18. You are a research assistant with access to Google search. Given the question from the user, your task is to use Google to search for the answer. Think step by step. Always output a valid JSON as follows: { "thought": describe your thoughts about the question, "tool": the search engine to use (must be Google), "input": important key phrases to give to search for, "observation": concise result from Google } MAKE BELIEVE
19. thought: This is about Star Trek, I will use Google search tool: Google input: Captain Kirk first name observation: James
21. WRONG
22. Retrieval-Augmented Generation
24. Generation
25. You are given a question from human and you have to answer it concisely using only the following reference document. Avoid using any external information or recalling from memory. Now use the above reference to answer this question.
27. This is the weather observation for Palo Alto. The current weather condition is broken clouds. * barometric pressure: 1013 mbars. * temperature (in Celcius): 17 °C. * temperature (in Fahrenheit): 63 °F. * humidity: 60%.
28. Retrieval Improvement
29. Query Transformation
33. You are a language assistant. The user specifies a conversation history followed by a statement. Your task is to rephrase the statement as a new inquiry, with only the relevant references in the conversation history. Explain the reason behind the rephrasing. Always respond in JSON format, such as: { "rephrased": a reformulated statement of the original, "explanation": reason to rephrase it that way }
34. User: What is a dwarf planet? Assistant: A dwarf planet is a celestial body that orbits the sun, is large enough to be rounded by its own gravity, but has not cleared the neighborhood around its orbit of other debris. User: Give an example! User: What is a dwarf planet? Assistant: A dwarf planet is a celestial body that orbits the sun, is large enough to be rounded by its own gravity, but has not cleared the neighborhood around its orbit of other debris. User: Can you provide an example of a dwarf planet?
35. rephrased: Can you provide an example of a dwarf planet? explanation: The user wants to know an example of a dwarf planet, as they have just learned about what a dwarf planet is.
38. HYDE Hypothetical Document Embeddings
39. 1 _ .. __ .. __ .. __ __ .. __ .. __ 2 _ .. __ .. __ .. __ __ .. __ .. __ 3 _ .. __ .. __ .. __ __ .. __ .. __ … __ .. __ .. __ .. __ __ .. __ .. __ .. __ 434 Dwarf planets are considered planets by some planetologists but not by the IAU. .. __ .. __ .. What is a dwarf planet? 77%
40. 1 _ .. __ .. __ .. __ __ .. __ .. __ 2 _ .. __ .. __ .. __ __ .. __ .. __ 3 _ .. __ .. __ .. __ __ .. __ .. __ … __ .. __ .. __ .. __ __ .. __ .. __ .. __ 433 A dwarf planet is a body orbiting the Sun that is massive enough to be made near-spherical by its own gravity but that has not cleared planetesimals from its neighborhood and is also not a satellite. .. __ .. __ .. A dwarf planet is a celestial body that orbits the sun, is large enough to be rounded by its own gravity, but has not cleared the neighborhood around its orbit of other debris. 87%
41. Keyphrases Search
42. Keyphrases Citations
44. thought: This is about Star Trek, I will use Google search tool: Google input: Captain Kirk first name observation: James
45. Knowledge Base First Retriever Relevant Passages Rerank
46. Intent Classifier
49. Reason-Act Thought: This is about weather and I will use weather action. Action: weather: Palo Alto Observation: No, it's not currently snowing in Palo Alto. Answer: It's not currently snowing in Palo Alto. 1
50. Reason-Act Thought: This is about weather and I will use weather action. Action: weather: Palo Alto Observation: The current weather in Palo Alto is clear sky at 17.5 C and humidity 48% Answer: No, it is not. 2 Weather API
55. Thank You speakerdeck.com/ariya