Poor Man’s BingChat – Building an Internet-connected Search Assistant from scratch*

Have you seen BingChat’s/ChatGPT’s ability to reason about current events, even though the underlying models have been trained with data only up to September 2021? Have you wondered how it works behind the scenes, what did they do to make it work?

During this talk Vlad will be talking about the ways to achieve this, including possible options such as fine-tuning and retrieval augmented generation, but also limitations such as models’ slow inference times and limited context window sizes. He will demo an app that can talk about current events, and you will learn how to build your own, from scratch*.

*from scratch = with nothing but Python and the Azure OpenAI APIs

Transcript

1. vladiliescu.net Poor Man’s BingChat vladiliescu.net
2. vladiliescu.net what do we want? to ask GPT-4 questions about events that have occurred after September 2021
3. vladiliescu.net how do we get it? well, uhhh….
4. vladiliescu.net think step by step
5. vladiliescu.net what are large language models
6. vladiliescu.net AI in a nutshell def do_ai(input): output = '' for i in range(0, max_words): context = input + output next_word = predict_next_word(context) output += next_word return output do_ai('Give me the list of speakers for Codecamp Iasi 2023')
7. vladiliescu.net Large Language Models are Zero-Shot Reasoners Kojima et al., 2022
8. vladiliescu.net why don’t we just retrain it
9. vladiliescu.net well…
10. vladiliescu.net a helpful visualization:
11. vladiliescu.net
12. vladiliescu.net Base models Q: How does Hamlet’s famous monologue start? A: What is the opening line of Romeo’s soliloquy in Act 2, Scene 2 of “Romeo and Juliet”? – What are the first words of the “Tears in rain” monologue from the film “Blade Runner”? – What are the first words of Mark Antony’s “Friends, Romans, countrymen” speech in “Julius Caesar”? –
13. vladiliescu.net Assistant models Q: How does Hamlet’s famous monologue start? A: Hamlet’s famous monologue, often referred to as the “To be or not to be” soliloquy, is from William Shakespeare’s play “Hamlet.” This monologue appears in Act 3, Scene 1 and delves into the protagonist’s contemplation of life, death, and the nature of existence. The opening line is as follows: To be, or not to be, that is the question
14. vladiliescu.net so, it doesn’t really make sense to do a full retrain
15. vladiliescu.net maybe just fine-tune the thing?
16. vladiliescu.net
17. vladiliescu.net how a fine-tuning dataset looks like
18. vladiliescu.net Open Orca
19. vladiliescu.net issue #1 gathering a dataset of all text on the Internet is non-trivial plus it takes like, a lot of time
20. vladiliescu.net issue #2 fine-tuning still takes some time to do
21. vladiliescu.net basically as soon as we’d fine-tune something it’s already obsolete (no real-time info)
22. vladiliescu.net back to the drawing board
23. vladiliescu.net
24. vladiliescu.net what if we include the info in the prompt ?
25. vladiliescu.net
26. vladili
28. vladiliescu.net say hello to context window sizes
29. vladiliescu.net Remember this? def do_ai(input): output = '' for i in range(0, max_words): context = input + output next_word = predict_next_word(context) output += next_word return output do_ai('Give me the list of speakers for Codecamp Iasi 2023')
30. vladiliescu.net some popular models Model Context window Company 4k tokens Meta 4k tokens OpenAI 8k tokens OpenAI 16k tokens OpenAI 32k tokens OpenAI 100k tokens Meta 100k tokens Anthropic Llama2 GPT-3.5-turbo GPT-4 GPT-3.5-turbo-16k GPT-4–32k Code Llama Claude 2
31. vladiliescu.net but they’re so small
32. vladiliescu.net what if we don’t send it EVERYTHING?
33. vladiliescu.net how do we know what to include in the prompt?
34. vladiliescu.net we include the most relevant info duh
35. vladiliescu.net pattern’s called r a g
36. vladiliescu.net pattern’s called retrieval augmented generation
37. vladiliescu.net
38. vladiliescu.net You are a search assistant that helps users find information from a series of curated documents. The user just asked you this question: {QUESTION}. You need to think carefully and find the best answer to the query. Do not make stuff up, and answer using only information from the following documents: {RELEVANT_INFO}
39. vladiliescu.net The thing about RAG is that it’s only as good as the quality of your search results. – Stack Overflow Blog
40. vladiliescu.net semantic vs lexical search
41. vladiliescu.net come to think of it, we don’t need all of that (we do, actually)
42. vladiliescu.net
43. vladiliescu.net
44. vladiliescu.net we can just use
45. vladiliescu.net let’s see how this works in practice
46. vladiliescu.net thank you! vladiliescu.net vlad.substack.com