Poor Man’s BingChat – Building an Internet-connected Search Assistant from scratch*

by Vlad Iliescu

Slide 1

Slide 1 text

vladiliescu.net Poor Man’s BingChat vladiliescu.net

Slide 2

Slide 2 text

vladiliescu.net what do we want? to ask GPT-4 questions about events that have occurred after September 2021

Slide 3

Slide 3 text

vladiliescu.net how do we get it? well, uhhh….

Slide 4

Slide 4 text

vladiliescu.net think step by step

Slide 5

Slide 5 text

vladiliescu.net what are large language models

Slide 6

Slide 6 text

vladiliescu.net AI in a nutshell def do_ai(input): output = '' for i in range(0, max_words): context = input + output next_word = predict_next_word(context) output += next_word return output do_ai('Give me the list of speakers for Codecamp Iasi 2023')

Slide 7

Slide 7 text

vladiliescu.net Large Language Models are Zero-Shot Reasoners Kojima et al., 2022

Slide 8

Slide 8 text

vladiliescu.net why don’t we just retrain it

Slide 9

Slide 9 text

vladiliescu.net well…

Slide 10

Slide 10 text

vladiliescu.net a helpful visualization:

Slide 11

Slide 11 text

vladiliescu.net

Slide 12

Slide 12 text

vladiliescu.net Base models Q: How does Hamlet’s famous monologue start? A: What is the opening line of Romeo’s soliloquy in Act 2, Scene 2 of “Romeo and Juliet”? – What are the first words of the “Tears in rain” monologue from the film “Blade Runner”? – What are the first words of Mark Antony’s “Friends, Romans, countrymen” speech in “Julius Caesar”? –

Slide 13

Slide 13 text

vladiliescu.net Assistant models Q: How does Hamlet’s famous monologue start? A: Hamlet’s famous monologue, often referred to as the “To be or not to be” soliloquy, is from William Shakespeare’s play “Hamlet.” This monologue appears in Act 3, Scene 1 and delves into the protagonist’s contemplation of life, death, and the nature of existence. The opening line is as follows: To be, or not to be, that is the question

Slide 14

Slide 14 text

vladiliescu.net so, it doesn’t really make sense to do a full retrain

Slide 15

Slide 15 text

vladiliescu.net maybe just fine-tune the thing?

Slide 16

Slide 16 text

vladiliescu.net

Slide 17

Slide 17 text

vladiliescu.net how a fine-tuning dataset looks like

Slide 18

Slide 18 text

vladiliescu.net Open Orca

Slide 19

Slide 19 text

vladiliescu.net issue #1 gathering a dataset of all text on the Internet is non-trivial plus it takes like, a lot of time

Slide 20

Slide 20 text

vladiliescu.net issue #2 fine-tuning still takes some time to do

Slide 21

Slide 21 text

vladiliescu.net basically as soon as we’d fine-tune something it’s already obsolete (no real-time info)

Slide 22

Slide 22 text

vladiliescu.net back to the drawing board

Slide 23

Slide 23 text

vladiliescu.net

Slide 24

Slide 24 text

vladiliescu.net what if we include the info in the prompt ?

Slide 25

Slide 25 text

vladiliescu.net

Slide 26

Slide 26 text

vladili

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

vladiliescu.net say hello to context window sizes

Slide 29

Slide 29 text

vladiliescu.net Remember this? def do_ai(input): output = '' for i in range(0, max_words): context = input + output next_word = predict_next_word(context) output += next_word return output do_ai('Give me the list of speakers for Codecamp Iasi 2023')

Slide 30

Slide 30 text

vladiliescu.net some popular models Model Context window Company 4k tokens Meta 4k tokens OpenAI 8k tokens OpenAI 16k tokens OpenAI 32k tokens OpenAI 100k tokens Meta 100k tokens Anthropic Llama2 GPT-3.5-turbo GPT-4 GPT-3.5-turbo-16k GPT-4–32k Code Llama Claude 2

Slide 31

Slide 31 text

vladiliescu.net but they’re so small

Slide 32

Slide 32 text

vladiliescu.net what if we don’t send it EVERYTHING?

Slide 33

Slide 33 text

vladiliescu.net how do we know what to include in the prompt?

Slide 34

Slide 34 text

vladiliescu.net we include the most relevant info duh

Slide 35

Slide 35 text

vladiliescu.net pattern’s called r a g

Slide 36

Slide 36 text

vladiliescu.net pattern’s called retrieval augmented generation

Slide 37

Slide 37 text

vladiliescu.net

Slide 38

Slide 38 text

vladiliescu.net You are a search assistant that helps users find information from a series of curated documents. The user just asked you this question: {QUESTION}. You need to think carefully and find the best answer to the query. Do not make stuff up, and answer using only information from the following documents: {RELEVANT_INFO}

Slide 39

Slide 39 text

vladiliescu.net The thing about RAG is that it’s only as good as the quality of your search results. – Stack Overflow Blog

Slide 40

Slide 40 text

vladiliescu.net semantic vs lexical search

Slide 41

Slide 41 text

vladiliescu.net come to think of it, we don’t need all of that (we do, actually)