Slide 23
Slide 23 text
Organizing Tokens & Storage
Vectorization
Chunks
The process of splitting an
article or document into
smaller segments, roughly
500 tokens
Why?
• LMs can only process
limited tokens at once
10320, 1821,
14156, 588,
7423, 257,
6294,
7271, 329,
4204, 15088,
…
562, 5894,
45697, 23641,
77512, 421,
458, 8745,
3659, 785412,
412, 8561,
3667, …
Converting each chunk into a
high-dimensional numeric
vector that captures its surface
level semantic meaning
How?
• Each chunk is passed
through an embedding
model
• Output is a vector: e.g.,
[0.012, -0.776,
..., 0.431]
• Stored in a vector database
for retrieval and surfacelevel similarity search