LLM 101: Exploring LLMs, Starting with Tokens.

This is the English version of “LLM 101: Exploring LLMs, Starting with Tokens.”

Transcript

1. LLM 101 Turning text into numbers and choosing the next token An introduction through Python scripts 01–29 01
2. The book behind this lecture Based on Build a Large Language Model (From Scratch) Sebastian Raschka, Manning, 2024 Local scripts Book chapters Focus 01–09 Chapter 2 Text data and embeddings 10–14 Chapter 3 Attention mechanisms 15–29 Chapters 4–5 GPT, training and pretrained weights This lecture follows the local study scripts 01–29. 02
3. A small model with inputs and outputs y=a×x+b 2×2+1=5 Symbol Meaning Example x Input value 2 a Weight that multiplies the input 2 b Bias added to the result 1 y Prediction computed by the model 5 The model computes y from x. Parameters a and b define its rule. 03
4. a controls slope, b controls the intercept Changing b with a = 2 12 12 10 10 8 8 Output y Output y Changing a with b = 1 6 4 6 4 2 2 0 0 0 1 a=1, b=1 2 Input x 3 4 a=2, b=1 0 1 a=2, b=1 2 Input x 3 4 a=2, b=3 a controls how fast y increases. b is the output at x = 0. 04
5. Training fits a and b to data Illustrative data and predictions At x = 2 10 Output y 8 First prediction: 2 6 Target: 5 4 2 0 0 1 2 Input x Before: y = x After: y = 2x + 1 3 4 After adjustment: 5 Observed targets Training adjusts weights and biases to improve predictions. 05
6. A linear layer works with vectors y = Wx + b Input x = [2, 1] Output Multiply inputs by weights, then add Result First value 2×2+1×1+1 6 Second value −1 × 2 + 3 × 1 + 0 1 The same idea extends from one number to a vector. In a linear layer, W contains weights and b contains biases. 06
7. GPT chooses one next token at a time Every effort moves you ___ Illustrative next-token probabilities 60% 50% 45% 40% 28% 30% 20% 20% 7% 10% 0% forward closer toward other Append one token, then repeat the calculation. 07
8. Lecture outline and a place to pause Section Scripts Main question Time Basics and text 01~09 How does text become numbers? ~30 min Token relationships 10~14 Which tokens contribute information? ~30 min Building GPT 15~23 How do the components connect? ~25 min Training and generation 24~29 How do we improve and use predictions? ~30 min Part 1 ends on slide 39. Continue with Part 2 next time if needed. 08
9. Reading tensor shapes Symbol Meaning Example B Sequences processed in one batch 2 sequences T Tokens in each sequence 4 tokens C Features representing each token 768 numbers [B, T, C] Sequences × tokens × features [2, 4, 768] B, T and C keep the same meaning throughout this lecture. 09
10. 01. Reading and splitting text 01_open_verdict.py Before Hello, world. This, is a test. After splitting Hello , world . This , is a test . Characters Total tokens Unique tokens 20,479 4,690 1,130 Total tokens count repeats. Unique tokens count each kind once. 10
11. 02. Giving each token an ID 02_tokenizer.py Token Vocabulary ID Process ! 0 Deduplicate, sort, then assign IDs I 53 Deduplicate, sort, then assign IDs the 988 Deduplicate, sort, then assign IDs you 1126 Deduplicate, sort, then assign IDs vocab = {token: integer for integer, token in enumerate(all_words)} An ID is a lookup index. Its size does not express meaning. 11
12. 03. Encoding and decoding 03_w_simple_tokenizer.py Text pieces the last he painted Encoded IDs 988 602 533 746 Decoded text the last he painted Text to IDs IDs to text encode() decode() Encoding maps text pieces to IDs. Decoding maps IDs back to text. 12
13. 04. Unknown tokens and document boundaries 04_special_token.py Case V1 V2 Known token Its token ID Its token ID Unknown token Lookup error ID of <|unk|> Joining documents Handle boundaries manually Use <|endoftext|> Document A <|endoftext|> Document B Each special token also has its own vocabulary ID. 13
14. 05. BPE splits unfamiliar words into pieces 05_bpe.py Akwirw ier Piece Ak w ir w space ier ID 33901 86 343 86 220 959 tiktoken.get_encoding("gpt2") GPT-2 uses byte-level BPE to represent text as smaller pieces. 14
15. 05. BPE merges frequent pairs 05_bpe.py Step Merge Resulting sequence Start None yet aaabdaaabac 1 aa becomes X XabdXabac 2 ab becomes Y XYdXYac 3 XY becomes Z ZdZac Script 05 uses the existing GPT-2 merges. It does not train new BPE rules. 15
16. 06. Targets shift one position ahead 06_sliding_window.py Position 0 1 2 3 Input x Every effort moves you Target y effort moves you forward Input IDs 6109 3626 6100 345 Target IDs 3626 6100 345 2651 At each position, the model predicts the next token. 16
17. 06. One sequence gives several predictions 06_sliding_window.py Visible input Next-token target Every effort Every effort moves Every effort moves you Every effort moves you forward A sequence of length 4 provides four next-token training targets. 17
18. 07. Sliding windows and stride 07_data_loader.py These illustrative IDs show how the input window moves. max_length sets the window length. stride sets how far it moves. 18
19. 07. A batch groups several input windows 07_data_loader.py Batch Token 0 Token 1 Token 2 Token 3 Input 1 40 367 2885 1464 Input 2 1807 3619 402 271 Input 3 10899 2138 257 7026 Input 4 15632 438 2016 257 Input 5 922 5891 1576 438 Input 6 568 340 373 645 Input 7 1049 5975 284 502 Input 8 284 3285 326 11 Both inputs and targets have shape [8, 4] in this example. 19
20. 08. Embeddings look up a vector by ID 08_token_embedding.py Feature 1 Feature 2 Feature 3 Vocabulary size: 6 ID 0 0.34 -0.18 -0.17 6 rows to choose from ID 1 0.92 1.58 1.30 ID 2 1.28 -0.20 -0.16 ID 3 -0.40 0.97 -1.15 Feature width: 3 3 numbers per token Input [2, 3, 5, 1] ID 4 -1.16 0.33 -0.63 Output shape [4, 3] ID 5 -2.84 -0.78 -1.41 Each integer ID selects one vector containing C numbers. 20
21. 09. Adding position vectors 09_position_encoding.py Same token Token vector Position vector Sum Position 0 [0.2, 0.4, 0.1] [0.1, 0.0, 0.3] [0.3, 0.4, 0.4] Position 1 [0.2, 0.4, 0.1] [0.0, 0.5, 0.1] [0.2, 0.9, 0.2] [8, 4, 256] + [4, 256] = [8, 4, 256] Position embeddings give the same token different inputs at different positions. 21
22. 09. The input embedding pipeline 09_position_encoding.py 1. Tokenization Text becomes integer IDs 2. Embedding lookup Retrieve one vector per ID 3. Add positions Vectors now include order Look up token vectors, then add information about their positions. 22
23. Token IDs and vectors Form Example Purpose Token Every A piece of text ID 6109 Index into the embedding table Vector [0.2, −0.5, …] Numbers for attention and neural layers IDs select rows. Vectors contain the numbers used in model calculations. 23
24. What self-attention does Unmasked example: journey gathers information from the same sequence. 'Self' means each token draws information from its own sequence. 24
25. 10. A small input for simplified attention 10_self_attention.py Feature 1 Feature 2 Feature 3 Your 0.43 0.15 0.89 journey 0.55 0.87 0.66 starts 0.57 0.85 0.64 with 0.22 0.58 0.33 6 tokens Your journey starts with one step 3 numbers per token Input shape [6, 3] Query token one 0.77 0.25 0.10 Second row: journey step 0.05 0.80 0.55 Teaching example: fixed vectors, before Q/K/V projections Fixed example vectors: one token per row, one feature per column. 25
26. 10. Direct dot products in the toy example 10_self_attention.py Direct dot-product scores for journey 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 1.50 1.48 1.09 0.95 Your 0.84 journey starts with 0.71 one step score(journey, starts) = 0.55×0.57 + 0.87×0.85 + 0.66×0.64 This toy example scores input vectors directly. Learned Q and K come next. 26
27. 10. Softmax turns scores into weights 10_self_attention.py Your journey starts with one step Each row sums to 1 Your 0.21 0.20 0.20 0.12 0.12 0.15 Interpret these as weights journey 0.14 0.24 0.23 0.12 0.11 0.16 starts 0.14 0.24 0.23 0.12 0.11 0.16 with 0.14 0.21 0.20 0.15 0.13 0.17 Darker = more weight Read the journey row Future tokens visible one 0.15 0.20 0.20 0.14 0.19 0.13 Script 12 adds a mask step 0.14 0.22 0.21 0.14 0.10 0.19 Rows receive information. Columns supply information. 27
28. 10. A weighted average creates a new vector 10_self_attention.py Source token Attention weight Input vector Your 13.9% [0.43, 0.15, 0.89] journey 23.8% [0.55, 0.87, 0.66] starts 23.3% [0.57, 0.85, 0.64] with 12.4% [0.22, 0.58, 0.33] one 10.8% [0.77, 0.25, 0.10] step 15.8% [0.05, 0.80, 0.55] New vector for journey = [0.442, 0.651, 0.568] The new vector mixes input information using these weights. 28
29. Simplified attention and GPT attention Script 10 isolates the arithmetic. GPT learns how to compare and combine. Operation Script 10 teaching example GPT attention Representations Fixed example input vectors Learned Q, K and V projections Scores xᵢ · xⱼ qᵢ · kⱼ / √dₖ Content to mix Original input vectors X Projected value vectors V Visible positions All positions in the example Current and earlier positions GPT head: Z = softmax(QKᵀ / √dₖ + M)V Q = XW_Q K = XW_K V = XW_V M masks future positions Direct dot products of input embeddings are a teaching simplification. 29
30. 11. Separate projections create Q, K and V 11_scaled_dot_product_attention.py Q: query K: key V: value to combine Input × W_query Input × W_key Input × W_value Q and K determine attention weights. Those weights mix V. 30
31. 11. Q, K and V shapes 11_scaled_dot_product_attention.py journey representation Feature values Width Input x₂ [0.55, 0.87, 0.66] 3 Q₂ [0.431, 1.455] 2 K₂ [0.443, 1.142] 2 V₂ [0.395, 1.004] 2 Each projection maps 3 input features to 2 output features. 31
32. 11. Scaling attention scores 11_scaled_dot_product_attention.py Attention weights before and after scaling 35% 30% 25% 20% 15% 10% 5% 0% Your journey starts Before scaling with one step After dividing by √2 Attention weights = softmax((Q × Kᵀ) / √dₖ) Divide scores by √dₖ before applying softmax. 32
33. 12. A causal mask hides future tokens 12_causal_attention.py Your journey starts with one step Purple: allowed Your 1 × × × × × Current and past tokens journey 1 1 × × × × starts 1 1 1 × × × with 1 1 1 1 × × Gray: blocked Future tokens Diagonal stays open one 1 1 1 1 1 × A token can use itself step 1 1 1 1 1 1 Each position can use itself and earlier tokens. 33
34. 12. Softmax after masking 12_causal_attention.py Your journey starts with one step First row: itself only Your 1.00 0.00 0.00 0.00 0.00 0.00 Weight = 1.00 journey 0.40 0.60 0.00 0.00 0.00 0.00 starts 0.25 0.38 0.37 0.00 0.00 0.00 with 0.23 0.28 0.28 0.21 0.00 0.00 Future weights = 0 Prevents target leakage Each row sums to 1 one 0.20 0.24 0.23 0.18 0.15 0.00 Normalize after masking step 0.16 0.21 0.20 0.14 0.11 0.18 Setting future scores to −∞ makes their softmax weights zero. 34
35. 13. Dropout randomly removes some values 13_dropout.py 1 2 3 4 5 6 p = 0.5 Row 1 2 2 0 2 2 0 About half become zero Row 2 0 0 0 2 0 2 Row 3 2 2 2 2 0 2 Row 4 0 2 2 0 0 2 Surviving values × 2 Keeps the expected scale Evaluation or generation Row 5 0 2 0 2 0 2 model.eval() Row 6 0 2 2 2 2 0 During training, dropout zeros some values and rescales the rest. 35
36. 14. Splitting 768 features into 12 heads 14_gpt_size.py + multi_head_attention.py Item Value Meaning Total feature width 768 Features per token Number of heads 12 Groups computed in parallel Features per head 64 768 ÷ 12 Maximum context length 1,024 Positions in one model input Each of the 12 heads works with 64 of the 768 features. 36
37. The computation inside each head 14_gpt_size.py + multi_head_attention.py Q–K scores Scaling and masking Attention weights Weighted sum of V Each head applies masking and a weighted sum. The results join afterward. 37
38. Tensor shapes in multi-head attention multi_head_attention.py Operation Shape Interpretation Each of Q, K and V [2, 4, 768] Batch, tokens, features Split and transpose heads [2, 12, 4, 64] Batch, heads, tokens, head features Q × Kᵀ [2, 12, 4, 4] 2 batches × 12 heads, each with a 4 × 4 tokenscore table Weighted sum, then join heads [2, 4, 768] 768 features per token After combining heads, the output shape returns to [B, T, 768]. 38
39. Part 1: token vectors with context Text pieces become IDs IDs become vectors with positions Attention adds context from earlier tokens Next: stacking these components to build GPT This is a natural place to pause and continue in the next session. 39
40. 15. A dummy GPT shows the overall shapes 15_dummy_gpt_model.py Stage Shape Role in this example Input IDs [2, 4] 2 sequences of 4 tokens Embeddings [2, 4, 768] Combine token and position vectors Dummy blocks and normalization [2, 4, 768] Pass-through placeholders Output logits [2, 4, 50257] Scores for 50,257 candidates at each position Scripts 16–21 replace the placeholder blocks with real components. 40
41. 16. Normalizing each token vector 16_layer_normalization.py Illustrative feature values for one token 2.0 1.0 0.0 -1.0 -2.0 Feature 1 Feature 2 Feature 3 Before normalization Feature 4 Feature 5 Feature 6 After normalization Layer normalization operates across the features of each token. 41
42. 17. Layer normalization has learned parameters 17_layer_norm.py + layer_norm.py Normalized value = (x − mean) / √(variance + ε) Output = scale × normalized value + shift Parameter Initial value Role ε 0.00001 Stabilizes the denominator scale / shift 1/0 Learns the output scale and offset After normalization, apply learned scale and shift parameters. 42
43. 18. GELU adds a smooth nonlinearity 18_gelu_test.py + gelu.py Horizontal: input x. Vertical: function output 3.0 2.5 2.0 1.5 1.0 0.5 0.0 -4 -3 -2 -1 -0.5 0 GELU 1 2 3 4 ReLU Nonlinear activations make stacked linear layers more expressive. 43
44. 19. The feed-forward network transforms features 19_feed_forward_test.py + feed_forward.py 768 features Input width 3,072 features Expand × 4, then apply GELU 768 features Return to the original width The same FFN acts on each position separately and keeps the token count. 44
45. 19. The FFN changes feature width 19_feed_forward_test.py Step Batch size B Tokens T Feature width C Input 2 3 768 First Linear 2 3 3,072 GELU 2 3 3,072 Second Linear 2 3 768 Both the input and final output have shape [2, 3, 768]. 45
46. 20. A residual is a change added to the input 20_example_deep_neural_network.py 13 − 10 = 3 Input and output with illustrative numbers 15 3 10 5 Input x = 10 Residual F(x) = +3 10 10 Output = 13 0 Input x Original input Output x + F(x) Added change In this illustrative example, adding a change of +3 turns 10 into 13. 46
47. 20. A residual connection creates two paths 20_example_deep_neural_network.py 10 + 3 = 13 The input and the change must have matching shapes. Add the computed change F(x) to the original input x. 47
48. 20. Gradients reaching the earlier layers 20_example_deep_neural_network.py Layer Without shortcuts With shortcuts Layer 1 0.000202 0.221698 Layer 2 0.000120 0.206941 Layer 3 0.000715 0.328970 Layer 4 0.001399 0.266573 Layer 5 0.005050 1.325854 First layer: 0.000202 / 0.221698 In this small experiment, the first layer's mean gradient is about 1,099× larger. 48
49. 21. Inside one transformer block 21_transformer_block_test.py + transformer_block.py Each block combines normalization, attention, an FFN and two shortcuts. 49
50. 21. A block preserves the tensor shape 21_transformer_block_test.py Input Inside the block Output [2, 4, 768] Context mixing + feature transformation [2, 4, 768] Same shape, updated vector values A consistent shape lets us stack several blocks of the same kind. 50
51. 22. The complete GPT architecture 22_gpt_model_test.py + gpt_model.py Input Token IDs with shape [B, T] 12 blocks Shape stays [B, T, 768] Output scores [B, T, 50257] Vector values in the diagram are illustrative. Embeddings feed 12 blocks, a final normalization and an output layer. 51
52. 22. The 124M model has 163M parameters here 22_gpt_model_test.py What we count Parameter count Explanation Actual total in this code 163,009,536 Separate embedding and output weights Total assuming weight tying 124,412,160 Count the shared matrix once fp32 weights only 621.8 MiB Training needs additional memory One embedding matrix: 50,257 × 768 This GPTModel stores token embeddings and output weights separately. 52
53. 22. Weight memory grows with model size 22_gpt_model_test.py Weight memory in this implementation (MiB) 7,000 6,248 6,000 5,000 4,000 3,198 3,000 2,000 1,000 1,550 622 0 small medium large XL These figures count fp32 weights only. 53
54. Numeric formats and weight memory Same weight count, different storage width 1 million weights: payload in MB 4.0 1 byte = 8 bits 4.0 FP32 BF16 8-bit 4-bit 3.0 2.0 2.0 1.0 1.0 4 bytes 2 bytes 1 byte ½ byte 0.5 0.0 FP32 BF16 8-bit Packed 4-bit 4-bit values need packing: two values per byte. Fewer bits reduce the weight payload. Runtime memory also includes other data. 54
55. Sign, mantissa and exponent Floating-point numbers separate the significant digits from their scale. 6.5 = (+1) × 1.625 × 2² Part Example Meaning Sign +1 Positive or negative Significand (mantissa) 1.625 Significant digits, including the leading 1 Exponent 2 2² = 4 sets the magnitude Normal binary value: significand = 1 + stored fraction More fraction bits give finer steps. More exponent bits allow a wider range. 55
56. FP32: 1 + 8 + 23 bits A 32-bit float stores a sign, a biased exponent and fraction bits. Sign s Stored exponent E Stored fraction bits 0 10000001 = 129 10100000000000000000000 FP32 bias = 127: e = 129 − 127 = 2 f = 0.625 6.5 = (+1) × (1 + 0.625) × 2² For normal FP32 values, the hidden leading 1 gives 24 bits of significand precision. 56
57. BF16: the exponent stays, the fraction shrinks BF16 uses 16 bits: 1 sign bit, 8 exponent bits and 7 fraction bits. Next number above 1 FP32 BF16 Value 1.0000001192… 1.0078125 BF16 keeps a similar magnitude range to FP32, with coarser spacing between values. 57
58. FP8: E4M3 and E5M2 E counts exponent bits. M counts stored fraction bits. Both include 1 sign bit. Raw format E4M3FN E5M2 Largest finite magnitude 448 57,344 Next number above 1 1.125 1.25 E4M3 has finer precision. E5M2 covers a wider magnitude range. 58
59. 8 bits or 4 bits can encode different types Integer formats and floating-point formats use their bits differently. Type Bit layout Raw values INT8 8-bit signed integer −128 to 127 INT4 4-bit signed integer −8 to 7 FP4 E2M1 1 sign + 2 exponent + 1 fraction 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6 INT4 example: q = 3 Its meaning as a weight depends on the accompanying scale factor. The bit width tells you the storage size. The format tells you which values it can store. 59
60. A scale factor connects the stored value to the weight Here, scale s means: reconstruct x̂ = q × s. Store: q = cast(x / s) Reconstruct: x̂ = q × s The scale extends the usable range. Rounding can still lose information. 60
61. INT8 and INT4: a rounding example Chosen scales: INT8 s = 0.01, INT4 s = 0.1. Zero point = 0. Original x INT8 q INT8 x̂ INT4 q INT4 x̂ −0.63 −63 −0.63 −6 −0.60 −0.24 −24 −0.24 −2 −0.20 0.33 33 0.33 3 0.30 0.68 68 0.68 7 0.70 q = clip(round(x / s), qmin, qmax) INT4 with s = 0.1 covers reconstructed values from −0.8 to +0.7. A smaller scale gives finer steps, but also reduces the range before clipping. 61
62. One scale per tensor, channel or block A shared scale can lose small values when another value is much larger. Data / encoding Small group Large group Original values 0.01, 0.03, 0.05, 0.07 1, 3, 5, 7 One scale: s = 1 Stored INT4 q 0, 0, 0, 0 1, 3, 5, 7 Two scales Stored INT4 q s = 0.01 1, 3, 5, 7 s=1 1, 3, 5, 7 Same stored q = 7 7 × 0.01 = 0.07 7×1=7 More local scales can preserve detail, while adding scale metadata. 62
63. 23. Choosing a token from the last position 23_text_generate.py + generate_text_simple.py Full output [B, T, 50257] Last position only [:, −1, :] Pick the largest score One token with argmax 50,257 candidates have IDs 0–50,256. Generation uses only the logits at the last input position. 63
64. 23. Appending the generated token 23_text_generate.py + generate_text_simple.py Script 23 repeats the generation loop to add six new tokens. 64
65. 24. Loss measures prediction error 24_text_eval.py + loss_util.py Horizontal: correct-token probability p. Vertical: loss 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 0 0.2 p = 0.1 gives loss 2.30 0.4 0.6 0.8 1 1.2 p = 0.8 gives loss 0.22 A higher probability for the correct token means a lower loss. 65
66. 24. Averaging loss across token positions 24_text_eval.py Item Before flattening After flattening Predicted logits [2, 3, 50257] [6, 50257] Target IDs [2, 3] [6] Loss Compare targets at each position Average over 6 predictions cross_entropy(logits.flatten(0, 1), targets.flatten()) CrossEntropyLoss takes logits as its input. 66
67. 24. Splitting training and validation text 24_text_eval.py Characters in the-verdict.txt 20,000 Training data 18,431 Used to update weights 15,000 Validation data 10,000 Used to check performance 5,000 2,048 0 Training characters Validation characters Context length: 256 Reduced from 1,024 here The code splits the original text by characters: first 90%, last 10%. 67
68. 24. The training loop 24_text_eval.py + train_util.py Step Action Code 1 Clear previous gradients optimizer.zero_grad() 2 Compute the current loss loss = calc_loss_batch(...) 3 Compute gradients for the weights loss.backward() 4 Update the weights optimizer.step() Compute the loss, then update weights in a direction that reduces it. 68
69. 24. Learning curves and overfitting 24_text_eval.py Illustrative overfitting curves (horizontal axis: epoch) 12.0 10.0 8.0 6.0 4.0 2.0 0.0 0 1 2 3 4 Training loss 5 6 7 8 9 10 Validation loss Compare training loss with validation loss. 69
70. 25. Greedy choice and random sampling 25_sampling.py Actual probabilities for the 9 candidates in script 25 70% 57.21% 60% 50% 40% 35.76% 30% 20% 10% 6.09% 0% closer 0.16% 0.01% every effort forward 0.34% 0.01% 0.01% inches moves pizza 0.40% toward you argmax picks the largest score. multinomial samples by probability. 70
71. 25. Temperature changes the distribution 25_sampling.py Actual distributions with the same logits and different temperatures 100% 80% 60% 40% 20% 0% closer every effort forward T = 0.1 inches T=1 moves pizza toward you T=5 Divide logits by temperature before computing probabilities. 71
72. 25. Top-k restricts the candidates 25_sampling.py Actual probabilities before and after filtering 70% 60% 50% 40% 30% 20% 10% 0% closer every effort forward inches Original probabilities moves pizza toward you After Top-3 Set logits outside the top k to −∞, then recompute probabilities. 72
73. 26. Training, sampling and saving 26_sampling2.py + generate_util.py Step Settings or actions in script 26 Training 10 epochs with a new model Generation Up to 15 tokens, Top-k 25, temperature 1.4 Save model state model_state_dict Save optimizer state optimizer_state_dict This script trains a new model before generating samples. 73
74. 27. Resuming a saved training state 27_model_load.py Step What to restore Why it matters 1 Same GPT_CONFIG Match the weight shapes 2 Model state_dict Restore trained weights 3 Optimizer state_dict Restore accumulated optimizer state 4 model.train(), then 1 epoch Continue training Recreate the architecture, then restore model and optimizer states. 74
75. 28. Downloading the helper code 28_download_gpt_download.py Scripts What it downloads or loads Next step Script 28 The code in gpt_download.py Prepare the download helper Script 29 GPT-2 settings and weights Load values into GPTModel 28: urlretrieve(...) 29: download_and_load_gpt2(...) Script 28 downloads gpt_download.py. Script 29 requests the model weights. 75
76. 29. Matching the pretrained architecture 29_load_pretrained_gpt.py Model Feature width Blocks Number of heads Width per head small 768 12 12 64 medium 1024 24 16 64 large 1280 36 20 64 xl 1600 48 25 64 Script 29 also sets context_length = 1024 and qkv_bias = True The current choice is GPT-2 XL, with context length 1,024 and QKV biases. 76
77. 29. Mapping checkpoint values to modules 29_load_pretrained_gpt.py + gpt_load_helper.py Checkpoint Destination in our code Required operation wte / wpe tok_emb / pos_emb Copy matching shapes c_attn weights and bias W_query / W_key / W_value Split into Q, K and V Linear-layer weights Linear layers in attention / FFN Transpose where needed All values assign(left, right) Error if shapes differ Match matrix names and axis order, check shapes, then copy the values. 77
78. 29. The same architecture, different weights 29_load_pretrained_gpt.py Step Model weights What the script demonstrates Script 23 Random initialization How the generation loop works Scripts 24 and 26 Trained on the-verdict.txt Loss reduction and generation changes Script 29 Public pretrained GPT-2 weights Weight conversion and loading The generation procedure stays the same after loading pretrained weights. 78
79. A sequence through the complete GPT pipeline Section Input Output Scripts Tokenization Text Integer IDs 01~05 Input representation IDs and positions Token vector 06~09 Context computation Token vector Updated token vectors 10~22 Generation Vocabulary scores One next token 23~29 Follow the text as it becomes IDs, vectors, scores and a new token. 79
80. Interactive visual references Transformer Explainer https://poloclub.github.io/transformer-explainer/ Explore tokenization, embeddings and changes in temperature. Brendan Bycroft's LLM Visualization https://bbycroft.net/llm Zoom into the matrices and repeated blocks of a language model. black7375's explanation (Korean) https://black7375.tistory.com/101 Review tokenizers, embeddings and the background of transformers. Explore the same computations interactively in these web visualizations. 80
81. Questions you can now explain How does a token ID become a vector? Why do we mask future tokens? Which numbers change during training? Why does generation repeat one token at a time? The appendix contains a script index and a glossary. 81
82. Appendix: script index 01–15 Scripts 01_open_verdict.py 02_tokenizer.py 03_w_simple_tokenizer.py 04_special_token.py 05_bpe.py 06_sliding_window.py 07_data_loader.py 08_token_embedding.py 09_position_encoding.py 10_self_attention.py 11_scaled_dot_product_attention.py 12_causal_attention.py 13_dropout.py 14_gpt_size.py 15_dummy_gpt_model.py Main concept Reading text and regex splitting Vocabulary and token IDs Encoding and decoding Unknown tokens and document boundaries GPT-2 BPE Inputs and targets shifted by one Windows and batches Token embeddings Learned position embeddings Simplified self-attention Q/K/V and scaling Causal masking Dropout Multi-head attention at GPT scale Dummy GPT architecture The concepts build on one another in script order. 82
83. Appendix: script index 16–29 Scripts 16_layer_normalization.py 17_layer_norm.py 18_gelu_test.py 19_feed_forward_test.py 20_example_deep_neural_network.py 21_transformer_block_test.py 22_gpt_model_test.py 23_text_generate.py 24_text_eval.py 25_sampling.py 26_sampling2.py 27_model_load.py 28_download_gpt_download.py 29_load_pretrained_gpt.py Main concept Normalization intuition LayerNorm class GELU and ReLU Feature transformation in the FFN Residual connections and gradients Transformer block Complete GPT model and size Generating one token at a time Loss, validation and training Temperature and Top-k Training, sampling and saving Checkpoint restoration Downloading the helper code Loading public GPT-2 weights The concepts build on one another in script order. 83
84. Appendix: glossary Term Meaning in this lecture Token A unit of text that the model processes Tensor An array of numbers organized along several axes Embedding A numeric vector associated with a token ID Attention A weighted combination of token information Logit A candidate-token score before softmax Loss A measure of prediction error for the target Gradient A derivative used to decide how to adjust weights Checkpoint A saved model or training state These terms connect the lecture to names used in the code. 84
85. Appendix: running the scripts Scripts Preparation or dependency 01~21 Run uv sync, then choose a standalone script 22 / 24 / 26 Prepare large-model or training runs before the lecture 27 Requires model_and_optimizer.pth from script 26 28 / 29 Prepare the helper, then download GPT-2 weights uv run python 05_bpe.py Run commands from the repository root, where the-verdict.txt is located. 85
86. Book and reference links Book: Build a Large Language Model (From Scratch) https://www.manning.com/books/build-a-large-language-model-from-scratch Official book code by Sebastian Raschka https://github.com/rasbt/LLMs-from-scratch Transformer Explainer https://poloclub.github.io/transformer-explainer/ Brendan Bycroft's LLM Visualization https://bbycroft.net/llm black7375's explanation (Korean) https://black7375.tistory.com/101 Click an underlined title or URL to open the source. 86