2026-10-08-belgium_nlp_meetup

This talk starts rather innocently introducing nanochat: a fully open, human-readable implementation of a simple LLM with GPT2 capabilities. We then use nanochat as an example to demonstrate how a bigger LLM can turn into an LLM engineer in an "autoresearch" loop. Finally, we'll show an early prototype of such an open-source LLM engineer, built by a human LLM engineer (me), let loose on a GPU cluster to perform genuinely open AI research.

Transcript

1. Open Source the LLM Engineer, not just the LLM Sofie Van Landeghem Human LLM engineer & Open-source maintainer October 2026, Belgium NLP meetup
2. nanochat • • • • Open-source implementation of an LLM Developed & maintained by Andrej Karpathy Code is minimal & hackable GPT-2 capabilities • Covers all major LLM stages: tokenization, pretraining, finetuning, evaluation, inference ◦ Not covered in depth in the repo: Reinforcement Learning ◦ Here we mostly focus on pretraining of a base model
3. https://github.com/karpathy/nanochat
4. Initial setup • Download the correct pytorch: • Base dir: ~/.cache/nanochat by default Will contain the training data, eval data, the tokenizer, model checkpoints and reports • Download the ClimbMix dataset
5. Track graphs & stats through W&B
6. Train a tokenizer • Separate rust project: https://github.com/karpathy/rustbpe • Similar to OpenAI's tiktoken, but supporting training as well
7. Train a base model • Pretraining: training the LLM to predict the next token • Not yet a chat bot (that would require supervised finetuning) • Use python (single GPU) or torchrun --standalone (multiple GPUs)
8. Evaluate the base model • Evaluate the model on the full validation set • val_bpb: bits per byte on the validation dataset (↓ is better) ◦ Cross-entropy loss, normalized per byte (vocab size independent) ◦ How accurately the model predicts the next token • CORE: bundles standard evaluation tasks (↑ is better) • Nanochat leaderboard: best wall-clock time to train a nanochat model to GPT-2 grade capability (CORE 0.2565)
9. Nanochat Leaderboard
10. The world is your oyster Fork the repo and experiment with anything! • • • • • • smaller/bigger models different tokenizer different data sets hyperparameter tuning improve the optimization … Try and beat the leaderboard!
11. … or have an LLM do it for you On this graph, each dot is an LLM-designed experiment that resulted in a lower val_bpb
12. Autoresearch • An even more compact version of the nanochat training code, condensed into one file: train.py • Also developed & maintained by Andrej Karpathy • Runs on a single GPU • Serves as a minimal playing ground to let LLMs experiment autonomously as actual LLM engineers • Comes with a program.md that explains the goal & setup https://github.com/karpathy/autoresearch
13. Applying autoresearch principles to nanochat
14. Faberon • • • • • • • • • Developed @Sevren: a new open-source, European AI lab Faberon lets an LLM performs autonomous LLM research Designed to be durable and auditable Works on top of a cluster or through local subprocesses Checkpoints every step to survive reboots/restarts Works with any LLM, including local and open ones Every LLM/human decision is recorded in a database Agnostic to work with any kind of repo that can use "optimization" Currently in a very early beta (v0.3.0) https://github.com/sevren-ai/faberon
15. Faberon architecture
16. Run Faberon • faberon serve starts the FastAPI server (with DBOS) that creates campaigns and runs research loops against a Slurm or local executor • Create a campaign from a research plan, which stipulates the target command (e.g. uv run train.py) and metric (e.g. val_bpb) • Use the CLI to list campaigns, show details, view events, etc.
17. Event ledger All details and the LLM's reasoning are recorded, too:
18. Faberon Chat • A Pi extension that has the Faberon context • Knows the same tools the CLI does, e.g. 'list campaigns" • Super user-friendly to summarize running campaigns, tried experiments etc.
19. Inject ideas • Faberon loops by itself and its LLM model proposes experiments • But humans can observe, analyse and propose ideas as well • The chat even guides you to that option when relevant! • Human ideas are recorded in the ledger for the LLM to pick up on
20. Thank you! Github: svlandeg Let's connect: https://www.linkedin.com/in/sofievanlandeghem My own-woman freelancing company: https://oxykodit.com/ "The Diff & The Merge" blog: https://oxykodit.substack.com