them. A data engineer’s excuse to dabble in data science for a few months. EDITORS VIEWERS THE NEWSROOM Find footage fast Relevant recommendations Know what we publish
finance crime / law / justice arts / culture / entertainment / media disaster / accident / emergency science & technology environment labour education lifestyle & leisure religion society conflict / war / peace human interest weather
automatic speech-to-text LANGUAGES 60+ “Ja, maar nee. Ik moet nog even wat dingetjes doen. Loopt hij? Ja, hij loopt. Ik sta aan de andere kant van de camera.” “Yeah, but no. I still need to sort a few things out. Is it rolling? Yeah, it’s rolling. I’m standing on the other side of the camera.” • 62% Dutch • 22% English • rest — the long tail Yes, this has a topic label. Kept intentionally messy, real-world data.
Anthropic majority vote Ties → settled by LLM confidence : each judge also reports how sure it is of its own answer, corrected for how generous each judge usually is.
or anthem on screen doesn’t make it political — the astronaut rule 4 Real politics stays politics elections, voting, coalition still count while avoiding overcorrection 2 Science stays science researchers discussing their own work, even if funding or officials come up 5 When in doubt, go specific two categories both fit? pick the narrower one — ‘labour’ over ‘economy,’ ‘weather’ over ‘environment’ 3 Label the subject, not stray words a video about heather bouquets is lifestyle, even if a minister is mentioned once 6 ‘Uncertain’ — an 18th label an escape hatch instead of a forced guess “An LLM-council is only as unbiased as the question you ask it.” Proof it worked — agreement with an independent off-the-shelf model: 9% 69%
OUR DATA accuracy — on their own written-news data classla — the open-source IPTC classifier by Taja Kuzman & Nikola Ljubešić (2025, IEEE Access) 69% accuracy — run completely unmodified same model, unmodified Decent. But we wanted more.
every time — only the starting point changes. Off-the-shelf From scratch Warm-start the classla model from before (Kuzman & Ljubešić) same backbone trained only on our data classla’s head start + our data
flooding a random sample; even religion (232 rows) keeps its seats in every split ▪ train 80% ▪ validation 10% ▪ test 10% Split once, reused for every experiment. Off-the-shelf classla only touches the test 10%. Languages checked, not assumed Dutch ≈ 59% in each of the three splits; the other top languages land just as evenly
fuzzy Weaker in two low-data languages Ground truth is still AI-labelled the first 512 tokens (≈400 words) humans only agree ~73% here too Arabic and Ukrainian human gold standard in progress
~200 hand-labelled videos to check our own homework Deeper IPTC levels from 17 broad topics to ~100 subcategories Multi-label topics some videos really are about two things at once
split, 3,859 rows · same model as the results slide · per-topic = share of that topic’s videos labelled correctly PER TOPIC PER LANGUAGE politics 87% conflict, war & peace 88% crime, law & justice 90% arts, culture & media 81% economy & finance 86% lifestyle & leisure 76% disaster & emergency 88% health 84% sport 92% society 67% science & technology human interest 53% religion English 86% German 89% French 83% Spanish 87% Ukrainian 73% 64% overall 84% 90% weather labour 84% Arabic 74% environment education Dutch 81% 74% 78% 74% overall 84% Most topics sit in the 80s. Weakest are the fuzzy ones — human interest and society — where humans also disagree, plus small classes like religion and education. Weakest languages: Arabic and Ukrainian, least training data.
not the model?” Fair: 9% is agreement, not accuracy. Agreement between two annotators is symmetric — if classla and the council disagree on 91% of rows, either one could be the broken one. Calling it “9% accuracy” already assumes the labels are right. Three things told us it was the labels: 1 · THE SHAPE 2 · THE ROWS 3 · THE FIX Every disagreement was one class Humans sided against the council Model unchanged, labels changed The council said “politics” for almost everything. One class swallowing all others is a labelling failure, not a classifier failure — a broken classifier scatters errors across 17 classes. We read the unanimous “politics” rows. The astronaut, the heather bouquets, the researchers — 4 out of 5 were not politics. A human settles that instantly. Same classla model, zero retraining. Against v1 labels: 9% agreement. Against v2 labels: 69% — right next to its own published 73%. Only the labels moved. Follow-up “Is the 84% also just agreement?” — Yes, with the corrected labels on a held-out split. Whether that is true accuracy is what the 200-row human gold standard is for.
a small model?” Because the two jobs are different. Labelling is a one-off; prediction runs on every new video, every day. Expensive, careful teacher — cheap, fast student. That is the title of the talk. ONE-OFF vs FOREVER COST & SPEED DETERMINISM Labelling happens once ~100× cheaper, milliseconds Same input, same label 40,000 rows, five models, a few days, a few hundred euros. Prediction is forever: every incoming video, in production, at pipeline speed. A five-model council on every video would be orders of magnitude slower and pricier than a 560M-parameter classifier on one GPU. The small model is deterministic and gives a calibrated confidence. LLM calls are not — and we need the confidence gate to route uncertain videos. Same recipe as Kuzman & Ljubešić — GPT-4o as teacher, XLM-RoBERTa as student — except we used five teachers and made them vote.
same prompts?” Individual LLM calls are non-deterministic, so a single model would flip some labels. That is exactly why we didn’t use a single model. THE MECHANISM WHICH ROWS MOVE HONEST GAP Five votes absorb one wobble The 3–2 votes No full re-run — yet A label only flips if the row was already contested. At 81% four-of-five agreement, most rows have a margin of at least two votes. The unstable rows are the split decisions — the fuzzy ones where two humans would disagree too. Vote margin is stored per row, so we know exactly which labels are fragile. We measured a bigger perturbation: changing the prompt moved 89% of labels. Sampling noise under fixed rules is far smaller. The 200-row gold standard is the fixed reference for any re-run. If asked for a number: “We haven’t re-run the full set. A 500-row re-run is on the list — it costs a few euros and gives an exact flip rate.”
No resampling, no class weights. The imbalance is real and we chose to see it rather than hide it. IMBALANCE IS REAL WHAT WE DID INSTEAD WHERE IT HURTS Politics is 22%, religion 0.5% Stratified split + better labels Fuzzy classes, not small ones Oversampling religion teaches the model to over-predict a topic that almost never occurs — the astronaut problem in reverse. Stratifying by label keeps religion’s ~20 test rows so weakness is visible. Fixing the labels moved agreement 9% → 69% — more than any sampling trick. The weak topics are human interest and society, not the rare ones. Resampling would not fix what is actually broken. Class weighting stays on the list if the gold standard says otherwise. CHECK before presenting: confirm no class weights were used in the loss. If they were, say so — the argument is the same.
of its scoring the model puts on the winning topic. All videos, sorted by the model’s confidence → Strict bar 96% accurate on ⅔ of videos Looser bar 91% accurate on 82% of videos
mark3t1ng
eitanlees
swwweet
codingconduct
samanthasiow
irinanazarova
soysaucechin
portentint
philnash
sergeychernyshev
thekraken
destraynor
chriscoyier
