Easy to Guess, Hard to Verify: Lessons from AIMO 3 for Olympiad-Level AI Mathematics

Recent LLMs are approaching and surpassing world-class mathematical ability. Identifying the correct answer given problems and traces that the average human cannot solve becomes a central challenge in achieving artificial superintelligence. Solving a problem with LLMs involves two steps: generating candidate solutions and verifying them. Through our AIMO 3 effort, we found that the relative difficulty of these steps is task-dependent and that competition mathematics is "easy to guess, hard to verify," in contrast to tasks like integer factorization or multi-hop QA benchmarks, where a single trace can be checked easily. This talk introduces existing approaches and discusses challenges for future AI-for-Math research. It also examines competition dynamics — why optimizing the mean score loses to optimizing the upper confidence bound — and the practical challenges of training math models, concluding with our publicly released model and SFT dataset.

Transcript

1. Easy to Guess, Hard to Verify: Lessons from AIMO 3 for Olympiad-Level AI Mathematics July 21, 2026 1
2. Self Introduction • Kosuke Nakago @corochann • Preferred Networks 2016~2024 ◦ Deep Learning for Manufacturing industry ◦ PLaMo-100B post-training • Sakana AI 2025~ ◦ 2025: AI Agent development for finance domain ◦ 2026: Namazu model development • Kaggle 2x Grandmaster (Competitions & Notebooks) 2
3. Agenda • Understanding task-specific behavior: “Easy to Guess, Hard to Verify” • Competition design is important: High variance approach was strong • FishMath model training approach: Improving gpt-oss-120b math performance with own dataset 3
4. AIMO 3 task understanding Possible to Guess, Hard to Verify characteristic 4
5. AIMO 3 Competition overview • Solving very difficult math problems up to International Math Olympiad level, which is global high-school math competitions. • Answer is integer [0, 99999], can be verified. • Overall Progress Prize Winner: $1.5M who achieved 47/50 score https://www.kaggle.com/competitions/ai-math ematical-olympiad-progress-prize-3 5
6. AIMO 3 Competition overview ✅ Can you solve the problem below? https://www.kaggle.com/competitions/ai-math ematical-olympiad-progress-prize-3 6
7. AIMO 3 Competition overview ✅ Can you solve the problem below? ✅ When you are given the solution approach, can you confidently verify that it is correct? https://www.kaggle.com/competitions/ai-math ematical-olympiad-progress-prize-3 ・・・ 7
8. Challenges • Recent LLMs are capable of solving math problems world-class difficulty • Normal human cannot solve the problem, nor verify its correctness • Current research topic is how to achieve superintelligence 8
9. Approach • Most common approach in AIMO 3, in time constraint problem setting • Parallel sampling → Majority voting (or entropy weighted voting) Majority Voting Entropy Voting Parallel Sampling 9
10. Pass@N VS Maj@N or Best of N • Pass@N is theoretical upper bound, not the actual achievable performance • Maj@N, Best of N is the actual achievable performance → How big is the difference? 10
11. Pass@20 was high in AIMO 3 • Even OSS Model can achieve grand prize level score 47/50 with Pass@20 • But nobody achieved point 47 in the competition, the bottleneck is the gap between Pass@N and Maj@N. https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559 11
12. Easy to Guess, Hard to Verify task Higher gap Pass@N - Maj@N NP Hard Problem AIMO 3 Ex. Traveling Salesman Hard to Verify Deep search task Ex. BrowseComp Easy to Verify NP Problem Ex. Integer Factorization Lower Pass@N Easy to Guess Hard to Guess 12
13. Easy to Guess, Hard to Verify task Higher gap Pass@N - Maj@N NP Hard Problem AIMO 3 Ex. Traveling Salesman Hard to Verify Synthetic Q&A generation is difficult Deep search task Synthetic Q&A generation is feasible Ex. BrowseComp Easy to Verify NP Problem Ex. Integer Factorization Lower Pass@N Easy to Guess Hard to Guess 13
14. Majority Voting characteristics • Majority voting converges to majority answer (A) • If the actual answer is minority (B or C), increasing N leads to score 0. Worse than single trace guessing. Monte carlo simulation when the LLM answer A, B or C in the specified probability and select each answer after majority voting 14
15. How to fill the gap? Challenge is to verify (select) the trace with the right answer • Approach 1. Model training → Talk later • Approach 2. Test time scaling, focusing on verification 15
16. Test time scaling approach - overcome majority voting • AIMO 2’s winner NVIDIA Team suggested GenSelect. ◦ https://arxiv.org/abs/2507.17797 [Message] It is important to compare the traces to find the right solution, instead of evaluate each trace separately. Latency is constrained by tree’s maximum depth Parallel Sampling (Ex. Majority voting) Tree-based Search (Ex. AB-MCTS) Low latency but no exploitation Computation budget efficient but not optimal for latency GenSelect Tournament-style selection; Effective but limited parallelism in later stages RSA / SAR (Ours) Highly parallelizable Best for latency-first workload 16
17. Test time scaling approach - overcome majority voting • Recursive Self Aggregation ◦ ◦ https://arxiv.org/abs/2509.26626 https://www.kaggle.com/competitions/ai-mathe matical-olympiad-progress-prize-3/writeups/rec ursive-self-aggregation-with-difficulty-aware-s Also utilizes multiple traces in previous round, to generate refined solution. Reported that the method can converge to Pass@N behavior with enough steps. Latency is constrained by tree’s maximum depth Parallel Sampling (Ex. Majority voting) Tree-based Search (Ex. AB-MCTS) Low latency but no exploitation Computation budget efficient but not optimal for latency GenSelect Tournament-style selection; Effective but limited parallelism in later stages RSA / SAR (Ours) Highly parallelizable Best for latency-first workload 17
18. Test time scaling approach - overcome majority voting • Blackboard approach by @ippeiogawa ◦ Execution intermediate sharing among multi-agents by sharing the code execution in push notification https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/aimo3test How to orchestrate multi-agent solutions? is the recent approach trend 18
19. Summary • AIMO 3 task is “Easy to guess, hard to verify” task • Recent research suggests that we need to compare multiple traces to select right answer in these type of questions. Discussion • How is the characteristic of proof competition? 19
20. AIMO 3 competition dynamics Why did high variance solution win? Optimizing UCB over expected mean score 20
21. High variance in AIMO 3 submission • Big shake occurred in the final result. Why? • The best scored Public Notebook had high variance https://www.kaggle.com/code/nihilis ticneuralnet/44-50-let-me-over-cook https://www.kaggle.com/competitions/ai-mathematica l-olympiad-progress-prize-3/discussion/683612 21
22. Why high variance, how to mitigate this? • Time management was difficult & highly stochastic ◦ Random sequence of problem, need to solve sequentially • What is the most lucky scenario? ◦ ◦ ◦ Easy problem: Use short time, just decide with few samples Difficulty Medium problem: Use longer time, get majority answer Difficult problem: Use relatively short time, pray the luck to obtain correct answer in first few samples, instead of waiting majority answer. • If you can allocate calculation budget more, score will stabilize a bit. ◦ Popular approach: Use Speculative Decoding (EAGLE3, DFlash) to have more calculation budget Aggressive time management High variance, lower mean Conservative time management Low variance, higher mean 22
23. Competition submit history of other participants • Increased expected mean and reduce variance https://www.kaggle.com/competitions/ai-mathematical-olympiadprogress-prize-3/writeups/every-improvement-failed-aimo3 https://www.kaggle.com/competitions/ai-mathematic al-olympiad-progress-prize-3/writeups/aimo3test 23
24. High variance solution VS Low variance solution • Which do you submit in private LB? 24
25. High variance solution VS Low variance solution • High variance solution wins in UCB • If 200 participants submit high variance approach, 4 people reach 45. There’s almost no chance that high-mean solution wins. Only here matters to win a prize 25
26. Other competitions • March Machine Learning Mania Competition ◦ Predicts NCAA Men's and Women's Division 1 college basketball tournaments results • Usual winning solution include “a bet” that which team wins in the earlier stage • You need to deliberately include luck factor to win the competition (=to have high UCB). 26
27. Other competitions • Market Prediction Competition: JPX competition etc. https://www.kaggle.com/competitions/jpx-tokyo-stock-exchange-prediction/discussion/335393 • Importance of competition design to reduce the luck factor ◦ https://note.com/tomoya_kitayama/n/nfe267fb95bab 27
28. Next AIMO Competition design ideas • Reduce variance factor in competition design Show all problem from the beginning, no sequential restriction Reconsider Answer format, evaluation metric • • Remove the “time-management” luck factor, which most reduces the execution uncertainty • • score variance • It can also allow maximize high-price GPU utilization. Only allow 1 answer increases the For example, let submitting multiple answer with their confidence • Cons: Not simple to understand Cons: LB probing hack 28
29. FishMath team solution Very quick summary of our approach and findings 29
30. Model Training - our approach • Team Writeup: “Pushing the Limits: Post-Training High-Capability Models under Strict Inference” • Speakerdeck presented at Kansai Kaggler event https://www.kaggle.com/competitions/ai-mathematical-olympiad-progr ess-prize-3/writeups/pushing-the-limits-post-training-high-capability https://speakerdeck.com/analokmaus/sotanosaranixian-he-yan-sii tui-lun-zhi-yue-xia-denogao-xing-neng-moderunopost-training 30
31. Model Training - our approach • Improving 20B class model is easy, 100B > was “very” difficult with the existing math datasets in research community • Gap between frontier labs and research areas SFT • RL (GRPO) Dataset: Re-generated traces by • frontier OSS models • • • Megatron-SWIFT, 4 node x 8 GPU TIS, No KL loss • → 10B tokens Rollout batch 64 * 16 samples = 1024 global batch Max seq 65k Global batch 256 x 10 epoch Algorithm GRPO, Clip-Higher, DAPO, • • AdamW (β=0.9/0.98), LR 1e-6 Partial rollout for long tail traces 31
32. Model Training - published model & SFT dataset • 🤗 Model: SakanaAI/gpt-oss-120b-sft-aimo3-fishmath • 🤗 SFT Dataset: SakanaAI/FishMath-SFT-Data ◦ Include strong model’s traces to improve gpt-oss-120b performance ◦ Curated difficult problems that even frontier model struggles, including 🤗 ycchen/Crystal-Math-Preview 32
33. Model Training - future work ”Easy to guess, Hard to verify” task specific training may be one interesting direction • E2E optimization of Majority voting metric in RL • Harness aware (Ex. Recursive Self Aggregation) RL 33
34. Conclusion • Understanding task-specific behavior ◦ “Easy to Guess, Hard to Verify” tendency in AIMO 3 tasks. • Competition design is important ◦ High variance approach was necessary to win in this time’s situation. • FishMath model training approach ◦ How to prepare difficult QA & how to properly evaluate the performance improvement were challenging these days 🐡🐠🐟 Thank you for listening! 🐡🐠🐟 34