What Works and What Fails: Lessons from Benchmarking Local Coding LLMs

Abstract. Evaluating Large Language Models (LLMs) for code generation often reveals a significant gap between theoretical leaderboard scores and day-to-day development environments. This talk shares the findings, methodologies, and raw experiences from practical experience in running and evaluating local LLMs in the research domain.
We discuss the tools, infrastructure design, challenges faced, the successes achieved, and the lessons learned from benchmarking various LLMs for code generation tasks.

Note. Slides presented at the "seLIA: I Jornada sobre Software Libre e Inteligencia Artificial Abierta".

Transcript

1. What Works and What Fails: Lessons from Benchmarking Local Coding LLMs Fuenlabrada Campus, Universidad Rey Juan Carlos July 06, 2026 Speaker: Giovanni Rosa Postdoctoral Researcher Universidad Rey Juan Carlos This presentation is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
2. Evaluating Code LLMs What? Measure model capabilities How? Verify output via unit tests
3. The origin: Codex and HumanEval Key concept: Measuring Python program synthesis directly from docstrings using manually defined unit tests Output: The corresponding function implementation Chen et. al 2021
4. SWE-Bench: Real-World Complexity Key concept: Evaluate LLMs on actual GitHub tasks (bugfix, regression, new features, etc.) Task: Generate a patch based on an issue and the full codebase Evaluation: The patch is verified on the patch unit tests and full test suite Jimenez et. al ICLR 2024
5. Main elements of a bench instance
6. Code generation with AlphaCodium Ridnik et. al 2024 Rosa et. al BENEVOL 2025 Core idea: Using TDD principles to guide LLM code generation
7. Input Problem 1580 A Description Public Tests
8. CodeContests Dataset *Y. Li et al., Competition-level code generation with AlphaCode, Science 378 (2022) Problem description Public and private test cases Correct and incorrect solutions etc…
9. The AlphaCodium approach Ridnik et. al 2024 - https://arxiv.org/abs/2401.08500
10. pass@k (k=5, n>=5) The probability that at least one of the top k generated code solutions is correct Inference statistics Evaluation Metrics Zero-shot prompting vs AlphaCodium Response Latency Token usage Num. requests
11. Local Inference Infrastructure Nvidia A100 80GB Nvidia A100 80GB OpenAI compatible REST APIs AlphaCodium w/ Tensor parallelism Server
12. Inference Engines: vLLM vs. Ollama Best for local development Easy setup GGUF format and cloud models Does not scale for heavy loads and parallel computation Production scale Multi-GPU and cluster support Full precision + quantized models (AWQ/BitsAndBytes) Usually requires custom config
13. Execution example (1x NVIDIA A100 80GB)
14. Advanced Infrastructure settings
15. Common benchmarking pitfalls Overfitting and Knowledge bias Impact of harness and unfair model comparisons Input/output parsing failures Token generation loops
16. Common benchmarking pitfalls Overfitting and Knowledge bias Impact of harness and unfair model comparisons Input/output parsing failures Fix: output parsers (pydantic), auto request retry Token generation Loops Fix: Implement inference timeouts
17. How to design better benchmarks? Consider the non-functional requirements, such as • Token usage • Bias and fairness • Code quality Static analysis tools can be a starting point (pylint, SonarQube) Sabra et. al 2025 (Sonar) Rodriguez-Cardenas et. al FORGE 2026
18. Summary Giovanni Rosa Postdoctoral Researcher @ URJC More at giovannirosa.com This presentation is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
19. Acknowledgments More at https://advise.codeberg.page/ This work is part of the ADVISE project (ADvanced Vision on Intelligent Software Engineering) ADVISE aims to integrate AI agents into software development to ensure code meets requirements and aligns with developer intent. Funded by the Spanish AEI, reference 2024/00416/002