Where does the rigor go? Research software and the future of trustworthy science.

Generative AI now makes it dramatically easier to produce something that looks like research: analysis code, figures, literature reviews, even whole papers. The mechanisms research relies on to establish trust were built for a slower, human-paced enterprise, and they are struggling with the volume. Software got here first: this community has a two-year head start on questions the rest of research is only beginning to ask, and RSE work is already changing shape, away from writing code for researchers and toward specification, verification, and the judgment that makes AI-assisted work trustworthy. I'll share observations from my perspectives as a funder, a journal editor, and a long-time member of this community and offer some thoughts on where research software engineering goes next.

Transcript

1. WHERE DOES THE RIGOUR GO? Research software and the future of trustworthy science Arfon Smith · RSECon26 · 9th September 2026
2. Rigour, rigor? https://aicoding.leaflet.pub/3mbrvhyye4k2e
3. Rigour, rigor?
4. In many domains, faster science is a good thing. Unverified science is not.
5. (Generative) AI is here to stay Even if model progress stopped completely today, the world has already changed permanently. I'm not mourning, but I am worried… Assumptions and biases This room likely contains grief, enthusiasm, anger, denial and exhaustion. The KPI of research software is research New knowledge is the key performance indicator, and not the software itself. Sorry. I'm “Claude-pilled” I use these tools heavily and find them genuinely useful. These slides do not represent the position of Schmidt Sciences, but I'm happy to discuss how our funding strategy is evolving.
6. OVERVIEW Rough plan for today – Part 1 Give a high level overview of where I think we are at. – Part 2 Provide some commentary on what this might mean. : : : – Part 3 Make recommendations on how and where to contribute in the future.
7. Part 1: Something has changed
8. PART 1 · SOMETHING HAS CHANGED AI capability progress is real
9. Task-completion time horizon https://metr.org/time-horizons
10. Task-completion time horizon https://metr.org/time-horizons
11. Task-completion time horizon https://metr.org/time-horizons
12. Stanford HAI (Human-Centered Artificial Intelligence) https://hai.stanford.edu/ai-index/2026-ai-index-report
13. Stanford HAI (Human-Centered Artificial Intelligence) https://hai.stanford.edu/ai-index/2026-ai-index-report
14. Evidence that models can do something like discovery… https://rowstron.github.io/ARIA/albert.html
15. Evidence that models can do something like discovery… https://rowstron.github.io/ARIA/albert.html
16. Evidence that models can do something like discovery… https://openai.com/index/model-disproves-discrete-geometry-conjecture/
17. PART 1 · SOMETHING HAS CHANGED The agentic coding threshold (a.k.a. the Claude Code moment)
18. Passing the agentic coding threshold – Late 2025 we had the “Claude Code moment”: Multi-step coding (and reasoning) becomes useful enough to change behaviour. – First model to break 80% on SWE-bench Verified: 80.9% (Opus 4.5) – Anthropic Economic Index: Coding dominates API use and shifts from repair tasks toward creation (i.e., higher order activities). – Implications for software engineering: Increasing fraction of task-level engineering work being automated. – Implications for (at least some RSE work): A suitably motivated PI can solve their own problems with less significantly less support. https://www.anthropic.com/research/economic-index-primitives
19. PART 1 · SOMETHING HAS CHANGED Communities (and platforms) are undergoing significant disruption
20. GitHub “Since April, monthly commits have grown from 1.4 billion to 2.9 billion.” https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/ https://metr.org/time-horizons
21. GitHub platform availability https://mrshu.github.io/github-statuses/
22. Increased controls over pull re uests (including disabling entirely) q https://github.blog/changelog/2026-02-13-new-repository-settings-for-configuring-pull-request-access/
23. Low uality bug bounty submissions q https://daniel.haxx.se/blog/2026/01/26/the-end-of-the-curl-bug-bounty/
24. Agents behaving badly (autonomous “scientific coders”) https://github.com/matplotlib/matplotlib/pull/31132
25. Agents behaving badly (autonomous “scientific coders”) https://github.com/matplotlib/matplotlib/pull/31132
26. Agents behaving badly (autonomous “scientific coders”) https://github.com/matplotlib/matplotlib/pull/31132
27. Impacts on individuals and teams https://newsletter.pragmaticengineer.com/p/the-impact-of-ai-on-software-engineers-2026
28. PART 1 · SOMETHING HAS CHANGED tl;dr – A dramatic increase in production, without a corresponding increase in the capacity of communities to absorb it
29. What type of developer are you? https://blog.lmorchard.com/2026/03/11/grief-and-the-ai-split/
30. What are you feeling? Grief, denial, enthusiasm, exhaustion? – AI is exposing a difference that has always been there, but historically was invisible. Essentially “craft lovers” and “results people” were indistinguishable. – Craft lovers really are mourning a loss (Nolan Lawson – “We Mourn Our Craft”). This is very real. – Some (of Orchard’s) early fears didn’t meet with reality (sees AI primarily as a change in abstraction level for solving problems). – Griefs that *are* sticking (for the author) centred around the open web ecosystem, AI training on the commons, consolidation of power, change in careers. https://blog.lmorchard.com/2026/03/11/grief-and-the-ai-split/
31. What happens when the capacity to produce software grows much faster than the capacity to review and absorb it?
32. PART 1 · SOMETHING HAS CHANGED …and this is now arriving in research
33. First (disclosed?) peer-reviewed scientific publication (2025) https://sakana.ai/ai-scientist-first-publication/
34. We are now benchmarking agents on doing scientific work Agent attempts to reproduce core scientific results Agent starts from the paper and attempts to reproduce the work
35. But “doing science” contains very different kinds of work TERMINAL-BENCH SCIENCE GENEBENCH PRO Can the agent execute a scientific Can the agent make good scientific workflow? judgments? Scope Real computational research tasks Messy real biological datasets Activity Terminal + code + scientific tools to Minimal instruction with requirement to generate known artifacts choose analyses and react to diagnostics Inspectable artifacts with objective Tests chains of scientific judgment What is it? Scoring basis graders GPT 5.6 → GPT 6 22.4% → 64.6% 28.7% → 37.8%
36. Progress most rapid where outputs can be verified – Andrej Karpathy: Verifiability is what drives the “jagged frontier” of AI capability. Software 1.0 automates what you can specify, Software 2.0 automates what you can verify (e.g., through mechanisms such as reinforcement learning). – Physical science requires real world: You can’t unit-test a wet-lab result. This is the rationale behind the huge investments in robotics and lab automation. – Model Hardware Standard (MHS): Anthropic and HHMI Janelia previewed a new specification that allows AI agents to operate laboratory instruments safely. – Dramatic progress with software because it’s unusually verifiable: What happens when the same production economics arrive in domains where verification is expensive and hard?
37. Great! So our job is to make all of science more verifiable so that AI can automate more of it?
38. Perhaps we don’t want AI to progress? Perhaps we don’t want to use AI? – Hogg argues astrophysics has no commercial value at all. – AI increasingly can do astrophysics. – Its purpose is the people who do it. – Automating it wouldn't accelerate the discipline, rather it would remove the point of doing it.
39. Much of science remains expensive to verify. Which parts of science we make cheaper to verify, and why, is a choice.
40. : Part 2 What breaks?
41. Production and verification have decoupled – Demirer, Musolff & Yang (2026): Show that the gains attenuate as you move up the production hierarchy (+180% commits, +50% completed projects, +30% releases). – Raw generation → realised value disconnect: Writing code has become dramatically cheaper. Shipping, verifying, and maintaining working software has not. – Rigour has to move elsewhere (Fowler thesis): Friction of production used to do some hidden rationing for us. When production becomes cheap, verification must become an explicitly engineered system or process. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools doi: 10.2139/ssrn.6843118
42. “AI is an accelerant, not the kindling. The kindling is publish or perish.” — Ivan Oransky, Retraction Watch https://cen.acs.org/policy/publishing/ai-fraud-science-journal-generative-ai/104/web/2026/07
43. We were already optimising proxies for good science – AI is an amplifier of existing behaviour and incentives: The incentive system already rewarded volume over rigour before AI arrived. – We already rewarded the most measurable things: Papers, citations, grant income all measurable, all celebrated. – These are imperfect stand-ins for the things we actually value: Trusted, reliable knowledge, insight, cumulative progress.
44. “We have allowed execution to substitute for thought” Large language models are not the problem (2026) — Hiranya Peiris https://arxiv.org/abs/2604.22071
45. The old trust signals are on life support – Signals we used: Well-written prose, thorough documentation, tidy repositories, passing CI, lots of tests, good overall presentation. – Many were always just proxies: Represented costly signals of effort (and care). Producing them implied: time, attention, familiarity with the work, dedication to a quality product. – What has changed: Generative tools can produce the outward signals cheaply, which means the information content drops for everyone. – Papers were always advertising ( Claerbout/Buckheit/Donoho): Advertising is becoming nearly free whereas the scholarly work is still expensive to verify.
46. The problem isn’t that AI will produce “wrong” science. It’s a dramatic increase in the number of possibly correct things to check* * Mollick
47. https://www.linkedin.com/posts/emollick_if-the-the-predictions-that-ai-will-create-share-7393306227373477889-wBs8/
48. Journals and conferences are living the software timeline DECEMBER 2022 OCTOBER 2025 Stack Overflow bans AI-generated answers arXiv CS restricts review/position papers R AT I O N A L E R AT I O N A L E Volunteer moderation being overwhelmed by cheap, plausible content. Implemented due to the unmanageable influx of review articles and position papers to arXiv CS
49. Journals and conferences are living the software timeline ~3 years DECEMBER 2022 OCTOBER 2025 Stack Overflow bans AI-generated answers arXiv CS restricts review/position papers R AT I O N A L E R AT I O N A L E Volunteer moderation being overwhelmed by cheap, plausible content. Implemented due to the unmanageable influx of review articles and position papers to arXiv CS
50. Journals and conferences are living the software timeline “This is the right time for the SE community to demand more rigor in the submission and reviewing process that would help reduce the number of submissions while maximizing the quality of the accepted papers.” Marcel Böhme https://www.linkedin.com/posts/marcelboehme1_icse-share-7483140119391195136-SmfT
51. Convergence around the generation ↔ verification gap – Google DeepMind – validation bottleneck: Recommends that funders should invest in experimental validation infrastructure and automated laboratories. – Auditability gap survey: Bottleneck is shifting from whether an agent can perform research tasks to whether reviewers can verify the claims the agent produces. – PLOS “checkability": Distinguishes checkability from correctness or guaranteed reproducibility. Checkability creates the conditions for verification. – Argonne / DOE scientific-computing workshops: Trust, validation and traceability as core design requirements; workforce development centred on judgment. https://deepmind.google/public-policy/conjecture-machines-ai-agents-and-the-new-validation-bottleneck-in-science https://arxiv.org/abs/2608.05179 https://explore.plos.org/practical-pathways-to-open-science https://arxiv.org/abs/2608.26519
52. Reproducibility is getting cheaper to implement V E R S I O N C O N T R O L → E N V I R O N M E N T → T E S T S → D O C S → PA C K A G E D W O R K F L O W Agents can now produce much of this mechanical layer quickly
53. So we can raise the bar: from presence to outcomes OLD BAR NEW BAR Evidence of good practice Evidence of a reproducible result Has code Has data Has tests Has environment Has docs Can another human or agent actually regenerate the result from what is provided?
54. When production scales, reproducibility becomes essential for progress
55. : Part 3 How and where to contribute
56. What is scarce is changing…
57. Supporting successful adoption of agentic tools by others “Helping people prompt" is not the job. Think the Carpentries curriculum, rebuilt for agentic workflows involving teams of humans and AIs. Curation is still a scarce good Curated skills lift agent pass rates from 34% to 50%, and self-generated skills provide no benefit. Models can't author the procedural knowledge they benefit from consuming. OPTION 1 Encode judgement Capture (and describe) critical knowledge MHS pilot: a human explained the physics of bubbles once and encoded it as a reusable liquid-handling skill. Only the person who knew the failure mode could have written it. Being embedded is key Domain-embedded RSEs are extremely well positioned because encoding judgment requires actually knowing the domain. Some fields will automate less, or resist on principle. This is totally legitimate, but "finding the place automation arrives last" feels risky.
58. Provenance, evidence traces, recorded and shared Proof of Insight (PoI) is a specification example for highly regulated domains. Build the composable scholarly record Figures, datasets, code, methods, claims and provenance as first-class versioned objects rather than context described only in prose. OPTION 2 Build verification infrastructure Benchmarks and datasets are infrastructure CASP and PDB are what made AlphaFold possible. Albert and ReplicationBench as examples from today. “Data Science at the Singularity” from Donoho is a great read here. More structured scientific process knowledge as public data Models learn from structured representations of process, not piles of finished content. See gpt-3.5-turbo-instruct’s chess performance. Does your science domain have an open process record? https://proofofinsight.org/spec/v0.7.0/ Data Science at the Singularity: https://arxiv.org/abs/2310.00865
59. Make the world checkable in-silico Better simulators and scientific foundation models that enable verification at compute speed instead of bench speed. Shorten the verification loop through the physical world Lab automation and instrument interfaces: MHS cut integration from weeks to hours, and feedback loops now run in parallel overnight. OPTION 3 Connect agents to reality Make more argument machine-checkable Formalisation tools such as Lean and mathlib are doing for maths proofs what CI did for code. Newly demands for new interfaces Instrument drivers, orchestration layers, encoding safety limits, and designing new protocols. None of these routes exists without engineers.
60. Nullius in verba (take nobody's word for it) Verification by an opaque authority reintroduces the trust in authority the scientific method exists to remove. Whoever holds the process data holds the future capability If all researchers used closed AI systems, those systems will accumulate capabilities the fastest. PRINCIPLES Keep the verification layer open The exoskeleton exists to build on The work here is maintenance and investment of things we already have (DOIs, Crossref, ORCID, OpenAlex, arXiv, Retraction Watch) – Tim O’Reilly. We must avoid scientific trust becoming a subscription product Elsevier, Clarivate, Springer Nature, Wiley are all building products to deal with the flood of AI-amplified human activity.
61. RSEs are unusually well placed to contribute A two-year head start, membership of several communities at once, and incentives that differ from academics'. tl;dr – the broader perspective needed. Add nuance to the conversation “Everyone for AI is too for it, everyone against is too against it” – Daniel Jalkut. RSEs see the capability and failure modes – share what you’re learning. PRINCIPLES Help establish norms The mathematicians just showed us all how it's done The Leiden Declaration:, published June 2026, now has 3700+ signatures. Recommendations for individuals, journals, funders, policymakers and AI companies Notice who was in the room Mathematicians, computer scientists, philosophers, historians, people working on formal verification methods. Cross-community people wrote the norms. https://leidendeclaration.ai
62. RSEs are unusually well placed to contribute A two-year head start, membership of several communities at once, and incentives that differ from academics'. tl;dr – the broader perspective needed. Add nuance to the conversation “Everyone for AI is too for it, everyone against is too against it” – Daniel Jalkut. RSEs see the capability and failure modes – share what you’re learning. PRINCIPLES Help establish norms The mathematicians just showed us all how it's done The Leiden Declaration:, published June 2026, now has 3700+ signatures. Recommendations for individuals, journals, funders, policymakers and AI companies Notice who was in the room Mathematicians, computer scientists, philosophers, historians, people working on formal verification methods. Cross-community people wrote the norms.
63. Research Software Catalyst Fund
64. Introducing the Research Software Catalyst Fund https://researchsoftwarecatalyst.fund
65. Research Software Catalyst Fund: Program Details – Eligibility: Individuals, informal teams, institutions, and non-profits, anywhere in the world. – Origin story: Generative AI in the RSE Workplace and Research Software Engineering in the Age of Generative AI: Building a Community Vision workshop are sources of inspiration but do not limit project scope or participation. – What it funds: Tools-building, writing sprints, curriculum development and training, conference workshops, supporting communities of practice. – Fiscally sponsored: Applicants who cannot receive funds through an institution can host their fund directly at Hack Club Bank. https://researchsoftwarecatalyst.fund
66. Research Software Catalyst Fund: Award Categories Award size Deadline Expected duration Expected # of projects EXPLORATORY FUND (COMING SOON) IMPACT FUND (PLANNED) Up to $50,000 Up to $200,000 Mid-November 2026 Q1 2027 3 — 6 months 6 — 12 months 10 — 20 ~3 — 5 Grants for the people building, verifying, and teaching research software – and for the norms, tools, methods, and training the community needs to use generative AI well. https://researchsoftwarecatalyst.fund
67. Wrapping up
68. Even though this looks scary, we are still early on this exponential
69. Science will no doubt produce more outputs Whether it compounds into more knowledge is up to people like you.
70. Thank you