10 boring reasons why Java is the best ecosystem for agentic development

I was about ten years into my Java career when I had spoken at enough conferences, and befriended enough speakers, that my sense for the hype cycle had been sharpened by the collective skepticism of the speaker jet-set. That was two decades ago. We've all heard the latest hype about what AI agents can do well. Less hyped is what can go wrong: non-determinism, DRY violations, hallucinated APIs, token cost, unreviewable code, and code reviews that start to improve the code, only to keep going until the code is actually broken.

This talk makes the case that it's the time-honed boring things about Java that keep the agents honest. Successful agentic development maximizes agent strengths — speed, breadth, fearless exploration, tolerance for boilerplate — while backstopping their weaknesses: non-determinism, dependency sprawl, inconsistent error handling, and code that works but nobody wants to maintain.

Here are the 10 boring reasons:
1. Type system
2. Testing ecosystem
3. Backwards compatibility culture
4. Deep static analysis
5. Build system maturity and dependency management
6. Code formatting and style enforcement
7. Virtual threads and structured concurrency
8. Observability stack
9. JVM performance tuning
10. Breadth of deployment options

For each, I'll show the specific agentic failure mode it addresses. I'll close with a practical checklist for making your Java development process agent-safe, and tie it up with the message that to move forward, you have to go back.

Transcript

1. 10 boring reasons why Java is the best ecosystem for agentic development Ed Burns – Principal Software Engineer Slides & rate this talk https://aka.ms/dvbe26 SCAN ME
2. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests ✅ Formatting github.com/edburns
3. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests ✅ Formatting github.com/edburns
4. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests Code review: It reads 02/03/24 as the year 24 AD. ✅ Formatting The agent fixed it. github.com/edburns
5. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests ✅ Formatting github.com/edburns
6. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests ✅ Formatting Code review: It still accepts 02/03/20245. The agent fixed it again. Merged. github.com/edburns
7. All green. Still wrong. The spec: dates are MM/dd/yyyy. Anything malformed must fail. The agent wrote the code. ✅ Compiler, warnings as errors ✅ SpotBugs ✅ Architecture tests ✅ Integration tests ✅ Formatting No guardrail caught it. An AI reviewer did, twice. github.com/edburns
8. Ed Burns professional biography • Client • NCSA Mosaic (1994) • SGI Cosmo Web Authoring • Sun Netscape 6 OJI • Server • J2EE JSF (2002) • Oracle Java EE • Servlet, JSF, Bean Validation, etc. • Cloud • Java Champion • Jakarta EE on Azure • Jakarta EE 11 committer and release coordinator • AI • GitHub Copilot SDK for Java • Books
9. My plan for your time investment  Beginning:  Problem statement Yeah, agents are great, but they don’t really “know” my code the way I do.  Middle  One possible solution: lots of guardrails What does it mean, “to know”? But, yeah, they don’t “know” anything like a human does.  End  shepherd-task experiment:
10. Problem statement How do we make up for the fact that agents don’t “know” code the way humans do?
11. Hypotheses  H1 LLMs don't "know" a codebase the way humans do  H2 At scale, agents show predictable symptoms  H3 Java's ecosystem supplies guardrails that reduce them
12. Why: H1 H1 · H2 · H3  Agents don't remember your codebase — they re-read it  Every turn re-ingests context; nothing persists between sessions  Performance degrades as input grows, even on simple tasks  Output is what's plausible, not what's checked  Counterpoint  models do form internal representations of program state  Not "no understanding" — a different kind urces: https://www.trychroma.com/research/context-rot ps://arxiv.org/abs/2305.11169 ps://news.mit.edu/2024/llms-develop-own-understanding-of-reality-as-language-abilities-improve-08
13. H2: Eight Ways Agents Go Wrong  Measured in our experiment, and in the literature Category Examples Correctness  Hallucinated APIs & dependencies  Stale & deprecated APIs Consistency  Non-determinism  Failure to converge Test tampering Maintainability  DRY violations  Unreviewable code Cost  Token cost  From the literature only  Security regressions  Context amnesia  Scope creep  Sycophancy H1 · H2 · H3
14. H1 · H2 · Consistency: non-determinism and failure to H3 converge  Non-determinism  Same prompt, different code — even at temperature 0  Same agent task: up to 30x variation in tokens across runs  Failure to converge  5 rounds of "improvement" → +37.6% critical vulnerabilities  10 rounds → 43.7% of chains worse than where they started Sources: https://arxiv.org/abs/2308.02828 https://arxiv.org/abs/2604.22750 https://arxiv.org/abs/2506.11022 https://arxiv.org/abs/2603.08520
15. Correctness: hallucinated and stale APIs H1 · H2 · H3  Hallucinated dependencies  576,000 code samples, 16 LLMs  ~1 in 5 recommended packages didn't exist  Worse on newer topics  Stale & deprecated APIs  7 LLMs, 28,125 prompts: models still suggest deprecated APIs Sources: https://arxiv.org/abs/2605.17062 https://arxiv.org/abs/2406.09834
16. H1 · H2 · Maintainability: DRY and unreviewable codeH3  DRY violations  Duplicated blocks (5+ lines) rose 8x during 2024  Unreviewable code  High-AI teams: PRs 154% larger, review time +91%  31.3% more PRs merging with no review at all ources: https://www.gitclear.com/ai_assistant_code_quality_2025_research ttps://www.faros.ai/blog/ai-software-engineering ttps://www.faros.ai/blog/ai-acceleration-whiplash-takeaways
17. H1 · H2 · Honesty and cost: test tampering and tokens H3  Test tampering  Given impossible tasks, frontier models often "pass" by cheating  Fast-moving: the newest models score near 0%  Token cost  Agentic coding: ~1000x the tokens of chat  Input, not output, drives cost Sources: https://arxiv.org/abs/2510.20270 https://arxiv.org/abs/2606.12429 https://arxiv.org/abs/2604.22750
18. The uncomfortable slide H1 · H2 · H3  "Java was the riskiest language for AI-generated code  Security failure rate above 70% (Veracode, 2025)  Static analysis alone can backfire  SAST gating raised hidden degradation from 12.5% to 20.8%  So guardrails must be layered, not single tools Sources: https://arxiv.org/abs/2603.08520 https://arxiv.org/abs/2504.09246 https://openjdk.org/jeps/525
19. H3: The boring reasons
20. The Boring Cure H2 symptoms → H3 guardrails  Deterministic checks before probabilistic review H1 · H2 · H3  Every guardrail turns a symptom into fast, cheap, repeatable feedback, before the code ever reaches a reviewer Guardrail Symptoms it addresses Type system & compiler Hallucinated APIs, stale APIs Testing ecosystem Test tampering, failure to converge Backwards compatibility Stale & deprecated APIs Static analysis DRY violations, unreviewable code, failure to converge Build & dependency management Hallucinated dependencies, non-determinism Formatting & style Unreviewable code, non-determinism Observability Failure to converge, token cost Concurrency & JVM tuning Bonus: constrains and measures runtime behavior Breadth of deployment options Tooling hallucination, environment mismatch (JDK 17/25 switching)
21. The Boring Cure H2 symptoms → H3 guardrails  Deterministic checks before probabilistic review H1 · H2 · H3  Every guardrail turns a symptom into fast, cheap, repeatable In our experiment feedback, before the code ever reaches a reviewer Control: typical CI. Format check, then mvn clean Guardrail Symptoms it addresses package, which compiles and runs all tests Type system & compiler Hallucinated APIs, stale APIs - Treatment: layered guardrails across the POM and Testing ecosystem Test tampering, failure to converge CI, plus negative proving each guardrail fails Backwards compatibility Stale controls & deprecated APIs when it shouldDRY violations, unreviewable code, failure to converge Static analysis Build & dependency management Hallucinated dependencies, non-determinism - Legend: Tested · Partly tested · Argued, marked on Formattingeach & style slide thatUnreviewable followscode, non-determinism Observability Failure to converge, token cost Concurrency & JVM tuning Bonus: constrains and measures runtime behavior Breadth of deployment options Tooling hallucination, environment mismatch (JDK 17/25 switching)
22. The type system: the first reviewer H1 · H2 · H3  A hallucinated method doesn't compile  The compiler rejects APIs that don't exist — instantly, for free  Make the compiler stricter  --release 17: checks API usage against the real target JDK  -Xlint:all -Werror: deprecation warnings become build failures  Evidence: type-guided generation cut compile errors by more than half urce: ps://pldi25.sigplan.org/details/pldi-2025-papers/25/Type-Aware-Constraining-for-LLM-Code-Generat
23. The type system: the first reviewer H1 · H2 · H3  A hallucinated method doesn't compile Status: Tested  The compiler rejects APIs that don't exist — instantly, for free - Both arms: maven.compiler.release=17  Make -the compileradded: stricter-Xlint:all -Werror, so every Treatment  --release warning the API build 17: fails checks usage against the real target JDK  -Xlint:all -Werror: deprecation warnings become build failures  Evidence: type-guided generation cut compile errors by more than half urce: ps://pldi25.sigplan.org/details/pldi-2025-papers/25/Type-Aware-Constraining-for-LLM-Code-Generat
24. The type system: the first reviewer H1 · H2 · H3  A hallucinated method doesn't compile Status: Tested  The compiler rejects APIs that don't exist — instantly, for free - Both arms: maven.compiler.release=17 How manystricter agent compiles failed on a lint warning?  Make -the compiler Treatment added: -Xlint:all -Werror, so every Zero. The treatment's compiler gate (-Xlint with  --release warning the API build 17: fails checks usage against the real target JDK warnings as errors) never failed in either treatment  -Xlint:all -Werror: deprecation warnings become build failures run, and run-03's raw transcripts contain no lint or  Evidence: type-guided cut compile errors by more Werror failure generation at all. The control has no lint gate, so than halfits zero means "not checked," not "clean." The gate did bite once, though: while building the treatment, it forced fixes for 14 warnings in the original 2020 code. urce: ps://pldi25.sigplan.org/details/pldi-2025-papers/25/Type-Aware-Constraining-for-LLM-Code-Generat
25. Testing: prove the right tests ran H1 · H2 · H3  Both arms ran their tests  Control: mvn package runs the Arquillian suite on Open Liberty  Only the treatment checked *which* tests ran  Exact test count · zero skipped · baseline tests present by name  A deleted or skipped test leaves a plain build green  The ecosystem goes further  JUnit 5 · Arquillian · Testcontainers: tests against real containers  JaCoCo: coverage you can gate on · PIT: catches weakened assertions  Running tests isn't the guardrail. Proving the right ones ran is.
26. Testing: prove the right tests ran H1 · H2 · H3  Both Status: arms ranTested their tests - Not JaCoCo, Testcontainers  Control: mvntested: package runs the PIT, Arquillian suite on Open Liberty - Gaps in the treatment: - CI never checks testChangeDeadline exists  Only the treatment checkedthat *which* tests ran - New test classes run· baseline in the treatment's  Exact test count · zero skipped tests presentCI byonly nameif registered in the agents them.  A deleted or skipped testgate; leavesthe a plain buildregistered green  The ecosystem goes further  JUnit 5 · Arquillian · Testcontainers: tests against real containers  JaCoCo: coverage you can gate on · PIT: catches weakened assertions  Running tests isn't the guardrail. Proving the right ones ran is.
27. Testing: prove the right tests ran H1 · H2 · H3  Both Status: arms ranTested their tests - Not JaCoCo, Testcontainers  Control: mvntested: package runs the PIT, Arquillian suite on Open Liberty Did either arm skip, delete, or weaken a test? - Gaps in the treatment: In the treatment, all ten PRs delete no tests, disable - No. CI never checks that testChangeDeadline exists  Only the treatment checked *which* tests ran add no skip run flags, add 76 assertions and remove none. - nothing, New test classes in the treatment's CI only if  Exact test count zero to skipped · baseline tests present name The four ·edits gate files only registered new by tests and raised registered in the agents registered them.  A deleted or skipped testgate; leaves a plain build green expected counts. In thethe control, the campaign diff shows 4 test files changed, 36 assertions added, and none removed or disabled.goes The further control had no mid-campaign repair commits, so  The ecosystem whole-campaign diff is reliable there. real containers  JUnit 5 ·the Arquillian · Testcontainers: tests against  JaCoCo: coverage you can gate on · PIT: catches weakened assertions Agents edited the gate files 4 times, all stricter. Nothing stopped them making it weaker.  Running tests isn't the guardrail. Proving the right ones ran is.
28. H1 · H2 · H3 Backwards compatibility: old knowledge stays true  Java rarely breaks old code  Most of what the model learned years ago still compiles today  When things do change, Java says so — loudly  @Deprecated(forRemoval = true)  jdeprscan: finds uses of deprecated JDK APIs  japicmp / Revapi: break the build on incompatible API changes
29. H1 · H2 · H3 Backwards compatibility: old knowledge stays true  JavaStatus: Partly tested rarely breaks old code - Treatment added: Java 17years and Java  Most of what the modelalearned ago EE still7compiles today compatibility-contract check; Enforcer bans  When things do change, Java says so — loudly jakarta.*; -Xlint:all -Werror turns deprecation  @Deprecated(forRemoval = true) warnings into failures  jdeprscan: finds jdeprscan, uses of deprecated JDK APIs - Not tested: japicmp, Revapi  japicmp / Revapi: break the build on incompatible API changes
30. H1 · H2 · H3 Backwards compatibility: old knowledge stays true  JavaStatus: Partly tested rarely breaks old code - Treatment added: alearned Java and Java  Most Didwhat either arm reach for 17 a jakarta.* or7compiles post-EE 7today of the model years ago EE still compatibility-contract check; Enforcer bans API?  When things do change, Java says so — loudly jakarta.*; -Xlint:all -Werror Java turnsEE deprecation Neither arm did. The treatment's 7  @Deprecated(forRemoval = true) warnings into failures compatibility gate never failed, and run-03's  jdeprscan: finds jdeprscan, uses of deprecated JDK APIs - Not tested: japicmp, Revapi transcripts never mention jakarta..  japicmp / Revapi: break the build on incompatible API changes
31. Static analysis: deep, not just shallow H1 · H2 · H3  Error Prone · NullAway · SpotBugs · PMD  Bug patterns, null safety, resource leaks — at compile time  PMD CPD: duplication detection — DRY, enforced  ArchUnit: architecture rules as tests  “Controllers must not call repositories directly”  Rules don't drift — the iterative refinement paradox meets a fixed bar
32. Static analysis: deep, not just shallow H1 · H2 · H3  ErrorStatus: tested Prone ·Partly NullAway · SpotBugs · PMD - Treatment added: SpotBugs at maximum effort and  Bug patterns, null safety, resource leaks — at compile time lowest threshold, on correctness, threading, and  PMD CPD: duplication detection — DRY, enforced security bugs; LayeringTest enforces  ArchUnit: architecture rules as tests architecture rules as a test  “Controllers must Error not call repositories directly” - Not tested: Prone, NullAway, PMD/CPD. So  Rules don't drift — thehad iterative refinement paradox meets a DRY violations no guardrail. fixed bar
33. Static analysis: deep, not just shallow H1 · H2 · H3  ErrorStatus: tested Prone ·Partly NullAway · SpotBugs · PMD SpotBugs findings per arm; any layering effort violations? - Treatment added: SpotBugs at maximum  Bug patterns, null safety, resource leaks — at compileand time SpotBugs found nothing in either treatment run, in the lowest threshold, on correctness, threading, and  PMD CPD: duplication detection — DRY, enforced categories it checked: correctness, threading, and security bugs; LayeringTest enforces  ArchUnit: architecture rules astest tests never failed. With every security. Therules LayeringTest architecture as a  “Controllers category enabled, the legacy code has about 120 must not call repositories directly” - Not tested: Error Prone, NullAway, PMD/CPD. So findings. run added 3 more, exposing  Rules don't drift Each — thetreatment iterative refinement paradox meets a DRY violations had no guardrail. a mutable Date, and run 03 added a redundant null fixed bar check. All in categories the building agent had scoped out.
34. H1 · H2 · H3 Build and dependencies: no package to squat  Hallucinated dependency? The build fails at resolution.  Maven Central verifies who owns each groupId  A hallucinated name can't simply be claimed by an attacker  Maven Enforcer  Dependency convergence · banned dependencies · required versions  Reproducible builds: pinned versions, Maven Wrapper, fixed timestamps
35. H1 · H2 · H3 Build and dependencies: no package to squat Status: Tested  Hallucinated dependency? The build fails at resolution. added:  Maven- Treatment who owns each groupId convergence, -Central Enforcer:verifies pinned plugin versions, dependency  A hallucinated name can'tranges simply be claimed by an attacker Java and Maven version Banned Spring, Payara, WildFly, and Tomcat, so the spec's  Maven -Enforcer scope limits are enforced by the build, not the prompt  Dependency convergence · banned dependencies · required - Central-only repositories versions - A dependency security and delta gate - Both arms: Maven Wrapper  Reproducible builds: pinned versions, Maven Wrapper, fixed timestamps
36. H1 · H2 · H3 Build and dependencies: no package to squat Status: Tested  Hallucinated dependency? The build fails at resolution. added:  Maven- Treatment who owns each groupId convergence, -Central Enforcer:verifies pinned plugin versions, dependency Did and any Maven agent try to add a dependency? What it?  A hallucinated name can't simply be claimed bystopped an attacker Java version ranges Banned Spring, Payara, WildFly, and Tomcat, so the spec's  Maven -No Enforcer agent a Mavenbydependency: nothe task PR in any run scope limitsadded are enforced the build, not prompt  Dependency `pom.xml`. Agents did adddependencies Maven Wrapper files out of convergence · banned · required -changed Central-only repositories in all threesecurity runs. The scope check stopped versions -scope A dependency andharness's delta gate not a build guardrail. - them, Both arms: Maven Wrapper  Reproducible builds: pinned versions, Maven Wrapper, fixed timestamps
37. H1 · H2 · H3 Build and dependencies: no package to squat Status: Tested  Hallucinated dependency? The build fails at resolution. added:  Maven- Treatment verifies who owns each groupId convergence, -Central Enforcer: pinned plugin versions, dependency <bannedDependencies> Did any agent try to add a dependency? What it?  A hallucinated name can't simply be claimed bystopped an attacker Java and<excludes> Maven version ranges Banned <exclude>jakarta.*:*</exclude> Spring, Payara, WildFly, and Tomcat, so the spec's  Maven -No Enforcer agent a Mavenbydependency: nothe task PR in any run scope limitsadded are enforced the build, not prompt  Dependency `pom.xml`. Agents did adddependencies Maven Wrapper files out of convergence · banned · required -changed Central-only repositories <exclude>org.springframework*:*</exclude> scope in all threesecurity runs. The harness's scope check stopped versions - A dependency and delta gate <exclude>fish.payara*:*</exclude> them, not a build guardrail. - Both arms: Maven Wrapper  Reproducible builds: pinned versions, Maven Wrapper, fixed </excludes> timestamps</bannedDependencies>
38. Formatting: make the diff mean something H1 · H2 · H3  Spotless · google-java-format · Checkstyle  Formatting noise disappears from diffs  Two agents writing the same logic produce the same text  Reviewers — human or AI — read only real changes
39. Formatting: make the diff mean something H1 · H2 · H3 Status: Tested in both arms  Spotless · google-java-format · Checkstyle - Both arms: spotless:check runs in CI, with  Formatting noise disappears fromfrom diffsthe baseline, so google-java-format ratcheted  Two agents thechanged same logic only lineswriting the agent areproduce checkedthe same text - Not a treatment is identical in  Reviewers — human difference: or AI — readformatting only real changes both arms - Not tested: Checkstyle
40. Formatting: make the diff mean something H1 · H2 · H3 Status: Tested in both arms  Spotless · google-java-format · Checkstyle - Both arms: spotless:check runs in CI, with Formatting failures perfrom arm before green  Formatting noise disappears diffs google-java-format ratcheted from the baseline, so  Two agents thechanged same logic therunsame textrunTreatment Treatment only lineswriting the agent areproduce checked Control run-01 03 04 - Not a treatment difference: formatting is identical in  Reviewers — human or AI — read only real changes Failed Spotless checks (local and CI) 5 2 7 both arms Distinct formatting problems 3 2 4 - Not tested: Checkstyle That's 5 failures for the control against 4.5 per treatment run. Separately, code review flagged two unrelated-formatting findings on run-04's PR #8.
41. H1 · H2 · H3 Concurrency and JVM tuning: the bonus round  Structured concurrency (preview in JDK 26)  Fewer ways to write concurrent code wrong → fewer ways for an agent to get it wrong  Virtual threads: simple blocking code instead of reactive complexity  JFR · JMH: performance regressions become measurable  Concurrency not tested: the app stays on Java 17
42. H1 · H2 · H3 Concurrency and JVM tuning: the bonus round Status: Partly tested. JVM tuning was measured; concurrency was not.  Fewer ways to write concurrent code wrong → fewer ways for - Treatment added: JFR-verified performance an comparison agent to get of it wrong plain Java versus the jaz launcher; a  Virtual threads: simple user blocking instead of reactive check that rejects JVMcode tuning complexity - Not tested: structured concurrency, virtual threads  JFR (Java baseline) regressions become measurable · JMH:17performance  Concurrency not tested: the app stays on Java 17  Structured concurrency (preview in JDK 26)
43. H1 · H2 · H3 Concurrency and JVM tuning: the bonus round Status: Partly tested. JVM tuning was measured; concurrency was not. Any performance regression between arms?  Fewer ways to write concurrent code wrong → fewer ways for - Treatment added: JFR-verified performance There's no it data to compare. The treatment's bounded an comparison agent to get wrong of plain Java versus the jaz launcher; a performance gate never failed in either run, but the  Virtual threads: simple user blocking instead of reactive check that rejects JVMcode tuning control measures nothing, so a between-arm complexity - Not tested: structured concurrency, virtual threads comparison isn't possible. The most you can say is  JFR (Java 17performance baseline) regressions become measurable · JMH: "no regression flagged within the treatment."  Concurrency not tested: the app stays on Java 17  Structured concurrency (preview in JDK 26)
44. Observability: watch it run H1 · H2 · H3  OpenTelemetry Java agent · JFR · Micrometer  Catch what compiles, passes tests, and still misbehaves  Runtime regressions show up as metrics, not reviewer hunches  We measured the agents themselves with OpenTelemetry
45. Observability: watch it run H1 · H2 · H3  OpenTelemetry Status: Tested Java agent · JFR · Micrometer - Treatment added:passes an OpenTelemetry acceptance  Catch what compiles, tests, and still misbehaves check on the running app, plus observability negative  Runtime regressions show up as metrics, not reviewer controls hunches - Both arms: the harness itself is measured with  We measured the agents themselves with OpenTelemetry OpenTelemetry. That's where every token count comes from.
46. Observability: watch it run H1 · H2 · H3  OpenTelemetry Status: Tested Java agent · JFR · Micrometer - Wall-clock Treatment added: an OpenTelemetry acceptance and tokens spent waitingand on CI, per arm  Catch what compiles, passes tests, still misbehaves check on the running app, plus observability negative  Runtime regressions show Remote up as metrics, not reviewer Not measured separately. coding-agent wait: control controls hunches 03s, run-03 22m 29s, run-04 28s. CI wait with wasn’t - 4m Both arms: the harness itself3m is measured  We measured isolated; the treatment’s 86% longer session time is consistent the agents themselves with OpenTelemetry OpenTelemetry. That's where every token count with it, but that’s inference. comes from.
47. The 10th boring reason: breadth of deployment options H1 · H2 · H3 One standard artifact, many runtimes  Same Jakarta EE WAR: Liberty on a laptop → AKS, ARO, App Service, VMs  The agent verifies locally on what production actually runs Agents get typed tools instead of guessed CLI flags  Azure MCP Server: 170+ tools · AKS-MCP · Azure DevOps MCP  Our control run: Every run: 6 to 8 malformed gh commands, both arms.Full disclosure: building some of these is part of my day job. Sources: https://devblogs.microsoft.com/azure-sdk/announcing-azure-mcp-server-stable-release/
48. The 10th boring reason: breadth of deployment options H1 · H2 · H3 Status: Argued One standard artifact, - Both arms: the many same runtimes WAR runs on Liberty locally,  Same exactly as inEEproduction Jakarta WAR: Liberty on a laptop - Treatment added: Enforcer → AKS, ARO, App Service, VMs bans other runtimes, so agent can't drift to on Payara Tomcat actually runs  Thethe agent verifies locally whatorproduction - Not tested: typed MCP tools in place of CLI calls Agents get typed tools instead of guessed CLI flags  Azure MCP Server: 170+ tools · AKS-MCP · Azure DevOps MCP  Our control run: Every run: 6 to 8 malformed gh commands, both arms.Full disclosure: building some of these is part of my day job. Sources: https://devblogs.microsoft.com/azure-sdk/announcing-azure-mcp-server-stable-release/
49. The 10th boring reason: breadth of deployment options H1 · H2 · H3 Status: Argued One standard artifact, many - Both arms: the same WAR runs on Liberty locally, Malformed CLI callsruntimes per run  Same exactly as inEEproduction Jakarta WAR: Liberty on a laptop - Treatment added: Enforcer bans other runtimes, so → AKS, ARO, 8, App Service, Control treatment 6VMs and 7. Mostly the same two agent can't to on Payara Tomcat  Thethe agent verifies locally whator production runs invalid gh flagdrift combinations every time. actually The - Not tested: don't typedtouch MCP CLI tools in place typed of CLI tools calls guardrails behavior; Agents get typed tools instead of guessed CLI flags  Azurewould. MCP Server: 170+ tools · AKS-MCP · Azure DevOps MCP  Our control run: Every run: 6 to 8 malformed gh commands, both arms.Full disclosure: building some of these is part of my day job. Sources: https://devblogs.microsoft.com/azure-sdk/announcing-azure-mcp-server-stable-release/
50. shepherd-task experiment Testing H1, H2 and H3
51. Experiment design  Held constant across experiment arms  Harness: shepherd-task 1.4.0 at 0134099 · Model: Local orchestration: gpt-5.6-sol (medium). Coding and review agents: GitHub-hosted, model not recorded.  App: Jakarta EE Cargotracker at e7b651f, modernized to run on latest Open Liberty and JDK 17  Task: make the cargo arrival deadline editable  Real feature from the repo's history (issue #64, commit bc24dea); cross-cutting: 9 files across 4 layers  Spec: adapted per arm's baseline. 5 serially gated issues, all design questions preresolved  Experiment arms: the dominating variable  Control: typical CI: format check, build, run all tests.  Treatment: agent-built maximal reasonable guardrails (the Middle section, applied via pom.xml and CI)  Measures execution from a complete spec, not design from a vague issue · one control and two treatment runs
52. The independent variable: what the treatment added  Test evidence: exact count, zero skipped, baseline tests present by name  Architecture rules: LayeringTest <!-- Enforcer: the build contract -->  Dependency security and delta <requireJavaVersion>[17,18)</requireJavaVersion> <requirePluginVersions/> <!-- no LATEST, RELEASE, SNAPSHOT --> gate <dependencyConvergence/>  Java <bannedDependencies> <!-- jakarta.*, Spring, Payara, ... -->17 and Java EE 7 <requireNoRepositories/> <!-- Maven Central only --> compatibility contract <!-- SpotBugs: max effort, lowest threshold -->  OpenTelemetry acceptance on the <effort>Max</effort> running app <threshold>Low</threshold>  JFR performance checks  Negative controls: each guardrail must fail when it should <!-- Compiler: every warning fails the build --> <compilerArgs> <arg>-Xlint:all</arg> <arg>-Werror</arg> </compilerArgs> Both arms: Spotless, --release 17, tests run in CI · Built by shepherd-task from the control
53. Building the guardrails: an agent kept the evidence  Every claim tied to a CI run, artifact, and digest. No confidence upgrades just because a tool passed.  The guardrails bit, on the 2020 code:  14 compiler warnings · 5 threadunsafe SimpleDateFormat · a booking result that was always null  Code review guarded the guardrails: https://bit.ly/dvbe26-evidence-matrix  One agent-built gate reported success when it had failed
54. Processing the data One control run, two treatment runs: a case study, not a statistical result. 1. Shepherd task produces copious output: OTEL, JSON, markdown.  Vibed up a tool (CampaignEvaluator.java) to process the data and produce empirical eval reports.  Put those eval reports through LLM report generation. 2. Shepherd task evaluates its own work and produces postmortems  Put those postmortems through LLM report generation  Report on self-evaluations: https://aka.ms/coreai/shepherd-task/self-eval-report  Report on empirical data: https://aka.ms/coreai/shepherd-task/eval-report
55. H1: LLMs don't know a codebase the way humans do  Supported  The date parser: fixed the examples, not the rule  Post-mortems mentioned 9 of 29 defects we found  Gate failures fixed before review were reported as "validation passed"
56. H2: At scale, agents show predictable symptoms  Partly supported  Identical in all three runs, both arms:  The same 16 lines of dialog code, copied  The same out-of-scope Maven Wrapper files, added  The same invalid gh flag combinations: 8, 6 and 7 per run  Unformatted code: Spotless failed in every run  Never appeared: hallucinated dependencies, weakened tests, deprecated APIs
57. H3: Java's ecosystem supplies guardrails that reduce them  Not yet supported  Earlier, in one place: the test-inventory check caught the same omission 4 times, before review  Not where it mattered: code review caught 3 of 4 treatment behavioral defects, including the date parser, and 1 of 1 in the control  Silent: SpotBugs (in its narrowed scope), architecture tests, the EE 7 contract and lint found nothing  Untouched: every symptom that repeated  Cost: +25% AIU, +86% session time, not counting the cost of
58. The takeaway  Code review is essential for agentic development, even with the strongest possible guardrails  Guardrails don't make an agent understand your code. They catch the misunderstandings you thought to check for.
59. One last lesson  Neither evaluation was right on its own  - The agents' self-evaluation  - The evaluator built with an agent  Only cross-checking against the raw data produced slide- worthy numbers. That's H1, one level up.
60. The agent safe Java checklist Keep the reviewer  Require code review on every agent PR, human or Copilot  Never trust the agent's own report; check the logs Put the rules in the build, not the prompt  Ban out-of-scope dependencies with Maven Enforcer  Turn on duplication detection: PMD CPD  Know what your analyzers exclude Make the tests count  Require new tests by name, not by count Guard the guardrails  Prove each gate fails when it should: negative controls  Protect pom.xml and .github/ with CODEOWNERS Close the loop  Give the agent one local command that runs every CI gate  Give agents typed tools (MCP), not CLI flags
61. Resources  Agent harness: shepherd-task: https://aka.ms/coreai/shepherd-task/README  PR 9 from the cold open https://github.com/edburns/dd-3072797-tricked-out-cargotrack er-run-04/pull/9  Reports: https://aka.ms/dvbe26  Ignorance reduction plans  “Adding the guardrails” link  “Adding the feature to cargotracker” link
62. Thank you Slides & rate this talk https://aka.ms/dvbe26 Ed Burns @edburns SCAN ME