The Unverified Developer: Threat Modeling AI Coding Agents in Secure Software Pipelines

Presented at Secure Software by Design 2026, hosted by Carnegie Mellon University's Software Engineering Institute.

AI coding agents have crossed from tools that assist developers into autonomous participants that generate source code, infrastructure, and build scripts with decreasing human review. That breaks a foundational assumption of secure software design: that every artifact entering a pipeline has a human author who understands its security implications. When the code generator and the code reviewer are the same entity, the trust boundaries our threat models were built on no longer hold.

This talk examines what breaks when AI coding agents enter the pipeline: where the attack surface expands, why controls built for human-authored code fail silently at the boundaries, and how established frameworks like NIST SSDF assume a human developer that no longer exists. It closes with concrete, per-stage adaptations across the development lifecycle.

Covers: trust-boundary analysis for AI-generated artifacts, a gap analysis of existing secure-by-design controls (SAST, SCA, provenance, code review), and adaptations from design through deployment.

Transcript

1. The Unverified Developer: Threat Modeling AI Coding Agents in Secure Software Pipelines Alok Tibrewala July 23, 2026 Carnegie Mellon University SEI Summer Series 2026, Secure Software By Design Virginia, USA
2. About Education MS Computer Science, The George Washington University, Washington DC Work Experience 10 years predominantly e-commerce and finance AI & Cybersecurity Research Secure Hybrid AI systems | Research Papers author
3. What Has Changed Post AI?
4. Before and After AI Coding Assistants BEFORE AFTER
5. Before and After AI Coding Assistants • Developers have shifted role from being an author to that of a reviewer • Most organizations made this shift in last 12 – 18 month • AI assisted developers produces 3 – 4x more commits that non-AI peers • 7000 developers, 62000 repositories studied source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
6. What AI Coding Agents Actually Do? • AI Agents • • • • • • • read your codebase for context plan multi step tasks write code across multiple files install packages and dependencies run commands in terminal maintain memory across sessions make choices on how to implement things – including security sensitive choices source: arxiv 2504.19956
7. AI makes a security choice, and it’s not random • Prompt: get all columns from table where id matches provided user_id • • • • • AI had 2 options Secure: use prepared statement (parameterized query) Insecure: use string concatenation AI chose string concatenation -> SQL Injection vulnerability (CWE-89) The developer got code working, but it is also exploitable source: Veracode, "2025 GenAI Code Security Report,"
8. This happens 45% of the time • 100+ LLMs, 80 coding tasks, 4 languages (Java, Python, JavaScript, C#) • 4 Vulnerability types – SQL Injection, XSS, broken crypto, log injection • • • • 45% of the time, model chose insecure implementation Syntax pass rate - ~50% (2023) -> ~96% (2025+) Security pass rate - ~55% (2023) -> ~55% (2025+) – FLAT, hasn’t improved Model size has no meaningful impact – 20B and 400B models cluster at same rate source: Veracode, "2025 GenAI Code Security Report,"
9. Java is the worst case • • Security pass rate – Python ~63% while Java ~29% Java fails roughly 7 out of 10 times and has not improved since 2025 report • Veracode hypothesis – models are overtrained on legacy Java patterns that predate modern security framework • • Enterprise codebases are disproportionately Java Models default to what appears most frequently in training data, not what is most secure source: Veracode, Spring 2026
10. AI is fixing the typos, but creating security nightmare • Apiiro studied Fortune 50 companies, tens of thousands of repos • AI assisted developers • • • • • Syntax error – down 76% Logic bugs – down 60% Privilege escalation - up 322% Architectural design flaws – up 153% Outcome? The shallow bugs got fixed, deep exploitable flaws however multiplied source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
11. The Scale Problem – 10,000+ new security findings per month • • • Same Fortune 50 dataset December 2024: baseline security findings rate June 2025: 10,000+ new security findings per month • • • 10x increase in 6 months PR volume dropped by one-third in same period Result – fewer PR, more code per PR, more vulnerabilities per PR source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
12. Why Existing Security Thinking Misses This?
13. What is a Trust Boundary? • Think of your office building • Front door – anyone can walk-in • Reception desk – visitor verified • Badge access floor – only authorized employees • Server room – restricted, logged, monitored • • • • Each door = a trust boundary Every time data crosses a boundary, it must be checked Food delivery person cannot walk straight to server room Software systems – same concept – data flows between components with different trust levels • Threat modeling – figuring out where these boundaries are and what can go wrong at each
14. AI agent is a new actor in your building A new employee who is incredibly fast but lacks security training
15. Four trust boundaries in the AI development workflow • • 4 boundaries in every AI development workflow Every boundary = a mandatory validation check 1. B1 – Developer Intent -> Agent Interpretation • Risk - prompt injection — someone puts malicious instructions in a ticket, doc, or code comment the agent reads 2. B2 - Agent Interpretation -> Code Generation • Risk - hallucination — agent generates plausible but wrong or insecure code, invents nonexistent packages 3. B3 – Generated Code -> Build Pipeline • Risk - supply chain — hallucinated or malicious dependencies get installed, secrets get committed 4. B4 – Build Artifacts -> Production • Risk - misconfiguration — insecure defaults in generated IaC, Dockerfiles, CI/CD configs reach production
16. The boundary does not stop at the commit • B5 - Production to Agent: • telemetry, logs, and incident data flow back into agent context • What production tells the agent shapes what the agent writes next • B6 - Agent to External Tool: • MCP servers, registries, retrieval indexes, and third-party integrations invoked at generation time • Both boundaries sit outside every artifact your pipeline currently scans • This is an open problem today and has not been solved yet
17. What your current Threat Model does not cover? 1. Hallucinated dependencies / slopsquatting • Agent invents a package name that does not exists • ~20% of AI code suggestions references non existent package names • Attacker register those names with malware inside 2. Training data as a trust input • Agent trained on million of lines of code – many contain vulnerabilities • Intentionally insecure projects are in training data • Agent treats insecure patterns as equally valid • This is baked in agent memory 3. Agent memory and context persistence • What agent saw In codebase yesterday influences what it writes today • Poison one document in its context -> every output that reference it is compromised • Traditional models do not account for an actor that accumulates state over time Slopsquatting hallucination rate source: arXiv:2406.10279
18. The Gap Analysis – Where Secure-by-Design Controls Fall Short
19. Controls that assume a human author • Code review assumes an author you can question or discuss with • AI generated code has no author to question • 3–4x more commits, fewer PRs review depth drops where volume rises • SAST finds known patterns, the growth is architectural and privilege flaws • SCA assumes the package name resolves to something real • Secrets scanning fires after the commit, at 2x the baseline rate
20. SSDF assumes a human developer • SSDF (Secure Software Development Framework) v1.1 organizes practices into • PO (Prepare the Organization), • PS (Protect the Software), • PW (Produce Well Secured Software), • RV (Respond to Vulnerabilities) • SP 800-218A adds AI practices for model producers not consumers • Neither covers the org consuming a coding agent • PW assumes a developer who can attest to intent • PS assumes artifacts with known authorship • The guidance is not wrong, however, the actor has changed Sources: NIST SP 800-218 v1.1 (2022) · NIST SP 800-218A (2024)
21. Slopsquatting • LLMs hallucinate package names that don't exist • Out of 756,000 code samples: ~20% referenced nonexistent packages • 58% of hallucinated packages repeat across queries — predictable, not random • Open-source models (CodeLlama, DeepSeek, WizardCoder, Mistral): ~20% hallucination rate • GPT-4: ~5% hallucination rate • The attack: • Researcher identifies commonly hallucinated package names • Registers them on PyPI/npm with malicious code • Waits for AI tools to recommend them • Developers run pip install / npm install without checking Slopsquatting hallucination rate source: arXiv:2406.10279
22. AI commits leak secrets at 2x the rate • 28.65M hardcoded secrets on public GitHub in 2025 (+34% YoY) • AI commits leak secrets at 3.2% vs 1.5% baseline - 2x the rate • AI credential leaks: +81% YoY (1.27M incidents) • 8 of 10 fastest-growing secret categories = AI-related • 24,008 secrets in MCP config files on public GitHub; 2,117 still valid • 64% of secrets from 2022 still exploitable in 2026 Number of secrets per 1000 commits source: GitGuardian, The State of Secrets Sprawl 2026
23. Adaptations across SDLC
24. Adaptations across the lifecycle • Design - model the agent as an actor, not merely as a tool • Generation - capture prompt, model version, context as metadata • Dependencies - verify existence before install, pin and allowlist • Build - treat generated IaC and CI configs as untrusted input • Review - mandatory human review at every trust boundary • Operate - keep a path from vulnerability back to generation context
25. 5 Layer Validation Pipeline • • Layer 1: SAST (Static Analysis) • Semgrep, CodeQL • Critical for XSS (86% AI failure rate) and log injection (88% failure) Layer 2: SCA (Software Composition Analysis) • Verify every dependency exists in the registry • • • • Layer 3: Secrets Scanning • GitLeaks, TruffleHog, GitGuardian • AI commits leak at 2x rate Layer 4: DAST (Dynamic Analysis) • • Block packages younger than 30 days or with <100 downloads Snyk, Socket.dev, npm audit, pip-audit Tests running application, helpful to catch business logic and runtime issues SAST misses Layer 5: Policy-as-Code • OPA/Rego gates for organizational standards • Enforce rules like "no new dependencies without SCA pass"
26. Generation context as an auditable artifact • A vulnerability reaches production. Can you reconstruct how it was written? • Capture: model version, prompt, retrieved context, tool permissions • No reproducible context means no investigation, only a patch • Track AI-generated and human-authored code as separate populations • SEI's FLARE-AI routes AI flaw reports to vendors and agencies Sources: CMU SEI, FLARE-AI (July 2026)
27. Takeaways
28. Takeaways
29. Thank You! Connect with me Talks & speaking - sessionize.com/aloktibrewala IEEE publications - ieeexplore.ieee.org/author/750919393611525 Email -