Threat Modeling Development Workflows with Autonomous Code Generation

Presented at OWASP Boston Application Security Conference (BASC) 2026.

AI coding agents don't just autocomplete, they plan, execute, install dependencies, and make security-sensitive implementation choices autonomously. This talk covers how to threat model your development workflow when an AI agent is a system actor in your pipeline.

Topics covered:
- Why AI-generated code is insecure 45% of the time and why bigger models don't help
- Four new trust boundaries (B1–B4) introduced by AI in the dev pipeline
- OWASP Agentic Top 10 mapped to real-world attack scenarios
- Slopsquatting, memory poisoning, and cascading multi-agent attacks
- A practical threat model template and 5-layer validation pipeline
- Org-level actions for AI code security

Transcript

1. Threat Modeling Development Workflows with Autonomous Code Generation Alok Tibrewala April 11, 2026 OWASP Boston Application Security Conference
2. About Education MS Computer Science, The George Washington University, Washington DC Work Experience 10+ years predominantly e-commerce and finance AI & Cybersecurity Research Secure Hybrid AI systems | Research Papers and Book author
3. What Has Changed Post AI?
4. Before and After AI Coding Assistants BEFORE After AFTER
5. Before and After AI Coding Assistants • Developers have shifted role from being an author to that of a reviewer • Most organizations made this shift in last 12 – 18 month • AI assisted developers produces 3 – 4x more commits that non-AI peers source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
6. What AI Coding Agents Actually Do? • AI Agents • read your codebase for context • plan multi step tasks • write code across multiple files • install packages and dependencies • run commands in terminal • maintain memory across sessions • make choices on how to implement things – including security sensitive choices source: arxiv 2504.19956
7. AI makes a security choice • Prompt: get all columns from table where id matches provided user_id • AI had 2 options • Secure: use prepared statement (parameterized query) • Insecure: use string concatenation • AI chose string concatenation -> SQL Injection vulnerability (CWE-89) • The developer got code working, but it is also exploitable source: Veracode, "2025 GenAI Code Security Report,"
8. And this happens 45% of the time • 100+ LLMs, 80 coding tasks • 4 languages – Java, Python, , JavaScript, C# • 4 Vulnerability types – SQL Injection, XSS, broken crypto, log injection • 45% of the time, model chose insecure implementation • Syntactic correctness improved over last 2 years • Security performance: flat. Hasn’t improved source: Veracode, "2025 GenAI Code Security Report,"
9. AI is fixing the typos, but creating security nightmare • Apiiro studied Fortune 50 companies, tens of thousands of repos • AI assisted developers • Syntax error – down 76% • Logic bugs – down 60% • Privilege escalation - up 322% • Architectural design flaws – up 153% • Outcome? The shallow bugs got fixed, deep exploitable flaws however multiplied source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
10. The Scale Problem – 10,000+ new security findings per month • Same Fortune 50 dataset • December 2024: baseline security findings rate • June 2025: 10,000+ new security findings per month • 10x increase in 6 months • PR volume dropped by one-third in same period • Result – fewer PR, more code per PR, more vulnerabilities per PR source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
11. Why Existing Security Thinking Misses This?
12. What is a Trust Boundary? • Think of your office building • Front door – anyone can walk-in • Reception desk – visitor verified • Badge access floor – only authorized employees • Server room – restricted, logged, monitored • Each door = a trust boundary • Every time data crosses a boundary, it must be checked • Food delivery person cannot walk straight to server room • Software systems – same concept – data flows between components with different trust levels • Threat modeling – figuring out where these boundaries are and what can go wrong at each
13. AI agent is a new actor in your building A new employee who is incredibly fast but lacks security training
14. We need New Trust Boundaries • 4 boundaries in every AI development workflow • Every boundary = a mandatory validation check 1. B1 – Developer Intent -> Agent Interpretation • Risk - prompt injection — someone puts malicious instructions in a ticket, doc, or code comment the agent reads 2. B2 - Agent Interpretation -> Code Generation • Risk - hallucination — agent generates plausible but wrong or insecure code, invents nonexistent packages 3. B3 – Generated Code -> Build Pipeline • Risk - supply chain — hallucinated or malicious dependencies get installed, secrets get committed 4. B4 – Build Artifacts -> Production • Risk - misconfiguration — insecure defaults in generated IaC, Dockerfiles, CI/CD configs reach production
15. What your current Threat Model does not cover? 1. Hallucinated dependencies / slopsquatting • Agent invents a package name that does not exists • ~20% of AI code suggestions references non existent package names • Attacker register those names with malware inside 2. Training data as a trust input • Agent trained on million of lines of code – many contain vulnerabilities • Intentionally insecure projects are in training data • Agent treats insecure patterns as equally valid • This is baked in agent memory 3. Agent memory and context persistence • What agent saw In codebase yesterday influences what it writes today • Poison one document in its context -> every output that reference it is compromised • Traditional models do not account for an actor that accumulates state over time Slopsquatting hallucination rate source: arXiv:2406.10279
16. A Framework to Reason About This
17. OWASP Agentic Top 10 • Published December 2025 by 100+ security researchers • Covers autonomous AI systems that plan, use tools and act • 3 main sections 1. Inputs – where attacks enter (prompts / APIs / connected data) 2. Integration / Processing – where attacks propagate (agent logic / tools / memory) 3. Outputs – where damage happens (code / actions / API calls) • 10 risk categories: ASI01 through ASI10
18. Three Most Common Scenarios
19. Five Threat Zones – places an attacker can touch your code 1. Zone 1 – Input Surfaces • Prompts, API calls, uploaded files, RAG retrieval results 2. Zone 2 – Planning & Reasoning • Goal decomposition, task breakdown, chain-of-thought • Attacker manipulates how the agent reasons 3. Zone 3 – Tool Execution • API calls, code execution, terminal commands, package installation • Attacker manipulates how the agent reasons, not just what it sees 4. Zone 4 – Memory & State • Session context, vector stores, accumulated knowledge • Poison here = persistent compromise across future sessions 5. Zone 5 – Inter – Agent Communication • Delegation chains between multiple agents • Each agent trusts the output of the previous one
20. How Attacks Cascade? source: Schneider, "Threat modeling agentic AI"
21. How Attacks Cascade? • A customer sends a support ticket with a hidden instruction embedded in the text • Triage Agent reads it - looks like a normal request, routes it forward • Knowledge Agent retrieves related docs - the hidden instruction survives • Transaction Agent receives what looks like an authorized action – it processes it • Supervisor Agent sees a completed workflow with no anomalies • Result: unauthorized wire transfer. No single agent saw the full attack. • Each agent validated its local context, however, nobody validated the chain.
22. Slopsquatting • LLMs hallucinate package names that don't exist • Out of 756,000 code samples: ~20% referenced nonexistent packages • 58% of hallucinated packages repeat across queries — predictable, not random • Open-source models (CodeLlama, DeepSeek, WizardCoder, Mistral): ~20% hallucination rate • GPT-4: ~5% hallucination rate • The attack: • Researcher identifies commonly hallucinated package names • Registers them on PyPI/npm with malicious code • Waits for AI tools to recommend them • Developers run pip install / npm install without checking Slopsquatting hallucination rate source: arXiv:2406.10279
23. AI commits leak secrets at 2x the rate • 28.65M hardcoded secrets on public GitHub in 2025 (+34% YoY) • AI commits leak secrets at 3.2% vs 1.5% baseline - 2x the rate • AI credential leaks: +81% YoY (1.27M incidents) • 8 of 10 fastest-growing secret categories = AI-related • 24,008 secrets in MCP config files on public GitHub; 2,117 still valid • 64% of secrets from 2022 still exploitable in 2026 Number of secrets per 1000 commits source: GitGuardian, The State of Secrets Sprawl 2026
24. Why models won’t self-correct? • Training data includes vulnerable code + intentionally insecure projects (WebGoat, DVWA) - not labeled secure vs. insecure • Models treat secure and insecure patterns as equally valid • Training data hasn't changed → security performance hasn't changed • All model sizes converge to ~51% security pass rate - bigger doesn't help Security Pass Rate vs LLM Release Dat source: Veracode, "2025 GenAI Code Security Report
25. The Threat Model Template
26. A Threat Model You Can Use
27. Make Your AI Code Environment Threat Ready
28. The Core Principle • Treat AI-generated code like untrusted 3rd party input • 45% has known vulnerabilities (Veracode, 100+ LLMs) • 322% more privilege escalation paths (Apiiro, Fortune 50) • 2x secret leak rate (GitGuardian) • No auto-merge. No auto- deploy. Security Pass Rate vs LLM Release Date source: Veracode 2025 GenAI Code Security Report CWE-327 – Weak Encryption CWE-89 – SQL Injection CWE-80 – Cross Site Scripting (XSS) CWE117 – Log Injection
29. 5 Layer Validation Pipeline • Layer 1: SAST (Static Analysis) • Semgrep, CodeQL • Critical for XSS (86% AI failure rate) and log injection (88% failure) • Layer 2: SCA (Software Composition Analysis) • Verify every dependency exists in the registry • Block packages younger than 30 days or with <100 downloads • Snyk, Socket.dev, npm audit, pip-audit • Layer 3: Secrets Scanning • GitLeaks, TruffleHog, GitGuardian • AI commits leak at 2x rate • Layer 4: DAST (Dynamic Analysis) • Tests running application, helpful to catch business logic and runtime issues SAST misses • Layer 5: Policy-as-Code • OPA/Rego gates for organizational standards • Enforce rules like "no new dependencies without SCA pass" Security Pass Rate vs LLM Release Date source: Veracode 2025 GenAI Code Security Report
30. Fighting Slopsquatting • Allowlist + block: approved packages only; block anything < 30 days old or < 100 downloads • Pin + verify: lock specific version of dependency, hash verification, registry API check before build • Scan in CI: Socket.dev / Snyk / OSV- Scanner - no model is at zero (GPT-4 still ~5%) Slopsquatting hallucination rate source: arXiv:2406.10279
31. Where Human Review Matters MOST • At the very least always review: • Auth, crypto, and session handling code • Input validation and output encoding • Infrastructure configs (IaC, Dockerfiles, K8s manifests) • Permission and access control logic • All new dependency additions • Anything crossing a trust boundary - B1 – B4 from earlier slides • AI code is shifting secret leaks toward cloud credentials - the most dangerous kind source: Apiiro, "4x Velocity, 10x Vulnerabilities," August 2025.
32. Org Level Actions
33. Takeaways
34. Takeaways
35. THANK YOU Connect with me on LinkedIn!