Merge with Caution: A Developer’s Guide to the Risks Hiding in AI-Generated Code

Presented at DevFest Kansas City 2026.

Merge with Caution: A Developer's Guide to the Risks Hiding in AI Generated Code

AI coding assistants now write a large share of the code developers merge. That code often looks correct but can carry hidden risks. This talk explains what has changed since AI joined the workflow and where things go wrong.

It uses a simple four-boundary model, published with OWASP: Developer to Agent, Agent to Repo, Repo to CI/CD, and CI/CD to Production. It covers insecure AI-generated code, hallucinated package names (slopsquatting), prompt injection through content the agent reads, and leaked secrets.

It ends with practical habits developers can start using to review and gate AI-generated code before it ships.

Covers: AI coding assistants, trust boundaries, threat modeling, prompt injection, slopsquatting, secrets, CI/CD security, software supply chain.

Transcript

1. Merge with Caution: A Developer’s Guide to the Risks Hiding in AI-Generated Code Alok Tibrewala DevFest Kansas City Nov 7, 2026
2. ~$ whoami Education MS Computer Science The George Washington University, Washington DC Work Experience 9+ professional years AI & Cybersecurity Research Secure Enterprise AI systems | 6 IEEE papers on AI security
3. Show of Hands Who merged an AI written PR recently?
4. Show of Hands Keep your hand up if you, …read every line
5. Show of Hands Keep your hand up if you, …read every line …read and verified every config file
6. Show of Hands Keep your hand up if you, …read every line …read and verified every config file …read and validated every new dependency
7. We review AI generated code like it’s from a trusted teammate, and that habit is where the risk lies
8. Today’s Agenda
9. What Has Changed Post AI?
10. Before and After AI Coding Assistants BEFORE AFTER
11. We Use It Daily, We Don’t Trust It Source: Stack Overflow 2025 Developer Survey
12. Code That Works Is Not Code That’s Safe Study Scope • 100+ LLMs • 80 coding tasks • 4 languages (Java, Python, JS, C#) Result • 45% of the time, AI generated code had security flaw Observation • Syntax pass rate – climbing close to 100% • Security pass rate – hovering around 55% Source: Veracode, "2025 GenAI Code Security Report"
13. Which Flaws, Which Languages Security Pass Rate • Python ~63% • Java ~29% Result • Java fails 7 out of 10 times • No improvement since 2025 Why • Veracode hypothesis: models are overtrained on legacy Java, written before modern security frameworks • Enterprise codebases are disproportionately Java • Models default to what’s most common, not what’s most secure Source: Veracode, Spring 2026 Report
14. Faster Output, More Findings Study Scope • Fortune 50 companies • Tens of thousands of repos Result • • • • Syntax errors: down 76% Logic bugs: down 60% Privilege escalations: up 322% Architectural design flaws: up 153% Observation • Shallow bugs got fixed • Deep, exploitable bugs multiplied source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
15. Fewer PRs, More Flaws per PR Study Scope • Same Fortune 50 companies Result • • • • December 2024: baseline finding June 2025: 10,000+ new security findings/month 10x increase in just 6 months PR volume dropped by 1/3rd in same period Observation • Fewer PRs, more code per PR • More security flaws per PR source: Apiiro, 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks," August 2025
16. Where Does Trust Break?
17. What is a Trust Boundary? • Think of your office building • Front door – anyone can walk-in • Reception desk – visitor verified • Badge access floor – only authorized employees • Server room – restricted, logged, monitored • Each door = a trust boundary • Every time data crosses a boundary, it must be checked • Software systems – same concept – data flows between components with different trust levels • Threat modeling – figuring out where these boundaries are and what can go wrong at each
18. B1–B4: The Pipeline View Every boundary = a mandatory validation check
19. AI agent is a new actor in your building A new eager employee who is incredibly fast but lacks security training
20. Can a Text File Hijack Your Agent?
21. Context Files are Code Now Adoption • 60,000+ open-source projects use AGENTS.md • 16–23% of active GitHub projects use coding agents Risk • Poisoned rule files hijacked Cursor and Copilot in 67–84% of attacks • 13.4% of public agent skills carry a critical security issue Observation • If the agent reads it, it can act on it Source: Linux Foundation AAIF announcement, Dec 2025; Robbes et al., arXiv:2601.18341, 2026; “Your AI, My Shell,” arXiv:2509.22040, 2025 Snyk ToxicSkills, Feb 2026
22. Every AI IDE Has Been Hit Incidents • Copilot: agent switched on its own auto-approve, then ran shell commands • Claude Code: repo config ran code and leaked the API key before the trust prompt • Cursor and Copilot: invisible Unicode in rule files Scale • 30+ flaws across 10+ AI IDEs, 24 CVEs • Every AI IDE tested was vulnerable Source: The Hacker News, “Researchers Uncover 30+ Flaws in AI Coding Tools,” Dec 2025; Embrace The Red, CVE-2025-53773, 2025; Check Point Research, “Caught in the Hook,” Feb 2026; Pillar Security via The Hacker News, Mar 2025
23. Is That Package REAL?
24. Slopsquatting • LLMs hallucinate package names that don’t exist • 19.7% of generated package references are hallucinated (205,474 unique fake names) • 58% of fake names repeat across queries: predictable, not random • Open-source models: ~22% hallucination rate • Commercial models: ~5% • How the attack happens? • Attacker collects commonly hallucinated names • Registers them on the package registry with malicious code • Waits for AI tools to recommend them • Developers install without checking source: arXiv:2406.10279
25. It Already Happened! Experiment • A researcher registered a name AI kept inventing, as an empty package – huggingfacecli • It got 30,000+ real downloads in 3 months Spread • A major tech company’s repo README told users to install it Observation • A hallucinated name spreads across models source: Lasso Security, “AI Package Hallucinations,” 2024; ITPro, 2024
26. The Same Attack Moves to Agent Skills Scale • 3,984 public agent skills scanned of 4,257 collected • 36.82% had at least one security flaw • 76 confirmed malicious payloads Observation • Skills are just text files that tell the agent what to do, so attackers now just needs to poison the instructions source: Snyk, “ToxicSkills,” Feb 2026
27. What Slipped Into Your Config?
28. Insecure Default Looks Normal Result • 35.8% of AI-generated Kubernetes manifests had at least one security smell • Most common: no CPU or memory limits Why Review Misses It • Model learnt from tutorials and the generated file looks like a common example from online tutorial source: Zhang et al., MSR 2024 (NSF PAR 10540424)
29. AI Commits Leak Secrets 2× Scale • 28.65M new secrets leaked on public GitHub in 2025, up 34% YoY • AI-assisted commits: 3.2% leak rate vs a 1.5% baseline (human committed) – 2x the rate • 8 of 10 fastest-growing secret leak categories = AI-related Why • AI generates more code and commits are bigger so there’s more room for a key to slip in Monthly AI assisted commits source: GitGuardian, State of Secrets Sprawl 2026
30. Agent Configs Leak Keys, and Keys Stay Valid Scale • 24,008 secrets in agent (MCP) config files; 2,117 still valid • AI-service keys leaked: 1.27M, up 81% YoY • 64% of secrets from 2022 still exploitable in 2026 Leaked Keys Don’t Expire source: GitGuardian, State of Secrets Sprawl 2026
31. How Does It Reach Production?
32. A Branch Name Hacked the Build The Flaw • The project’s build bot copied the branch name into a command and ran it • The bot had the keys to publish release The Attack • The attacker opened an empty PR and made the branch name a command: “download my script and run it” • The build bot ran it, and the attacker’s code went into the official release • 4 bad releases shipped a crypto miner to a package with ~60M downloads source: ReversingLabs, Dec 2024
33. A Branch Name Hacked the Build source: ReversingLabs, Dec 2024
34. The Malware Used Your AI Agent What It Did • A hacked code-editor plugin started every AI agent on the machine, with all safety checks off • A 2,000-word prompt told the agent to play “investigator” and find passwords and keys • The agent uploaded what it found to GitHub, logged in as the victim Impact • Severity score 10 out of 10, the highest possible • It tried 5 different AI agents and quietly skipped any that weren’t installed GitHub Security Advisory source: Socket, Mar 2026; CVE-2026-28353
35. What Should You Check Before Merge? HOUSTON NEEDS ACTION PLAN!
36. The Pre-Merge Checklist
37. How Do You Contain The Agent? LIMITTING THE DAMAGE SHOULD AGENT GET COMPROMISED
38. Limit What It Can See Sandbox • Lock down both - the files it can touch and the sites it can reach • Without network limits, a hijacked agent can send your keys out • Without file limits, it can break out of the sandbox Bonus • Sandboxing cut permission prompts by 84% source: Anthropic Engineering, “Beyond permission prompts,” Oct 2025
39. Limit What It Can Do • Keep safety checks on, except in a throwaway sandbox • Settings files can run code, so check them like code • Run nothing from a new repo until you trust it • Give agents keys that expire fast and valid only for one job source: Check Point Research, Feb 2026; Socket, Mar 2026; GitGuardian, 2026
40. Tell It, Then Check It Anyway Instruct • Put security rules in the agent’s rules file • Ask it to “find problems in your answer,” then “improve it” Don’t • Tell it to “act as a security expert.” OpenSSF doesn’t currently recommend it. Why Checking Wins • The best AI model still fails 32% of security tests • Bigger models barely help: large 53% vs small 51% pass rate • Adding scanning improved secure code by up to 47% source: OpenSSF, “Security-Focused Guide for AI Code Assistant Instructions,” Aug 2025; Veracode, 2026 GenAI Code Security Report, Jul 2026; arXiv:2603.22717, 2026
41. How Do You Automate The Safety Net?
42. Be Lazy, Wait Before You Install A 14-day wait skips the attack window • Most attacks are caught within hours or days of release • A 7-day wait would have blocked most of the 10 big attacks studied; 14 days blocked all but 1 • One major package’s malicious versions were caught in about 3 hours source: Woodruff, “We should all be using dependency cooldowns,” Nov 2025; Datadog Security Labs, Apr 2026
43. Put Scanning In The Pipeline • Five checks on every PR: code, dependencies, secrets, infrastructure, policy • A failed check should block the merge, irrespective of the author of the PR – human / AI • Most of these tools are free, and most teams already run one or two
44. Keep a Record of How AI Code Was Made • A bug reaches production. Can you trace how it was written? • Save four things when code is generated: the model version, the prompt, the files it read, and the tools it could use • Without that record, you can only patch and RCA investigation will be difficult • Track AI-written and human-written code separately, so you can compare their bug rates source: CMU SEI, FLARE-AI, Jul 2026
45. Ready for Homework?
46. Homework for Monday!
47. AI output is untrusted input, at every boundary. Thank You! Alok Tibrewala Connect aloktibrewala.com Session Feedback