Mr. Brokebot: Lethal language attacks against AI agents

This talk is a fast, practical tour of real-world AI hacking: not sci-fi doom, but hands-on exploits against LLMs and agents that are already embedded in developer tools, browsers, chatbots, and cloud systems.

NOTE: The Firefox example is a PROTOTYPE.

I'm not sure how well it will come across as a slides-only deck.

Transcript

1. Luke Crouch, 2026 eps0.1_leth4l_langu4ge_att4cks.md MR. BROKEBOT
2. speakerdeck.com/groovecoder
3. eps0.2_my_s0urces
4. my own tinkering
5. Lethal Trifecta Simon Willison https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
6. Month of AI Bugs Johann Rehberger https://embracethered.com/blog/posts/2025/wrapping-up-month-of-ai-bugs/
7. https://www.youtube.com/watch?v=TWhKGqYQT9g
8. eps0.3_ag3nda
9. 1. large language models (3m) 2. LLM attacks (20m) 3. agents (3m) 4. agent attacks (3m)
10. eps1.0_l4rge_langu4ge_m0dels
11. https://www.youtube.com/watch?v=LPZh9BOjkQs
12. https://www.youtube.com/watch?v=LPZh9BOjkQs
13. https://www.youtube.com/watch?v=LPZh9BOjkQs
14. https://www.youtube.com/watch?v=LPZh9BOjkQs
15. https://www.youtube.com/watch?v=LPZh9BOjkQs
16. https://www.youtube.com/watch?v=LPZh9BOjkQs
17. https://www.youtube.com/watch?v=LPZh9BOjkQs
18. https://www.youtube.com/watch?v=LPZh9BOjkQs
19. https://www.youtube.com/watch?v=LPZh9BOjkQs
20. https://www.youtube.com/watch?v=LPZh9BOjkQs
21. https://www.youtube.com/watch?v=LPZh9BOjkQs
22. https://www.youtube.com/watch?v=LPZh9BOjkQs
23. https://www.youtube.com/watch?v=LPZh9BOjkQs
24. input output
25. input output
26. 1. large language models: ✓ questions? 2. llm attacks 3. agents 4. agent attacks
27. 
 eps2.0_llm_att4cks
28. 1. hallucination (5m) 2. jailbreak (5m) 3. prompt injection (10m) llm attacks
29. 
 eps2.1_hallucinati0n
30. llm attack hallucination • generates incorrect or incomplete content • factual inaccuracies, fabricated information, contradictions, omissions, etc. • inherit characteristic of LLMs • larger models tend to hallucinate less https://cacm.acm.org/practice/the-price-of-intelligence/
31. input output with hallucinations
32. is this really an “attack”?
33. it’s de fi nitely a problem
34. https://futurism.com/local-restaurant-exhausted-google-ai
35. llm risk hallucination • medical misinformation [1] • mental health misinformation [2] • legal or policy citations [3] • software
36. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or
37. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or
38. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or
39. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or yet
40. so, attackers create packages with names that LLMs hallucinate
51. 
 eps2.1.1_slop_squ4tting
52. https://arxiv.org/abs/2406.10279
53. https://arxiv.org/abs/2406.10279
54. https://arxiv.org/abs/2406.10279
55. https://arxiv.org/abs/2406.10279
56. how to stop hallucinations
57. how to stop reduce hallucinations remember: these are an inherit characteristic of llms
58. 1. retrieval-augmented generation 2. self-re fi nement 3. fi ne-tuning https://arxiv.org/abs/2406.10279 llm attack hallucination
59. https://arxiv.org/abs/2406.10279
60. https://arxiv.org/abs/2406.10279
63. 
 eps2.1_hallucinati0n questions?
64. 1. hallucination ✓ 2. jailbreak 3. prompt injection llm attack
65. 
 eps2.2_j4ilbreak
66. input output
67. input output behavior alignment
68. large language model behavior alignment • aka “alignment layer” or “safety layer” • trained into the model • by supervised fi ne-tuning • to shape an llm’s output - especially “safe” output … https://cacm.acm.org/practice/the-price-of-intelligence/
69. For example …
71. llm attack jailbreak • manipulate an llm into violating its safety alignment • exploits LLMs’ lack of situational awareness or adversarial intention https://cacm.acm.org/practice/the-price-of-intelligence/
72. input output behavior alignment jailbreak llm attack malicious unsafe
73. A look at AI Security with Mark Russinovich
74. 
 eps2.2.1_skel3t0n_k3y
75. A look at AI Security with Mark Russinovich
76. 1. skeleton key ✓ 2. context compliance 3. crescendo llm attacks jailbreak
77. 
 eps2.2.2_cont3xt_compli4nce
78. A look at AI Security with Mark Russinovich
79. https://x.com/itsandrewgao/status/1964117887943094633
80. https://x.com/itsandrewgao/status/1964117887943094633
81. https://x.com/itsandrewgao/status/1964117887943094633
82. 1. skeleton key ✓ 2. context compliance ✓ 3. crescendo llm attacks jailbreak
83. 
 eps2.2.3_cresc3nd0
84. A look at AI Security with Mark Russinovich
85. A look at AI Security with Mark Russinovich
86. A look at AI Security with Mark Russinovich
87. A look at AI Security with Mark Russinovich
88. A look at AI Security with Mark Russinovich
89. How to stop jailbreaks?
90. How to stop jailbreaks: don’t
91. How to stop jailbreaks: don’t leave it to model-makers
92. https://d2jud02ci9yv69.cloudfront.net/2025-04-28-do-not-write-jaibreak-papers-57/blog/do-not-write-jaibreak-papers/
93. 1. hallucination ✓ 2. jailbreak ✓ 3. prompt injection llm attacks
94. 
 eps2.3_pr0mpt_inj3ction
95. 
 eps2.3_pr0mpt_inj3ction if you’re building AI, I feel bad for you son, you got 99 probLLMs and can’t patch this one
96. pretty much everyone says this is the foundational problem
97. I (and many others) say this is the foundational problem: mixing input data and instructions
98. user: hiring manager
99. user: hiring manager intent: evaluate resumes
100. evaluation resume
101. user: hiring manager intent: evaluate resumes attacker: job applicant
102. https://www.linkedin.com/posts/anmol-gupta7_this-resume-can-bypassall-llm-based-candidate-activity-7335636441978281984-EPRc/
103. https://www.linkedin.com/posts/anmol-gupta7_this-resume-can-bypassall-llm-based-candidate-activity-7335636441978281984-EPRc/
105. llm attack fake evaluation malicious resume prompt injection
106. another hypothetical example …
107. another hypothetical example … WE DID NOT DO THIS!
108. user: job applicant
109. user: job applicant intent: make perfect application
111. job posting job application
112. user: job applicant intent: make perfect application counter-attacker: hiring manager
120. job posting job application llm attack prompt injection malicious fl agged
121. again: WE DID NOT DO THIS!
122. Firefox example
 (⚠ lots of code ahead)
123. 
 aux_sql_inj3ction
124. https://xkcd.com/327/
125. mixing input data and sql instructions
128. Structured Query Language
129. Malicious
 Structured Query Language Injection
130. Malicious
 English Language Injection
132. PROTOTYPE
133. PROTOTYPE
134. PROTOTYPE
135. PROTOTYPE
136. PROTOTYPE
137. PROTOTYPE
138. PROTOTYPE
139. https://github.com/Firefox-AI/ fi refox-prototypes/blob/6d4eeae66d4e1d8997e6798ce4adf678111d3a3f/browser/components/smartwindow/content/suggestions.mjs#L468-L499
140. https://github.com/Firefox-AI/ fi refox-prototypes/blob/6d4eeae66d4e1d8997e6798ce4adf678111d3a3f/browser/components/smartwindow/content/suggestions.mjs#L468-L499
141. https://github.com/Firefox-AI/ fi refox-prototypes/blob/6d4eeae66d4e1d8997e6798ce4adf678111d3a3f/browser/components/smartwindow/content/suggestions.mjs#L468-L499
149. PROTOTYPE
150. PROTOTYPE
151. I (and many others) say this is the foundational problem: mixing input data and instructions
153. PROTOTYPE
154. how to stop prompt injections
155. how to stop reduce prompt injections they’re possible because LLMs are probabilistic
156. 
 aux_ev4ls
170. how to stop reduce prompt injections
171. 1. sanitize inputs 2. harden prompts 3. add guardrails llm attack prompt injection
173. 
 eps2.3.1_sanit1ze_1nputs
180. 
 eps2.3.2_hard3n_pr0mpts
181. https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html#structured-prompts-with-clear-separation
189. 
 eps2.3.3_guardra1ls
190. input output guard rail guard rail
191. 1. static regex 2. LLM-as-judge 3. custom classi fi er model llm attack prompt injection guardrails
203. 1. static regex (Deterministic) 2. LLM-as-judge (LLM Judge) 3. custom classi fi er model (HF Semantic) llm attack prompt injection guardrails
214. 
 eps2.3.3_guardra1ls
215. 
 guardra1ls 
 
 do not work
216. 
 guardra1ls 
 
 do not work … on their own
218. https://arxiv.org/html/2510.09023v1
219. https://arxiv.org/html/2510.09023v1
221. 1. sanitize inputs ✓ 2. harden prompts ✓ 3. add guardrails ✓ llm attack prompt injection
222. 1. hallucination ✓ 2. jailbreak ✓ 3. prompt injection ✓ questions? llm attacks
223. 1. large language models ✓ 2. llm attacks ✓ questions? 3. agents 4. agent attacks
224. eps3.0_ag3nts
225. https://simonwillison.net/2025/Sep/18/agents/
226. https://simonwillison.net/2025/Sep/18/agents/
227. input output
228. input 🛠 tools output
229. input 🛠 tools result output
230. input 🛠 tools result planning reasoning output
231. input 🛠 tools result planning reasoning output goal achieved?
232. input 🛠 tools result planning reasoning output goal achieved?
233. input 🛠 tools result planning reasoning output goal achieved?
237. eps3.1_t00ls
239. https://chatgpt.com/features/connectors
240. https://chatgpt.com/features/connectors
242. 
 eps4.0_ag3nt_att4cks
243. https://labs.zenity.io/p/agent fl ayer-chatgpt-connectors-0click-attack-5b41
248. https://labs.zenity.io/p/agent fl ayer-chatgpt-connectors-0click-attack-5b41
252. Lethal Trifecta Simon Willison https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
254. https://chatgpt.com/features/connectors
263. how to stop agent attacks
265. https://github.com/mozilla/bugbug/blob/6c876a6c5e87a2f82b33ef7b6cd345e351dcd1f4/mcp/src/bugbug_mcp/server.py#L35
266. https://github.com/mozilla/bugbug/blob/master/bugbug/tools/code_review/prompts.py#L11
267. https://github.com/mozilla/bugbug/blob/6c876a6c5e87a2f82b33ef7b6cd345e351dcd1f4/mcp/src/bugbug_mcp/server.py#L105
268. https://phabricator.services.mozilla.com/D270778
276. https://arxiv.org/html/2510.09023v1
278. 1. large language models ✓ 2. llm attacks ✓ 3. agents ✓ 4. agent attacks ✓
285. hah, gotcha!
287. wait … an iPhone in Mexico?
290. 1. large language models ✓ 2. llm attacks ✓ 3. agents ✓ 4. agent attacks ✓ questions?