Mr. Brokebot: Lethal language attacks against AI agents

Transcript

1. Mr. BrokeBot Lethal language attacks against AI agents and how to stop them Luke Crouch, 2025-12-05
2. Me Web dev: 2001-2017 Security: 2017-Present
3. My Sources
4. My own tinkering Luke Crouch
5. AI as Normal Technology Arvind Narayanan, Sayash Kapoor https://knightcolumbia.org/content/ai-as-normal-technology
6. Lethal Trifecta Simon Willison https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
7. The Price of Intelligence Mark Russinovich, Ahmed Salem, Santiago Zanella- Béguelin, and Yonatan Hunger https://cacm.acm.org/practice/the-price-of-intelligence/
8. A look at AI security Mark Russinovich https://events.zoom.us/ejl/AjNT9jigixu5ee0EB-r5hLj8jU_QKnazopGtDDEVMg1-OQxZK03J~A5LjepLzotXy8SjW-eouBEZsyLOwdn9Sm-S9GItE6TSecDjfCM-dCQH5_32Pgpln6VO-2uxAXe1PIaOqa7Tz-1Z0kDaji/_Wa1TEuUQj-8IvG47DQ7SA/video/VeEF3rkpSb2lpEZXpHzxEg?_dlaccessid=g1TcNbLRTV6P-90seGZYuw&directLinkOpen=1
9. My own hacking Luke Crouch
10. Month of AI Bugs Johann Rehberger https://embracethered.com/blog/posts/2025/wrapping-up-month-of-ai-bugs/
11. Agenda
12. 1. Large Language Models 2. LLM attacks 3. Agents 4. Agent Attacks
13. Large Language Models
14. https://www.youtube.com/watch?v=LPZh9BOjkQs
15. https://www.youtube.com/watch?v=LPZh9BOjkQs
16. https://www.youtube.com/watch?v=LPZh9BOjkQs
17. https://www.youtube.com/watch?v=LPZh9BOjkQs
18. https://www.youtube.com/watch?v=LPZh9BOjkQs
19. https://www.youtube.com/watch?v=LPZh9BOjkQs
20. https://www.youtube.com/watch?v=LPZh9BOjkQs
21. https://www.youtube.com/watch?v=LPZh9BOjkQs
22. https://www.youtube.com/watch?v=LPZh9BOjkQs
23. https://www.youtube.com/watch?v=LPZh9BOjkQs
24. https://www.youtube.com/watch?v=LPZh9BOjkQs
25. https://www.youtube.com/watch?v=LPZh9BOjkQs
26. https://www.youtube.com/watch?v=LPZh9BOjkQs
27. https://www.youtube.com/watch?v=LPZh9BOjkQs
28. https://www.youtube.com/watch?v=LPZh9BOjkQs
29. Input Output
30. Input Output
31. 1. Large Language Models ✓ 2. LLM attacks 3. Agents 4. Agent Attacks
32. LLM Attacks
33. The Price of Intelligence Mark Russinovich, Ahmed Salem, Santiago Zanella- Béguelin, and Yonatan Hunger
34. 1. Hallucination 2. Jailbreak 3. Prompt Injection
35. Hallucination
36. LLM Attacks Hallucination • generates incorrect or incomplete content • factual inaccuracies, fabricated information, contradictions, omissions, etc. • inherit characteristic of LLMs • larger models tend to hallucinate less https://cacm.acm.org/practice/the-price-of-intelligence/
37. Input Output with Hallucinations
38. Is this really an “attack”?
39. I say no - an attack requires an attacker
40. “Harmless” Hallucination
41. “Harmless” Hallucination still can be risks …
42. Hallucination specials that don’t exist https://futurism.com/local-restaurant-exhausted-google-ai
43. “Harmful” Hallucination?
44. LLM Attacks Risk Hallucination • Medical misinformation [1] • Mental health misinformation [2] • False legal or policy citations [3]
45. Hallucination Attack
46. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or ask ChatGP for help integrating arangodb in node.js …
47. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or … it will tell you which npm package to install …
48. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or … even if that package doesn’t exist
49. https://securaize.substack.com/p/can-you-trust-chatgpts-package-or … even if that package doesn’t exist yet
50. https://futurism.com/local-restaurant-exhausted-google-ai Slopsquatting registering a non-existent software package name that a large language model (LLM) may hallucinate in its output, whereby someone unknowingly may copy- paste and install the software package without realizing it is fake
51. registering a non-existent software package name that a large language model (LLM) may hallucinate in its output, whereby someone unknowingly may copy- paste and install the software package without realizing it is fake https://arxiv.org/abs/2406.10279 Slopsquatting
52. Input Output with malware package
53. How to stop hallucinations
54. How to stop reduce hallucinations remember: these are an inherit characteristic of LLMs
55. despite mitigation efforts, AI hallucination rates still generally vary from as low as 2% in some models for short summarization tasks and as high as 50% for more complex tasks and speci fi c domains, such as law and healthcare https://cacm.acm.org/practice/the-price-of-intelligence/#sec5
56. https://cacm.acm.org/practice/the-price-of-intelligence/#sec5 Reduce hallucinations
57. 1. External Groundedness checkers 2. Fact Correction 3. Improved RAG Systems 4. Ensemble methods https://cacm.acm.org/practice/the-price-of-intelligence/#sec5
58. 1. Hallucination ✓ 2. Jailbreak 3. Prompt Injection
59. Jailbreak
60. Input Output
61. Input Output Behavior Alignment
62. Large Language Model Behavior Alignment • aka “Outer alignment”, “Safety layer”, “Alignment layer” • part of training the model • Supervised fi ne-tuning, Reinforcement learning from human feedback, etc. • shape an LLM’s output - especially “safe” output … https://cacm.acm.org/practice/the-price-of-intelligence/
63. Large Language Model Behavior Alignment • aka “Outer alignment”, “Safety layer”, “Alignment layer” • part of training the model • Supervised fi ne-tuning, Reinforcement learning from human feedback, etc. • shape an LLM’s output - especially “safe” output … • … according to whomever is training the model! https://cacm.acm.org/practice/the-price-of-intelligence/
64. For example …
66. LLM Attacks Jailbreak • manipulate an LLM into violating its established guidelines, ethical constraints, or trained alignments • exploit the fl exibility and contextual understanding capabilities of LLMs • e.g., some military or law enforcement need to understand how pipe bombs are made … • but some military or law enforcement should NOT … • but maybe they’re writing a documentary … • but … https://cacm.acm.org/practice/the-price-of-intelligence/
67. LLM Attacks Jailbreak Behavior Alignment unsafe Output malicious Input
68. A look at AI Security with Mark Russinovich
69. A look at AI Security with Mark Russinovich
70. A look at AI Security with Mark Russinovich
71. A look at AI Security with Mark Russinovich
72. A look at AI Security with Mark Russinovich
73. A look at AI Security with Mark Russinovich
74. LLM Attacks Jailbreak Behavior Alignment unsafe Output malicious Input
75. How to stop jailbreaks
76. It’s on the model-makers, IMO, so let’s move on …
77. 1. Hallucination ✓ 2. Jailbreak ✓ 3. Prompt Injection
78. Prompt injection
79. I say this is the foundational problem
80. SQL injection
81. https://xkcd.com/327/
82. mixing input data and sql instructions
83. LLM Attacks prompt injection • LLM follows instructions in the data, rather than the user’s instructions • exploits the LLM mixing input data and language instructions https://cacm.acm.org/practice/the-price-of-intelligence/
84. https://cacm.acm.org/opinion/llms-data-control-path-insecurity/
85. LLM Attacks Malicious Output malicious Input prompt injection
86. LLM Attacks Malicious Output malicious Input prompt injection may not violate the LLM but violates the user intent
87. For example …
89. https://x.com/itsandrewgao/status/1964117887943094633
90. https://x.com/itsandrewgao/status/1964117887943094633
91. 1. Large Language Models ✓ 2. LLM attacks ✓ 3. Agents 4. Agent Attacks
92. Agents
93. Input Output
94. Input Output
95. Input 🛠 Tools Output
96. Input 🛠 Tools Result Output
97. Input 🛠 Tools Result Output
98. Input 🛠 Tools Result Planning/ Reasoning Output
99. Input 🛠 Tools Result Planning/ Reasoning Output Prompt not complete
100. Input 🛠 Tools Result Planning/ Reasoning Output Prompt complete
101. 1. Large Language Models ✓ 2. LLM attacks ✓ 3. Agents ✓ 4. Agent Attacks
102. Agent Attacks
103. Input Output
104. LLM Attacks Malicious Input unintended Output
105. 🛠 Tools Result Planning/ Reasoning Prompt complete Malicious Input unintended Output Agent Attacks
106. Input 🛠 Tools Result Planning/ Reasoning Output Prompt not complete
107. Input 🛠 Tools Result Planning/ Reasoning Output Prompt not complete
108. https://chatgpt.com/features/connectors
110. Ability to
 change data Agent Tools
111. Ability to
 change data Agent Tools
112. Ability to
 change data Agent Tools Fetch
 more content
113. Lethal Language Attacks Exposure to
 untrusted content Ability to
 change data Lethal Pair
114. External communication Ability to
 change data Agent Tools Fetch
 more content
115. Access to
 private data External communication Ability to
 change data Agent Tools Fetch
 more content
116. Lethal Language Attacks Exposure to untrusted content Access to
 private data External communication Ability to
 change data Lethal Trifecta
117. Lethal Language Attacks Exposure to untrusted content Access to
 private data External communication Ability to
 change data Lethal Trifecta Lethal Pair