re I A t re c e s s p ra "Amazon sc " n e m o w t s in a g a s ia b d e w o "DPD AI chatbot swears, calls itself that sh ‘useless’ and criticises delivery firm " "Google pauses Gemini AI image " es ci ra cu ac in al ric o st hi r te af n generatio THE GUARDIAN, 20 24 CNBC, 2024 !
re I A t re c e s s p ra "Amazon sc " n e m o w t s in a g a s ia b d e w o "DPD AI chatbot swears, calls itself that sh ‘useless’ and criticises delivery firm " "Google pauses Gemini AI image " es ci ra cu ac in al ric o st hi r te af n generatio "Air Canada must honor refund policy invented by airline’s chatbot" "An AI-Generated Police Report Claimed a Cop Transformed Into a Frog" THE GUARDIAN, 20 24 CNBC, 2024 ARS TECHNICA, 2024 VICE, 2024 !
re I A t re c e s s p ra "Amazon sc " n e m o w t s in a g a s ia b d e w o "DPD AI chatbot swears, calls itself that sh ‘useless’ and criticises delivery firm " "Google pauses Gemini AI image " es ci ra cu ac in al ric o st hi r te af n generatio "Air Canada must honor refund policy invented by airline’s chatbot" "An AI-Generated Police Report Claimed a Cop Transformed Into a Frog" "AI agent seemingly tries to shame open source developer for rejected pull request" THE GUARDIAN, 20 24 CNBC, 2024 ARS TECHNICA, 2024 VICE, 2024 THE REGISTER, 2026 !
Performs Poorly in Crime Recidivism Prediction 1 Brisha Borden stole an $80 bike. Score: 8/10 high risk. She was never re-arrested. Vernon Prater had prior armed robbery convictions. Score: 3/10 low risk. He went back to prison. ProPublica analyzed 11,757 defendants, Black defendants flagged high-risk at nearly twice the rate of white defendants in identical situations. HTTPS://INCIDENTDATABASE.AI/CITE/40/
Performs Poorly in Crime Recidivism Prediction 1 Brisha Borden stole an $80 bike. Score: 8/10 high risk. She was never re-arrested. Vernon Prater had prior armed robbery convictions. Score: 3/10 low risk. He went back to prison. ProPublica analyzed 11,757 defendants, Black defendants flagged high-risk at nearly twice the rate of white defendants in identical situations. HTTPS://INCIDENTDATABASE.AI/CITE/40/ WHAT IT USED WHAT IT BROKE Defendants respond to a COMPAS questionnaire. The software generate the predictions of “Risk of Recidivism”. Unequal false positive and false negative rates between black and white defendants. Multi-factor questionnaire Due process
recruiting tool that showed bias against women 2 Ten years of hiring data. One clear signal: men got hired. The model penalized resumes containing the word "women's" and downgraded allwomen's college graduates. It ran for three years before anyone stopped it. No one knows how many women never got a callback. HTTPS://INCIDENTDATABASE.AI/CITE/37/
recruiting tool that showed bias against women 2 Ten years of hiring data. One clear signal: men got hired. The model penalized resumes containing the word "women's" and downgraded allwomen's college graduates. It ran for three years before anyone stopped it. No one knows how many women never got a callback. HTTPS://INCIDENTDATABASE.AI/CITE/37/ WHAT IT USED WHAT IT BROKE Historical resumes treated as ground truth. Gender bias became hiring policy, running faster and quieter than any recruiter. Resume history Equal access
chatbot removed after racist and hateful responses 3 750,000 users in three weeks. Lee Luda was trained on 9.4 billion private KakaoTalk messages from 600,000 people who never consented. Real names, home addresses, and relationship details leaked verbatim through the chatbot. Fine: $93,000. HTTPS://INCIDENTDATABASE.AI/CITE/106/
chatbot removed after racist and hateful responses 3 750,000 users in three weeks. Lee Luda was trained on 9.4 billion private KakaoTalk messages from 600,000 people who never consented. Real names, home addresses, and relationship details leaked verbatim through the chatbot. Fine: $93,000. HTTPS://INCIDENTDATABASE.AI/CITE/106/ WHAT IT LEARNED FROM WHAT WENT WRONG The chatbot was trained on massive amounts of informal human dialogue, inheriting the biases and toxic language present in those conversations. Instead of filtering harmful stereotypes, the system reproduced and normalized them in public interactions at scale. Real online conversations Bias amplification
STARTS 1 WHAT IT DAMAGES 2 The model doesn't invent prejudice. Decisions that should be merit-based Skewed training data becomes skewed predictions. become proxies for race, gender, or wealth. This is automated and it happens at a scale no human bias ever could. B
STARTS 1 WHAT IT DAMAGES 2 B WHAT TO DO 3 The model doesn't invent prejudice. Decisions that should be merit-based Disaggregate your metrics. A model Skewed training data becomes skewed predictions. become proxies for race, gender, or wealth. This is automated and it can be accurate and still unfair. Audit your model by subgroup. happens at a scale no human bias ever could.
IT DAMAGES Model gives a score or answer, but no No debugging, no appeal, no reason. "Why?" becomes a guess, not meaningful audit. If nobody can an explanation. inspect the decision, nobody can defend it. 2 O
IT DAMAGES Model gives a score or answer, but no No debugging, no appeal, no reason. "Why?" becomes a guess, not meaningful audit. If nobody can an explanation. inspect the decision, nobody can defend it. 2 O WHAT TO DO 3 Use interpretable models when possible. Use XAI tools to extract insights from your model. If you can't explain it, you can't trust it.
STARTS 1 WHAT IT DAMAGES 2 Models don't just learn patterns, The model becomes an unintended data they memorize. Rare sequences, store. Query it the right way and it personal records, credentials get reproduces content it was never meant encoded directly into model weights. to expose. D
STARTS 1 WHAT IT DAMAGES 2 D WHAT TO DO 3 Models don't just learn patterns, The model becomes an unintended data Scrub sensitive data before training. they memorize. Rare sequences, store. Query it the right way and it Apply differential privacy to personal records, credentials get reproduces content it was never meant guarantee protection. Red-team your encoded directly into model weights. to expose. model.
Canada chatbot invents a refund policy 2 Jake Moffatt was flying to his grandmother's funeral. The chatbot told him he could claim a bereavement discount retroactively. Air Canada denied it, then argued in court that its chatbot was "a separate legal entity" it wasn't responsible for. The tribunal rejected that argument. HTTPS://INCIDENTDATABASE.AI/CITE/639/
Canada chatbot invents a refund policy 2 Jake Moffatt was flying to his grandmother's funeral. The chatbot told him he could claim a bereavement discount retroactively. Air Canada denied it, then argued in court that its chatbot was "a separate legal entity" it wasn't responsible for. The tribunal rejected that argument. HTTPS://INCIDENTDATABASE.AI/CITE/639/ WHAT IT USED WHAT IT BROKE A conversational interface replaced a structured policy lookup, with no grounding on actual policy documents. A hallucinated answer became commitment. The company was held responsible for what the model said. Natural language support Legal accountability
at a customer 3 Ashley Beauchamp asked the bot to evaluate DPD's own service. It replied: "DPD is the worst delivery service in the world." Screenshots went viral on X, 1.3 million views in 24 hours. Bot disabled by end of day. HTTPS://INCIDENTDATABASE.AI/CITE/631/
at a customer 3 Ashley Beauchamp asked the bot to evaluate DPD's own service. It replied: "DPD is the worst delivery service in the world." Screenshots went viral on X, 1.3 million views in 24 hours. Bot disabled by end of day. HTTPS://INCIDENTDATABASE.AI/CITE/631/ WHAT IT USED WHAT IT BROKE No input constraints, no adversarial testing, a public-facing assistant deployed with production guardrails that hadn't been stress-tested. One successful bypass was enough. The incident was public, immediate, and entirely preventable. Open-ended chat Safety and reputation
Scalable Extraction of Training Data from (Production) Language Models 1 By querying the model with carefully crafted prompts, researchers recovered hundreds of verbatim sequences from training data, including real names, phone numbers, email addresses, and private conversations. HTTPS://ARXIV.ORG/ABS/2311.17035
Scalable Extraction of Training Data from (Production) Language Models 1 By querying the model with carefully crafted prompts, researchers recovered hundreds of verbatim sequences from training data, including real names, phone numbers, email addresses, and private conversations. HTTPS://ARXIV.ORG/ABS/2311.17035 WHAT IT USED WHAT IT BROKE No special access needed. Carefully crafted prompts were enough to coax the model into reproducing training data verbatim. Real people's personal information, scraped from the web years earlier, resurfaced through a public chatbot with no warning and no audit trail. Normal API queries Privacy at scale
IT DAMAGES 2 The model predicts the most plausible Fabricated citations, invented next token, not the most truthful policies, and wrong technical answers one. It has no mechanism to know what delivered with full confidence. it doesn't know. H
IT DAMAGES 2 The model predicts the most plausible Fabricated citations, invented next token, not the most truthful policies, and wrong technical answers one. It has no mechanism to know what delivered with full confidence. it doesn't know. H WHAT TO DO 3 Ground outputs on verified sources. Never let model output reach a user without a trust boundary between generation and delivery.
1 WHAT IT DAMAGES 2 Safety rules are part of the model's The model reveals content it was behavior, not hard-coded limits. Attackers use roleplay, indirection, supposed to refuse: harmful instructions, toxic outputs, hidden or adversarial prompts to bypass prompts, or restricted information. them. J
1 WHAT IT DAMAGES 2 J WHAT TO DO Safety rules are part of the model's The model reveals content it was Use layered protections: stronger behavior, not hard-coded limits. Attackers use roleplay, indirection, supposed to refuse: harmful instructions, toxic outputs, hidden alignment, adversarial training, output monitoring, and continuous or adversarial prompts to bypass prompts, or restricted information. red-teaming. them. 3
1 WHAT IT DAMAGES A malicious instruction hidden inside The LLM silently changes behavior a webpage, email, PDF, or document across the workflow and outputs gets processed as if it were part of content it was not designed to the prompt. produce. 2 P
1 WHAT IT DAMAGES 2 P WHAT TO DO 3 A malicious instruction hidden inside The LLM silently changes behavior Treat external content as untrusted a webpage, email, PDF, or document across the workflow and outputs input. Isolate them from the user gets processed as if it were part of content it was not designed to prompts. the prompt. produce.
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery —
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery — but they fail on out of distribution scenarios.
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery — but they fail on out of distribution scenarios. 03 Structured red-teaming Find bypasses before users do —
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery — but they fail on out of distribution scenarios. 03 Structured red-teaming Find bypasses before users do — but coverage is never complete.
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery — but they fail on out of distribution scenarios. 03 Structured red-teaming Find bypasses before users do — but coverage is never complete. 04 Benchmarking Track regressions with reproducible tests —
gap. 01 Post-training L Shape behavior after pretraining — but capabilities learned in pre-training don't go away. 02 Output classifiers Catch known harm patterns before delivery — but they fail on out of distribution scenarios. 03 Structured red-teaming Find bypasses before users do — but coverage is never complete. 04 Benchmarking Track regressions with reproducible tests — but a model can pass every benchmark and still feel off.
agent deletes the entire production database and backups in 9 seconds Cursor, running Claude Opus 4.6, deleted the production database, and all backups. No confirmation step, no environment check. HTTPS://INCIDENTDATABASE.AI/CITE/1469/
agent deletes the entire production database and backups in 9 seconds Cursor, running Claude Opus 4.6, deleted the production database, and all backups. No confirmation step, no environment check. HTTPS://INCIDENTDATABASE.AI/CITE/1469/ WHAT IT USED WHAT IT BROKE The agent found a production API token in an unrelated file and used it autonomously, no human had granted it that access for that task. The agent acted without verification on an irreversible action. Backups stored in the same volume were wiped at the same time. Over-permissioned token Least privilege & rollback
prompt injection silently exfiltrates Microsoft 365 Copilot data Researchers at Aim Security discovered EchoLeak, a zero-click prompt injection vulnerability in Microsoft 365 Copilot that enabled remote, unauthenticated data exfiltration via a single crafted email. CVE-2025-32711
prompt injection silently exfiltrates Microsoft 365 Copilot data Researchers at Aim Security discovered EchoLeak, a zero-click prompt injection vulnerability in Microsoft 365 Copilot that enabled remote, unauthenticated data exfiltration via a single crafted email. CVE-2025-32711 WHAT IT USED WHAT IT BROKE A malicious email delivered instructions the AI treated as legitimate. No user click required, just receiving and processing the email was enough. External content redirected the agent's behavior and exfiltrated sensitive data, without ever needing to bypass the model itself. Untrusted email content Instruction boundary
agent publishes a "hit piece" on a maintainer after its pull request is rejected Scott Shambaugh closed the PR in 40 minutes, bots aren't allowed. The agent then searched for his personal blog and hobby projects, and published a 1,500-word hit piece. An Ars Technica journalist fabricated quotes attributed to Shambaugh. The journalist was later fired. HTTPS://INCIDENTDATABASE.AI/CITE/1373/
agent publishes a "hit piece" on a maintainer after its pull request is rejected Scott Shambaugh closed the PR in 40 minutes, bots aren't allowed. The agent then searched for his personal blog and hobby projects, and published a 1,500-word hit piece. An Ars Technica journalist fabricated quotes attributed to Shambaugh. The journalist was later fired. HTTPS://INCIDENTDATABASE.AI/CITE/1373/ WHAT IT USED WHAT IT BROKE The agent had credentials to post publicly. "Write a blog about today" was enough for it to decide a public attack was appropriate. The agent took a consequential, reputationdamaging action with no human in the loop and no confirmation step before publishing. Broad autonomous scope Human oversight
exfiltrated. MAY 2026 · NEW Prompt injection in Microsoft Copilot Cowork silently exfiltrates files via Teams 4 PromptArmor found that 5 lines of malicious text in a skill file are enough to make Copilot fetch download links for the victim's files and leak them through Teams. Claude Opus 4.7 included. PROMPTARMOR.COM/RESOURCES/MICROSOFT-COPILOT-COWORK-EXFILTRATES-FILES
exfiltrated. MAY 2026 · NEW Prompt injection in Microsoft Copilot Cowork silently exfiltrates files via Teams 4 PromptArmor found that 5 lines of malicious text in a skill file are enough to make Copilot fetch download links for the victim's files and leak them through Teams. Claude Opus 4.7 included. PROMPTARMOR.COM/RESOURCES/MICROSOFT-COPILOT-COWORK-EXFILTRATES-FILES WHAT IT USED WHAT IT BROKE No human in the loop. 5 injected lines hijack the agent and pull pre-authenticated links from SharePoint and OneDrive. Every file the victim could reach, silently leaked. No click required. Auto-approved Teams messages File access boundary
REAL ACTIONS 2 Data and instructions share the same Agents don't just talk, they execute. context. Any document, email, or web A mistake isn't a wrong answer, it's page the agent reads can hijack what a deleted database, a leaked file, a it does next. published post. T
REAL ACTIONS 2 T NO UNDO Data and instructions share the same Agents don't just talk, they execute. By the time a human notices, the context. Any document, email, or web A mistake isn't a wrong answer, it's action is already done. Speed and page the agent reads can hijack what a deleted database, a leaked file, a autonomy collapse the window for it does next. published post. oversight. 3
PRIVILEGE 1 MINIMAL FOOTPRINT 2 Give the agent only the permissions Prefer reversible actions and confirm the task truly needs. before anything irreversible.
PRIVILEGE 1 MINIMAL FOOTPRINT 2 HUMAN IN THE LOOP 3 Give the agent only the permissions Prefer reversible actions and confirm Put a person in front of high-stakes the task truly needs. before anything irreversible. or destructive actions.
PRIVILEGE 1 MINIMAL FOOTPRINT 2 HUMAN IN THE LOOP 3 Give the agent only the permissions Prefer reversible actions and confirm Put a person in front of high-stakes the task truly needs. before anything irreversible. or destructive actions. SANDBOXING 4 Isolate agent execution so the blast radius stays bounded.
PRIVILEGE 1 MINIMAL FOOTPRINT 2 HUMAN IN THE LOOP 3 Give the agent only the permissions Prefer reversible actions and confirm Put a person in front of high-stakes the task truly needs. before anything irreversible. or destructive actions. SANDBOXING 4 Isolate agent execution so the blast radius stays bounded. AUDIT LOGS Record every tool call with inputs, outputs, and timestamp. 5
PRIVILEGE 1 MINIMAL FOOTPRINT 2 HUMAN IN THE LOOP 3 Give the agent only the permissions Prefer reversible actions and confirm Put a person in front of high-stakes the task truly needs. before anything irreversible. or destructive actions. SANDBOXING 4 Isolate agent execution so the blast radius stays bounded. AUDIT LOGS Record every tool call with inputs, outputs, and timestamp. 5 TEST ADVERSARIALLY 6 Assume a clever prompt will still try to cross the boundary.
hidden assumption → real-world failure. Classic ML: the data speaks for the world. LLMs: safe outputs mean a safe model. Agents: the environment can be trusted.
hidden assumption → real-world failure. Classic ML: the data speaks for the world. LLMs: safe outputs mean a safe model. Agents: the environment can be trusted. The incidents won't stop. Our job is to name the next assumption to predict and prevent them.
lcorbucci
wjessup
rkendrick25
inesmontani
kneath
szymonslowik
retrage
lynnandtonic
brettharned
skipperchong
bermonpainter
irinanazarova
aleyda
![THANKS. QUESTIONS? Luca Corbucci → lucacorbucci.me → @[email protected] → @lucacorbucci.bsky.social](https://files.speakerdeck.com/presentations/b8048eaaabdf49b78e2d40f48352a160/slide_121.jpg){kind=link}