The Tale of Leo: Brave Lion and Curious Little Bug

Browser Crash Club #1
https://browsercrashclub.connpass.com/event/350203/

The original slides are here. You can see gif working correctly :)
https://docs.google.com/presentation/d/1Vg3Qj0GF57Y3wBajf46lR1619dV3moWi9Vf6jTsYICg/edit?usp=sharing

1. The Tale of Leo: Brave Lion🦁 and Curious Little Bug🐞
2. myself :) - canalun (@i_am_canalun) - Interested in developing and crashing browsers :) - Helping Firefox dev mainly on Web Animations - Finding vulns for Edge, Brave etc. https://www.youtube.com/watch?v=kEs6LHdHTI0
3. Do you know CommitStyles API?? With a lot of help by Birtles...!! (spec editor of Web Animations) https://groups.google.com/a/mozilla.org/g/dev-platform/c/7p11iesCdbA
4. Let’s talk about Leo, Brave browser’s AI Sidebar!!
5. What it looks like?? - Basic AI sidebar (or tab) - Can refer to the page - Conversation UI
6. With advanced features! 1. You can set any AI model (“BYOM”) 2. It recognizes various contents
7. With advanced features! 1. You can set any AI model (“BYOM”) - endpoint - context size - API key - system prompt (model will be initialized by it)
8. With advanced features! 2. It can recognize various contents - google docs (=canvas) - youtube (=video) - pdf
9. Google docs is rendered on canvas.
10. https://brave.com/blog/leo-docsupport/
11. 👶<Let’s see the internal
12. First, let’s see processes. Tips: `chrome:// process- internal` is nice.
14. Sidebar = 2 renderers (1 main frame + 1 OOPIF)
15. 👶<Let’s see more!!! dive!!!!
17. It’s original fig. not official one
18. send prompt to browser proc using Mojo. It’s original fig. not official one
19. 1 conversation per time. It’s original fig. not official one
20. Sanitize user’s prompt. Remove special tags depending on Model. (e.g. Llama’s special <SYS> tag) It’s original fig. not official one
21. https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-2/
22. Get page content from renderer. Basically, “innerHTML”. So frames are not included :( (Some content type might probably not need renderer) It’s original fig. not official one
23. Sanitize page. Again, remove special tags depending on Model. It’s original fig. not official one
24. Time of AI🤖 It’s original fig. not official one
25. Save conversation into browser internal DB. It’s original fig. not official one
26. Send the result to OOPIF using Mojo and it’s displayed. It’s original fig. not official one
27. It’s original fig. not official one
28. 👶<source??
29. self-XSS-ish... It’s original fig. not official one
30. 👶<interesting parts??
31. AI DB privileged process web interface It’s original fig. not official one
32. AI DB privileged process web interface What an educational…😂😂 It’s original fig. not official one
33. Let's see one by one!! - Storage - AI - Web Interface - Privileged Process
34. Let's see one by one!! - Storage - AI - Web Interface - Privileged Process
35. Browsers, at least FF and Chromium, use SQLite.
36. Browsers, at least FF and Chromium, use SQLite.
37. Of course, prepared stmt👶 https://github.com/brave/brave-core/blob/aee7757a76fd86aa70fb0d86eb6937d0c26e91d1/components/ai_chat/core/browser/ai_chat_database.cc#L500
38. Let's see one by one!! - Storage - AI - Web Interface - Privileged Process
39. Checking AI attack categories. Based on OWASP AI threats overview... - Leak model params or test data - Leak user data like input or history - Manipulate AI to deceive the user - Break and Disable AI - Any attacks on non AI-specific assets, with AI https://owaspai.org/docs/ai_security_overview/#threat-model
40. Checking AI attack categories. Based on OWASP AI threats overview... - Leak model params or test data - Leak user data like input or history - Manipulate AI to deceive the user - Break and Disable AI - Any attacks on non AI-specific assets, with AI No Idea :( https://owaspai.org/docs/ai_security_overview/#threat-model
41. Brave’s AI is privacy-limited and XS-leaks seems to be difficult. - one conversation can refer to only one document (page reload/nav leads to new conversation!!) - page content is got by innerHTML, so framed content cannot be read by model.
42. Brave’s AI is privacy-limited and XS-leaks seems to be difficult. - one conversation can refer to only one document (page reload/nav leads to new conversation!!) - page content is got by innerHTML, so framed content cannot be read by model. the conversation is either - with trusted content, or - with untrusted content **no cross-over between them!!**
43. Checking AI attack categories. Based on OWASP AI threats overview... - Leak model params or test data - Leak user data like input or history - Manipulate AI to deceive the user - Break and Disable AI - Any attacks on non AI-specific assets, with AI https://owaspai.org/docs/ai_security_overview/#threat-model
44. Checking AI attack categories. Based on OWASP AI threats overview... - Leak model params or test data - Leak user data like input or history - Manipulate AI to deceive the user - Break and Disable AI - Any attacks on non AI-specific assets, with AI Let’s inject prompt👶 https://owaspai.org/docs/ai_security_overview/#threat-model
45. Oh, “Role Prompting”...! - Leo uses “role” in prompt JSON. - Set “role” to each prompt like “system”, “user” - This makes it easier to set a security/privilege boundary between prompts. - kind of RBAC in prompt world. - Anthropic (Claude), OpenAI seems to adopt it. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/system-prompts?q=role https://platform.openai.com/docs/guides/text#message-roles-and-instruction-following
46. Oh, “Role Prompting”...! - Leo uses “role” in prompt JSON. - Set “role” to each prompt like “system”, “user” - This makes it easier to set a security/privilege boundary between prompts. - kind of RBAC in prompt world. - Anthropic (Claude), OpenAI seems to adopt it. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/system-prompts?q=role https://platform.openai.com/docs/guides/text#message-roles-and-instruction-following If models take “role” seriously, prompt injection is not effective. JSON injection or something is needed.
47. Found no vuln here. - malicious input WITH user role didn’t work. - AI detected the fact that I was trying to deceive them as user😂 - JSON escape module used here is from chromium and well-tested (or battle-tested)👏
48. Let's see one by one!! - Storage - AI - Web Interface - Privileged Process
49. web interface It’s original fig. not official one
50. brave://leo-ai chrome-untrusted:// leo-ai-conversation-entries
51. chrome:// vs chrome-untrusted:// what is “chrome://”?? - schema for WebUI, like a web interface for browser - chrome://history, chrome://about - WebUI is privileged. For example...
52. chrome:// can use camera w.o. permission prompt.
53. chrome:// vs chrome-untrusted:// what is “chrome-untrusted://”? - Tool for WebUI to handle untrusted resources. - WebUI can use - iframe to embed untrusted web page, or - the schema to combine untrusted resources - No privilidge
54. iframe in WebUI: Edge Copilot from “Piloting Edge Copilot” by Jun Kokatsu https://speakerdeck.com/shhnjk/piloting-edge-copilot
55. brave://leo-ai chrome-untrusted:// leo-ai-conversation-entries AI’s response is from external world, and untrusted. So it’s sandboxed.
56. What about CSPs?? You can see it from DevTools Application tab :)
57. Secure...!!(though not “Strict CSP”) - default-src: ‘none’ - script-src: ‘self’ chrome://resources - frame-ancestors - parent: ‘none’ - child: chrome://leo-ai(=parent) - child-src - parent: chrome-untrusted://...(=child) - child: ‘none’ - OTHERS: base-uri, connect-src, font-src, object-src, image-src, trusted types, etc.
58. And also HTML tags in conversation area are white-listed by react-markdown.
59. No idea😭😭😭
60. Let's see one by one!! - Storage - AI - Web Interface - Privileged Process UaF or something?🤔
61. finally found nullptr deref☺ https://hackerone.com/reports/2958097
62. finally found nullptr deref☺ https://hackerone.com/reports/2958097 Not Exploitable, but they gave me $100👏
63. This was about parsing AI’s response. - Parse process for AI’s response assumes a specific JSON structure. - But BYOM model can responds in any format - So unexpected field leads to nullptr deref.
64. found by spending a lot of time on reading the code... Tips - CLion is good i think - Please use compile_commands.json - I asked them how to generate. please refer to it :) https://github.com/brave/brave-browser/issues/44239
65. 👶＜That’s it??
66. It’s original fig. not official one
67. There might be more opportunities in page fetch...! - some contents needs complex way - pdf: traversing a11y tree - google docs: generating print preview(?) - video: parsing xml script doc
68. There might be more opportunities in page fetch...! - some contents needs complex way - pdf: traversing a11y tree - google docs: generating print preview - video: parsing xml script doc Anyone have idea??👶 please try and tell me the result :)
69. That’s it! Thanks :)