How We Built a Secure Sandbox Platform for AI

Transcript

1. How We Built a Secure
 Sandbox Platform for AI Agents AIエージェントSaaSを安全に
 提供するための自社サンドボックス基盤 CODE BLUE 2025 Software Engineer @pizzacat83
2. © 2025 GMO Flatt Security Inc. All Rights Reserved. $ whoami @pizzacat83 SWE @ GMO Flatt Security Inc. Developing “Takumi byGMO” Formerly security engineer セキュリティ若手の会 幹事 (第1期) seccamp ‘20 alumnus
3. 🤖 Make a cool blog app The of AI Agents has made various operations ––As far as it requires no human intervention. But autonomy comes with ... AI Agents interacts with the env, maybe in an unintended way Press a dangerous button Upload confidential info to public Delete or modify important data “Require human approval” ruins autonomy! autonomy scalable risks Autonomous interaction © 2025 GMO Flatt Security Inc. All Rights Reserved.
4. Restrict the access to the env so that the worst possible damage is still acceptable 🤖 What internal info can be read? What external communication is possible? Confidential docs Public web pages Confidential docs Full internet access Full internet access No internet access = Acceptable! = Not acceptable × × × = Acceptable! © 2025 GMO Flatt Security Inc. All Rights Reserved. To Achieve Autonomy without Sacrificing Security What action can be made against internal resources?
5. Local AI Agents Cloud-hosted, multi-tenant AI Agents AI may have access to
 whatever human has AI may interfere with
 AI of another tenant AI may have access to internal services filesystem filesystem internal systems A internal services © 2025 GMO Flatt Security Inc. All Rights Reserved. AI Agents with Compute environment Run code, read files, browse the web, ... So we sandbox them! 🤖 🤖 B 🤖
6. Tenant A Tenant A Data A Data A Data B Data B Tenant B Tenant B 🤖 🤖 e.g., If A must not access data of B, MUST NOT write data of B to where A can read MUST NOT allow A to read where B can write data → A’s agent and B’s agent must be isolated Permission boundary gives a hint. © 2025 GMO Flatt Security Inc. All Rights Reserved. Isolate per what?
7. User X User X Data A Data A Data B Data B User Y User Y 🤖 🤖 © 2025 GMO Flatt Security Inc. All Rights Reserved. Isolate per what? e.g., If A must not access data of B, MUST NOT write data of B to where A can read MUST NOT allow A to read where B can write data → A’s agent and B’s agent must be isolated Permission boundary gives a hint.
8. git checkout branch-a git checkout branch-b Same user “Please review PR #123” “Please review PR #987” Without properly isolating the workspaces, even benign AI Agents can interfere each other. © 2025 GMO Flatt Security Inc. All Rights Reserved. Ideally, isolate per agent––not just for security! 🤖 💥 🤖
9. per-agent isolation (In some usecases) VM-level isolation multi-tenant resource sharing scalability Technically challenging! Start working in seconds
 (for chat UX) © 2025 GMO Flatt Security Inc. All Rights Reserved. Requirements for sandbox, for AI Agent as a Service
10. Takumi – AI-Powered AppSec Auditor CVE-2025-29768 potential data loss with zip.vim and special crafted zip files in Vim < v9.1.1198 vim/vim CVE-2025-30218 x-middleware-subrequest-id may be leaked to external hosts
 vercel/next.js CVE-2025-31483 Stored XSS in Miniflux Media Proxy due to improper Content-Security- Policy configuration miniflux/v2 We faced this technical challenge of sandboxing during the development of our AI Agent product “Takumi byGMO”. © 2025 GMO Flatt Security Inc. All Rights Reserved.
11. Whitebox Blackbox Review Exploit User input unescaped! Let’s send <script>... © 2025 GMO Flatt Security Inc. All Rights Reserved. What Takumi can do
12. Whitebox Blackbox Review Exploit User input unescaped! Let’s send <script>... exploit.py Needs a sandbox! © 2025 GMO Flatt Security Inc. All Rights Reserved. How Takumi works
13. “Review PR #123” Spawn a sandbox Here you go! © 2025 GMO Flatt Security Inc. All Rights Reserved. So we developed a sandbox platform! Sandbox platform Backend server
14. More isolated More overhead Docker can run inside sandbox Less isolated Less overhead Can seamlessly fallback to non-sandboxed exec ‘s choice! © 2025 GMO Flatt Security Inc. All Rights Reserved. Isolation technologies Anthropic Sandbox Runtime:
 bubblewrap, Seatbelt Codex: Landlock, seccomp, Seatbelt Gemini CLI: Seatbelt Virtual machines Containers Others Claude Code provides
 official Dev Container
15. Node (VM pool) apiserver VMM: Firecracker Many VMs in 1 Node Forward
 request Give me a VM! Horizontally scalable …… © 2025 GMO Flatt Security Inc. All Rights Reserved. Architecture of our sandbox platform
16. OSS virtualization technology maintained by AWS Core technology of AWS Lambda Implemented in Rust Has just minimal features for Lambda-like workload Performance benefit: Low overhead! Boots in <1 sec Thousands of Firecracker VMs can run in one machine Security benefit: Small attack surface! © 2025 GMO Flatt Security Inc. All Rights Reserved. Firecracker – Security of VMs, Speed like containers
17. “ in box” style Agent “ in box” style Action 🧠 🧠 Container Use Devin Takumi Claude Code in Dev Container © 2025 GMO Flatt Security Inc. All Rights Reserved. Architectures for sandboxing AI Agents LLM API LLM API How does this work?
18. User message Tool use request Execute tool Tool use result Agent implementation © 2025 GMO Flatt Security Inc. All Rights Reserved. Internals of AI Agents LLM API “Summarize errors in ./logs” Tool name: Bash Args: ls ./logs Tool name: Bash Args: head logs/01.log bash -c ${args} Tool output: 01.log 02.log ... LLM itself doesn’t run commands! Actual execution happens here 1 2 3 4 5 6 reply prompt stdout reply args resp2 prompt reply stdout := := := ... callLLMAPI exec callLLMAPI ([ ]); ( , , [ ]. ); ([ , , ])); "bash" "-c" "Bash" substitutable!
19. User message Tool use request Execute tool Tool use result Agent implementation © 2025 GMO Flatt Security Inc. All Rights Reserved. Example: Ask for human before exec LLM API “Summarize errors in ./logs” Tool name: Bash Args: head logs/01.log Ask for human’s approval and then:
 bash -c ${args} Tool output: 01.log 02.log ... 1 2 3 4 5 6 7 8 reply prompt reply args throw stdout reply args resp2 prompt reply stdout := ! := := ... callLLMAPI confirmHuman exec callLLMAPI ([ ]); ( [ ]. ) { } ( , , [ ]. ); ([ , , ])); if "Bash" "rejected by human" "bash" "-c" "Bash" Ask for human approval
20. User message Tool use request Execute tool Tool use result Agent implementation © 2025 GMO Flatt Security Inc. All Rights Reserved. Example: “Action in box” style (container) LLM API “Summarize errors in ./logs” Tool name: Bash Args: head logs/01.log docker run ubuntu
 bash -c ${args} Tool output: 01.log 02.log ... 1 2 3 4 5 6 7 reply prompt stdout reply args resp2 prompt reply stdout := := := ... callLLMAPI exec callLLMAPI ([ ]); ( , , , , , [ ]. ); ([ , , ])); "docker" "run" "ubuntu" "bash" "-c" "Bash" Exec in container!
21. User message Tool use request Execute tool Tool use result Agent implementation The risk is NOT the LLM output itself What to isolate is the real action taken based on it © 2025 GMO Flatt Security Inc. All Rights Reserved. Example: “Action in box” style (VM) LLM API “Summarize errors in ./logs” Tool name: Bash Args: head logs/01.log ssh user@$VM_IP
 bash -c ${args} Tool output: 01.log 02.log ... 1 2 3 4 5 6 7 reply prompt stdout reply args resp2 prompt reply stdout := := := ... callLLMAPI exec callLLMAPI ([ ]); ( , , , , [ ]. ); ([ , , ])); "ssh" `user@${vmIP}` "bash" "-c" "Bash" Exec in VM!
22. “ in box” style Agent “ in box” style Action 🧠 🧠 Container Use Devin Takumi Claude Code in Dev Container © 2025 GMO Flatt Security Inc. All Rights Reserved. Architectures for sandboxing AI Agents LLM API LLM API Why choose this style?
23. “Agent in box” style “Action in box” style 🧠 🧠 LLM API Key can be leaked Key is protected © 2025 GMO Flatt Security Inc. All Rights Reserved. Architectures for sandboxing AI Agents 1 2 3 4 5 reply prompt stdout reply args resp2 prompt reply := := := callLLMAPI exec callLLMAPI ([ ]); ( , , , , [ ]. ); ([ , , "ssh" `user@${vmIP}` "bash" "-c" "Bash" LLM API key LLM API key LLM API LLM API
24. “Agent in box” style “Action in box” style 🧠 🧠 © 2025 GMO Flatt Security Inc. All Rights Reserved. Architectures for sandboxing AI Agents LLM API key LLM API key LLM API LLM API “Agent in box” style minimizes what the untrusted sandbox has access to
25. “ in box” style Agent “ in box” style Action 🧠 🧠 Good choice when implementing
 Agent as a Service (esp. multi-tenant) Good choice when running closed-source AI Agent (i.e. when you can’t isolate the internals) © 2025 GMO Flatt Security Inc. All Rights Reserved. Architectures for sandboxing AI Agents LLM API LLM API Make sure all untrusted actions
 happen in isolated env
26. VM Affinity Ensure the Brain VM and the Compute VM of the same agent
 are spawned in the same Node VM snapshots Make snapshots with the dev env set up e.g., apt install, clone repositories Boot VMs with zero-copied snapshots (Copy-on-Write) …… © 2025 GMO Flatt Security Inc. All Rights Reserved. Topics we won't cover today: Techniques for speed & efficiency
27. Isolation is critical for AI Agents, for security, and more We showed various approaches to isolation, and when to choose which Provisioning sandboxes for multi-tenant, cloud-hosted AI Agents is technically challenging,
 and we built a secure and performant sandbox platform © 2025 GMO Flatt Security Inc. All Rights Reserved. Conclusion
28. We are the backbone of engineers. GMO Flatt Security, based in Tokyo, offers expert security assessments and penetration testing for software. Our seasoned professionals deliver proven, top-tier services. We also provide tools to help you internalize cutting edge, state of the art security practices.
29. 脆弱性診断・ペネトレーションテストを プロフェッショナルサービス / AIで多角的に提供 専門家・高度診断 コード・仕様の分析も行い、 専門家が脆弱性を網羅的に発見 AI・継続診断 継続的なセキュリティレビューを
 AIエージェントで簡単に実現 提供サービス 提供サービス
30. Tomorrow, Nov. 19th 10:00 / 11:30 / 13:00 / 14:30 Track 2 (HALL A) Visit Our Booth! Join Our Workshop! Thank you! AI x Pentesting Forefront: A Hands-on Workshop with "Takumi byGMO" Meet our team! Learn more about
 our company and our services. We’re hiring! recruit.flatt.tech © 2025 GMO Flatt Security Inc. All Rights Reserved. What’s next? Learn more & Get hands-on