Exploring Abstract Design Tools to Turn Spatial App Feedback into Codex (En)

Transcript

1. Exploring Abstract Design Tools to Turn Spatial App Feedback into Codex Copyright © 2026 Yorihiroi Frontend. Kansock Industries / Broader Frontends Kazuhiro Hara / @kara_d visionOS Engineer Meetup: vol.13 Lightning Talk: Online
2. Speaker Kazuhiro Hara @kara_d CEO, Kansock Industries Organizer, Broader Frontends Started learning Swift after ChatGPT was released Die-hard OpenAI fan (Personal Pro Subscriber) Photo by Rin Yano
3. Agenda I will talk about my exploration while creating development tools for AI x XR, such as Seiro MCP and the tool I am announcing today. I hope this serves as a reference when building AI tools. • Developing daily with Seiro MCP • New frustrations • Conceiving the Abstract Layout Tool (tentative) • DEMO • Future developments
4. Developing daily with Seiro MCP
5. A brief introduction to Seiro MCP https://github.com/karad/seiro-mcp • visionOS + Codex CLI / App Assists autonomous coding ◦ MCP / Skills / CLI ◦ Available as OSS (MIT License) ◦ First version released last December ◦ Current version v0.4.0 • XcodeBuildMCP existed, but didn't work in my environment ◦ Decided to create something simple specialized for spatial computing app development • In February, the official Xcode MCP was also released ◦ If I were planning it now, I might not have started making it Waiting for your ★!!
6. Seiro MCP handles these parts • Automatic builds of visionOS projects via MCP with Codex CLI / App • Capture detailed errors in addition to standard error logs • Includes Skills and CLI tools
7. UI Study with Codex + Seiro MCP
8. UI Study with Codex + Seiro MCP
9. New Frustrations
10. New Frustrations (1) Once "if I can do this" is achieved, you realize there is even more ahead In the case of Seiro MCP... • Abstract errors make the AI unsure of what to do ◦ Resolved by digging into more detailed logs • Similar errors occur repeatedly, requiring re-solving the same problems ◦ The same mistakes are made often, and the errors tend to be consistent ◦ These steps are taken every time an error occurs • How to centralize solution know-how is one of the current challenges To be addressed in the next Seiro MCP
11. Drawbacks of the Feedback Loop • The feedback loop works when the AI understands the correct state but the implementation is incorrect • If the AI doesn't understand the correct answer to begin with, it cannot reach a solution • Giving instructions for spatial content is inherently difficult ◦ Describing spatial relationships is challenging, even for skilled writers ◦ Providing corrective instructions is also difficult
12. New Frustrations (2) Giving visual instructions in 3D space is quite difficult Buttons are placed horizontally, but if you try to press them from the front... It often doesn't work out well This is the theme for today
13. I just want to make adjustments easier...
14. Coming up with the Abstract Layout Tool (tentative)
15. What is the Abstract Layout Tool (tentative)? • A feedback tool specialized for communication with AI • Composed of MCP server, Skills, desktop app, and SQLite • Specialized in abstract layout instructions without detailed shape directions • Something to use roughly and quickly
16. Usage Style of Abstract Layout Tool (tentative) 1. Save the layout from the AI (via MCP) 2. Review and edit the layout in the desktop app 1. Position 2. Rotation 3. Size 4. Free comments 3. Layout revisions are managed upon saving 4. AI can load the edited layout (via MCP)
17. How do we provide feedback to this tool?
18. 1. Save the layout of a specific state from the AI Instruct the AI to save the layout using a Skill from Codex
19. 2. Review and edit the layout in the desktop app
20. 3. AI loads and modifies the edited layout AI can automatically retrieve Layout IDs and the latest Revision IDs
21. Why an Abstract Design Tool?
22. Why an Abstract Design Tool? • Abstracting shapes keeps the tool simple. • Motivation: It's enough to be able to point out abstract-level designs. • By separating from implementation, we can extract parts that the AI considers "correct". • In other words, feedback is possible even for layouts not defined in code: ◦ Expected layout after animation movement ◦ Objects displayed after user actions ◦ Temporary placement of elements not yet implemented • Abstraction also offers the advantage of platform independence: ◦ Originally for visionOS, but usable as a WebXR tool.
23. Not just visionOS, but also WebXR
24. Not just visionOS, but also WebXR • Platform-independent because it handles abstract layout information rather than detailed rendering of individual objects • Since it communicates via JSON and the interpretation layer is AI, it can be used for both visionOS and WebXR (theoretically usable for all XR systems) • Uses standard technologies such as MCP and Skills
25. DEMO
26. Development for visionOS Prepare a project where three objects are arranged in a row
27. Development for WebXR Prepare a project similar to the visionOS project Made with Three.js
28. Future Development
29. Future Development • Make the screen more pop and intuitive • Make the installation process easier • Intuitive feedback via drag, pinch, etc. • Enhanced security features • Documentation preparation • To be released as OSS
30. If you find your own challenges , let's turn them into a tool!!
31. Exploring Abstract Design Tools to Turn Spatial App Development Feedback into Codex Copyright © 2026 Yorihiroi Frontend. Kansock Industries / Yorihiroi Frontend Kazuhiro Hara / @kara_d visionOS Engineer Meetup: vol.13 Short Talk Session: Online Thank You