Share
Embed
Exploring Abstract Design Tools to Turn Spatial App Feedback into Codex (En)
kazuhiro hara
May 10, 2026
Transcript
31 slides
-
Exploring Abstract Design Tools to Turn Spatial App Feedback into
Codex Copyright © 2026 Yorihiroi Frontend. Kansock Industries / Broader Frontends Kazuhiro Hara / @kara_d visionOS Engineer Meetup: vol.13 Lightning Talk: Online -
Speaker Kazuhiro Hara @kara_d CEO, Kansock Industries Organizer, Broader Frontends
Started learning Swift after ChatGPT was released Die-hard OpenAI fan (Personal Pro Subscriber) Photo by Rin Yano -
Agenda I will talk about my exploration while creating development
tools for AI x XR, such as Seiro MCP and the tool I am announcing today. I hope this serves as a reference when building AI tools. • Developing daily with Seiro MCP • New frustrations • Conceiving the Abstract Layout Tool (tentative) • DEMO • Future developments -
A brief introduction to Seiro MCP https://github.com/karad/seiro-mcp • visionOS +
Codex CLI / App Assists autonomous coding ◦ MCP / Skills / CLI ◦ Available as OSS (MIT License) ◦ First version released last December ◦ Current version v0.4.0 • XcodeBuildMCP existed, but didn't work in my environment ◦ Decided to create something simple specialized for spatial computing app development • In February, the official Xcode MCP was also released ◦ If I were planning it now, I might not have started making it Waiting for your ★!! -
Seiro MCP handles these parts • Automatic builds of visionOS
projects via MCP with Codex CLI / App • Capture detailed errors in addition to standard error logs • Includes Skills and CLI tools -
New Frustrations (1) Once "if I can do this" is
achieved, you realize there is even more ahead In the case of Seiro MCP... • Abstract errors make the AI unsure of what to do ◦ Resolved by digging into more detailed logs • Similar errors occur repeatedly, requiring re-solving the same problems ◦ The same mistakes are made often, and the errors tend to be consistent ◦ These steps are taken every time an error occurs • How to centralize solution know-how is one of the current challenges To be addressed in the next Seiro MCP -
Drawbacks of the Feedback Loop • The feedback loop works
when the AI understands the correct state but the implementation is incorrect • If the AI doesn't understand the correct answer to begin with, it cannot reach a solution • Giving instructions for spatial content is inherently difficult ◦ Describing spatial relationships is challenging, even for skilled writers ◦ Providing corrective instructions is also difficult -
New Frustrations (2) Giving visual instructions in 3D space is
quite difficult Buttons are placed horizontally, but if you try to press them from the front... It often doesn't work out well This is the theme for today -
What is the Abstract Layout Tool (tentative)? • A feedback
tool specialized for communication with AI • Composed of MCP server, Skills, desktop app, and SQLite • Specialized in abstract layout instructions without detailed shape directions • Something to use roughly and quickly -
Usage Style of Abstract Layout Tool (tentative) 1. Save the
layout from the AI (via MCP) 2. Review and edit the layout in the desktop app 1. Position 2. Rotation 3. Size 4. Free comments 3. Layout revisions are managed upon saving 4. AI can load the edited layout (via MCP) -
1. Save the layout of a specific state from the
AI Instruct the AI to save the layout using a Skill from Codex -
3. AI loads and modifies the edited layout AI can
automatically retrieve Layout IDs and the latest Revision IDs -
Why an Abstract Design Tool? • Abstracting shapes keeps the
tool simple. • Motivation: It's enough to be able to point out abstract-level designs. • By separating from implementation, we can extract parts that the AI considers "correct". • In other words, feedback is possible even for layouts not defined in code: ◦ Expected layout after animation movement ◦ Objects displayed after user actions ◦ Temporary placement of elements not yet implemented • Abstraction also offers the advantage of platform independence: ◦ Originally for visionOS, but usable as a WebXR tool. -
Not just visionOS, but also WebXR • Platform-independent because it
handles abstract layout information rather than detailed rendering of individual objects • Since it communicates via JSON and the interpretation layer is AI, it can be used for both visionOS and WebXR (theoretically usable for all XR systems) • Uses standard technologies such as MCP and Skills -
Development for visionOS Prepare a project where three objects are
arranged in a row -
Development for WebXR Prepare a project similar to the visionOS
project Made with Three.js -
Future Development • Make the screen more pop and intuitive
• Make the installation process easier • Intuitive feedback via drag, pinch, etc. • Enhanced security features • Documentation preparation • To be released as OSS -
If you find your own challenges , let's turn them
into a tool!! -
Exploring Abstract Design Tools to Turn Spatial App Development Feedback
into Codex Copyright © 2026 Yorihiroi Frontend. Kansock Industries / Yorihiroi Frontend Kazuhiro Hara / @kara_d visionOS Engineer Meetup: vol.13 Short Talk Session: Online Thank You