A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation

Presentation given at DIS 2026, International Conference on Designing Interactive Systems, Singapore, Republic of Singapore, June 2026.

ABSTRACT: 3D environments offer powerful capabilities for simulating and monitoring complex systems. However, they remain inaccessible to non-expert users who lack specialised training. We present the Layered Customisation System (LCS), a hybrid interface that combines GUI-based direct manipulation with LLM-powered natural language interaction for customising 3D scenes. Through a controlled study with 12 participants, each completing three sessions with different interfaces (GUI-only, LLM-only and hybrid), we investigated how interaction types affect performance, errors and user preferences. Key findings include a reduced learning effect across sessions for users starting with the LLM interface and a trend where participants who first experienced GUI-based interaction showed higher rates of recognising LLM errors later. We contribute empirical evidence of GUI-LLM modality trade-offs in 3D interaction, the selector-layer architecture for hybrid interfaces, and design recommendations, including GUI-first onboarding, enabling prompts as discovery tools and supporting prompt-then-refine workflows.

Research paper: https://beatsigner.com/publications/a-hybrid-gui-llm-interface-paradigm-for-3d-scene-customisation.pdf

Transcript

1. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez - jvaladez@vub.be - VUB Irving Aviña - irving@madmonkeys.io Beat Signer - bsigner@vub.be - VUB A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
2. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez - jvaladez@vub.be - VUB Irving Aviña - irving@madmonkeys.io Beat Signer - bsigner@vub.be - VUB A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
3. 3D scenes can be hard to use, let’s try and make it easier A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
4. What was our study about? A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
5. Summary What we did We conducted an experiment with 12 participants We asked them to complete 3 groups of tasks using three different interaction modalities: A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
6. 1 Graphical User Interface A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation 2 LLM-assisted Interface 3 Hybrid Interface Isaac Valadez
7. The Experiment... A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
8. The Experiment... A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
9. So, how did it look like? A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
10. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
11. Features The Prototype’s Functionality Users can augment or simplify the scene by changing: How objects are rendered (size, colours). Where they are positioned (in a map, in a grid). Representation itself (original shape or a different one). Things to be added (labels with data, lines connecting objects). A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
12. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
13. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Selectors: group objects based on shared attributes. Isaac Valadez
14. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Selectors: group objects based on shared attributes. Layers: group objects based on shared attributes. Isaac Valadez
15. Rearrange and colour Working example 1 “Place buildings next to each other and change the colour of all houses to blue.” A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
16. Rearrange and colour Working example 1 “Place buildings next to each other and change the colour of all houses to blue.” A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
17. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
18. Select and connect Working example 2 “Paint houses in blue if they have more than 2 floors. Also, draw a line from each house to their closest hospital.” A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
19. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
20. Observations What did we find? Speed versus control: The LLM-only approach was the fastest one, but not the preferred interface. Automation versus mental model: Users who started with the GUI first were better at spotting errors produced by the LLM interface. Efficiency versus effort: The cognitive load of prompting can be higher than manually doing the task. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
21. Outcomes Design Recommendations DR1: Consider GUI-first Onboarding DR2: Design for High AI Error Rate DR3: Leverage User Error Detection Capability. DR4: Support the Prompt-Then-Refine Workflow. DR5: Enable Prompts as Discovery Tools. DR6: Support Vocabulary Learning. DR7: Design for Control, Not Just Speed. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
22. In addition... The Prototype’s Implementation Functionality is decoupled from interaction. The Layered Customisation System is just one of many possible implementations. Framework is designed to be expanded. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez
23. Conclusion Hybrid GUI-LLM interfaces are promising to ease interaction with 3D environments. LLM interfaces can hinder learning the inner works of a system. A hybrid interface can be a good tool to show users how things work. LLM interfaces can reveal hard to find functionality. DR1: Consider GUI-first Onboarding DR2: Design for High AI Error Rate DR3: Leverage User Error Detection Capability. DR4: Support the Prompt-Then-Refine Workflow. DR5: Enable Prompts as Discovery Tools. And more! (in the paper) DR6: Support Vocabulary Learning. DR7 : Design for Control, N ot Ju st S peed. A Hybrid GUI-LLM Interface Paradigm for 3D Scene Customisation Isaac Valadez