2 December 2005 As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction Prof. Beat Signer beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel

edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Cross-Media Information Spaces

Cross-Media Linking Fluid Document Formats Cross-Media User Interfaces

RSL Hypermedia Metamodel & Link Server ipaper iwebfirefox movieplugin B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proc. of ER 2007 B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proc. of ER 2007

Data-centric approach for cross-media information interaction based on a hypermedia metamodel

Cross-Media PIM Interaction S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring andRe-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016

Presentation Platform ▪ Flexible representation of presentations ▪ content-based approach ▪ use of structural RSL links ▪ cross-media content reuse via transclusion ▪ Extensible platform ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI, 3 (Issue EICS), 2019 R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI, 3(Issue EICS), 2019

Non-Linear Cross-Media Narrative ▪ Writing and drawing as a collaborative act of story- telling ▪ Captured handwriting is linked to existing digital and physical content ▪ Link server driving the augmentation process ▪ active components - ambient sound - ambient light - … cosmo A. Vogelsang and B. Signer, The Lost Cosmonaut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proc. of ICVS 2005 A red star with black text Description automatically generated

eSPACE: Adaptive Distributed Hybrid UIs ▪ End-User UI design ▪ use of navigational and structural RSL links via transclusion A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proc. of RCIS 2019 A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proc. of RCIS 2019

Hypermedia Desktop Engine Graphical user interface Description automatically generated B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross- Media Concepts to Modern Desktop Environments, Proc. of Hypertext 2021

Mixed Reality (MR) Solutions Meta Quest 3 Meta Quest 3 Varjo XR-4 Varjo XR-4 Apple Vision Pro Apple Vision Pro Microsoft HoloLens 2 Microsoft HoloLens 2 Wikitude video see-through on smartphone Wikitude video see-through on smartphone

edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Cross-Media Information Spaces

Cross-Media PIM Interaction Revisited S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring andRe-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016

“Linking” in IoT Environments Y. Yu, Natural Mixed-Reality Interactions for IoT Environments, Master's Thesis VUB (supervisor Prof. B. Signer), September 2024

Hypertext principles can play a major role in realising a general infrastructure for human- information interaction in mixed reality environments

General Infrastructure for Mixed Reality ▪ W3C WebXR Device API ▪ cross-platform compatibility ▪ web-based applications running in a browser ▪ W3C Web of Things(WoT) Thing Description (TD) ▪ metadata and interfaces of IoT devices & passive things ▪ Solid Project ▪ decentralised Solid Pods in addition to the Web ▪ collaborative authoring and sharing of information Solid M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proc. of SoSy 2024 M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proc. of SoSy 2024

General Infrastructure for Mixed Reality … ▪ Anchoring of hypertext structures in the real world ▪ fixed location (geospatial anchoring) ▪ marker-based ▪ vision-based ▪ Hypertext principles to link and interact with informa- tion across digital and physical spaces ▪ exploit innovative human- information interaction and user interfaces ipaper B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proc. of ER 2010 B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proc. of ER 2010

Hypertext principles in combination with technologies such as WebXR, WoT or Solid can play a major role in realising a general infrastructure for human-information interaction in mixed reality environments Mixed reality offers opportunities for innovative forms of human-information interaction and user interfaces across digital and physical spaces

References ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Second Edition, ISBN 978-3-8370-2713-6, August 2017 cover2.jpg ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf

References ... ▪ B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross- Media Content Composition and Reuse, Proceedings of the 29th International Conference on Conceptual Mod- eling (ER 2010), Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf ▪ R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, Proceedings of the ACM on Human-Computer Interaction (PACMHCI), 2019 ▪ https://beatsigner.com/publications/roels_PACMHCI2019.pdf

References ... ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf ▪ A. Vogelsang and B. Signer, The Lost Cosmonaut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of the International Conference on Virtual Storytelling 2005, Strasbourg, France, December 2005 ▪ https://beatsigner.com/publications/vogelsang_ICVS2005.pdf

References ... ▪ A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/sanctorum_RCIS2019.pdf ▪ B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021, August 2021 ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf

References ... ▪ M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proceedings of SoSy 2024 (Poster), 2nd Solid Symposium, Leuven, Belgium, May 2024 ▪ https://beatsigner.com/publications/vanDeWynckel_SoSy2024.pdf ▪ B. Signer, As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction, Proceedings of HUMAN 2024, Poznań, Poland, September 2024 ▪ https://beatsigner.com/publications/signer_HUMAN2024.pdf

References ... ▪ MindXpres Website ▪ https://mindxpres.com ▪ The Lost Cosmonaut Video ▪ https://www.youtube.com/watch?v=fu27Cmr1aX4