As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction

Keynote given at HUMAN 2024.

ABSTRACT: Long before the advent of personal computing, Vannevar Bush envisioned the Memex as a solution to address information overload by enhancing the management and refinding of information through associative trails. While other hypertext pioneers like Douglas Engelbart and Ted Nelson introduced advanced hypertext concepts to create more flexible document structures and augment the human intellect, some of their original ideas are still absent in our daily interaction with documents and information systems. Today, many digital document formats mimic paper documents without fully leveraging the opportunities offered by digital media and documents are often organised in hierarchical file structures. In this keynote, we explore how cross-media technologies, such as the resource-selector-link (RSL) hypermedia metamodel, can be used to organise and interact with information across digital and physical spaces. While emerging wearable mixed reality (MR) headsets offer new possibilities to augment the human intellect, we discuss how hypermedia research, in combination with other technologies, could play a major role in providing the necessary linked data and hypertext infrastructure for this augmentation process. We outline the challenges and opportunities for next-generation multimodal human-information interaction enabled by flexible cross-media information spaces and document structures in combination with upcoming mixed and virtual reality solutions.

Research paper: https://beatsigner.com/publications/as-we-may-interact-challenges-and-opportunities-for-next-generation-human-information-interaction.pdf

Transcript

1. 2 December 2005 As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction Prof. Beat Signer beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel
2. edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Cross-Media Information Spaces
3. Cross-Media Linking Fluid Document Formats Cross-Media User Interfaces
4. RSL Hypermedia Metamodel & Link Server ipaper iwebfirefox movieplugin B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proc. of ER 2007 B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proc. of ER 2007
5. Data-centric approach for cross-media information interaction based on a hypermedia metamodel
6. Cross-Media PIM Interaction S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring andRe-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016
7. Presentation Platform ▪ Flexible representation of presentations ▪ content-based approach ▪ use of structural RSL links ▪ cross-media content reuse via transclusion ▪ Extensible platform ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI, 3 (Issue EICS), 2019 R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI, 3(Issue EICS), 2019
8. Non-Linear Cross-Media Narrative ▪ Writing and drawing as a collaborative act of story- telling ▪ Captured handwriting is linked to existing digital and physical content ▪ Link server driving the augmentation process ▪ active components - ambient sound - ambient light - … cosmo A. Vogelsang and B. Signer, The Lost Cosmonaut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proc. of ICVS 2005 A red star with black text Description automatically generated
9. eSPACE: Adaptive Distributed Hybrid UIs ▪ End-User UI design ▪ use of navigational and structural RSL links via transclusion A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proc. of RCIS 2019 A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proc. of RCIS 2019
10. Hypermedia Desktop Engine Graphical user interface Description automatically generated B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross- Media Concepts to Modern Desktop Environments, Proc. of Hypertext 2021
11. Mixed Reality (MR) Solutions Meta Quest 3 Meta Quest 3 Varjo XR-4 Varjo XR-4 Apple Vision Pro Apple Vision Pro Microsoft HoloLens 2 Microsoft HoloLens 2 Wikitude video see-through on smartphone Wikitude video see-through on smartphone
12. edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Cross-Media Information Spaces
13. Cross-Media PIM Interaction Revisited S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring andRe-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proc. of AVI 2016
14. “Linking” in IoT Environments Y. Yu, Natural Mixed-Reality Interactions for IoT Environments, Master's Thesis VUB (supervisor Prof. B. Signer), September 2024
15. Hypertext principles can play a major role in realising a general infrastructure for human- information interaction in mixed reality environments
16. General Infrastructure for Mixed Reality ▪ W3C WebXR Device API ▪ cross-platform compatibility ▪ web-based applications running in a browser ▪ W3C Web of Things(WoT) Thing Description (TD) ▪ metadata and interfaces of IoT devices & passive things ▪ Solid Project ▪ decentralised Solid Pods in addition to the Web ▪ collaborative authoring and sharing of information Solid M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proc. of SoSy 2024 M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proc. of SoSy 2024
17. General Infrastructure for Mixed Reality … ▪ Anchoring of hypertext structures in the real world ▪ fixed location (geospatial anchoring) ▪ marker-based ▪ vision-based ▪ Hypertext principles to link and interact with informa- tion across digital and physical spaces ▪ exploit innovative human- information interaction and user interfaces ipaper B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proc. of ER 2010 B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proc. of ER 2010
18. Hypertext principles in combination with technologies such as WebXR, WoT or Solid can play a major role in realising a general infrastructure for human-information interaction in mixed reality environments Mixed reality offers opportunities for innovative forms of human-information interaction and user interfaces across digital and physical spaces
19. References ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Second Edition, ISBN 978-3-8370-2713-6, August 2017 cover2.jpg ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf
20. References ... ▪ B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross- Media Content Composition and Reuse, Proceedings of the 29th International Conference on Conceptual Mod- eling (ER 2010), Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf ▪ R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, Proceedings of the ACM on Human-Computer Interaction (PACMHCI), 2019 ▪ https://beatsigner.com/publications/roels_PACMHCI2019.pdf
21. References ... ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf ▪ A. Vogelsang and B. Signer, The Lost Cosmonaut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of the International Conference on Virtual Storytelling 2005, Strasbourg, France, December 2005 ▪ https://beatsigner.com/publications/vogelsang_ICVS2005.pdf
22. References ... ▪ A. Sanctorum and B. Signer, A Unifying Reference Framework and Model for Adaptive Distributed Hybrid User Interfaces, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/sanctorum_RCIS2019.pdf ▪ B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021, August 2021 ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf
23. References ... ▪ M. Van de Wynckel and B. Signer, Discoverable and Interoperable Augmented Reality Environments Through Solid Pods, Proceedings of SoSy 2024 (Poster), 2nd Solid Symposium, Leuven, Belgium, May 2024 ▪ https://beatsigner.com/publications/vanDeWynckel_SoSy2024.pdf ▪ B. Signer, As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction, Proceedings of HUMAN 2024, Poznań, Poland, September 2024 ▪ https://beatsigner.com/publications/signer_HUMAN2024.pdf
24. References ... ▪ MindXpres Website ▪ https://mindxpres.com ▪ The Lost Cosmonaut Video ▪ https://www.youtube.com/watch?v=fu27Cmr1aX4