2 December 2005 Next Generation User Interfaces Information Architectures Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - [email protected] 2 February 17, 2025 Information Architectures ▪ Information architectures (IAs) address the organisation, structuring and labelling of content ▪ Modelling of information spaces ▪ maintenance ▪ linking ▪ navigation ▪ search ▪ presentation IA Content Users Context

Beat Signer - Department of Computer Science - [email protected] 3 February 17, 2025 Personal Information Management (PIM) ▪ PIM is about keeping, organising and re-finding information Personal information management or PIM is both the practice and the study of the activities people perform to acquire, organise, maintain, retrieve, use, and control the distribution of information items such as documents (paper-based and digital), web pages, and email messages for everyday use to complete tasks (work- related and not) and to fulfil a person's various roles (as parent, employee, friend, member of a community, etc.). William Jones & Jaime Teevan, Personal Information Management, University of Washington Press, 2007

Beat Signer - Department of Computer Science - [email protected] 4 February 17, 2025 Filers and Pilers ▪ Study by Malone (1983) revealed two strategies of information organisation in offices ▪ Filers: Instances are explicitly titled and arranged in some systematic order and these structures may themselves be explicitly titled and systematically organised ▪ Pilers: Piles tend not to have internal structure, other than access frequency; their spatial location is often the key to finding them ▪ Both files and piles are ways of collecting groups of elements into larger units ▪ These strategies are also applied in the digital world ▪ e.g.to organise emails, bookmarks or files

Beat Signer - Department of Computer Science - [email protected] 5 February 17, 2025 Long Term Memory ▪ Permanent memory store intended for the long-term storage of information (everything that we know including experiential knowledge, procedural skills, etc.) ▪ huge (if not unlimited) capacity ▪ relatively slow access time ▪ forgetting (when it occurs) happens much slower than with short-term memory ▪ Three long-term memory subsystems ▪ episodic memory: memory of events and experiences in a serial form (chronology) ▪ semantic memory: structured record of facts and concepts that we have acquired ▪ procedural skills: "know-how" memory (skills, procedures)

Beat Signer - Department of Computer Science - [email protected] 6 February 17, 2025 "As We May Think" (1945) When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used; one has to have rules as to which path will locate it, and the rules are cumbers- ome. Having found one item, moreover, one has to emerge from the system and re-enter on a new path. The human mind does not work that way. It operates by association. ... Vannevar Bush

Beat Signer - Department of Computer Science - [email protected] 7 February 17, 2025 "As We May Think" (1945) … ... It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another. This is the essential feature of the memex. The process of tying two items together is the important thing. ... Vannevar Bush, As We May Think, Atlanic Monthly, July 1945 Vannevar Bush

Beat Signer - Department of Computer Science - [email protected] 8 February 17, 2025 "As We May Think" (1945) … ▪ Bush's article As We May Think (1945) is often seen as the "origin" of hypertext ▪ The article introduces the Memex ▪ memory extender ▪ store and access information ▪ follow cross-references in the form of associative trails between pieces of information (microfilms) ▪ prototypical hypertext machine ▪ trail blazers are those who find delight in the task of establishing useful trails Memex

Beat Signer - Department of Computer Science - [email protected] 9 February 17, 2025 Beyond the Desktop Metaphor ▪ Desktop metaphor ▪ WIMP interaction ▪ "filing cabinets" with hierarchical folders ▪ Paper paradigm ▪ "What You See Is What You Get" (WYSIWYG) ▪ what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]

Beat Signer - Department of Computer Science - [email protected] 10 February 17, 2025 Digital Documents as a Paper Simulator? Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! This re-froze the computer document into a closed rectangular object which cannot be penetrated by outside markings (curtailing what you could do with paper). No marginal notes, no sticky notes, no crossouts, no insertions, no overlays, no highlighting - PAPER UNDER GLASS. Ted Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009 Ted Nelson

Beat Signer - Department of Computer Science - [email protected] 11 February 17, 2025 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adaptation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation

Beat Signer - Department of Computer Science - [email protected] 12 February 17, 2025 Lifestreams ▪ Avoid naming and classification efforts ▪ time-ordered stream as storage model ▪ motivated by studies of Malone ▪ Stream ▪ represents the past, present and future (e.g.reminders) ▪ stream filters (virtual directories) to further classify documents in substreams Freeman & Gelernter, 1996

Beat Signer - Department of Computer Science - [email protected] 13 February 17, 2025 MyLifeBits ▪ Lifelogging research project by Microsoft ▪ inspired by Bush's Memex ▪ towards "total recall" ▪ Database of resources and links ▪ multiple classification via collections ▪ dynamic collections (queries) ▪ annotations ▪ full-text search ▪ multiple visualisations Gemmell et al., 2002

Beat Signer - Department of Computer Science - [email protected] 14 February 17, 2025 Microsoft SenseCam ▪ Take a picture when one of the sensors is active ▪ light, motion or temperature ▪ around 4000 pictures a day ▪ Studies have shown that rich image sets do not promote "total recall" ▪ Commercial products Narrative Clip 2

Beat Signer - Department of Computer Science - [email protected] 15 February 17, 2025 Presto ▪ Placeless Documents ▪ more natural and fluid forms of interaction with a document space ▪ Semantic information no longer encoded in a file's path ▪ separation of document content, storage and properties ▪ Single document model ▪ integrate content from various sources Dourish et al., 1999

Beat Signer - Department of Computer Science - [email protected] 16 February 17, 2025 Presto … ▪ User-level properties ▪ key/value pairs added to document "proxies" ▪ Documents can be grouped in collections ▪ multiple classification ▪ dynamic collections ("fluid" collections) - membership defined based on predicates (presence and value of document properties) - furthermore, there exists an inclusion and exclusion list ▪ static collections - implemented as dynamic collections where only inclusion list is non-empty ▪ Lessons learned ▪ "One of the primary experiences of property-based document interaction is uniformity. […] This uniformity has proven invaluable."

Beat Signer - Department of Computer Science - [email protected] 17 February 17, 2025 Haystack ▪ PIM solution using RDF as data model ▪ predefined basic ontology ▪ personalised ontologies ▪ Most data imported ▪ special extractors ▪ Collections ▪ multiple classification and query-based collections ▪ User interfaces also modelled in RDF ▪ highly customisable UI Adar et al., 1999

Beat Signer - Department of Computer Science - [email protected] 18 February 17, 2025 Stuff I've Seen ▪ Index over multiple repositories ▪ email, file system, … ▪ Different forms of search ▪ full-text search ▪ contextual cues - time, user, … ▪ Platform for different visualisations ▪ "Predecessor" of Windows Search and Spotlight Dumais et al., 2003

Beat Signer - Department of Computer Science - [email protected] 19 February 17, 2025 edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Fluid Cross-Media Information Spaces

Beat Signer - Department of Computer Science - [email protected] 20 February 17, 2025 RSL Hypermedia Metamodel & Link Server iwebfirefox movieplugin ipaper B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x

Beat Signer - Department of Computer Science - [email protected] 21 February 17, 2025 iserverNew_blue Users B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x

Beat Signer - Department of Computer Science - [email protected] 22 February 17, 2025 Bridging the Paper-Digital Divide picture1.png PIC00008.jpg map1.jpg IMG_0738.JPG

Beat Signer - Department of Computer Science - [email protected] 23 February 17, 2025 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector) Gregory Cardone

Beat Signer - Department of Computer Science - [email protected] 24 February 17, 2025 Next Generation File Manager Graphical user interface Description automatically generated B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021 B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021

Beat Signer - Department of Computer Science - [email protected] 25 February 17, 2025 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ OC2 PIM model ▪ based on RSL hypermedia metamodel ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities

Beat Signer - Department of Computer Science - [email protected] 26 February 17, 2025 Object-Concept-Context (OC2) Framework object layer structural link concept layer context layer navigational link associative link weighted link Object 1 Object n Object 2 Context 1 Context n Concept 1 Concept 2 Concept n extent link entity link Links selector resource (1,*) (1,*) (1,1) (0,*) (0,*) (0,*) RefersTo HasTarget HasSource partition Entities (0,*) link Navigational Links link Extent Links partition Resources Selectors context Contexts link Structural Links link Associative Links resource Physical Objects resource Digital Objects resource Concepts resource Objects HasAssoc Target (0,*) (1,*) HasExt Target HasExt Source HasAssoc Source (1,1) (1,*) (0,*) (0,*) (0,*) (1,1) disjoint partition translation OC2 conceptual model RSL-based metamodel

Beat Signer - Department of Computer Science - [email protected] 27 February 17, 2025 PimVis Setup S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016

Beat Signer - Department of Computer Science - [email protected] 28 February 17, 2025 PimVis Document View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016

Beat Signer - Department of Computer Science - [email protected] 29 February 17, 2025 PimVis Focus View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016

Beat Signer - Department of Computer Science - [email protected] 30 February 17, 2025 PimVis Context View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016 S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016

Beat Signer - Department of Computer Science - [email protected] 31 February 17, 2025 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide and monolithic documents ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation

Beat Signer - Department of Computer Science - [email protected] 32 February 17, 2025 MindXpres Presentation Platform ▪ Flexible representation of presentations ▪ use of structural RSL links ▪ separation of content and structure ▪ Extensible platform ▪ content-based approach ▪ cross-media content reuse ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014 R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014

Beat Signer - Department of Computer Science - [email protected] 33 February 17, 2025 Interactive Source Code Plug-in R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform , CCIS 583, 2016 R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform , CCIS 583, 2016

Beat Signer - Department of Computer Science - [email protected] 34 February 17, 2025 Interactive Data Visualisation R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017 R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017

Beat Signer - Department of Computer Science - [email protected] 35 February 17, 2025 Interactive Data Visualisation ... R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017 R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017

Beat Signer - Department of Computer Science - [email protected] 36 February 17, 2025 MindXpres Document Representation R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019 R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019

Beat Signer - Department of Computer Science - [email protected] 37 February 17, 2025 MindXpres (Video)

Beat Signer - Department of Computer Science - [email protected] 38 February 17, 2025 Open Cross-Media Document Linking local visual plug-ins external visual plug-ins communicate Link Browser A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Proceedings of DocEng 2018 A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Proceedings of DocEng 2018

Beat Signer - Department of Computer Science - [email protected] 39 February 17, 2025 Open Cross-Media Document Linking … ▪ Integration of a new resource type includes ▪ development of a data plug-in ▪ implementation of a visual plug-in ▪ registration with the resource plug-in repository ▪ Link browser loads plug-ins on demand ▪ internal or external (third party application) visualisation ▪ Third-party applications use the link browser for visualisation ▪ similar to the integration of third-party applications with web browsers

Beat Signer - Department of Computer Science - [email protected] 40 February 17, 2025 RSL-based Hypermedia Engine RSL-based Hypermedia Engine Legacy Applications File System Driver File System Driver Hypermedia-enabled Applications HT2021.docx HT2021.docx … B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proc. of Hypertext 2021 B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proc. of Hypertext 2021

Beat Signer - Department of Computer Science - [email protected] 41 February 17, 2025 Homework ▪ Read the following paper that is available on Canvas (Papers/Signer2019) ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019

Beat Signer - Department of Computer Science - [email protected] 42 February 17, 2025 Exercise 1 ▪ Phidgets, Arduino and ESP32

Beat Signer - Department of Computer Science - [email protected] 43 February 17, 2025 References ▪ V. Bush, As We May Think, Atlantic Monthly 176(1), July 1945 ▪ https://dx.doi.org/10.1145/227181.227186 (reprint) ▪ T. Malone, How do People Organize their Desks? Impli- cations for the Design of Office Information Systems, ACM Transactions on Information Systems (TOIS) 1(1), January 1983 ▪ https://dx.doi.org/10.1145/357423.357430 ▪ T. Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, ISBN-13: 978-0578004389, Mindful Press 2009

Beat Signer - Department of Computer Science - [email protected] 44 February 17, 2025 References … ▪ B. Signer, What is Wrong with Digital Docu- ments? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of ER 2010, 29th International Conference on Conceptual Modeling, Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf

Beat Signer - Department of Computer Science - [email protected] 45 February 17, 2025 References … ▪ E. Freeman and D. Gelernter, Lifestreams: A Storage Model for Personal Data, ACM SIGMOD Record 25(1), March 1996 ▪ https://dx.doi.org/10.1145/381854.381893 ▪ J. Gemmell et al., MyLifeBits: Fulfilling the Memex Vision, Proceedings of Multimedia 2002, Juan-les-Pins, December 2002 ▪ https://dx.doi.org/10.1145/641007.641053 ▪ P. Dourish et al., Presto: An Experimental Architecture for Fluid Interactive Document Spaces, ACM Trans- actions on Computer-Human Interaction 6(2), June 1999 ▪ https://dx.doi.org/10.1145/319091.319099

Beat Signer - Department of Computer Science - [email protected] 46 February 17, 2025 References … ▪ E. Adar, D. Karger and L.A. Stein, Haystack: Per-User Information Environments, Proceedings of CIKM 1999, 8th International Conference on Information and Knowledge Management, Kansas, USA, November 1999 ▪ https://dx.doi.org/10.1145/319950.323231 ▪ S. Dumais et al., Stuff I’ve Seen: A System for Personal Information Retrieval and Re-Use, Proceedings of SIGIR 2003, 26th Annual International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, Toronto, Canada, August 2003 ▪ https://dx.doi.org/10.1145/860435.860451

Beat Signer - Department of Computer Science - [email protected] 47 February 17, 2025 References … ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Pro- ceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf ▪ B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environ- ments, Proceedings of Hypertext 2021, International Conference on Hypertext and Social Media, 2021 ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf

Beat Signer - Department of Computer Science - [email protected] 48 February 17, 2025 References … ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf ▪ S. Trullemans, P. Ebrahimi and B. Signer, Crossing Spaces: Towards Cross-Media Personal Information Management User Interfaces, Proceedings of AVI 2018, International Working Conference on Advanced Visual Interfaces, Grosseto, Italy, May 2018 ▪ https://beatsigner.com/publications/trullemans_AVI2018a.pdf

Beat Signer - Department of Computer Science - [email protected] 49 February 17, 2025 References … ▪ A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Pro- ceedings of DocEng 2018, 18th ACM Symposium on Document Engineering, Halifax, Canada, August 2018 ▪ https://beatsigner.com/publications/tayeh_DocEng2018.pdf ▪ R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th International Confer- ence on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ https://beatsigner.com/publications/roels_WISE2014.pdf

Beat Signer - Department of Computer Science - [email protected] 50 February 17, 2025 References … ▪ R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf ▪ R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, Communications in Computer and Information Science (CCIS), 739, 2017 ▪ https://beatsigner.com/publications/roels_CCIS2017.pdf

Beat Signer - Department of Computer Science - [email protected] 51 February 17, 2025 References … ▪ R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, Proceedings of the ACM on Human-Computer Interaction (PACMHCI), 3(Issue EICS), June 2019 ▪ https://beatsigner.com/publications/roels_PACMHCI2019.pdf ▪ MindXpres Presentation Platform ▪ https://mindxpres.com

2 December 2005 Next Lecture Multimodal Interaction