Information Architectures - Lecture 2 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

Transcript

1. 2 December 2005 Next Generation User Interfaces Information Architectures Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com
2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 February 18, 2024 Information Architecture ▪ Information architecture (IA) addresses the organisation, structuring and labelling of content ▪ Modelling of information spaces ▪ maintenance ▪ linking ▪ navigation ▪ search ▪ presentation IA Content Users Context
3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 February 18, 2024 Personal Information Management (PIM) ▪ PIM is about keeping, organising and re-finding information Personal information management or PIM is both the practice and the study of the activities people perform to acquire, organise, maintain, retrieve, use, and control the distribution of information items such as documents (paper-based and digital), web pages, and email messages for everyday use to complete tasks (work- related and not) and to fulfil a person's various roles (as parent, employee, friend, member of a community, etc.). William Jones & Jaime Teevan, Personal Information Management, University of Washington Press, 2007
4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 February 18, 2024 Filers and Pilers ▪ Study by Malone (1983) revealed two strategies of information organisation in offices ▪ Filers: Instances are explicitly titled and arranged in some systematic order and these structures may themselves be explicitly titled and systematically organised ▪ Pilers: Piles tend not to have internal structure, other than access frequency; their spatial location is often the key to finding them ▪ Both files and piles are ways of collecting groups of elements into larger units ▪ These strategies are also applied in the digital world ▪ e.g. to organise emails, bookmarks or files
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 February 18, 2024 Long Term Memory ▪ Permanent memory store intended for the long-term storage of information (everything that we know including experiential knowledge, procedural skills, etc.) ▪ huge (if not unlimited) capacity ▪ relatively slow access time ▪ forgetting (when it occurs) happens much slower than with short-term memory ▪ Three long-term memory subsystems ▪ episodic memory: memory of events and experiences in a serial form (chronology) ▪ semantic memory: structured record of facts and concepts that we have acquired ▪ procedural skills: "know-how" memory (skills, procedures)
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 February 18, 2024 "As We May Think" (1945) When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used; one has to have rules as to which path will locate it, and the rules are cumbers- ome. Having found one item, moreover, one has to emerge from the system and re-enter on a new path. The human mind does not work that way. It operates by association. ... Vannevar Bush
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 February 18, 2024 "As We May Think" (1945) … ... It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another. This is the essential feature of the memex. The process of tying two items together is the important thing. ... Vannevar Bush, As We May Think, Atlanic Monthly, July 1945 Vannevar Bush
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 February 18, 2024 "As We May Think" (1945) … ▪ Bush's article As We May Think (1945) is often seen as the "origin" of hypertext ▪ The article introduces the Memex ▪ memory extender ▪ store and access information ▪ follow cross-references in the form of associative trails between pieces of information (microfilms) ▪ prototypical hypertext machine ▪ trail blazers are those who find delight in the task of establishing useful trails Memex
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 February 18, 2024 Beyond the Desktop Metaphor ▪ Desktop metaphor ▪ WIMP interaction ▪ "filing cabinets" with hierarchical folders ▪ Paper paradigm ▪ "What You See Is What You Get" (WYSIWYG) ▪ what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10 February 18, 2024 Digital Documents as a Paper Simulator? Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! This re-froze the computer document into a closed rectangular object which cannot be penetrated by outside markings (curtailing what you could do with paper). No marginal notes, no sticky notes, no crossouts, no insertions, no overlays, no highlighting - PAPER UNDER GLASS. Ted Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009 Ted Nelson
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11 February 18, 2024 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adaptation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 February 18, 2024 Lifestreams ▪ Avoid naming and classification efforts ▪ time-ordered stream as storage model ▪ motivated by studies of Malone ▪ Stream ▪ represents the past, present and future (e.g. reminders) ▪ stream filters (virtual directories) to further classify documents in substreams Freeman & Gelernter, 1996
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 February 18, 2024 MyLifeBits ▪ Lifelogging research project by Microsoft ▪ inspired by Bush's Memex ▪ towards "total recall" ▪ Database of resources and links ▪ multiple classification via collections ▪ dynamic collections (queries) ▪ annotations ▪ full-text search ▪ multiple visualisations Gemmell et al., 2002
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 February 18, 2024 Microsoft SenseCam ▪ Take a picture when one of the sensors is active ▪ light, motion or temperature ▪ around 4000 pictures a day ▪ Studies have shown that rich image sets do not promote "total recall" ▪ Commercial products Narrative Clip 2
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 February 18, 2024 Presto ▪ Placeless Documents ▪ more natural and fluid forms of interaction with a document space ▪ Semantic information no longer encoded in a file's path ▪ separation of document content, storage and properties ▪ Single document model ▪ integrate content from various sources Dourish et al., 1999
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 February 18, 2024 Presto … ▪ User-level properties ▪ key/value pairs added to document "proxies" ▪ Documents can be grouped in collections ▪ multiple classification ▪ dynamic collections ("fluid" collections) - membership defined based on predicates (presence and value of document properties) - furthermore, there exists an inclusion and exclusion list ▪ static collections - implemented as dynamic collections where only inclusion list is non-empty ▪ Lessons learned ▪ "One of the primary experiences of property-based document interaction is uniformity. […] This uniformity has proven invaluable."
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 February 18, 2024 Haystack ▪ PIM solution using RDF as data model ▪ predefined basic ontology ▪ personalised ontologies ▪ Most data imported ▪ special extractors ▪ Collections ▪ multiple classification and query-based collections ▪ User interfaces also modelled in RDF ▪ highly customisable UI Adar et al., 1999
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 February 18, 2024 Stuff I've Seen ▪ Index over multiple repositories ▪ email, file system, … ▪ Different forms of search ▪ full-text search ▪ contextual cues - time, user, … ▪ Platform for different visualisations ▪ "Predecessor" of Windows Search and Spotlight Dumais et al., 2003
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 February 18, 2024 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 February 18, 2024 RSL Hypermedia Metamodel & Link Server B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 February 18, 2024 Users B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007x
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 February 18, 2024 Bridging the Paper-Digital Divide
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 February 18, 2024 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector) Gregory Cardone
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 February 18, 2024 Next Generation File Manager B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 February 18, 2024 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ OC2 PIM model ▪ based on RSL hypermedia metamodel ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 February 18, 2024 Object-Concept-Context (OC2) Framework object layer structural link concept layer context layer navigational link associative link weighted link Object 1 Object n Object 2 Context 1 Context n Concept 1 Concept 2 Concept n extent link entity link Links selector resource (1,*) (1,*) (1,1) (0,*) (0,*) (0,*) RefersTo HasTarget HasSource partition Entities (0,*) link Navigational Links link Extent Links partition Resources Selectors context Contexts link Structural Links link Associative Links resource Physical Objects resource Digital Objects resource Concepts resource Objects HasAssoc Target (0,*) (1,*) HasExt Target HasExt Source HasAssoc Source (1,1) (1,*) (0,*) (0,*) (0,*) (1,1) disjoint partition translation OC2 conceptual model RSL-based metamodel
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 February 18, 2024 PimVis Setup S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 February 18, 2024 PimVis Document View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 February 18, 2024 PimVis Focus View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 February 18, 2024 PimVis Context View S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 February 18, 2024 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide and monolithic documents ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation
32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 February 18, 2024 MindXpres Presentation Platform ▪ Flexible representation of presentations ▪ use of structural RSL links ▪ separation of content and structure ▪ Extensible platform ▪ content-based approach ▪ cross-media content reuse ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014
33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 February 18, 2024 Interactive Source Code Plug-in R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform , CCIS 583, 2016
34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 February 18, 2024 Interactive Data Visualisation R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017
35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 February 18, 2024 Interactive Data Visualisation ... R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, CCIS, 739, 2017
36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 February 18, 2024 MindXpres Document Representation R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019
37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 February 18, 2024 MindXpres (Video)
38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 February 18, 2024 Open Cross-Media Document Linking local visual plug-ins external visual plug-ins communicate Link Browser A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Proceedings of DocEng 2018
39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 February 18, 2024 Open Cross-Media Document Linking … ▪ Integration of a new resource type includes ▪ development of a data plug-in ▪ implementation of a visual plug-in ▪ registration with the resource plug-in repository ▪ Link browser loads plug-ins on demand ▪ internal or external (third party application) visualisation ▪ Third-party applications use the link browser for visualisation ▪ similar to the integration of third-party applications with web browsers
40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 February 18, 2024 RSL-based Hypermedia Engine RSL-based Hypermedia Engine Legacy Applications File System Driver Hypermedia-enabled Applications HT2021.docx HT2021.docx … B. Signer et al., Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proc. of Hypertext 2021
41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 February 18, 2024 Homework ▪ Read the following paper that is available on Canvas (Papers/Signer2019) ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019
42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 February 18, 2024 Exercise 1 ▪ Phidgets, Arduino and ESP32
43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 February 18, 2024 References ▪ V. Bush, As We May Think, Atlantic Monthly 176(1), July 1945 ▪ https://dx.doi.org/10.1145/227181.227186 (reprint) ▪ T. Malone, How do People Organize their Desks? Impli- cations for the Design of Office Information Systems, ACM Transactions on Information Systems (TOIS) 1(1), January 1983 ▪ https://dx.doi.org/10.1145/357423.357430 ▪ T. Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, ISBN-13: 978-0578004389, Mindful Press 2009
44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 February 18, 2024 References … ▪ B. Signer, What is Wrong with Digital Docu- ments? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of ER 2010, 29th International Conference on Conceptual Modeling, Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf
45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45 February 18, 2024 References … ▪ E. Freeman and D. Gelernter, Lifestreams: A Storage Model for Personal Data, ACM SIGMOD Record 25(1), March 1996 ▪ https://dx.doi.org/10.1145/381854.381893 ▪ J. Gemmell et al., MyLifeBits: Fulfilling the Memex Vision, Proceedings of Multimedia 2002, Juan-les-Pins, December 2002 ▪ https://dx.doi.org/10.1145/641007.641053 ▪ P. Dourish et al., Presto: An Experimental Architecture for Fluid Interactive Document Spaces, ACM Trans- actions on Computer-Human Interaction 6(2), June 1999 ▪ https://dx.doi.org/10.1145/319091.319099
46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46 February 18, 2024 References … ▪ E. Adar, D. Karger and L.A. Stein, Haystack: Per-User Information Environments, Proceedings of CIKM 1999, 8th International Conference on Information and Knowledge Management, Kansas, USA, November 1999 ▪ https://dx.doi.org/10.1145/319950.323231 ▪ S. Dumais et al., Stuff I’ve Seen: A System for Personal Information Retrieval and Re-Use, Proceedings of SIGIR 2003, 26th Annual International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, Toronto, Canada, August 2003 ▪ https://dx.doi.org/10.1145/860435.860451
47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47 February 18, 2024 References … ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Pro- ceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf ▪ B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environ- ments, Proceedings of Hypertext 2021, International Conference on Hypertext and Social Media, 2021 ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf
48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48 February 18, 2024 References … ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf ▪ S. Trullemans, P. Ebrahimi and B. Signer, Crossing Spaces: Towards Cross-Media Personal Information Management User Interfaces, Proceedings of AVI 2018, International Working Conference on Advanced Visual Interfaces, Grosseto, Italy, May 2018 ▪ https://beatsigner.com/publications/trullemans_AVI2018a.pdf
49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49 February 18, 2024 References … ▪ A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Pro- ceedings of DocEng 2018, 18th ACM Symposium on Document Engineering, Halifax, Canada, August 2018 ▪ https://beatsigner.com/publications/tayeh_DocEng2018.pdf ▪ R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th International Confer- ence on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ https://beatsigner.com/publications/roels_WISE2014.pdf
50. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50 February 18, 2024 References … ▪ R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf ▪ R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, Communications in Computer and Information Science (CCIS), 739, 2017 ▪ https://beatsigner.com/publications/roels_CCIS2017.pdf
51. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 51 February 18, 2024 References … ▪ R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, Proceedings of the ACM on Human-Computer Interaction (PACMHCI), 3(Issue EICS), June 2019 ▪ https://beatsigner.com/publications/roels_PACMHCI2019.pdf ▪ MindXpres Presentation Platform ▪ https://mindxpres.com
52. 2 December 2005 Next Lecture Multimodal Interaction