Information Architectures - Lecture 4 - Next Generation User Interfaces (4018166FNR)

1135dc242dcff3b90ae46fc586ff4da8?s=47 Beat Signer
October 11, 2020

Information Architectures - Lecture 4 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

1135dc242dcff3b90ae46fc586ff4da8?s=128

Beat Signer

October 11, 2020
Tweet

Transcript

  1. 2 December 2005 Next Generation User Interfaces Information Architectures Prof.

    Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com
  2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2

    October 12, 2020 Information Architecture ▪ Information architecture (IA) addresses the organisation, structuring and labelling of content ▪ Modelling of information spaces ▪ maintenance ▪ linking ▪ navigation ▪ search ▪ presentation IA Content Users Context
  3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3

    October 12, 2020 Personal Information Management (PIM) ▪ PIM is about keeping, organising and re-finding information Personal information management or PIM is both the practice and the study of the activities people perform to acquire, organise, maintain, retrieve, use, and control the distribution of information items such as documents (paper-based and digital), web pages, and email messages for everyday use to complete tasks (work- related and not) and to fulfil a person's various roles (as parent, employee, friend, member of a community, etc.). William Jones & Jaime Teevan, Personal Information Management, University of Washington Press, 2007
  4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4

    October 12, 2020 Filers and Pilers ▪ Study by Malone (1983) revealed two strategies of information organisation in offices ▪ Filers: Instances are explicitly titled and arranged in some systematic order and these structures may themselves be explicitly titled and systematically organised ▪ Pilers: Piles tend not to have internal structure, other than access frequency; their spatial location is often the key to finding them ▪ Both files and piles are ways of collecting groups of elements into larger units ▪ These strategies are also applied in the digital world ▪ e.g. to organise emails, bookmarks or files
  5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5

    October 12, 2020 Long Term Memory ▪ Permanent memory store intended for the long-term storage of information (everything that we know including experiential knowledge, procedural skills, etc.) ▪ huge (if not unlimited) capacity ▪ relatively slow access time ▪ forgetting (when it occurs) happens much more slowly than with short-term memory ▪ Three long-term memory subsystems ▪ episodic memory: memory of events and experiences in a serial form (chronology) ▪ semantic memory: structured record of facts and concepts that we have acquired ▪ procedural skills: "know-how" memory (skills, procedures)
  6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6

    October 12, 2020 "As We May Think" (1945) When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used; one has to have rules as to which path will locate it, and the rules are cumbers- ome. Having found one item, moreover, one has to emerge from the system and re-enter on a new path. The human mind does not work that way. It operates by association. ... Vannevar Bush
  7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7

    October 12, 2020 "As We May Think" (1945) … ... It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another. This is the essential feature of the memex. The process of tying two items together is the important thing. ... Vannevar Bush, As We May Think, Atlanic Monthly, July 1945 Vannevar Bush
  8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8

    October 12, 2020 "As We May Think" (1945) … ▪ Bush's article As We May Think (1945) is often seen as the "origin" of hypertext ▪ The article introduces the Memex ▪ memory extender ▪ store and access information ▪ follow cross-references in the form of associative trails between pieces of information (microfilms) ▪ prototypical hypertext machine ▪ trail blazers are those who find delight in the task of establishing useful trails Memex
  9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9

    October 12, 2020 Beyond the Desktop Metaphor ▪ Desktop metaphor ▪ WIMP interaction ▪ "filing cabinets" with hierarchical folders ▪ Paper paradigm ▪ "What You See Is What You Get" (WYSIWYG) ▪ what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
  10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10

    October 12, 2020 Digital Documents as a Paper Simulator? Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! This re-froze the computer document into a closed rectangular object which cannot be penetrated by outside markings (curtailing what you could do with paper). No marginal notes, no sticky notes, no crossouts, no insertions, no overlays, no highlighting - PAPER UNDER GLASS. Ted Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009 Ted Nelson
  11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11

    October 12, 2020 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adaptation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation
  12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12

    October 12, 2020 Lifestreams ▪ Avoid naming and classification efforts ▪ time-ordered stream as storage model ▪ motivated by studies of Malone ▪ Stream ▪ represents the past, present and future (e.g. reminders) ▪ stream filters (virtual directories) to further classify documents Freeman & Gelernter, 1996
  13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13

    October 12, 2020 MyLifeBits ▪ Lifelogging research project by Microsoft ▪ inspired by Bush's Memex ▪ towards "total recall" ▪ Database of resources and links ▪ multiple classification via collections ▪ dynamic collections (queries) ▪ annotations ▪ full-text search ▪ multiple visualisations Gemmell et al., 2002
  14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14

    October 12, 2020 Microsoft SenseCam ▪ Take a picture when one of the sensors is active ▪ light, motion or temperature ▪ around 4000 pictures a day ▪ Studies have shown that rich image sets do not promote "total recall" ▪ Commercial products Narrative Clip 2
  15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15

    October 12, 2020 Presto ▪ Placeless Documents ▪ more natural and fluid forms of interaction with a document space ▪ Semantic information no longer encoded in a file's path ▪ separation of document content, storage and properties ▪ Single document model ▪ integrate content from various sources Dourish et al., 1999
  16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16

    October 12, 2020 Presto … ▪ User-level properties ▪ key/value pairs added to document "proxies" ▪ Documents can be grouped in collections ▪ multiple classification ▪ dynamic collections ("fluid" collections) - membership defined based on predicates (presence and value of document properties) - furthermore, there exists an inclusion and exclusion list ▪ static collections - implemented as dynamic collections where only inclusion list is non-empty ▪ Lessons learned ▪ "One of the primary experiences of property-based document interaction is uniformity. […] This uniformity has proven invaluable."
  17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17

    October 12, 2020 Haystack ▪ PIM solution using RDF as data model ▪ predefined basic ontology ▪ personalised ontologies ▪ Most data imported ▪ special extractors ▪ Collections ▪ multiple classification and query-based collections ▪ User interfaces also modelled in RDF ▪ highly customisable UI Adar et al., 1999
  18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18

    October 12, 2020 Semantic Desktops ▪ Apply Semantic Web tech- nologies to personal infor- mation management (PIM) ▪ inter-application data sharing ▪ enhancement of limited filesystem functionality - add document metadata ▪ machine-readable data ▪ Nepomuk is a semantic desktop solution ▪ semantic search in KDE 4.0 ▪ later replaced by Baloo Nepomuk Integration with Dolphin (KDE 4.0)
  19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19

    October 12, 2020 Stuff I've Seen ▪ Index over multiple repositories ▪ email, file system, … ▪ Different forms of search ▪ full-text search ▪ contextual cues - time, user, … ▪ Platform for different visualisations ▪ "Predecessor" of Windows Search and Spotlight Dumais et al., 2003
  20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20

    October 12, 2020 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces
  21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21

    October 12, 2020 RSL Hypermedia Metamodel and iServer Global Information Systems Group, ETH Zurich
  22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22

    October 12, 2020 Links Users Layers Plug-ins Structure Active Content Global Information Systems Group, ETH Zurich
  23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23

    October 12, 2020 Bridging the Paper-Digital Divide
  24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24

    October 12, 2020 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector) Gregory Cardone
  25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25

    October 12, 2020 RSL-based Associative File System ... Gregory Cardone
  26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26

    October 12, 2020 PIM@WISE ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ OC2 PIM model ▪ based on RSL hypermedia metamodel ▪ Cross-media PIM system ▪ explicit as well as implicit associations between entities Sandra Trullemans
  27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27

    October 12, 2020 Object-Concept-Context (OC2) Framework object layer structural link concept layer context layer navigational link associative link weighted link Object 1 Object n Object 2 Context 1 Context n Concept 1 Concept 2 Concept n extent link entity link Links selector resource (1,*) (1,*) (1,1) (0,*) (0,*) (0,*) RefersTo HasTarget HasSource partition Entities (0,*) link Navigational Links link Extent Links partition Resources Selectors context Contexts link Structural Links link Associative Links resource Physical Objects resource Digital Objects resource Concepts resource Objects HasAssoc Target (0,*) (1,*) HasExt Target HasExt Source HasAssoc Source (1,1) (1,*) (0,*) (0,*) (0,*) (1,1) disjoint partition translation OC2 conceptual framework RSL-based metamodel
  28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28

    October 12, 2020 Cross-Media PIM (PimVis) Sandra Trullemans Audrey Sanctorum
  29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29

    October 12, 2020 PimVis Setup Sandra Trullemans
  30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30

    October 12, 2020 PimVis Document View Audrey Sanctorum
  31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31

    October 12, 2020 PimVis Focus View Audrey Sanctorum
  32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32

    October 12, 2020 PimVis Context View Audrey Sanctorum
  33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33

    October 12, 2020 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation
  34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34

    October 12, 2020 MindXpres Presentation Platform ▪ Flexible representation of presentations ▪ use of structural RSL links ▪ separation of content and structure ▪ Extensible platform ▪ content-based approach ▪ cross-media content reuse ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types Reinout Roels
  35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35

    October 12, 2020 Interactive Source Code Plug-in Paul Mestereaga
  36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36

    October 12, 2020 Interactive Data Visualisation Plug-in
  37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37

    October 12, 2020 Interactive Data Visualisation Plug-in ...
  38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38

    October 12, 2020 MindXpres (Video)
  39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39

    October 12, 2020 Open Cross-Media Document Linking local visual plug-ins external visual plug-ins communicate Link Browser
  40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40

    October 12, 2020 Open Cross-Media Document Linking … ▪ Integration of a new resource type includes ▪ development of a data plug-in ▪ implementation of a visual plug-in ▪ registration with the resource plug-in repository ▪ Link browser loads plug-ins on demand ▪ internal or external (third party application) visualisation ▪ Third-party applications use the link browser for visualisation ▪ similar to the integration of third-party applications with web browsers
  41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41

    October 12, 2020 Next Generation Cross-Media Storage RSL-based Hypermedia Engine … Legacy Applications File System Driver Hypermedia-enabled Applications RCIS2019.docx RCIS2019.docx
  42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42

    October 12, 2020 Homework ▪ Read the following paper that is available on Canvas (Papers/Signer2019) ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019
  43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43

    October 12, 2020 References ▪ V. Bush, As We May Think, Atlantic Monthly 176(1), July 1945 ▪ https://dx.doi.org/10.1145/227181.227186 (reprint) ▪ T. Malone, How do People Organize their Desks? Impli- cations for the Design of Office Information Systems, ACM Transactions on Information Systems (TOIS) 1(1), January 1983 ▪ https://dx.doi.org/10.1145/357423.357430 ▪ T. Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, ISBN-13: 978-0578004389, Mindful Press 2009
  44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44

    October 12, 2020 References … ▪ B. Signer, What is Wrong with Digital Docu- ments? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of ER 2010, 29th International Conference on Conceptual Modeling, Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, 13th International Conference on Research Challenges in Information Science, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf
  45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45

    October 12, 2020 References … ▪ E. Freeman and D. Gelernter, Lifestreams: A Storage Model for Personal Data, ACM SIGMOD Record 25(1), March 1996 ▪ https://dx.doi.org/10.1145/381854.381893 ▪ J. Gemmell et al., MyLifeBits: Fulfilling the Memex Vision, Proceedings of Multimedia 2002, Juan-les-Pins, December 2002 ▪ https://dx.doi.org/10.1145/641007.641053 ▪ P. Dourish et al., Presto: An Experimental Architecture for Fluid Interactive Document Spaces, ACM Trans- actions on Computer-Human Interaction 6(2), June 1999 ▪ https://dx.doi.org/10.1145/319091.319099
  46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46

    October 12, 2020 References … ▪ E. Adar, D. Karger and L.A. Stein, Haystack: Per-User Information Environments, Proceedings of CIKM 1999, 8th International Conference on Information and Knowledge Management, Kansas, USA, November 1999 ▪ https://dx.doi.org/10.1145/319950.323231 ▪ S. Dumais et al., Stuff I’ve Seen: A System for Personal Information Retrieval and Re-Use, Proceedings of SIGIR 2003, 26th Annual International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, Toronto, Canada, August 2003 ▪ https://dx.doi.org/10.1145/860435.860451
  47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47

    October 12, 2020 References … ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Pro- ceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf ▪ S. Trullemans and B. Signer, Towards a Conceptual Framework and Metamodel for Context-Aware Personal Cross-Media Information Management Systems, Proceedings of ER 2014, 33rd International Conference on Conceptual Modelling, Atlanta, USA, October, 2014 ▪ https://beatsigner.com/publications/trullemans_ER2014.pdf
  48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48

    October 12, 2020 References … ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf ▪ S. Trullemans, P. Ebrahimi and B. Signer, Crossing Spaces: Towards Cross-Media Personal Information Management User Interfaces, Proceedings of AVI 2018, International Working Conference on Advanced Visual Interfaces, Grosseto, Italy, May 2018 ▪ https://beatsigner.com/publications/trullemans_AVI2018a.pdf
  49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49

    October 12, 2020 References … ▪ A.A.O Tayeh, P. Ebrahimi and Beat Signer, Cross-Media Document Linking and Navigation, Pro- ceedings of DocEng 2018, 18th ACM Symposium on Document Engineering, Halifax, Canada, August 2018 ▪ https://beatsigner.com/publications/tayeh_DocEng2018.pdf ▪ R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th International Confer- ence on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ https://beatsigner.com/publications/roels_WISE2014.pdf
  50. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50

    October 12, 2020 References … ▪ R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf ▪ R. Roels, Y. Baeten and B. Signer, Interactive and Narrative Data Visualisation for Presentation-based Knowledge Transfer, Communications in Computer and Information Science (CCIS), 739, 2017 ▪ https://beatsigner.com/publications/roels_CCIS2017.pdf
  51. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 51

    October 12, 2020 References … ▪ MindXpres Presentation Platform ▪ https://mindxpres.com
  52. 2 December 2005 Next Lecture Multimodal Interaction