Cross-Media Information Systems - Quo Vadis?

Seminar presentation about cross-media information spaces given at EPFL.

Transcript

1. 2 December 2005 Cross-Media Information Systems - Quo Vadis? Prof. Beat Signer beatsigner.com Web & Information Systems Engineering Lab (WISE) Department of Computer Science Vrije Universiteit Brussel
2. 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces
3. Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Dr. Bruno Dumas Multimodal Interaction, Cross- Media Information Architectures Lode Hoste Multimodal Interaction, Midas and Mudra Framework Brecht De Rooms Implicit Context-aware Human- Computer Interaction, HCII CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
4. Sandra Trullemans Personal Cross-Media Information Management Reinout Roels MindXpres Next Generation Presentation Tool Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 PARC User Interface (Mid 70s) ▪ Desktop metaphor ▪ WIMP interaction ▪ "filing cabinets" with hierarchical folders ▪ Paper paradigm ▪ "What You See Is What You Get" (WYSIWYG) ▪ what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Digital Documents as a Paper Simulator? Vannevar Bush Ted Nelson Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association. As We May Think, Atlantic Monthly, July 1945 Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adaptation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options!
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 RSL Metamodel and iServer
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Links Users Layers Plug-ins Structure Active Content
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Bridging the Paper-Digital Divide
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Interactive Paper Applications ▪ Enhanced Reading ▪ interactive paper maps ▪ interactive festival brochures ▪ educational materials ▪ auction brochures ▪ scientific publications ▪ … ▪ Enhanced Writing ▪ photo album ▪ researcher's notebook ▪ mammography annotation ▪ query by sketching ▪ … ▪ Paper-based Interfaces ▪ PaperPoint ▪ document proof-editing ▪ digital libraries ▪ video analysis tool ▪ educational games ▪ Art Installations ▪ Lost Cosmonaut ▪ Generosa Enterprise ▪ Interactive Tabletops ▪ photo browser ▪ collaborative sketch tool
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Interactive Festival Guide (EdFest)
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 The Lost Cosmonaut ▪ Writing as a collaborative act of memory and story- telling ▪ Reading, writing and narrative as an act of making sense ▪ Handwriting as a tool for human-computer interaction In collaboration with Axel Vogelsang, Artists in Labs Programme
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector)
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 RSL-based Associative File System ...
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 MindXpres Presentation Tool ▪ HTML5-based extensible core with three types of JavaScript plug-ins ▪ components - source code, graph, video, ... ▪ containers - slide, image gallery, ... ▪ structures - grid, graph, clustered, ... ▪ Presentation as RSL- based overlay structure ▪ enables navigation across different presentations
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 MindXpres Presentation Tool ...
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 MindXpres Presentation Tool ... ▪ Functional MindXpres prototype ▪ Ongoing work ▪ visual authoring tool - alternative to XML-based authoring ▪ rich media plug-ins - videos with subtitles, annotations etc. - statistical data with alternative visualisations - live sensor data - ... ▪ audience-driven presentations ▪ collaborative annotations and information sharing ▪ multimodal interaction (e.g. integration of PaperPoint concepts) ▪ import of existing slideware formats
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Fluid Multimodal Cross-Media Interfaces ▪ Investigation of mobile and distributed document formats ▪ Mobile Cross-Media Applications (MobiCraNT) ▪ Passive or active information units (crosslets) ▪ processing and storage of sensor data ▪ binding of specific multimodal interactions to resources ▪ Implicit context-aware human-computer interaction (HCII) ▪ Personal Cross-Media Information Management
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 ▪ Midas/Mudra Framework ▪ declarative definition of multimodal and multi-touch interactions ▪ rule-based language approach ▪ rapid prototyping and application development ▪ iGesture Workbench ▪ create and test gesture sets and algorithms ▪ different modalities: digital pen, Wii remote, … ▪ open source (www.igesture.org) ▪ SpeeG ▪ multimodal speech- and body gesture-based text input ▪ real-time correction of recognised voice input Multimodal and Multi-Touch Interaction
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Open Cross-Media Linking ▪ Four existing classes of cross-media link systems ▪ A: single source and single target type ▪ B: single source and multiple target types ▪ C: multiple source and single target type ▪ D: multiple source and multiple target types ▪ Our proposed open cross- media link approach covers the whole space # source types 1 2 3 4 5 ... 1 2 3 4 5 ... # target types B1 B2 D2 open cross-media linking D1 A1 ... An C1 C2 C3
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Open Cross-Media Authoring Cross-Media Link Service Client Application Cross-Media Browser & Editor Visual Plug-ins Resource Plug-in Repository Data Plug-ins Visual Plug-ins Data Plug-ins Visual Plug-ins
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Open Cross-Media Authoring ... ▪ Integration of a new resource type includes ▪ development of data and visual plug-ins ▪ registration with the resource plug-in repository ▪ Cross-media browser loads plug-ins on demand ▪ internal or external (third party application) visualisation ▪ Third party applications delegate the visualisation of a link target to the cross-media browser ▪ similar to "integration" of third-party applications in web browsers ▪ Non-monolithic link model and authoring tool ▪ resource-specific plug-ins
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Open Cross-Media Browser and Editor Cross-media browser & editor Client application showEntity(), getSelectedEntity() showEntity() C R U D R S L Menu Source Window Target Window
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Visual Plug-ins ▪ Cross-media browser deals only with RSL-specific data but not with any media-specific details ▪ create/delete links (CRUD), user management, … ▪ Definition of link source and target anchors (resources or selectors) handled by visual plug-ins ▪ separation of concerns ▪ Two basic purposes ▪ rendering of specific resource types (resources and selectors) ▪ creation and deletion of resources and selectors ▪ Well-defined visual plug-in API
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be November 2, 2012 Conclusions ▪ Time to question existing document formats which are based on the simulation of paper on desktop computers ▪ RSL-based representation of open and fluid cross-media information spaces ▪ interactive paper solutions ▪ associative file system ▪ MindXpres presentation tool ▪ crosslet-based fluid multimodal interfaces ▪ open cross-media information browser and editor ▪ Interlinked structural components rather than a document-centric representation of information ▪ data no longer "owned" by applications