Paper-Digital User Interfaces - Applications, Frameworks and Future Challenges

Invited talk given at the User Interface Colloquium, Otto-von-Guericke University Magdeburg, Germany, November 2, 2009

While there have been dramatic increases in the use of digital technologies for information storage, processing and delivery over the last few decades, the affordances of paper have ensured its retention as a key information medium. Despite predictions of the paperless office, paper is ever more present in our daily work. However, there is a gap between the paper and digital worlds: information present in paper documents cannot be seamlessly transferred to digital media and digital services are not easily accessible from the paper world.

ABSTRACT: In this talk I will present an information-centric approach for integrating paper with digital as well as physical media based on a general cross-media information platform (iServer). Some details about the architecture and implementation of the iServer platform as well as the underlying resource-selector-link (RSL) metamodel for cross-media linking will be highlighted. A selection of interactive paper applications that have been developed based on this platform over the past nine years will be presented, including the EdFest interactive paper guide for the Edinburgh festivals, the PaperPoint presentation tool as well as the PaperProof proof-editing solution. Challenges and solutions for novel forms of interactive paper and cross-media publishing are discussed based on the presented applications. This includes specific extensions of the iServer platform and RSL model as well as the application of our solution in new domains such as digital libraries, cross-media annotation and retrieval or personal cross-media information management that goes beyond the hierarchical information management imposed by the desktop metaphor.

Transcript

1. Otto-von-Guericke-Universität Magdeburg, User Interface Colloquium, November 2, 2009 Paper-Digital User Interfaces Beat Signer Department of Computer Science Vrije Universiteit Brussel Applications, Frameworks and Future Challenges http://www.beatsigner.com
2. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Bridging the Paper-Digital Divide
3. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Personal Background ▪ Research on paper-digital and cross-media solutions over the last 8 years ▪ realised more than 20 interactive paper applications ▪ Involved in 2 European projects ▪ Paper++ (2001-2004) - focus on enhanced reading ▪ PaperWorks (2005-2008) - enhanced reading, writing and publishing
4. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Applications ▪ Enhanced Reading ▪ interactive paper maps ▪ interactive festival brochures ▪ educational materials ▪ auction brochures ▪ scientific publications ▪ … ▪ Enhanced Writing ▪ photo album ▪ researcher's notebook ▪ mammography annotation ▪ query by sketching ▪ … ▪ Paper-based Interfaces ▪ PaperPoint ▪ document proof-editing ▪ digital libraries ▪ video analysis tool ▪ educational games ▪ Art Installations ▪ Lost Cosmonaut ▪ Generosa Enterprise ▪ Interactive Tabletops ▪ photo browser ▪ collaborative sketch tool
5. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be EdFest Project
6. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be EdFest Documents
7. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Design Challenges
8. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Rough Guide
9. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be iGesture Framework ▪ iGesture Workbench ▪ create/test gesture sets and algorithms ▪ Different modalities ▪ digital pen, tablet PC, mouse, Wii remote, … ▪ multimodal gestures ▪ Open Source ▪ http://www.igesture.org
10. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Print-n-Link ▪ Handle existing PDF documents ▪ e.g. research papers ▪ Automatic link autho- ring at print time ▪ applicable to other domains
11. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be PaperPoint Presentation Tool ▪ Mobile PowerPoint presentation tool ▪ non-linear presentations ▪ real-time annotations ▪ digital whiteboard ▪ ... ▪ Multi-pen support ▪ brainstorming
12. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be PaperPoint Stickers ▪ Empty stickers linked to specific PowerPoint slides containing ▪ text ▪ images ▪ movies, ... ▪ Effective tool for inter- active paper mockups
13. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be PaperProof: Paper-Digital Proof-Editing
14. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be 14 Mammography Annotations
15. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be The Lost Cosmonaut ▪ Writing as a collaborative act of memory and story- telling ▪ Reading, writing and narrative as an act of making sense ▪ Handwriting as a tool for human computer interaction In collaboration with Axel Vogelsang, Artists in Labs Programme
16. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be iPaper Research Link Metamodel (RSL) Tools Cross-Media Frameworks Applications
17. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be RSL Metamodel and iServer
18. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Links Users Layers Plug-ins Structure Active Content
19. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Active Components Button Slider Slider2D DateSelector Browser MapSelector Rating Capture CaptureAndSpeak Image Sound Movie SQLClient XCMRequest ClientCtrlRequest PaperPointControl PaperPointAnnot ...
20. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Cross-Media Authoring and Publishing ▪ Manual link authoring ▪ Content-driven publishing ▪ e.g. EdFest brochure ▪ Automatic link authoring ▪ e.g. Print-n-Link ▪ Collaborative authoring
21. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Content-driven Cross-Media Publishing XCM content publishing XML data of festival venues and events iServer cross-media link server import publish PDF XML link definition data iPublish publishing framework Structure Style XSL CSS Information-centric approach
22. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Interactive Paper 2.0 consumers / publishers books publisher web pages videos presentation slides
23. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Interactive Paper 2.0 ... ▪ Global and open architecture instead of isolated solutions ▪ Two main challenges to be addressed ▪ unqiue addressing and resolving of interactive paper services - requires a global paper lookup service (PLS) similar to the DNS on the Internet ▪ service oriented architecture to enable cross- application interaction and composition
24. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Interactive Paper 2.0 ... ▪ It is not only about enabling interactions accross different interactive paper solutions ▪ integegration with existing paperless information spaces (e.g. Web 2.0) ▪ personal cross-media information management ▪ Are we missing a basic layer in existing interactive paper architectures?
25. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Future Challenges ▪ Design ▪ visual encoding of interactive areas and functionality ▪ interaction design - online/offline processing, lack of modal dialogues, ... ▪ Interactive Paper 2.0 ▪ global publishing and interaction architecture ▪ collaborative authoring and information sharing ▪ seamless integration with personal cross-media information management environments
26. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Conclusions ▪ Interactive paper framework (iPaper) ▪ information-centric approach supports different technologies, functionalities and applications ▪ separation of authoring and programming ▪ rapid prototyping ▪ Innovative authoring and publishing tools ▪ manual, content-driven, automatic and collaborative ▪ Future Interactive Paper 2.0 challenges!
27. User Interface Colloquium, November 2, 2009 Beat Signer, bsigner@vub.ac.be Further Information ▪ Presentation available on ▪ http://www.slideshare.net/signer/ ▪ Research papers about the presented applications and frameworks are available on ▪ http://vub.academia.edu/BeatSigner ▪ Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Beat Signer, ISBN 978-3-8370-2713-6, May 2008