The Remediation of Paper - Creating Affinities between Paper and Cross-Media Information Spaces

Inaugural lecture given at the Vrije Universiteit Brussel, Brussels, Belgium, April 2009

ABSTRACT: While there have been dramatic increases in the use of digital technologies for information storage, processing and delivery over the last few decades, the affordances of paper have ensured its retention as a key information medium. Despite predictions of the paperless office, paper is ever more present in our daily work. However, there is a gap between the paper and digital worlds: information present in paper documents cannot be transferred seamlessly to digital media and digital services are not easily accessible from the paper world.

In this seminar talk I will present an information-centric approach for integrating paper with digital as well as physical media based on a general cross-media information platform (iServer). Some details about the architecture and implementation of the iServer platform as well as the underlying RSL metamodel for cross-media linking will be highlighted. A selection of interactive paper applications that have been developed based on this platform over the past few years will be presented, including the EdFest interactive paper guide for the Edinburgh festivals and the PaperPoint tool for paper-driven PowerPoint presentations. Challenges and solutions for novel forms of interactive paper and cross-media publishing are discussed based on the presented applications.

The second part of my talk will address future directions for research on interactive paper and cross-media information management. This includes specific extensions of the iServer platform and RSL model as well as the application of our solution in new domains such as digital libraries, cross-media annotation and retrieval or personal cross-media information management that goes beyond the hierarchical information management imposed by the desktop metaphor.

Transcript

1. Cross-Media Information Spaces, April 22, 2009
   The Remediation of Paper
   Beat Signer
   Department of Computer Science
   Vrije Universiteit Brussel
   Creating Affinities between Paper and
   Cross-Media Information Spaces
   

   View Slide

2. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   The Paperless Office
   

   View Slide

3. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   The Myth of the Paperless Office
   

   View Slide

4. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   Bridging the Paper-Digital Divide
   

   View Slide

5. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   EdFest Project
   

   View Slide

6. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   Digital Pen and Paper
   ▪ Anoto offers
   ▪ camera technology
   ▪ pattern license
   (virtual paper space
   of 60 million km2)
   ▪ Pen manufacturers
   ▪ Nokia
   ▪ Maxell
   ▪ Adapx
   ▪ Livescribe
   

   View Slide

7. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   EdFest Demo
   

   View Slide

8. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   PaperPoint Presentation Tool
   ▪ Mobile presentation tool
   ▪ Non-linear presentations
   ▪ Real-time annotations
   ▪ Digital whiteboard
   ▪ Multi-pen support
   ▪ brainstorming
   

   View Slide

9. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
   Applications
   ▪ Enhanced Reading
   ▪ interactive paper maps
   ▪ interactive festival brochures
   ▪ educational materials
   ▪ auction brochures
   ▪ scientific publications
   ▪ …
   ▪ Enhanced Writing
   ▪ photo album
   ▪ researcher's notebook
   ▪ mammography
   ▪ query by sketching
   ▪ …
   ▪ Paper-based Interfaces
   ▪ PaperPoint
   ▪ document proof-editing
   ▪ digital libraries
   ▪ video analysis tool
   ▪ educational games
   ▪ Artists' Installations
   ▪ Lost Cosmonaut
   ▪ Generosa Enterprise
   ▪ Interactive Tabletops
   ▪ photo browser
   ▪ collaborative sketch tool
   

   View Slide

10. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Research
    Link Metamodel (RSL) Tools
    Cross-Media Server / Publishing Applications
    

    View Slide

11. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Cross-Media Server (iServer)
    

    View Slide

12. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Active Page Areas and Layers
    image
    video clip
    

    View Slide

13. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    RSL Metamodel and iServer
    

    View Slide

14. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Links
    Users
    Layers
    Plug-ins
    Structure
    Active
    Content
    

    View Slide

15. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    iServer Resource Plug-ins
    Resource
    document page
    XHTML document
    mpeg file, avi file etc.
    mpeg file, avi file etc.
    mp3 file, wav file etc.
    gif file, jpeg file etc.
    RFID space
    ...
    Medium
    paper
    web page
    movie
    movie
    sound
    image
    physical object
    ...
    Selector
    shape
    XPointer
    time span
    shape
    time span
    shape
    RFID tag
    ...
    

    View Slide

16. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Active Components
    ▪ Execute code (services) on the client and server side
    Active Component
    A
    Active Component
    B
    Active Component
    C
    iServer
    AC Manager
    Server
    Request Sender
    Active Component
    Stub B
    AC Names
    ACS Manager
    Client Device
    AC Names
    Client
    HTTP
    

    View Slide

17. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Active Components …
    Button
    Slider
    Slider2D
    DateSelector
    Browser
    MapSelector
    Rating
    Capture
    CaptureAndSpeak
    Image
    Sound
    Movie
    SQLClient
    XCMRequest
    ClientCtrlRequest
    PaperPointControl
    PaperPointAnnot
    ...
    

    View Slide

18. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Cross-Media Authoring and Publishing
    ▪ Manual link authoring
    ▪ Content-driven publishing
    ▪ e.g. EdFest brochure
    ▪ Automatic link authoring
    ▪ e.g. Print-n-Link
    ▪ Collaborative authoring
    

    View Slide

19. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Manual Authoring Tool
    

    View Slide

20. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Cross-Media Publishing Framework
    XCM
    content publishing
    XML data
    of festival
    venues
    and events
    iServer
    cross-media link server
    • paper link data
    • active areas on paper
    • layers
    • digital resources
    • user model
    import
    publish
    PDF
    XML link
    definition
    data
    iPublish
    publishing framework
    Structure Style
    XSL CSS
    Information-centric approach
    

    View Slide

21. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    EdFest Architecture Overview
    iPaper Client
    XCM
    content publishing
    platform
    Metadata DB Appln DB
    iServer/iPaper
    cross-media
    link server
    Link DB
    Context
    Engine
    Context DB
    Client
    Controller
    Active
    Components
    Text-to-Speech
    Engine ICR Software
    handwriting
    recognition
    

    View Slide

22. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Print-n-Link
    ▪ Citations build the basis
    for a scientific web of
    publications
    ▪ Use digital services to
    search / retrieve cited
    publications
    ▪ Automatic authoring
    

    View Slide

23. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Collaborative Authoring
    consumers /
    publishers
    books
    publisher
    web pages
    videos
    presentation slides
    

    View Slide

24. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Paper-Digital Proof-Editing (PaperProof)
    

    View Slide

25. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    iGesture Framework
    ▪ Gesture recognition tool
    ▪ create/test gesture sets
    ▪ create/test algorithms
    ▪ Device independent
    ▪ digital pen, tablet PC,
    mouse, Wii remote, …
    ▪ Open Source
    ▪ initial release in Feb 2007
    ▪ >4500 downloads
    http://www.igesture.org
    

    View Slide

26. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Rough Guide
    

    View Slide

27. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Interactive Table (iTable)
    Pen-Based Google Earth Browser Pen-Based Drawing Tool
    Photo Browser Photo Browser (Slider)
    

    View Slide

28. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Interactive Table (iTable) …
    

    View Slide

29. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Pulse Smartpen
    ▪ Record and replay
    synchronised ink & audio
    ▪ 1 GB memory
    ▪ 16'000 pages
    ▪ 200 hours voice recording
    ▪ 96x18 OLED Display
    ▪ Open application platform
    

    View Slide

30. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Ongoing and Future Work
    ▪ Cross-Media Information Spaces and
    Architectures (CISA) research team
    ▪ cross-media lab space ("meeting room of the future")
    ▪ General and extensible representations for
    cross-media information spaces
    ▪ interlinked structural components rather than
    document-centric representation of information
    ▪ data no longer "owned" by applications
    ▪ replacement of traditional hierarchical file systems
    

    View Slide

31. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    CISA Research
    ▪ New forms of cross-media publishing
    methodologies and tools
    ▪ global publishing infrastructures and architectures
    ▪ personal cross-media information management
    ▪ information sharing and collaborative authoring
    ▪ cross-media annotations
    ▪ Innovative multimodal user interfaces (digital
    and physical) to cross-media information spaces
    ▪ includes further research on interactive paper
    

    View Slide

32. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Conclusions
    ▪ Interactive paper framework (iPaper)
    ▪ information-centric approach supports different
    technologies, functionalities and applications
    ▪ rapid prototyping (reusable active components)
    ▪ separation of authoring and programming
    ▪ Extensible platform for cross-media information
    management (iServer) based on RSL metamodel
    ▪ Cross-media authoring and publishing tools
    ▪ manual, content-driven, automatic and collaborative
    

    View Slide

33. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Acknowledgements
    ▪ GlobIS group @ ETH Zurich
    ▪ in particular Prof. Moira C. Norrie, Nadir Weibel and
    Adriana Ispas
    ▪ Paper++ and PaperWorks members
    ▪ ... and many other collaborators in individual
    projects
    

    View Slide

34. Cross-Media Information Spaces, April 22, 2009 Beat Signer, bsigner@vub.ac.be
    Further Information
    ▪ http://vub.academia.edu/BeatSigner
    ▪ Fundamental Concepts for Interactive Paper
    and Cross-Media Information Spaces, Beat
    Signer, ISBN 978-3-8370-2713-6, May 2008
    

    View Slide