What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse

Presentation given at ER 2010, 29th International Conference on Conceptual Modeling, Vancouver, Canada.

ABSTRACT: Many of today’s digital document formats are strongly based on a digital emulation of printed media. While such a paper simulation
might be appropriate for the visualisation of certain digital content, it is generally not the most effective solution for digitally managing and storing information. The oversimplistic modelling of digital documents as monolithic blocks of linear content, with a lack of structural semantics, does not pay attention to some of the superior features that digital media offers in comparison to traditional paper documents. For example, existing digital document formats adopt the limitations of paper documents by unnecessarily replicating content via copy and paste operations, instead of digitally embedding and reusing parts of digital documents via structural references. We introduce a conceptual model for structural cross-media content composition and highlight how the proposed solution not only enables the reuse of content via structural relationships, but also supports dynamic and context-dependent document adaptation, structural content annotations as well as the integration of arbitrary nontextual media types. We further discuss solutions for the fluid navigation and cross-media content publishing based on the proposed structural cross-media content model.

Research paper: https://beatsigner.com/publications/what-is-wrong-with-digital-documents-a-conceptual-model-for-structural-cross-media-content-composition-and-reuse.pdf

Transcript

1. ER 2010, November 4, 2010
   What is Wrong with Digital Documents?
   Beat Signer, http://www.beatsigner.com
   Web & Information Systems Engineering Lab (WISE)
   Department of Computer Science
   Vrije Universiteit Brussel
   A Conceptual Model for Structural Cross-Media
   Content Composition and Reuse
   

   View Slide

2. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   PARC User Interface (Mid 70s)
   ▪ Desktop metaphor
   ▪ WIMP interaction
   ▪ "filing cabinets" with
   hierarchical folders
   ▪ Paper paradigm
   ▪ "What You See Is What
   You Get" (WYSIWYG)
   ▪ what about richer
   document formats that
   existed at that time?
   Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
   

   View Slide

3. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   Digital Documents as a Paper Simulator?
   Vannevar Bush
   Ted Nelson
   Most people don't understand the logic of the
   concept: "What You See Is What You Get" is
   based on printing the document out ("get"
   means "get WHEN YOU PRINT IT OUT"). And that
   means a metaphysical shift: a document can
   only consist of what can be printed! [...] No
   overlays [...] – PAPER UNDER GLASS.
   When data of any sort are placed in storage,
   they are filed alphabetically or numerically,
   and information is found (when it is) by
   tracing it down from subclass to subclass. It
   can be in only one place, unless duplicates
   are used [...] The human mind does not work
   that way. It operates by association.
   As We May Think, Atlantic Monthly, July 1945
   Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
   

   View Slide

4. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   Paper-based Work Practices
   Marcel Proust, À la recherche du temps perdu
   Du côté de chez Swann [http://expositions.bnf.fr/brouillons/grand/45.htm]
   Marginal
   annotations
   versioning
   Cut and paste
   #
   

   View Slide

5. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   oN-Line System (NLS)
   ▪ Collaborative System
   by D. Engelbart et al.
   demonstrated in 1968
   ▪ addressing and linking
   parts of files
   ▪ flexible file views
   ▪ overlay file structures
   ▪ cross-file editing
   ▪ ...
   

   View Slide

6. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   010010001001001
   0101001100010010100010010
   0100100010010001000100001001000
   100010001100001001000110010101010110
   1100010010100010010010010100010101011001
   1010011000011001 010101010100
   11000100101000100 110101010101
   010000100100 01010101000
   01001001001 1111111111
   111 0000000010
   0 11111111
   0101001011
   010101010
   111010101010101
   00101001011010
   Fluid Cross-media Information Spaces
   

   View Slide

7. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   Challenges
   ▪ Representation of cross-media relationships
   ▪ navigational relationships
   ▪ structural relationships
   - within document structure
   - external document structure
   ▪ Content reuse across media boundaries
   ▪ cross-media transclusion
   ▪ Distribution of data and metadata across
   multiple devices and media formats
   

   View Slide

8. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   Structure Metamodel (RSL)
   entity
   selector resource
   (1,*)
   (1,*)
   (1,1) (0,*)
   (0,*) (0,*)
   RefersTo
   HasTarget
   HasSource
   partition
   HasProperties
   parameter
   Properties
   (0,*)
   (0,*)
   HasResolver
   contextResolver
   Context
   Resolvers
   Entities
   (0,*)
   (0,*)
   user
   Users
   parameter
   Preferences
   group
   Groups
   (0,*)
   (0,*)
   (0,*)
   (0,*)
   HasMembers
   AccessibleTo
   individual
   Individuals
   (0,*)
   (0,*)
   (1,1)
   partition
   Has
   Preferences
   (0,*)
   layer
   Layers
   OnLayer |HasLayers|
   (1,1)
   (0,*) (0,*)
   (0,*)
   (1,*)
   (0,*)
   link
   Structural
   Links
   partition
   CreatedBy
   Resources
   Selectors
   |HasChild|
   link
   Navigational
   Links
   link
   Links
   structure
   Structures HasElements
   (0,*) (1,1)
   B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems. In Proc. of ER 2007, Nov 2007
   OM notation
   (developed at ETH Zurich)
   

   View Slide

9. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
   Associative File System (RBAF)
   signer
   My Pictures
   Vancouver ER 2010
   Granville
   Island
   Vancouver
   Harbour
   Public
   Market
   Keynote
   J. Thorp
   Olga and
   Lamia
   Dinner
   ER 2010
   multiple classification
   structural link
   navigational link
   My Videos
   Vancouver
   Aquarium
   ER 2010
   Keynote
   cross-media association
   My Presentations
   ER 2010 WISE 2009
   cross-media
   transclusion
   single slide
   transclusion
   (via OOXML
   RSL selector)
   

   View Slide

10. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    Associative File System (RBAF) ...
    

    View Slide

11. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    Structure Metamodel Features
    ▪ Multiple overlay structures can be defined on top
    of a set of entities (views)
    ▪ Structural relationships between parts of
    resources (fragments) via selector concept
    ▪ Structural relationships as first class entities
    ▪ tagging via properties (key/value pairs)
    ▪ navigational links to other resources
    ▪ context-dependent adaptive structures via context
    resolvers
    

    View Slide

12. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    Cross-media Information Browser & Editor
    ▪ How to build a cross-media structure browser & editor?
    

    View Slide

13. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    Conclusions
    ▪ Existing document formats are very much
    based on the simulation of paper affordances
    on static desktop computers
    ▪ Structure metamodel for the representation of
    open and fluid cross-media information spaces
    ▪ associative file system example
    ▪ cross-media information browser & editor
    ▪ Remediation of the "paper simulation" approach
    ▪ WYSIWYG is only one out of many options!
    

    View Slide

14. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    Acknowledgments
    ▪ Colleagues at ETH Zurich for collaborating on
    the RSL link metamodel; in particular
    Prof. Dr. Moira C. Norrie
    ▪ Gregory Cardone for implementing the initial
    prototype of the RSL-based associative file
    system (RBAF)
    

    View Slide

15. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be
    References
    ▪ B. Signer and M.C. Norrie, As We May Link: A General
    Metamodel for Hypermedia Systems, Proc. of ER 2007,
    Auckland, New Zealand, November 2007, 359–374
    ▪ V. Bush, As We May Think, Atlantic Monthly 176(1), 1945
    ▪ T. Nelson, Geeks Bearing Gifts: How the Computer World
    Got This Way, Mindful Press, 2009
    ▪ B. Signer, Fundamental Concepts for Interactive
    Paper and Cross-Media Information Spaces,
    ISBN 978-3-8370-2713-6, May 2008
    

    View Slide