What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse

Presentation given at ER 2010, 29th International Conference on Conceptual Modeling, Vancouver, Canada.

ABSTRACT: Many of today’s digital document formats are strongly based on a digital emulation of printed media. While such a paper simulation
might be appropriate for the visualisation of certain digital content, it is generally not the most effective solution for digitally managing and storing information. The oversimplistic modelling of digital documents as monolithic blocks of linear content, with a lack of structural semantics, does not pay attention to some of the superior features that digital media offers in comparison to traditional paper documents. For example, existing digital document formats adopt the limitations of paper documents by unnecessarily replicating content via copy and paste operations, instead of digitally embedding and reusing parts of digital documents via structural references. We introduce a conceptual model for structural cross-media content composition and highlight how the proposed solution not only enables the reuse of content via structural relationships, but also supports dynamic and context-dependent document adaptation, structural content annotations as well as the integration of arbitrary nontextual media types. We further discuss solutions for the fluid navigation and cross-media content publishing based on the proposed structural cross-media content model.

Research paper: https://beatsigner.com/publications/what-is-wrong-with-digital-documents-a-conceptual-model-for-structural-cross-media-content-composition-and-reuse.pdf

Transcript

1. ER 2010, November 4, 2010 What is Wrong with Digital Documents? Beat Signer, http://www.beatsigner.com Web & Information Systems Engineering Lab (WISE) Department of Computer Science Vrije Universiteit Brussel A Conceptual Model for Structural Cross-Media Content Composition and Reuse
2. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be PARC User Interface (Mid 70s) ▪ Desktop metaphor ▪ WIMP interaction ▪ "filing cabinets" with hierarchical folders ▪ Paper paradigm ▪ "What You See Is What You Get" (WYSIWYG) ▪ what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
3. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Digital Documents as a Paper Simulator? Vannevar Bush Ted Nelson Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association. As We May Think, Atlantic Monthly, July 1945 Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
4. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Paper-based Work Practices Marcel Proust, À la recherche du temps perdu Du côté de chez Swann [http://expositions.bnf.fr/brouillons/grand/45.htm] Marginal annotations versioning Cut and paste #
5. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be oN-Line System (NLS) ▪ Collaborative System by D. Engelbart et al. demonstrated in 1968 ▪ addressing and linking parts of files ▪ flexible file views ▪ overlay file structures ▪ cross-file editing ▪ ...
6. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be 010010001001001 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 1010011000011001 010101010100 11000100101000100 110101010101 010000100100 01010101000 01001001001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101010101 00101001011010 Fluid Cross-media Information Spaces
7. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Challenges ▪ Representation of cross-media relationships ▪ navigational relationships ▪ structural relationships - within document structure - external document structure ▪ Content reuse across media boundaries ▪ cross-media transclusion ▪ Distribution of data and metadata across multiple devices and media formats
8. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Structure Metamodel (RSL) entity selector resource (1,*) (1,*) (1,1) (0,*) (0,*) (0,*) RefersTo HasTarget HasSource partition HasProperties parameter Properties (0,*) (0,*) HasResolver contextResolver Context Resolvers Entities (0,*) (0,*) user Users parameter Preferences group Groups (0,*) (0,*) (0,*) (0,*) HasMembers AccessibleTo individual Individuals (0,*) (0,*) (1,1) partition Has Preferences (0,*) layer Layers OnLayer |HasLayers| (1,1) (0,*) (0,*) (0,*) (1,*) (0,*) link Structural Links partition CreatedBy Resources Selectors |HasChild| link Navigational Links link Links structure Structures HasElements (0,*) (1,1) B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems. In Proc. of ER 2007, Nov 2007 OM notation (developed at ETH Zurich)
9. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Associative File System (RBAF) signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector)
10. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Associative File System (RBAF) ...
11. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Structure Metamodel Features ▪ Multiple overlay structures can be defined on top of a set of entities (views) ▪ Structural relationships between parts of resources (fragments) via selector concept ▪ Structural relationships as first class entities ▪ tagging via properties (key/value pairs) ▪ navigational links to other resources ▪ context-dependent adaptive structures via context resolvers
12. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Cross-media Information Browser & Editor ▪ How to build a cross-media structure browser & editor?
13. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Conclusions ▪ Existing document formats are very much based on the simulation of paper affordances on static desktop computers ▪ Structure metamodel for the representation of open and fluid cross-media information spaces ▪ associative file system example ▪ cross-media information browser & editor ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options!
14. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be Acknowledgments ▪ Colleagues at ETH Zurich for collaborating on the RSL link metamodel; in particular Prof. Dr. Moira C. Norrie ▪ Gregory Cardone for implementing the initial prototype of the RSL-based associative file system (RBAF)
15. ER 2010, November 4, 2010 Beat Signer - Vrije Universiteit Brussel - bsigner@vub.ac.be References ▪ B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proc. of ER 2007, Auckland, New Zealand, November 2007, 359–374 ▪ V. Bush, As We May Think, Atlantic Monthly 176(1), 1945 ▪ T. Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press, 2009 ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, ISBN 978-3-8370-2713-6, May 2008