Bridging the Gap: Managing and Interacting with Information Across Media Boundaries

Presentation given in the Personal Information Systems (PIM) seminar organised by Prof. Ofer Bergman, online, February 2023.

Transcript

1. 2 December 2005
   Bridging the Gap: Managing and Interacting
   with Information Across Media Boundaries
   Prof. Beat Signer
   beatsigner.com
   Web & Information Systems Engineering Lab
   Department of Computer Science
   Vrije Universiteit Brussel
   

   View Slide

2. 01001000100
   0101001100010010100010010
   0100100010010001000100001001000
   100010001100001001000110010101010110
   1100010010100010010010010100010101011001
   101001100001100 101010100
   0010100 100 110101010101
   00100 01010101000
   1001 1111111111
   111 0000000010
   0 11111111
   0101001011
   010101010
   111010101
   001010010110
   Cross-Media Information Spaces
   

   View Slide

3. Fragmentation of information
   across different applications,
   devices and digital & physical
   information environments
   

   View Slide

4. Information
   Systems &
   Management
   Information
   Visualisation
   & Navigation
   Information
   Visualisation
   nd Navigation
   Human-Machine &
   Human-Information
   Interaction
   CISA
   RSL
   MindXpres
   OC2
   associative file system
   data-driven storytelling
   XIMA
   cross-device interaction
   iServer/iPaper
   ViDaX
   data physicalisation
   OpenHPS
   non-linear storytelling
   PaperProof
   iGesture
   PaperSketch
   TangHo
   Context Modelling Toolkit
   Midas
   SpeeG2
   PaperPoint
   Print-n-Link
   digital libraries
   Mudra mixed reality
   ArtVis
   open cross-media linking
   PimVis
   DocTr
   EdFest
   source code visualisation
   INFEX
   technology-enhanced learning
   Cross-Media Technologies
   WEB & INFORMATION
   SYSTEMS ENGINEERING
   

   View Slide

5. Data-centric approach for
   cross-media information
   solutions based on a
   hypermedia metamodel
   

   View Slide

6. Cross-Media Linking
   Flexible Document Representation
   Cross-Media User Interfaces
   

   View Slide

7. RSL Hypermedia Metamodel & Link Server
   B. Signer and M.C. Norrie, As We May Link: A General Meta-
   model for Hypermedia Systems, Proceedings of ER 2007x
   

   View Slide

8. RSL Hypermedia Metamodel
   B. Signer and M.C. Norrie, As We May Link: A General Meta-
   model for Hypermedia Systems, Proceedings of ER 2007x
   

   View Slide

9. Interactive Paper (EdFest)
   B. Signer, M. Grossniklaus and M.C. Norrie, Interactive Paper as a Mobile
   Client for a Multi-Channel Web Information System WWWJ 10(4), 2007 x
   

   View Slide

10. PimVis: Personal Information Management
    S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding
    Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
    

    View Slide

11. PimVis Setup
    S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding
    Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
    

    View Slide

12. PimVis Document View
    S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding
    Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
    

    View Slide

13. PimVis Focus View
    S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding
    Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
    

    View Slide

14. Cross-Media Linking
    Flexible Document Representation
    Cross-Media User Interfaces
    

    View Slide

15. What is Wrong with Digital Documents?
    Most people don't understand the logic of the concept:
    "What You See Is What You Get" is based on printing
    the document out ("get" means "get WHEN YOU
    PRINT IT OUT"). And that means a metaphysical shift:
    a document can only consist of what can be printed!
    [...] No overlays [...] – PAPER UNDER GLASS.
    Ted Nelson, Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
    Existing document formats often based on
    the simulation of paper affordances on
    desktop computers
    

    View Slide

16. What is Wrong with Slideware?
    ▪ Simulation of physical slides
    ▪ limited space due to slide concept
    ▪ linear navigation
    ▪ monolithic documents
    ▪ difficult to reuse content and embed rich media types
    

    View Slide

17. MindXpres Presentation Platform
    ▪ Flexible representation
    of presentations
    ▪ use of structural RSL links
    ▪ separation of content and
    structure
    ▪ Extensible platform
    ▪ content-based approach
    ▪ cross-media content reuse
    via transclusion
    ▪ non-linear navigation and
    zoomable user interface
    ▪ associative linking
    ▪ rich media types
    R. Roels and B. Signer, MindXpres: An Extensible Content-driven
    Cross-Media Presentation Platform, Proceedings of WISE 2014
    

    View Slide

18. MindXpres Document Representation
    R. Roels and B. Signer, A Conceptual Framework and Content Model for
    Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019
    

    View Slide

19. MindXpres (Video)
    

    View Slide

20. Different context-dependent
    adaptive overlay structures on top
    of (parts of) content
    Clear separation between content
    and structure enables cross-media
    content reuse (transclusion)
    

    View Slide

21. Cross-Media Linking
    Flexible Document Representation
    Cross-Media User Interfaces
    

    View Slide

22. PimVis: Paper-Digital User Interface
    S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding
    Documents in Cross-Media Information Spaces, Proceedings of AVI 2016
    

    View Slide

23. ArtVis Tangible User Interface
    B. Dumas, B. Moerman, S. Trullemans and B. Signer, ArtVis: Combining Advanced Visualisation and Tangible
    Interaction for the Exploration, Analysis and Browsing of Digital Artwork Collections, Proceedings of AVI 2014
    

    View Slide

24. Dynamic Data Physicalisation
    B. Signer, P. Ebrahimi, T.J. Curtin and A.K.A. Abdullah, Towards a Framework for Dynamic Data
    Physicalisation, International Workshop Toward a Design Language for Data Physicalization, 2018
    

    View Slide

25. eSPACE Framework for Hybrid UIs
    A. Sanctorum and B. Signer, A Unifying Reference Framework and Model
    for Adaptive Distributed Hybrid User Interfaces, Proceedings of RCIS 2019
    x
    

    View Slide

26. Unified Cross-Media Information Spaces
    

    View Slide

27. Common underlying
    representation of data, cross-
    media links and structures via
    RSL-based hypermedia engine
    supporting legacy applications
    

    View Slide

28. RSL-based Hypermedia Engine
    RSL-based Hypermedia Engine
    Legacy Applications
    File System Driver
    Hypermedia-enabled Applications
    HT2021.docx
    HT2021.docx
    …
    B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hyper-
    media and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021
    

    View Slide

29. RSL-based Hypermedia Engine …
    signer
    My Pictures
    Vancouver ER 2010
    Granville
    Island
    Vancouver
    Harbour
    Public
    Market
    Keynote
    J. Thorp
    Olga and
    Lamia
    Dinner
    ER 2010
    multiple classification
    structural link
    navigational link
    My Videos
    Vancouver
    Aquarium
    ER 2010
    Keynote
    cross-media association
    My Presentations
    ER 2010 WISE 2009
    cross-media
    transclusion
    single slide
    transclusion
    (via OOXML
    RSL selector)
    

    View Slide

30. Next Generation File Manager
    B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hyper-
    media and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021
    

    View Slide

31. System-wide hypermedia engine
    at operating system level, enabling
    cross-media information manage-
    ment across application boundaries
    

    View Slide

32. References
    ▪ B. Signer, Fundamental Concepts for
    Interactive Paper and Cross-Media Information
    Spaces, Second Edition, ISBN 978-3-8370-2713-6,
    August 2017
    ▪ B. Signer and M.C. Norrie, As We May Link: A General
    Metamodel for Hypermedia Systems, Proceedings of
    ER 2007, 26th International Conference on Conceptual
    Modeling, Auckland, New Zealand, November 2007
    ▪ https://beatsigner.com/publications/signer_ER2007.pdf
    

    View Slide

33. References ...
    ▪ B. Signer, M. Grossniklaus and M.C. Norrie,
    Interactive Paper as a Mobile Client for a Multi-Channel
    Web Information System, World Wide Web Journal
    (WWW), Vol. 10, No. 4, Springer, December 2007
    ▪ https://beatsigner.com/publications/signer_WWWJ2007.pdf
    ▪ S. Trullemans, A. Sanctorum and B. Signer,
    PimVis: Exploring and Re-finding Documents in Cross-
    Media Information Spaces, Proceedings of AVI 2016,
    International Working Conference on Advanced Visual
    Interfaces, Bari, Italy, June 2016
    ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf
    

    View Slide

34. References ...
    ▪ R. Roels and B. Signer, MindXpres: An
    Extensible Content-driven Cross-Media Presentation
    Platform, Proceedings of WISE 2014, 15th International
    Conference on Web Information System Engineering,
    Thessaloniki, Greece, October 2014
    ▪ https://beatsigner.com/publications/roels_WISE2014.pdf
    ▪ R. Roels and B. Signer, A Conceptual Framework and
    Content Model for Next Generation Presentation
    Solutions, Proceedings of the ACM on Human-Computer
    Interaction (PACMHCI), 2019
    ▪ https://beatsigner.com/publications/roels_PACMHCI2019.pdf
    

    View Slide

35. References ...
    ▪ B. Signer, What is Wrong with Digital
    Documents? A Conceptual Model for Structural Cross-
    Media Content Composition and Reuse, Proceedings of
    ER 2010, 29th International Conference on Conceptual
    Modeling, Vancouver, Canada, November 2010
    ▪ https://beatsigner.com/publications/signer_ER2010.pdf
    ▪ B. Dumas, B. Moerman, S. Trullemans and B. Signer,
    ArtVis: Combining Advanced Visualisation and Tangible
    Interaction for the Exploration, Analysis and Browsing of
    Digital Artwork Collections, Proceedings of AVI 2014,
    Como, Italy, May 2014
    ▪ https://beatsigner.com/publications/dumas_AVI2014.pdf
    

    View Slide

36. References ...
    ▪ B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed
    K.A. Abdullah, Towards a Framework for Dynamic Data
    Physicalisation, International Workshop Toward a Design
    Language for Data Physicalisation, Berlin, Germany,
    October 2018
    ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf
    ▪ A. Sanctorum and B. Signer, A Unifying Reference
    Framework and Model for Adaptive Distributed Hybrid
    User Interfaces, Proceedings of RCIS 2019, 13th
    International Conference on Research Challenges in
    Information Science, Brussels, Belgium, May 2019
    ▪ https://beatsigner.com/publications/sanctorum_RCIS2019.pdf
    

    View Slide

37. References ...
    ▪ B. Signer, R. Roels, R. van Barlingen and
    B. Willems, Back to the Future: Bringing Original
    Hypermedia and Cross-Media Concepts to Modern
    Desktop Environments, Proceedings of Hypertext 2021,
    Virtual Event, August 2021
    ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf
    

    View Slide

38. Prof. Dr. Beat Signer
    Cross-MediaTechnology, Interac-
    tive Paper, Data Physicalisation
    Dr. Audrey Sanctorum
    User-defined XDI and IoT Inter-
    action, Human-AI Interaction
    CISA
    Human-Machine &
    Human-Information
    Interaction
    Information
    Systems &
    Management
    Information
    Visualisation
    & Navigation
    WEB & INFORMATION
    SYSTEMS ENGINEERING
    CROSS-MEDIA INFORMATION SPACES
    AND ARCHITECTURES (CISA)
    Maxim Van de Wynckel
    Hybrid Positioning, Implicit
    Human-Computer Interaction
    Yoshi Malaise
    Technology-enhanced Learning,
    Content-driven Presentations
    

    View Slide

39. WEB & INFORMATION
    SYSTEMS ENGINEERING
    CISA
    Human-Machine &
    Human-Information
    Interaction
    Information
    Systems &
    Management
    Information
    Visualisation
    & Navigation
    CROSS-MEDIA INFORMATION SPACES
    AND ARCHITECTURES (CISA)
    Ekene Attoh
    IoT Middleware, Context-aware
    Computing, Implicit HCI
    Isaac Valadez
    Knowledge Physicalisation and
    Augmentation, Tangible UIs
    Xuyao Zhang
    Extensible Platform for Dynamic
    Data Physicalisation
    Ingela Rossing
    Dynamic Data Physicalisation
    Framework and Guidelines
    

    View Slide

40. WEB & INFORMATION
    SYSTEMS ENGINEERING
    CISA
    Human-Machine &
    Human-Information
    Interaction
    Information
    Systems &
    Management
    Information
    Visualisation
    & Navigation
    CROSS-MEDIA INFORMATION SPACES
    AND ARCHITECTURES (CISA)
    Arun Sojan
    Physicalisation for Digital Twins
    Piet Van Der Paelt
    Julia-based Framework for
    Simulation and Optimisation
    Evan Cole
    Technology-enhanced Learning,
    Study Lenses
    Migdeily Cantera
    End-User Development, Mixed
    Reality IoT UIs, Intelligibility
    

    View Slide