Cross-Media Technologies and Applications - Future Directions for Personal Information Management

Webinar given at icity Lab Talks - The Digital Value Chain

In this talk, I will first provide an overview of the lab’s research on a general data-driven approach for cross-media information system and architectures based on the resource-selector-link (RSL) hypermedia metamodel. We will then have a look at several cross-media applications for personal information management and next-generation presentation solutions (MindXpres). Finally, I will outline the lab’s most recent research on tangible interaction and dynamic data physicalisation.

Transcript

1. 2 December 2005
   Cross-Media Technologies and Applications
   Future Directions for Personal Information Management
   Prof.Dr.Beat Signer
   beatsigner.com
   Web & Information Systems Engineering Lab
   Department of Computer Science
   Vrije Universiteit Brussel
   WEB & INFORMATION
   SYSTEMS ENGINEERING
   speakerdeck.com/signer/icity
   

   View Slide

2. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   

   View Slide

3. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   01001000100
   0101001100010010100010010
   0100100010010001000100001001000
   100010001100001001000110010101010110
   1100010010100010010010010100010101011001
   101001100001100 101010100
   0010100 100 110101010101
   00100 01010101000
   1001 1111111111
   111 0000000010
   0 11111111
   0101001011
   010101010
   111010101
   001010010110
   Cross-Media Information Spaces and Architectures
   

   View Slide

4. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   Information
   Systems &
   Management
   Information
   Visualisation
   & Navigation
   Information
   Visualisation
   nd Navigation
   Human-Machine &
   Human-Information
   Interaction
   CISA
   RSL [4]
   MindXpres [8]
   OC2 [5]
   associative file system [2]
   data-driven storytelling [11]
   XIMA [13]
   cross-device interaction [9]
   iServer/iPaper [3]
   ViDaX [16]
   data physicalisation [32]
   OpenHPS [18]
   non-linear storytelling [27]
   PaperProof [29]
   iGesture [23]
   PaperSketch [30]
   TangHo [31]
   Context Modelling Toolkit [10]
   Midas [21]
   SpeeG2 [24]
   PaperPoint [26]
   Print-n-Link [17]
   digital libraries [12]
   Mudra [22] mixed reality [28]
   ArtVis [15]
   open cross-media linking [1]
   PimVis [14]
   DocTr [6]
   EdFest [25]
   source code visualisation [19]
   INFEX [7]
   technology-enhanced learning [20]
   Cross-Media Information Spaces and Architectures
   

   View Slide

5. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   RSL Hypermedia Metamodel
   Global Information Systems Group, ETH Zurich
   

   View Slide

6. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   Personal Information Management (PIM)
   ▪ Keeping, organising and
   re-finding information
   ▪ digital and physical
   ▪ Study of human-
   information interaction
   ▪ files, piles, mixtures, …
   ▪ Cross-Media PIM system
   ▪ explicit as well as implicit
   associations between entities
   

   View Slide

7. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   PimVis Setup
   

   View Slide

8. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   Mixed Reality in IoT Environments
   ▪ eSPACE IoT platform
   ▪ end-user authoring
   ▪ mashup tool for digital and
   physical (IoT) components
   ▪ Context Modelling Toolkit
   (CMT)
   ▪ OpenHPS hybrid
   positioning system
   ▪ stream-based processing
   ▪ combination of existing and
   new positioning techniques
   ▪ decentralised sensor fusion
   Elena Zambon
   

   View Slide

9. Beat Signer - Department of Computer Science - bsigner@vub.be
   June 9, 2022
   Interactive Paper and Tangible Interfaces
   ▪ Bridging the paper-digital
   divide
   ▪ seamless transition
   ▪ multimodal interaction
   ▪ new interaction and design
   patterns
   ▪ Advanced visualisation
   techniques in combination
   with tangible user
   interfaces (TUI)
   ▪ e.g. ArtVis
   Bram Moerman
   Global Information Systems Group, ETH Zurich
   

   View Slide

10. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Tangible Hologram (TangHo) Prototype
    

    View Slide

11. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Collaborative TangHo Interaction
    

    View Slide

12. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Dynamic Data Physicalisation
    ▪ Physical objects used for
    input as well as output
    ▪ Dynamic data physicali-
    sation
    ▪ use physical variables such
    as temperature or texture
    ▪ exploration of big data sets
    ▪ dynamic data physicalisation
    framework
    - data physicalisation grammar
    

    View Slide

13. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    What is Wrong with Slideware?
    ▪ Simulation of physical slides
    ▪ limited space due to the slide concept
    ▪ linear navigation from slide to slide
    ▪ difficult to reuse content and embed rich media types
    ▪ MindXpres addresses these issues
    ▪ unlimited canvas with zoomable user interface
    ▪ non-linear navigation and associative linking
    ▪ content-based approach with automatic visualisation
    

    View Slide

14. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    MindXpres Presentation Platform
    ▪ Extensible prototyping
    platform for novel presen-
    tation concepts
    ▪ content-based approach
    - separation of content and presen-
    tation (automatic visualisation)
    ▪ cross-media content reuse
    ▪ non-linear navigation via
    zoomable user interface
    ▪ connectivity and interactivity
    ▪ Rich-media plug-ins
    ▪ e.g. source code or
    interactive data visualisation
    

    View Slide

15. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Interactive Source Code Plug-in
    Paul Mestereaga
    

    View Slide

16. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Personalised Learning Environments
    ▪ Suggestion of learning
    content
    ▪ knowledge graphs
    ▪ user knowledge graph
    ▪ learning paths
    ▪ Diagnostic assessments
    to detect knowledge gaps
    ▪ Automatic content adap-
    tation
    ▪ based on user preferences
    and constraints
    Enhanced Curriculum
    Assessments
    Tagged Learning Content
    Knowledge Graphs
    Adaptive Learning
    Content
    Learning Paths
    Knowledge Representation Framework
    Assessment Module
    Assessment Analyser Knowledge Gap Analyser
    Adaptive Learning Environment
    Suggestion Module
    User Knowledge
    Graph
    Personalised
    Presentation Module
    Learner Constraints
    Groups Module
    

    View Slide

17. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    RSL-based Hypermedia Engine
    RSL-based Hypermedia Engine
    Legacy Applications
    File System Driver
    Hypermedia-enabled Applications
    HT2021.docx
    HT2021.docx
    …
    

    View Slide

18. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    Models, Frameworks and Platforms
    dynamic data physicalisation framework
    RSL hypermedia metamodel eSPACE IoT platform iGesture framework
    OpenHPS hybrid positioning system MindXpres presentation platform
    

    View Slide

19. View Slide

20. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References
    [4] B. Signer and M.C. Norrie, As We May Link: A General
    Metamodel for Hypermedia Systems, Proceedings of
    ER 2007, 26th International Conference on Conceptual
    Modeling, Auckland, New Zealand, November 2007
    ▪ https://beatsigner.com/publications/signer_ER2007.pdf
    [8] R. Roels and B. Signer, MindXpres: An Exten-
    sible Content-driven Cross-Media Presentation
    Platform, Proceedings of WISE 2014, 15th Interna-
    tional Conference on Web Information System
    Engineering, Thessaloniki, Greece, October, 2014
    ▪ https://beatsigner.com/publications/roels_WISE2014.pdf
    

    View Slide

21. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [10] S. Trullemans and B. Signer, A Multi-layered
    Context Modelling Approach for End Users, Expert
    Users and Programmers, Proceedings of SERVE
    2016, International Workshop on Smart Ecosystems
    cReation by Visual dEsign, Bari, Italy, June 2016
    ▪ https://beatsigner.com/publications/trullemans_SERVE2016.pdf
    [14] S. Trullemans, A. Sanctorum and B. Signer,
    PimVis: Exploring and Re-finding Documents in Cross-
    Media Information Spaces, Proceedings of AVI 2016,
    International Working Conference on Advanced Visual
    Interfaces, Bari, Italy, June 2016
    ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf
    

    View Slide

22. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [15] B. Dumas, B. Moerman, S. Trullemans and
    B. Signer, ArtVis: Combining Advanced Visualisation
    and Tangible Interaction for the Exploration, Analysis
    and Browsing of Digital Artwork Collections, Proceed-
    ings of AVI 2014, Como, Italy, May 2014
    ▪ https://beatsigner.com/publications/dumas_AVI2014.pdf
    [18] M. Van de Wynckel and B. Signer, OpenHPS: An
    Open Source Hybrid Positioning System, Technical
    Report WISE Lab, WISE-2020-01, December 2020
    ▪ https://beatsigner.com/publications/vanDeWynckel_CoRR2020.pdf
    

    View Slide

23. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [19] R. Roels, P. Mestereaga and B. Signer, An
    Interactive Source Code Visualisation Plug-in for the
    MindXpres Presentation Platform, Communications in
    Computer and Information Science (CCIS), 583, 2016
    ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf
    [20] Y. Malaise and B. Signer, Personalised Learning
    Environments Based on Knowledge Graphs and the
    Zone of Proximal Development, Proceedings of
    CSEDU 2022, International Conference on Computer
    Supported Education, Online Event, April 2022
    ▪ https://beatsigner.com/publications/malaise_CSEDU2022.pdf
    

    View Slide

24. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [15] B. Dumas, B. Moerman, S. Trullemans and
    B. Signer, ArtVis: Combining Advanced Visualisation
    and Tangible Interaction for the Exploration, Analysis
    and Browsing of Digital Artwork Collections, Proceed-
    ings of AVI 2014, Como, Italy, May 2014
    ▪ https://beatsigner.com/publications/dumas_AVI2014.pdf
    [18] M. Van de Wynckel and B. Signer, OpenHPS: An
    Open Source Hybrid Positioning System, Technical
    Report WISE Lab, WISE-2020-01, December 2020
    ▪ https://beatsigner.com/publications/vanDeWynckel_CoRR2020.pdf
    

    View Slide

25. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [25] B. Signer, M. Grossniklaus and M.C. Norrie,
    Interactive Paper as a Mobile Client for a Multi-
    Channel Web Information System, World Wide Web
    Journal (WWW), Vol. 10, No. 4, Springer,
    December 2007
    ▪ https://beatsigner.com/publications/signer_WWWJ2007.pdf
    [28] L. Hoste and B. Signer, Expressive Control of Indirect
    Augmented Reality During Live Music Performances,
    Proceedings of NIME 2013, 13th International Confe-
    rence on New Interfaces for Musical Expression,
    Daejeon, Korea Republic, May 2013
    ▪ https://beatsigner.com/publications/hoste_NIME2013.pdf
    

    View Slide

26. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    [31] B. Signer and T.J. Curtin, Tangible Holograms:
    Towards Mobile Physical Augmentation of Virtual
    Objects, Technical Report WISE Lab, WISE-2017-01,
    March 2017
    ▪ https://beatsigner.com/publications/signer_arXiv2017.pdf
    [32] B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed
    K.A. Abdullah, Towards a Framework for Dynamic
    Data Physicalisation, International Workshop Toward
    a Design Language for Data Physicalisation, Berlin,
    Germany, October 2018
    ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf
    

    View Slide

27. Beat Signer - Department of Computer Science - bsigner@vub.be
    June 9, 2022
    References ...
    ▪ B. Signer, Towards Cross-Media Information
    Spaces and Architectures, Proceedings of RCIS 2019,
    Brussels, Belgium, May 2019
    ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf
    ▪ MindXpres Project
    ▪ https://mindxpres.com
    ▪ OpenHPS
    ▪ https://openhps.org
    ▪ iGesture
    ▪ http://igesture.org
    

    View Slide