Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From Interactive Paper to Dynamic Data Physical...

From Interactive Paper to Dynamic Data Physicalisation

Invited talk given at the GlobIS Alumni Workshop, ETH Zurich, Switzerland, June 1, 2018

Beat Signer

June 01, 2018
Tweet

More Decks by Beat Signer

Other Decks in Research

Transcript

  1. 2 December 2005 From Interactive Paper to Dynamic Data Physicalisation

    Prof. Beat Signer http://www.beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel WEB & INFORMATION SYSTEMS ENGINEERING
  2. 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101

    00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Cross-Media Information Spaces and Architectures (CISA)
  3. Beat Signer - Department of Computer Science - [email protected] 3

    June 1, 2018 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adapatation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation
  4. Beat Signer - Department of Computer Science - [email protected] 4

    June 1, 2018 Information Systems & Management Information Visualisation & Navigation Information Visualisation nd Navigation Human-Machine & Human-Information Interaction CISA RSL [4] MindXpres [8] OC2 [5] associative file system [2] data-driven storytelling [11] XIMA [13] cross-device interaction [9] iServer/iPaper [3] ViDaX [16] data physicalisation [32] map-based interaction [18] non-linear storytelling [27] PaperProof [29] iGesture [23] Query by Sketch [30] PaperSketch [31] Context Modelling Toolkit [10] Midas [21] SpeeG2 [24] PaperPoint [26] Print-n-Link [17] digital libraries [12] Mudra [22] mixed reality [28] ArtVis [15] open cross-media linking [1] PimVis [14] DocTr [6] EdFest [25] source code visualisation [19] INFEX [7] collaborative filtering [20] Cross-Media Information Spaces and Architectures
  5. Beat Signer - Department of Computer Science - [email protected] 5

    June 1, 2018 Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Audrey Sanctorum User-defined Cross-Device and Cross-Media Interaction Reinout Roels MindXpres: Extensible Content- driven Presentation Platform CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Payam Ebrahimi Dynamic Data Physicalisation
  6. Beat Signer - Department of Computer Science - [email protected] 6

    June 1, 2018 WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Steven Vanden Broucke Dynamic Data Physicalisation Jan Maushagen Learning Analytics, Adaptive Persuasive ICT Tools Dr. Sandra Trullemans Cross-Media Personal Information Management Dr. Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats
  7. Beat Signer - Department of Computer Science - [email protected] 7

    June 1, 2018 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities ▪ Document Tracking Framework (DocTr)
  8. Beat Signer - Department of Computer Science - [email protected] 9

    June 1, 2018 Context-aware Applications ▪ Implicit context-aware human-computer interaction (HCII) ▪ Beyond simple "if this then that" rules ▪ Context Modelling Toolkit ▪ multi-layered context modelling approach ▪ programmer, expert user and end user ▪ Increase trust via intelligibility
  9. Beat Signer - Department of Computer Science - [email protected] 10

    June 1, 2018 Multimodal and Multi-Touch Interaction ▪ Midas/Mudra Framework ▪ declarative definition of multimodal and multi-touch interactions ▪ rule-based language approach (Midas) ▪ rapid protoyping and application development ▪ iGesture Workbench ▪ create and test gesture sets and algorithms ▪ different modalities: digital pen, Wii remote, …
  10. Beat Signer - Department of Computer Science - [email protected] 11

    June 1, 2018 SpeeG2: Multimodal Text Input User Speech recognition (Microsoft SAPI 5.4) Skeletal tracking (Microsoft Kinect) 5 4 2 3 SpeeG2 GUI 6 1 Sven De Kock
  11. Beat Signer - Department of Computer Science - [email protected] 12

    June 1, 2018 Augmented Reality (Mental Nomad) Toon Duwee
  12. Beat Signer - Department of Computer Science - [email protected] 13

    June 1, 2018 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues
  13. Beat Signer - Department of Computer Science - [email protected] 14

    June 1, 2018 MindXpres Presentation Platform ▪ Extensible prototyping platform for novel presen- tation concepts ▪ content-based approach - separation of content and pre- sentation (automatic visualisation) ▪ cross-media content reuse ▪ non-linear navigation via zoomable user interface ▪ connectivity and interactivity ▪ Rich-media plug-ins ▪ e.g. source code or interactive data visualisation
  14. Beat Signer - Department of Computer Science - [email protected] 15

    June 1, 2018 Interactive Source Code Plug-in Paul Mestereaga
  15. Beat Signer - Department of Computer Science - [email protected] 16

    June 1, 2018 Interactive Data Visualisation
  16. Beat Signer - Department of Computer Science - [email protected] 17

    June 1, 2018 Interactive Data Visualisation ...
  17. Beat Signer - Department of Computer Science - [email protected] 19

    June 1, 2018 ArtVis: TUI-based Information Exploration Bram Moerman
  18. Beat Signer - Department of Computer Science - [email protected] 20

    June 1, 2018 ArtVis: TUI-based Information Exploration ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information Bram Moerman
  19. Beat Signer - Department of Computer Science - [email protected] 21

    June 1, 2018 TRANSFORM TRANSFORM, MIT, Ishii et al.
  20. Beat Signer - Department of Computer Science - [email protected] 22

    June 1, 2018 Dynamic Data Physicalisation ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform
  21. Beat Signer - Department of Computer Science - [email protected] 23

    June 1, 2018 Collaborative TangHo Interaction
  22. Beat Signer - Department of Computer Science - [email protected] 24

    June 1, 2018 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ Replaceable end effector for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system is body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin
  23. Beat Signer - Department of Computer Science - [email protected] 27

    June 1, 2018 Dynamic Data Physicalisation ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Data physicalisation design guidelines ▪ what are the just-perceptual differences that a physical variable can convey? ▪ Dynamic data physicalisation framework ▪ data physicalisation grammar ▪ data processing with device-independent output ▪ software drivers for different physicalisations (e.g. TangHo) ▪ New application domains Jacques Bertin, 1967