2 December 2005 From Interactive Paper to Dynamic Data Physicalisation Prof. Beat Signer http://www.beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel WEB & INFORMATION SYSTEMS ENGINEERING

01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Cross-Media Information Spaces and Architectures (CISA)

Beat Signer - Department of Computer Science - [email protected] 3 June 1, 2018 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adapatation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation

Beat Signer - Department of Computer Science - [email protected] 4 June 1, 2018 Information Systems & Management Information Visualisation & Navigation Information Visualisation nd Navigation Human-Machine & Human-Information Interaction CISA RSL [4] MindXpres [8] OC2 [5] associative file system [2] data-driven storytelling [11] XIMA [13] cross-device interaction [9] iServer/iPaper [3] ViDaX [16] data physicalisation [32] map-based interaction [18] non-linear storytelling [27] PaperProof [29] iGesture [23] Query by Sketch [30] PaperSketch [31] Context Modelling Toolkit [10] Midas [21] SpeeG2 [24] PaperPoint [26] Print-n-Link [17] digital libraries [12] Mudra [22] mixed reality [28] ArtVis [15] open cross-media linking [1] PimVis [14] DocTr [6] EdFest [25] source code visualisation [19] INFEX [7] collaborative filtering [20] Cross-Media Information Spaces and Architectures

Beat Signer - Department of Computer Science - [email protected] 5 June 1, 2018 Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Audrey Sanctorum User-defined Cross-Device and Cross-Media Interaction Reinout Roels MindXpres: Extensible Content- driven Presentation Platform CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Payam Ebrahimi Dynamic Data Physicalisation

Beat Signer - Department of Computer Science - [email protected] 6 June 1, 2018 WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Steven Vanden Broucke Dynamic Data Physicalisation Jan Maushagen Learning Analytics, Adaptive Persuasive ICT Tools Dr. Sandra Trullemans Cross-Media Personal Information Management Dr. Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats

Beat Signer - Department of Computer Science - [email protected] 7 June 1, 2018 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities ▪ Document Tracking Framework (DocTr)

Beat Signer - Department of Computer Science - [email protected] 8 June 1, 2018 PimVis Setup

Beat Signer - Department of Computer Science - [email protected] 9 June 1, 2018 Context-aware Applications ▪ Implicit context-aware human-computer interaction (HCII) ▪ Beyond simple "if this then that" rules ▪ Context Modelling Toolkit ▪ multi-layered context modelling approach ▪ programmer, expert user and end user ▪ Increase trust via intelligibility

Beat Signer - Department of Computer Science - [email protected] 10 June 1, 2018 Multimodal and Multi-Touch Interaction ▪ Midas/Mudra Framework ▪ declarative definition of multimodal and multi-touch interactions ▪ rule-based language approach (Midas) ▪ rapid protoyping and application development ▪ iGesture Workbench ▪ create and test gesture sets and algorithms ▪ different modalities: digital pen, Wii remote, …

Beat Signer - Department of Computer Science - [email protected] 11 June 1, 2018 SpeeG2: Multimodal Text Input User Speech recognition (Microsoft SAPI 5.4) Skeletal tracking (Microsoft Kinect) 5 4 2 3 SpeeG2 GUI 6 1 Sven De Kock

Beat Signer - Department of Computer Science - [email protected] 12 June 1, 2018 Augmented Reality (Mental Nomad) Toon Duwee

Beat Signer - Department of Computer Science - [email protected] 13 June 1, 2018 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues

Beat Signer - Department of Computer Science - [email protected] 14 June 1, 2018 MindXpres Presentation Platform ▪ Extensible prototyping platform for novel presen- tation concepts ▪ content-based approach - separation of content and pre- sentation (automatic visualisation) ▪ cross-media content reuse ▪ non-linear navigation via zoomable user interface ▪ connectivity and interactivity ▪ Rich-media plug-ins ▪ e.g. source code or interactive data visualisation

Beat Signer - Department of Computer Science - [email protected] 15 June 1, 2018 Interactive Source Code Plug-in Paul Mestereaga

Beat Signer - Department of Computer Science - [email protected] 16 June 1, 2018 Interactive Data Visualisation

Beat Signer - Department of Computer Science - [email protected] 17 June 1, 2018 Interactive Data Visualisation ...

Beat Signer - Department of Computer Science - [email protected] 18 June 1, 2018

Beat Signer - Department of Computer Science - [email protected] 19 June 1, 2018 ArtVis: TUI-based Information Exploration Bram Moerman

Beat Signer - Department of Computer Science - [email protected] 20 June 1, 2018 ArtVis: TUI-based Information Exploration ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information Bram Moerman

Beat Signer - Department of Computer Science - [email protected] 21 June 1, 2018 TRANSFORM TRANSFORM, MIT, Ishii et al.

Beat Signer - Department of Computer Science - [email protected] 22 June 1, 2018 Dynamic Data Physicalisation ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform

Beat Signer - Department of Computer Science - [email protected] 23 June 1, 2018 Collaborative TangHo Interaction

Beat Signer - Department of Computer Science - [email protected] 24 June 1, 2018 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ Replaceable end effector for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system is body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 25 June 1, 2018 TangHo Interactions

Beat Signer - Department of Computer Science - [email protected] 26 June 1, 2018 TangHo Interactions …

Beat Signer - Department of Computer Science - [email protected] 27 June 1, 2018 Dynamic Data Physicalisation ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Data physicalisation design guidelines ▪ what are the just-perceptual differences that a physical variable can convey? ▪ Dynamic data physicalisation framework ▪ data physicalisation grammar ▪ data processing with device-independent output ▪ software drivers for different physicalisations (e.g. TangHo) ▪ New application domains Jacques Bertin, 1967