Slide 1

Slide 1 text

2 December 2005 From Interactive Paper to Dynamic Data Physicalisation Prof. Beat Signer http://www.beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel WEB & INFORMATION SYSTEMS ENGINEERING

Slide 2

Slide 2 text

01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Cross-Media Information Spaces and Architectures (CISA)

Slide 3

Slide 3 text

Beat Signer - Department of Computer Science - [email protected] 3 June 1, 2018 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adapatation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation

Slide 4

Slide 4 text

Beat Signer - Department of Computer Science - [email protected] 4 June 1, 2018 Information Systems & Management Information Visualisation & Navigation Information Visualisation nd Navigation Human-Machine & Human-Information Interaction CISA RSL [4] MindXpres [8] OC2 [5] associative file system [2] data-driven storytelling [11] XIMA [13] cross-device interaction [9] iServer/iPaper [3] ViDaX [16] data physicalisation [32] map-based interaction [18] non-linear storytelling [27] PaperProof [29] iGesture [23] Query by Sketch [30] PaperSketch [31] Context Modelling Toolkit [10] Midas [21] SpeeG2 [24] PaperPoint [26] Print-n-Link [17] digital libraries [12] Mudra [22] mixed reality [28] ArtVis [15] open cross-media linking [1] PimVis [14] DocTr [6] EdFest [25] source code visualisation [19] INFEX [7] collaborative filtering [20] Cross-Media Information Spaces and Architectures

Slide 5

Slide 5 text

Beat Signer - Department of Computer Science - [email protected] 5 June 1, 2018 Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Audrey Sanctorum User-defined Cross-Device and Cross-Media Interaction Reinout Roels MindXpres: Extensible Content- driven Presentation Platform CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Payam Ebrahimi Dynamic Data Physicalisation

Slide 6

Slide 6 text

Beat Signer - Department of Computer Science - [email protected] 6 June 1, 2018 WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Steven Vanden Broucke Dynamic Data Physicalisation Jan Maushagen Learning Analytics, Adaptive Persuasive ICT Tools Dr. Sandra Trullemans Cross-Media Personal Information Management Dr. Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats

Slide 7

Slide 7 text

Beat Signer - Department of Computer Science - [email protected] 7 June 1, 2018 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities ▪ Document Tracking Framework (DocTr)

Slide 8

Slide 8 text

Beat Signer - Department of Computer Science - [email protected] 8 June 1, 2018 PimVis Setup

Slide 9

Slide 9 text

Beat Signer - Department of Computer Science - [email protected] 9 June 1, 2018 Context-aware Applications ▪ Implicit context-aware human-computer interaction (HCII) ▪ Beyond simple "if this then that" rules ▪ Context Modelling Toolkit ▪ multi-layered context modelling approach ▪ programmer, expert user and end user ▪ Increase trust via intelligibility

Slide 10

Slide 10 text

Beat Signer - Department of Computer Science - [email protected] 10 June 1, 2018 Multimodal and Multi-Touch Interaction ▪ Midas/Mudra Framework ▪ declarative definition of multimodal and multi-touch interactions ▪ rule-based language approach (Midas) ▪ rapid protoyping and application development ▪ iGesture Workbench ▪ create and test gesture sets and algorithms ▪ different modalities: digital pen, Wii remote, …

Slide 11

Slide 11 text

Beat Signer - Department of Computer Science - [email protected] 11 June 1, 2018 SpeeG2: Multimodal Text Input User Speech recognition (Microsoft SAPI 5.4) Skeletal tracking (Microsoft Kinect) 5 4 2 3 SpeeG2 GUI 6 1 Sven De Kock

Slide 12

Slide 12 text

Beat Signer - Department of Computer Science - [email protected] 12 June 1, 2018 Augmented Reality (Mental Nomad) Toon Duwee

Slide 13

Slide 13 text

Beat Signer - Department of Computer Science - [email protected] 13 June 1, 2018 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues

Slide 14

Slide 14 text

Beat Signer - Department of Computer Science - [email protected] 14 June 1, 2018 MindXpres Presentation Platform ▪ Extensible prototyping platform for novel presen- tation concepts ▪ content-based approach - separation of content and pre- sentation (automatic visualisation) ▪ cross-media content reuse ▪ non-linear navigation via zoomable user interface ▪ connectivity and interactivity ▪ Rich-media plug-ins ▪ e.g. source code or interactive data visualisation

Slide 15

Slide 15 text

Beat Signer - Department of Computer Science - [email protected] 15 June 1, 2018 Interactive Source Code Plug-in Paul Mestereaga

Slide 16

Slide 16 text

Beat Signer - Department of Computer Science - [email protected] 16 June 1, 2018 Interactive Data Visualisation

Slide 17

Slide 17 text

Beat Signer - Department of Computer Science - [email protected] 17 June 1, 2018 Interactive Data Visualisation ...

Slide 18

Slide 18 text

Beat Signer - Department of Computer Science - [email protected] 18 June 1, 2018

Slide 19

Slide 19 text

Beat Signer - Department of Computer Science - [email protected] 19 June 1, 2018 ArtVis: TUI-based Information Exploration Bram Moerman

Slide 20

Slide 20 text

Beat Signer - Department of Computer Science - [email protected] 20 June 1, 2018 ArtVis: TUI-based Information Exploration ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information Bram Moerman

Slide 21

Slide 21 text

Beat Signer - Department of Computer Science - [email protected] 21 June 1, 2018 TRANSFORM TRANSFORM, MIT, Ishii et al.

Slide 22

Slide 22 text

Beat Signer - Department of Computer Science - [email protected] 22 June 1, 2018 Dynamic Data Physicalisation ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform

Slide 23

Slide 23 text

Beat Signer - Department of Computer Science - [email protected] 23 June 1, 2018 Collaborative TangHo Interaction

Slide 24

Slide 24 text

Beat Signer - Department of Computer Science - [email protected] 24 June 1, 2018 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ Replaceable end effector for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system is body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin

Slide 25

Slide 25 text

Beat Signer - Department of Computer Science - [email protected] 25 June 1, 2018 TangHo Interactions

Slide 26

Slide 26 text

Beat Signer - Department of Computer Science - [email protected] 26 June 1, 2018 TangHo Interactions …

Slide 27

Slide 27 text

Beat Signer - Department of Computer Science - [email protected] 27 June 1, 2018 Dynamic Data Physicalisation ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Data physicalisation design guidelines ▪ what are the just-perceptual differences that a physical variable can convey? ▪ Dynamic data physicalisation framework ▪ data physicalisation grammar ▪ data processing with device-independent output ▪ software drivers for different physicalisations (e.g. TangHo) ▪ New application domains Jacques Bertin, 1967