2 December 2005 Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration Prof. Beat Signer beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel WEB & INFORMATION SYSTEMS ENGINEERING

Beat Signer - Department of Computer Science - [email protected] 2 June 2, 2017 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces

Beat Signer - Department of Computer Science - [email protected] 3 June 2, 2017 Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Audrey Sanctorum User-defined Cross-Device and Cross-Media Interaction Reinout Roels MindXpres: Extensible Content- driven Presentation Tool CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Payam Ebrahimi Data Physicalisation

Beat Signer - Department of Computer Science - [email protected] 4 June 2, 2017 Dr. Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Kushal Soni Smart Environments, Implicit Human-Computer Interaction Jan Maushagen Learning Analytics, Adaptive Persuasive ICT Tools Sandra Trullemans Personal Cross-Media Information Management

Beat Signer - Department of Computer Science - [email protected] 5 June 2, 2017 WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Lars Van Holsbeeke Smart Environments, Implicit Human-Computer Interaction

Beat Signer - Department of Computer Science - [email protected] 6 June 2, 2017 Cross-Media Information Spaces and Architectures Information Systems & Management Information Visualisation & Navigation Information Visualisation nd Navigation Human-Machine & Human-Information Interaction CISA RSL [4] MindXpres [7] OC2 [5] associate file system [2] data-driven storytelling [10] XIMA [12] cross-device interaction [8] iServer/iPaper [3] ViDaX [15] data physicalisation [30] map-based interaction [17] non-linear storytelling [25] PaperProof [27] iGesture [21] Query by Sketch [28] PaperSketch [29] Context Modelling Toolkit [9] Midas [19] SpeeG2 [22] PaperPoint [24] Print-n-Link [16] digital libraries [11] Mudra [20] mixed reality [26] ArtVis [14] open cross-media linking [1] PimVis [13] DocTr [6] EdFest [23] source code visualisation [18]

Beat Signer - Department of Computer Science - [email protected] 7 June 2, 2017 ArtVis: TUI-based Information Exploration ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information Bram Moerman

Beat Signer - Department of Computer Science - [email protected] 8 June 2, 2017 ArtVis Setup Bram Moerman

Beat Signer - Department of Computer Science - [email protected] 9 June 2, 2017 ArtVis Visualisation Bram Moerman

Beat Signer - Department of Computer Science - [email protected] 10 June 2, 2017 Tangible Bits (Ishii & Ullmer, 1997) ▪ Beyond GUIs ("painted bits") ▪ Tangible User Interfaces (TUIs) augment the physical space by coupling digital information to everyday objects and environments ▪ physical instantiation of GUI elements in TUI ▪ Key concepts ▪ interactive surfaces ▪ unidirectional coupling of atoms and bits ▪ ambient media (inspired by Life Wire) ▪ Tangible bits mainly act as input controls for digital information and services (output on computer screens) Hiroshi Ishii Brygg Ullmer

Beat Signer - Department of Computer Science - [email protected] 11 June 2, 2017 Radical Atoms (Ishii et al., 2012) ▪ Vision taking a leap beyond Tangible Bits ▪ assuming a hypothetical generation of materials that can dynamically change their form and appearance ▪ vision-driven design research ▪ Radical Atoms is ▪ a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) - changes to digital model are reflected in physical representation and vice versa ▪ a future material that can transform its shape, conform to constraints and inform the users of its affordances ▪ a vision for the future of human-material interaction and Material User Interfaces (MUIs) in which all digital information has a physical manifestation so that we can directly interact with it

Beat Signer - Department of Computer Science - [email protected] 12 June 2, 2017 Radical Atoms … ▪ Transform ▪ interface can transform its shape to modify the model and reflect changes in the computational model ▪ Conform ▪ interface has to conform to some physical laws and user constraints (e.g. for safety) ▪ Inform ▪ user has to be informed about changing interface affordances

Beat Signer - Department of Computer Science - [email protected] 13 June 2, 2017 Interactions with Radical Atoms ▪ Direct touch and gestural interaction ▪ gestures coupled with direct touch create an interaction appropriate for Radical Atoms since users are able to rapidly reform dynamic materials at all scales ▪ Context-aware transformation ▪ context-aware transformations of the tool/interface - e.g. screwdriver adapting to the type of screw it is operating on ▪ Shape-memory clay: Perfect Red ▪ Perfect Red is a fictional material that can be sculpted like and responds according to rules inspired by CAD operations - e.g. if we split a piece in two even halves, then the operations performed on one part are mirrored on the other part

Beat Signer - Department of Computer Science - [email protected] 14 June 2, 2017 TRANSFORM TRANSFORM, MIT ▪ Dynamic shape display that can physically render 3D content ▪ tangible interaction with digital content - geospatial data - 3D modelling - … ▪ TRANSFORM display can also interact with the physical world around it ▪ remote users can be displayed physically ▪ Step towards MIT’s vision of Radical Atoms

Beat Signer - Department of Computer Science - [email protected] 15 June 2, 2017 Video: TRANSFORM

Beat Signer - Department of Computer Science - [email protected] 16 June 2, 2017 Data Physicalisation (Jansen et al., 2015) ▪ Physical objects used for output ▪ often static encoding of digital information ▪ physical visualisations can outperform their virtual counterparts ▪ How can we achieve dynamic data physicalisation with dynamic affordances ▪ e.g. 3D visualisation with additional dimensions represented by physical variables such as temperature, texture, weight or plasticity ▪ TRANSFORM is one possible approach; tangible holograms are another experimental mobile solution - can be used for multidimensional collaborative exploration and analysis of big data sets

Beat Signer - Department of Computer Science - [email protected] 17 June 2, 2017 Mixed Reality "The perceived congruence concerns both visual as well as tactile perception. A perfect hologram will appear embodied but trying to touch it will destroy the illusion." Microsoft HoloLens Jansen, 2014 [34]

Beat Signer - Department of Computer Science - [email protected] 18 June 2, 2017 Tangible Hologram (TangHo) Setup Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 19 June 2, 2017 Tangible Graph Hologram

Beat Signer - Department of Computer Science - [email protected] 20 June 2, 2017 Collaborative TangHo Interaction

Beat Signer - Department of Computer Science - [email protected] 21 June 2, 2017 Robotic Arm Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ replaceable sphere for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system will be body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 22 June 2, 2017 TangHo Prototype Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 23 June 2, 2017 TangHo Interactions Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 24 June 2, 2017 Data Physicalisation Framework ▪ Inspired by research agenda of Jansen et al. [34] ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Understanding the perceptual effectiveness ▪ e.g. what are the just-noticeable differences that a physical variable can convey? ▪ Framework for dynamic data physicalisation ▪ data processing with device-independent output - TextDIO: text for static or dynamic reports - PhysDIO: data points, data variables and physical variables ▪ text and visualisation drivers for representation on specific devices - e.g. tangible hologram prototype

Beat Signer - Department of Computer Science - [email protected] 25 June 2, 2017 Conclusions ▪ Tangible holograms for data physicalisation and simulation of radical atoms ▪ physical augmentation of highly configurable virtual objects rather than programming of physical material (programmable matter) ▪ mobile solution that is no longer bound to a specific place such as a the TRANSFORM table ▪ modular solution that can be configured for specific domains (e.g. different spheres for various physical variables) ▪ Data physicalisation framework for investigating new opportunities in dynamic data physicalisation ▪ new data physicalisation design guidelines (as in InfoVis) ▪ new evaluation techniques ▪ …

Beat Signer - Department of Computer Science - [email protected] 26 June 2, 2017 References ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, ISBN 978-3-8370-2713-6, May 2008 ▪ ArtVis Project ▪ http://www.beatsigner.com/flyers/ArtVis.pdf ▪ Tangible Holograms ▪ http://beatsigner.com/tangibleHolograms.html ▪ TRANSFORM ▪ https://www.youtube.com/watch?v=lCARHatJQJA

Beat Signer - Department of Computer Science - [email protected] 27 June 2, 2017 References ... [1] B. Signer and M.C. Norrie, A Model and Architecture for Open Cross-Media Annotation and Link Services, Information Systems 36(3), Elsevier, May 2011 ▪ http://beatsigner.com/publications/signer_IS2011.pdf [2] B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of the 29th International Conference on Conceptual Modeling (ER 2010), Vancouver, Canada, November 2010 ▪ http://beatsigner.com/publications/signer_ER2010.pdf

Beat Signer - Department of Computer Science - [email protected] 28 June 2, 2017 References ... [3] B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Dissertation ETH No. 16218, Zurich, Switzerland, 2005 ▪ http://beatsigner.com/publications/signer2006.pdf [4] B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of the 26th International Conference on Conceptual Modeling (ER 2007), Auckland, New Zealand, November 2007 ▪ http://beatsigner.com/publications/signer_ER2010.pdf

Beat Signer - Department of Computer Science - [email protected] 29 June 2, 2017 References ... [5] S. Trullemans and B. Signer, Towards a Con- ceptual Framework and Metamodel for Context-Aware Personal Cross-Media Information Management Systems, Proceedings of ER 2014, 33rd International Conference on Conceptual Modelling, Atlanta, USA, October, 2014 ▪ http://beatsigner.com/publications/trullemans_ER2014.pdf [6] S. Trullemans, A. Vercruysse and B. Signer, DocTr: A Unifying Framework for Tracking Physical Docu- ments and Organisational Structures, Proceedings of EICS 2016, Brussels, Belgium, June 2016 ▪ http://beatsigner.com/publications/trullemans_EICS2016.pdf

Beat Signer - Department of Computer Science - [email protected] 30 June 2, 2017 References ... [7] R. Roels and B. Signer, MindXpres: An Exten- sible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th Interna- tional Conference on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ http://beatsigner.com/publications/roels_WISE2014.pdf [8] A. Sanctorum and B. Signer, Towards User-defined Cross-Device Interaction, Proceedings of DUI 2016, 5th Workshop on Distributed User Interfaces, Lugano, Switzerland, June 2016 ▪ http://beatsigner.com/publications/sanctorum_DUI2016.pdf

Beat Signer - Department of Computer Science - [email protected] 31 June 2, 2017 References ... [9] S. Trullemans and B. Signer, A Multi-layered Context Modelling Approach for End Users, Expert Users and Programmers, Proceedings of the Interna- tional Workshop on Smart Ecosystems cReation by Visual dEsign (SERVE 2016), Bari, Italy, June 2016 ▪ http://beatsigner.com/publications/trullemans_SERVE2016.pdf [10] R. Roels, Y. Baeten and B. Signer, An Interactive Data Visualisation Approach for Next Generation Presentation Tools: Towards Rich Presentation-based Data Exploration and Storytelling, Proceedings of CSEDU 2016, Rome, Italy, April, 2016 ▪ http://beatsigner.com/publications/roels_CSEDU2016.pdf

Beat Signer - Department of Computer Science - [email protected] 32 June 2, 2017 References ... [11] M.C. Norrie, B. Signer and N. Weibel, Interactive Paper as a Reading Medium in Digital Libraries, Pro- ceedings of ECDL 2008, 12th European Conference on Research and Advanced Technology for Digital Libraries, Aarhus, Denmark, September 2008 ▪ http://beatsigner.com/publications/norrie_ECDL2008.pdf [12] B. Signer, M.C. Norrie, P. Geissbuehler and D. Heiniger, Telephone Interface for Avalanche War- nings based on Information Server for Adaptable Con- tent Delivery, Proceedings of Pervasive 2002, Zurich, Switzerland, August 2002 ▪ http://beatsigner.com/publications/signer_Pervasive2002.pdf

Beat Signer - Department of Computer Science - [email protected] 33 June 2, 2017 References ... [13] S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of the Inter- national Working Conference on Advanced Visual Interfaces (AVI 2016), Bari, Italy, June 2016 ▪ http://beatsigner.com/publications/trullemans_AVI2016.pdf [14] B. Dumas, B. Moerman, S. Trullemans and B. Signer, ArtVis: Combining Advanced Visualisation and Tan- gible Interaction for the Exploration, Analysis and Browsing of Digital Artwork Collections, Proceedings of AVI 2014, Como, Italy, May 2014 ▪ http://beatsigner.com/publications/dumas_AVI2014.pdf

Beat Signer - Department of Computer Science - [email protected] 34 June 2, 2017 References ... [15] B. Dumas, T. Broché, L. Hoste and B. Signer, ViDaX: An Interactive Semantic Data Visualisation and Exploration Tool, Proceedings of AVI 2012, Internatio- nal Working Conference on Advanced Visual Inter- faces, Capri Island, Italy, May 2012 ▪ http://beatsigner.com/publications/dumas_AVI2012.pdf [16] Moira C. Norrie, Beat Signer and Nadir Weibel, Print-n-Link: Weaving the Paper Web, Proceedings of DocEng 2006, ACM Symposium on Document Engineering, Amsterdam, The Netherlands, October 2006 ▪ http://beatsigner.com/publications/norrie_DocEng2006.pdf

Beat Signer - Department of Computer Science - [email protected] 35 June 2, 2017 References ... [17] M. Grossniklaus, M.C. Norrie, B. Signer and N. Weibel ,Putting Location-Based Services on the Map, Proceedings of W2GIS 2006, Hong Kong, China, December 2006 ▪ http://beatsigner.com/publications/grossniklaus_W2GIS2006.pdf [18] R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ http://beatsigner.com/publications/roels_CCIS2016.pdf

Beat Signer - Department of Computer Science - [email protected] 36 June 2, 2017 References ... [19] C. Scholiers, L. Hoste, B. Signer and W. De Meuter, Midas: A Declarative Multi-Touch Interaction Framework, Proceedings of TEI 2011, 5th Interna- tional Conference on Tangible, Embedded and Em- bodied Interaction, Funchal, Portugal, January 2006 ▪ http://beatsigner.com/publications/scholliers_TEI2011.pdf [20] L. Hoste, B. Dumas and B. Signer, Mudra: A Unified Multimodal Interaction Framework, Proceedings of ICMI 2011, 13th International Conference on Multi- modal Interaction (ICMI 2011), Alicante, Spain, November 2011 ▪ http://beatsigner.com/publications/scholliers_TEI2011.pdf

Beat Signer - Department of Computer Science - [email protected] 37 June 2, 2017 References ... [21] B. Signer, U. Kurmann and M.C. Norrie, iGesture: A General Gesture Recognition Framework, Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil, September 2007 ▪ http://beatsigner.com/publications/signer_ICDAR2007.pdf [22] L. Hoste and B. Signer, SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry, Proceedings of ICMI 2013, 15th Inter- national Conference on Multimodal Interaction, Sydney, Australia, December 2013 ▪ http://beatsigner.com/publications/hoste_ICMI2013.pdf

Beat Signer - Department of Computer Science - [email protected] 38 June 2, 2017 References ... [23] B. Signer, M. Grossniklaus and M.C. Norrie, Interactive Paper as a Mobile Client for a Multi- Channel Web Information System, World Wide Web Journal (WWW), Vol. 10, No. 4, Springer, December 2007 ▪ http://beatsigner.com/publications/signer_WWWJ2007.pdf [24] B. Signer and M.C. Norrie, PaperPoint: A Paper- based Presentation and Interactive Paper Prototyping Tool, Proceedings of TEI 2007, First International Conference on Tangible and Embedded Interaction, Baton Rouge, USA, February 2007 ▪ http://beatsigner.com/publications/signer_TEI2007.pdf

Beat Signer - Department of Computer Science - [email protected] 39 June 2, 2017 References ... [25] A. Vogelsang and B. Signer, The Lost Cosmo- naut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of the Inter- national Conference on Virtual Storytelling 2005, Strasbourg, France, December 2005 ▪ http://beatsigner.com/publications/vogelsang_ICVS2005.pdf [26] L. Hoste and B. Signer, Expressive Control of Indirect Augmented Reality During Live Music Performances, Proceedings of NIME 2013, 13th International Confe- rence on New Interfaces for Musical Expression, Daejeon, Korea Republic, May 2013 ▪ http://beatsigner.com/publications/hoste_NIME2013.pdf

Beat Signer - Department of Computer Science - [email protected] 40 June 2, 2017 References ... [27] N. Weibel, A. Ispas, B. Signer and M.C. Norrie, PaperProof: A Paper-Digital Proof-Editing System, Proceedings of CHI 2008, 26th International Confe- rence on Human Factors in Computing Systems (Interactivity Track), Florence, Italy, April 2008 ▪ http://beatsigner.com/publications/weibel_CHI2008.pdf [28] A. Ispas, H. Schuldt, M.C. Norrie and B. Signer, Towards Query by Sketch, Michael Springmann, Second DELOS Conference on Digital Libraries, Pisa, Italy, December 2007 ▪ http://beatsigner.com/publications/springmann_DELOS2007.pdf

Beat Signer - Department of Computer Science - [email protected] 41 June 2, 2017 References ... [29] N. Weibel, B. Signer, M.C. Norrie, H. Hofstetter, H.-C. Jetter and H. Reiterer, PaperSketch: A Paper- Digital Collaborative Remote Sketching Tool, Proceed- ings of IUI 2011, International Conference on Intelli- gent User Interfaces, Palo Alto, USA, February 2011 ▪ http://beatsigner.com/publications/weibel_IUI2011.pdf [30] B. Signer and T.J. Curtin, Tangible Holograms: Towards Mobile Physical Augmentation of Virtual Objects, Technical Report WISE Lab, WISE-2017-01, March 2017 ▪ http://beatsigner.com/publications/signer_arXiv2017.pdf

Beat Signer - Department of Computer Science - [email protected] 42 June 2, 2017 References ... [31] H. Ishii and B. Ullmer, Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms, Proceedings of CHI 1997, ACM Conference on Human Factors in Computing Systems, Atlanta, USA, March 1997 ▪ http://dx.doi.org/10.1145/258549.258715 [32] H. Ishii, D. Lakatos, L. Bonanni and J.-B. Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions, 19(1), January 2012 ▪ http://dx.doi.org/10.1145/2065327.2065337

Beat Signer - Department of Computer Science - [email protected] 43 June 2, 2017 References ... [33] Y. Jansen et al., Opportunities and Challenges for Data Physicalizations, Proceedings of CHI 2015, ACM Conference on Human Factors in Computing Systems, Seoul, Korea, April 2015 ▪ https://doi.org/10.1145/2702123.2702180 [34] Y. Jansen, Physical and Tangible Information Visualization, 2014, Ph.D. Dissertation, Université Paris-Sud ▪ https://tel.archives-ouvertes.fr/tel-00981521