2 December 2005 Next Generation User Interfaces Data Physicalisation Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 April 22, 2024 Mesopotamian Clay Tokens (8000BCE)

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 April 22, 2024 Quipu (2500BC)

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 April 22, 2024 Data Physicalisation ▪ “A data physicalization (or simply physicalization) is a physical artifact whose geometry or material properties encode data.” [Jansen et al., 2015] ▪ “Data physicalisation as a research area that examines how computer-supported, physical representations of data (i.e., physicalizations), can support cognition, communication, learning, problem solving, and decision making.” [Jansen et al., 2015] ▪ Physical visualisation sometimes used as a synonym for data physicalisations

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 April 22, 2024 Data Physicalisation … ▪ Data sculptures refer to artistic data physicalisations ▪ “Data sculpture as a data-based physical artifact, possessing both artistic and functional qualities, that aims to augment a nearby audience’s understanding of data insights and any socially relevant issues that underlie it.” [Zhao and Vande Moere] ▪ Multisensory data representation ▪ displaying data in a way that engages multiple senses, such as sight, hearing, touch, etc., to enhance comprehension and interpretation of complex datasets ▪ instead of relying only on visual cues, multisensory approaches incorporate other sensory modalities, allowing us to represent more variables and relationships simultaneously

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 April 22, 2024 Domains ▪ The use of physical artefacts in data physicalisation is based on various domains ▪ information visualisation and scientific visualisation ▪ tangible user interfaces (TUIs) ▪ shape-changing interfaces ▪ human-computer interaction and design ▪ electrical engineering ▪ … ▪ Similar to information visualisation solutions, data physicalisation can be used for the following tasks ▪ discover ▪ present ▪ enjoy

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 April 22, 2024 Video: Global Population Growth

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 April 22, 2024 Advantages of Data Physicalisation ▪ Tactile exploration ▪ embodied interaction ▪ embodied cognition - physical manipulation of data can lead to deeper insights and understanding ▪ Active participation and engagement ▪ Accessibility ▪ users with disabilities (e.g.visual impairment) might be supported via alternative modalities ▪ Efficiency ▪ interactions with data physicalisations can be more efficient than with digital visualisations (e.g.3D bar chart example shown later)

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 April 22, 2024 Efficiency of Data Physicalisation ▪ Study shows that physical 3D bar charts outperform on- screen 3D bar charts ▪ touch plays a major role in interacting with the physical bar chart ▪ fingers marks parts of the chart - external cognitive and visual aids

Beat Signer - Department of Computer Science - bsigner@vub.ac.be September 26, 2016 Activity Sculptures ▪ Data sculptures to represent data from running activities ▪ Passive (calm) physicalisation as a tool for self reflection ▪ quantified self ▪ social dynamics (discussion about runs) ▪ …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be September 26, 2016 Enabling Technologies ▪ Digital fabrication tools ▪ 3D printers ▪ laser cutters ▪ CNC machines ▪ … ▪ Actuated tangible interfaces ▪ physical objects changing their form, position or other physical properties ▪ Shape-changing displays ▪ Olfactory output devices ▪ …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 April 22, 2024 Video: EMERGE

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 April 22, 2024 Video: inFORM/TRANSFORM

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 April 22, 2024 DataPhys for Geo-tagged Data ▪ Use of multiple physical variables ▪ shape change (3D pixels) ▪ vibration for each 3D pixel ▪ temperature for each 3D pixel (via Peltier modules) ▪ Physicalisation of Covid-19 dataset for Europe Ghassan Saliby

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 April 22, 2024 Video: Braille Smartwatch

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 April 22, 2024 Video: Dot Pad Tactile Display

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 April 22, 2024 DataLev: Mid-air Data Physicalisation

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 April 22, 2024 Video: Taste the TV (TTTV)

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 April 22, 2024 Video: 3D Printed Food

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 April 22, 2024 Video: Olfactory Interfaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 April 22, 2024 Video: Weart TouchDIVER Haptic Glove

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 April 22, 2024 Video: Dynamic Composite DataPhys

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 April 22, 2024 SensorBricks

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 April 22, 2024 Research Challenges ▪ Most existing data physicalisation applications are static ▪ need for new forms of dynamic data physicalisation ▪ How to enable data-driven physicalisations? ▪ requires the formal and fundamental definition of data physicalisation guidelines ▪ Better understanding of the design space for dynamic data physicalisation ▪ physical encoding of information (e.g. physical variables) ▪ perceptual effectiveness of encodings and the combination of different physical variables ▪ More systematic approach via conceptual data physicalisation framework

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 April 22, 2024 Research Challenges … ▪ Need for data physicalisation software frameworks ▪ supports rapid prototyping of design alternatives ▪ evaluation of the effectiveness of physical encodings ▪ … ▪ Interaction design guidelines for combining physical with synthetic interactions ▪ new forms of multi-user interactions ▪ …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 April 22, 2024 Dynamic Data Physicalisation @WISE ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 April 22, 2024 Dynamic Data Physicalisation Framework ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Understanding the perceptual effectiveness ▪ e.g. what are the just-noticeable differences that a physical variable can convey? ▪ Framework for dynamic data physicalisation ▪ data processing with device-independent output - PhysDIO: data points, data variables and physical variables ▪ drivers for representation on specific devices (e.g.TangHo) ▪ new data physicalisation design guidelines (as in InfoVis) ▪ New application domains Jacques Bertin, 1967

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 April 22, 2024 Dynamic Data Physicalisation Framework

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 April 22, 2024 Physical Variables ▪ Individual physical variables as foundation for more complex multimodal/multisensory data physicalisations ▪ Which physical variables do exist? ▪ temperature ▪ friction ▪ smoothness ▪ … ▪ What do we have to know about individual variables ▪ domain (e.g.min and max value) ▪ human perception/sensitivity for individual variables ▪ interaction/interference with other variables ▪ ergonomics ▪ …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 April 22, 2024 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ replaceable sphere for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system will be body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 April 22, 2024 TangHo Interactions

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 April 22, 2024 Real-Time Cross-Media Data Exploration

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 April 22, 2024 Real-Time Cross-Media Data Exploration ...

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 April 22, 2024 Dynamic Data Physicalisation @ WISE ▪ Conceptual and software dataphys framework ▪ rapid prototyping ▪ drivers for physicalisations (e.g. TangHo) ▪ Dataphys grammar ▪ data-driven physicalisation ▪ exploration of large multi-dimensional datasets ▪ Dynamic dataphys design and interaction guidelines

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 April 22, 2024 Multisensory Data Representation ▪ Design space ▪ projects classified across multiple dimensions

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 April 22, 2024 Radical Atoms (2012) Revisited ▪ Vision taking a leap beyond Tangible Bits ▪ assuming a hypothetical generation of materials that can change their form and appearance dynamically ▪ Radical Atoms is about ▪ a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) ▪ the future material that can transform its shape, conform to constraints and inform the users of its affordances ▪ a vision for the future of human-material interaction, in which all digital information has a physical manifestation so that we can interact directly with it ▪ a new Material User Interface (MUI) Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni and Jean-Baptiste Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions 19(1), January 2012

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 April 22, 2024 References ▪ P. Dragicevic, Y. Jansen and A. Vande Moere, Data Physicalization, Handbook of Human Computer Interaction, May 2021 ▪ https://doi.org/10.1007/978-3-319-27648-9_94-1 ▪ H. Brombacher, R. Van Koningsbruggen, S. Vos and S. Houben, SensorBricks: A Collaborative Tangible Sensor Toolkit to Support the Development of Data Literacy, Proceedings of TEI 2024, February 2024 ▪ https://doi.org/10.1145/3623509.3633378

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 April 22, 2024 References … ▪ S.S. Bae, C. Zheng, M.E. West, Y.-L. Do and S. Huron, Making Data Tangible: A Cross-disciplinary Design Space for Data Physicalization, Proceedings of CHI 2022, April 2022 ▪ https://doi.org/10.1145/3491102.3501939 ▪ F. Taher, J. Hardy, A. Karnik, C. Weichel, Y. Jansen, K. Hornbæk and J. Alexander, Exploring Interactions with Physically Dynamic Bar Charts, Proceedings of CHI 2015, April 2015 ▪ https://doi.org/10.1145/2702123.2702604

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 April 22, 2024 References … ▪ M. Le Goc, C. Perin, S. Follmer, J.-D. Fekete and P. Dragicevic, Dynamic Composite Data Physicali- zation Using Wheeled Micro-Robots, IEEE Transactions on Visualization and Computer Graphics 25(1), January 2019 ▪ https://doi.org/10.1109/TVCG.2018.2865159 ▪ B. Signer, P. Ebrahimi, T.J. Curtin and A.K.A. Abdullah, Towards a Framework for Dynamic Data Physicalisation, International Workshop Toward a Design Language for Data Physicalization, October 2018 ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 April 22, 2024 References … ▪ G. Saliby, Dynamic Data Physicalisation for Exploring Geo-tagged Datasets, Master’s Thesis, WISE lab, VUB, 2020 ▪ https://wise.vub.ac.be/thesis/dynamic-data-physicalisation-exploring-geo- tagged-datasets ▪ T. Hogan and E. Hornecker, Towards a Design Space for Multisensory Data Representation, Interacting with Computers, 29(2), 2017 ▪ https://doi.org/10.1093/iwc/iww015

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 April 22, 2024 References … ▪ Data Physicalisation Wiki ▪ http://dataphys.org ▪ List of Physical Visualisations ▪ http://dataphys.org/list/ ▪ EMERGE: A Physical and Interactive Bar Chart ▪ https://www.youtube.com/watch?v=xA4jOwCHO8I ▪ Dynamic Composite Data Physicalization Using Wheeled Micro Robots ▪ https://www.youtube.com/watch?v=2GmWR617YeI&t=186s ▪ Hans Rosling: Global Population Growth, Box by Box ▪ https://www.youtube.com/watch?v=fTznEIZRkLg

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 April 22, 2024 References … ▪ Braille Smartwatch ▪ https://www.youtube.com/watch?v=96O82Uh5lQo ▪ Dot Pad Real-time Tactile Graphics Display ▪ https://www.youtube.com/watch?v=iSmRM2PUBzA ▪ Taste the TV (TTTV) ▪ https://www.youtube.com/watch?v=LJD-uPTIgx8 ▪ 3D Printed Food ▪ https://www.youtube.com/watch?v=2kCjSq_l-0s ▪ Olfactory Interfaces ▪ https://www.youtube.com/watch?v=2kCjSq_l-0s

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 April 22, 2024 References … ▪ Weart TouchDIVER Haptic Glove ▪ https://www.youtube.com/watch?v=DPiG9EiClwU ▪ inFORM ▪ https://www.youtube.com/watch?v=lvtfD_rJ2hE

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 April 22, 2024 References … ▪ Factfulness: Ten Reasons We're Wrong About The World – And Why Things Are Better Than You Think, Hans Rosling et al., April 2018, ISBN-13: 978-1119547259

2 December 2005 Next Lecture Implicit and Cross-Device Interaction