Data Physicalisation - Lecture 9 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

Transcript

1. 2 December 2005 Next Generation User Interfaces Data Physicalisation Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com
2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 April 28, 2025 Mesopotamian Clay Tokens (8000BCE)
3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 April 28, 2025 Quipu (2500BC)
4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 April 28, 2025 Data Physicalisation ▪ “A data physicalization (or simply physicalization) is a physical artifact whose geometry or material properties encode data.” [Jansen et al., 2015] ▪ “Data physicalisation as a research area that examines how computer-supported, physical representations of data (i.e., physicalizations), can support cognition, communication, learning, problem solving, and decision making.” [Jansen et al., 2015] ▪ Physical visualisation sometimes used as a synonym for data physicalisations
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 April 28, 2025 Data Physicalisation … ▪ Data sculptures refer to artistic data physicalisations ▪ “Data sculpture as a data-based physical artifact, possessing both artistic and functional qualities, that aims to augment a nearby audience’s understanding of data insights and any socially relevant issues that underlie it.” [Zhao and Vande Moere] ▪ Multisensory data representation ▪ displaying data in a way that engages multiple senses, such as sight, hearing, touch, etc., to enhance comprehension and interpretation of complex datasets ▪ instead of relying only on visual cues, multisensory approaches incorporate other sensory modalities, allowing us to represent more variables and relationships simultaneously
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 April 28, 2025 Domains ▪ The use of physical artefacts in data physicalisation is based on various domains ▪ information visualisation and scientific visualisation ▪ tangible user interfaces (TUIs) ▪ shape-changing interfaces ▪ human-computer interaction and design ▪ electrical engineering ▪ … ▪ Similar to information visualisation solutions, data physicalisation can be used for the following tasks ▪ discover ▪ present ▪ enjoy
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 April 28, 2025 Video: Global Population Growth
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 April 28, 2025 Advantages of Data Physicalisation ▪ Tactile exploration ▪ embodied interaction ▪ embodied cognition - physical manipulation of data can lead to deeper insights and understanding ▪ Active participation and engagement ▪ Accessibility ▪ users with disabilities (e.g. visual impairment) might be supported via alternative modalities ▪ Efficiency ▪ interactions with data physicalisations can be more efficient than with digital visualisations (e.g. 3D bar chart example shown later)
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 April 28, 2025 Efficiency of Data Physicalisation ▪ Study shows that physical 3D bar charts outperform on- screen 3D bar charts ▪ touch plays a major role in interacting with the physical bar chart ▪ fingers marks parts of the chart - external cognitive and visual aids
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be September 26, 2016 Activity Sculptures ▪ Data sculptures to represent data from running activities ▪ Passive (calm) physicalisation as a tool for self reflection ▪ quantified self ▪ social dynamics (discussion about runs) ▪ …
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be September 26, 2016 Enabling Technologies ▪ Digital fabrication tools ▪ 3D printers ▪ laser cutters ▪ CNC machines ▪ … ▪ Actuated tangible interfaces ▪ physical objects changing their form, position or other physical properties ▪ Shape-changing displays ▪ Olfactory output devices ▪ …
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 April 28, 2025 Video: EMERGE
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 April 28, 2025 Video: inFORM/TRANSFORM
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 April 28, 2025 DataPhys for Geo-tagged Data ▪ Use of multiple physical variables ▪ shape change (3D pixels) ▪ vibration for each 3D pixel ▪ temperature for each 3D pixel (via Peltier modules) ▪ Physicalisation of Covid-19 dataset for Europe Ghassan Saliby
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 April 28, 2025 Video: Braille Smartwatch
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 April 28, 2025 Video: Dot Pad Tactile Display
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 April 28, 2025 DataLev: Mid-air Data Physicalisation
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 April 28, 2025 Video: Taste the TV (TTTV)
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 April 28, 2025 Video: 3D Printed Food
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 April 28, 2025 Video: Olfactory Interfaces
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 April 28, 2025 Video: Weart TouchDIVER Haptic Glove
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 April 28, 2025 Video: Dynamic Composite DataPhys
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 April 28, 2025 SensorBricks
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 April 28, 2025 Research Challenges ▪ Most existing data physicalisation applications are static ▪ need for new forms of dynamic data physicalisation ▪ How to enable data-driven physicalisations? ▪ requires the formal and fundamental definition of data physicalisation guidelines ▪ Better understanding of the design space for dynamic data physicalisation ▪ physical encoding of information (e.g. physical variables) ▪ perceptual effectiveness of encodings and the combination of different physical variables ▪ More systematic approach via conceptual data physicalisation framework
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 April 28, 2025 Research Challenges … ▪ Need for data physicalisation software frameworks ▪ supports rapid prototyping of design alternatives ▪ evaluation of the effectiveness of physical encodings ▪ … ▪ Interaction design guidelines for combining physical with synthetic interactions ▪ new forms of multi-user interactions ▪ …
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 April 28, 2025 Dynamic Data Physicalisation @WISE ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 April 28, 2025 Dynamic Data Physicalisation Framework ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Understanding the perceptual effectiveness ▪ e.g.what are the just-noticeable differences that a physical variable can convey? ▪ Framework for dynamic data physicalisation ▪ data processing with device-independent output - PhysDIO: data points, data variables and physical variables ▪ drivers for representation on specific devices (e.g. TangHo) ▪ new data physicalisation design guidelines (as in InfoVis) ▪ Modular and extensible hardware platform for dynamic data physicalisation Jacques Bertin, 1967
28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 April 28, 2025 Dynamic Data Physicalisation Framework
29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 April 28, 2025 Physical Variables ▪ Individual physical variables as foundation for more complex multimodal/multisensory data physicalisations ▪ Which physical variables do exist? ▪ temperature ▪ friction ▪ smoothness ▪ … ▪ What do we have to know about individual variables ▪ domain (e.g. min and max value) ▪ human perception/sensitivity for individual variables ▪ interaction/interference with other variables ▪ ergonomics ▪ …
30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 April 28, 2025 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ replaceable sphere for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system will be body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin
31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 April 28, 2025 TangHo Interactions
32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 April 28, 2025 Real-Time Cross-Media Data Exploration
33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 April 28, 2025 Real-Time Cross-Media Data Exploration ...
34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 April 28, 2025 Dynamic Data Physicalisation @ WISE ▪ Conceptual and software dataphys framework ▪ rapid prototyping ▪ drivers for physicalisations (e.g.TangHo) ▪ Dataphys grammar ▪ data-driven physicalisation ▪ exploration of large multi-dimensional datasets ▪ Dynamic dataphys design and interaction guidelines
35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 April 28, 2025 Multisensory Data Representation ▪ Design space ▪ projects classified across multiple dimensions Hogan and Hornecker, 2017
36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 April 28, 2025 Radical Atoms (2012) Revisited ▪ Vision taking a leap beyond Tangible Bits ▪ assuming a hypothetical generation of materials that can change their form and appearance dynamically ▪ Radical Atoms is about ▪ a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) ▪ the future material that can transform its shape, conform to constraints and inform the users of its affordances ▪ a vision for the future of human-material interaction, in which all digital information has a physical manifestation so that we can interact directly with it ▪ a new Material User Interface (MUI) Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni and Jean-Baptiste Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions 19(1), January 2012 Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni and Jean-Baptiste Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions 19(1), January 2012
37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 April 28, 2025 References ▪ P. Dragicevic, Y. Jansen and A. Vande Moere, Data Physicalization, Handbook of Human Computer Interaction, May 2021 ▪ https://doi.org/10.1007/978-3-319-27648-9_94-1 ▪ H. Brombacher, R. Van Koningsbruggen, S. Vos and S. Houben, SensorBricks: A Collaborative Tangible Sensor Toolkit to Support the Development of Data Literacy, Proceedings of TEI 2024, February 2024 ▪ https://doi.org/10.1145/3623509.3633378
38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 April 28, 2025 References … ▪ S.S. Bae, C. Zheng, M.E. West, Y.-L. Do and S. Huron, Making Data Tangible: A Cross-disciplinary Design Space for Data Physicalization, Proceedings of CHI 2022, April 2022 ▪ https://doi.org/10.1145/3491102.3501939 ▪ F. Taher, J. Hardy, A. Karnik, C. Weichel, Y. Jansen, K. Hornbæk and J. Alexander, Exploring Interactions with Physically Dynamic Bar Charts, Proceedings of CHI 2015, April 2015 ▪ https://doi.org/10.1145/2702123.2702604
39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 April 28, 2025 References … ▪ M. Le Goc, C. Perin, S. Follmer, J.-D. Fekete and P. Dragicevic, Dynamic Composite Data Physicali- zation Using Wheeled Micro-Robots, IEEE Transactions on Visualization and Computer Graphics 25(1), January 2019 ▪ https://doi.org/10.1109/TVCG.2018.2865159 ▪ B. Signer, P. Ebrahimi, T.J. Curtin and A.K.A. Abdullah, Towards a Framework for Dynamic Data Physicalisation, International Workshop Toward a Design Language for Data Physicalization, October 2018 ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf
40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 April 28, 2025 References … ▪ X. Zhang, M. Ilić and B. Signer, A Modular and Extensible Hardware Platform Prototype for Dynamic Data Physicalisation, WISE-2025-01, March 2025 ▪ https://doi.org/10.48550/arXiv.2503.06583 ▪ G. Saliby, Dynamic Data Physicalisation for Exploring Geo-tagged Datasets, Master’s Thesis, WISE lab, VUB, 2020 ▪ https://wise.vub.ac.be/thesis/dynamic-data-physicalisation-exploring-geo- tagged-datasets
41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 April 28, 2025 References … ▪ T. Hogan and E. Hornecker, Towards a Design Space for Multisensory Data Representation, Interacting with Computers, 29(2), 2017 ▪ https://doi.org/10.1093/iwc/iww015 ▪ Data Physicalisation Wiki ▪ http://dataphys.org ▪ List of Physical Visualisations ▪ http://dataphys.org/list/ ▪ EMERGE: A Physical and Interactive Bar Chart ▪ https://www.youtube.com/watch?v=xA4jOwCHO8I
42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 April 28, 2025 References … ▪ Dynamic Composite Data Physicalization Using Wheeled Micro Robots ▪ https://www.youtube.com/watch?v=2GmWR617YeI&t=186s ▪ Hans Rosling: Global Population Growth, Box by Box ▪ https://www.youtube.com/watch?v=fTznEIZRkLg ▪ Braille Smartwatch ▪ https://www.youtube.com/watch?v=96O82Uh5lQo ▪ Dot Pad Real-time Tactile Graphics Display ▪ https://www.youtube.com/watch?v=iSmRM2PUBzA ▪ Taste the TV (TTTV) ▪ https://www.youtube.com/watch?v=LJD-uPTIgx8
43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 April 28, 2025 References … ▪ 3D Printed Food ▪ https://www.youtube.com/watch?v=2kCjSq_l-0s ▪ Olfactory Interfaces ▪ https://www.youtube.com/watch?v=2kCjSq_l-0s ▪ Weart TouchDIVER Haptic Glove ▪ https://www.youtube.com/watch?v=DPiG9EiClwU ▪ inFORM ▪ https://www.youtube.com/watch?v=lvtfD_rJ2hE
44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 April 28, 2025 References … ▪ Factfulness: Ten Reasons We're Wrong About The World – And Why Things Are Better Than You Think, Hans Rosling et al., April 2018, ISBN-13: 978-1119547259
45. 2 December 2005 Next Lecture Implicit and Cross-Device Interaction