Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tangible, Embedded and Embodied Interaction - Lecture 9 - Next Generation User Interfaces (4018166FNR)

Tangible, Embedded and Embodied Interaction - Lecture 9 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

1135dc242dcff3b90ae46fc586ff4da8?s=128

Beat Signer
PRO

May 02, 2022
Tweet

More Decks by Beat Signer

Other Decks in Education

Transcript

  1. 2 December 2005 Next Generation User Interfaces Tangible, Embedded and

    Embodied Interaction Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com
  2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2

    May 2, 2022 Marble Answering Machine Revisited ▪ Marble answering machine ▪ incoming messages repre- sented by physical marbles ▪ Differences ▪ familiar physical objects show the number of messages ▪ aesthetically pleasing and enjoyable to use ▪ one step actions to perform a task ▪ simple but elegant design with less functionality ▪ anyone can listen to any of the messages ▪ Might not be robust enough to be used in public space ▪ important to take into account where a product is going to be used Marble answering machine, Durell Bishop, 1992
  3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3

    May 2, 2022 Graspable User Interfaces (1995) ▪ Direct control of electronic or virtual objects through physical handles (bricks) ▪ A brick is a new user inter- face that is tightly coupled to a virtual object ▪ Bricks are used on top of a large horizontal display surface known as the ActiveDesk ▪ GraspDraw is one particular application GraspDraw on ActiveDesk George W. Fitzmaurice, Hiroshi Ishii and William Buxton, Bricks: Laying the Foundations for Graspable User Interfaces, Proceedings of CHI 1995, ACM Conference on Human Factors in Computing Systems, Denver, USA, May 1995
  4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4

    May 2, 2022 Graspable User Interfaces (1995) … A graspable object Floor planner: multiple objects with handles and spline with multiple handles Two bricks attached to a single digital object (one acting as an anchor) Moving and rotating both bricks at the same time
  5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5

    May 2, 2022 Graspable User Interfaces (1995) … ▪ Advantages of Graspable UI design ▪ encourages two-handed interactions ▪ shifts to more specialised context-sensitive input devices ▪ allows for more parallel input specification ▪ makes use of our skills for physical object manipulations - affordances of artefacts define how we use the interface ▪ externalises traditionally internal computer representations ▪ facilitates interactions by making interface elements more "direct and manipulable" by using physical artefacts ▪ takes advantage of our spatial reasoning skills ▪ affords multi-person collaborative use ▪ … ▪ Foundations of tangible interaction
  6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6

    May 2, 2022 Affordances ▪ Term affordance introduced in 1977 by psychologist James J.Gibson in the 'The Theory of Affordances' ▪ originally defined as all possible actions with an object in an environment independent of an individual's ability to recognise these actions ▪ Don Norman refined the term affordances in the context of human-machine interaction ▪ only those possible actions with an object that can be recognised by an individual ▪ an affordance of an object tells us something (gives us a clue) about how to use the object ▪ good interaction design will take affordances and the related discoverability into account
  7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7

    May 2, 2022 Video: The Norman Door
  8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8

    May 2, 2022 Definition of Tangible Interaction ▪ Tangible interaction is an umbrella term for ▪ graspable user interfaces ▪ tangible user interfaces ▪ embodied interaction ▪ Tangible Interaction encompasses user interfaces and interaction approaches that emphasise ▪ tangibility and materiality of the interface ▪ physical embodiment of data ▪ whole-body interaction ▪ embedding of the interface and user interaction in real spaces and contexts ▪ physical objects as representation and control for digital information
  9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9

    May 2, 2022 Tangible Bits (1997) ▪ Beyond GUIs ("painted bits") ▪ Tangible User Interfaces (TUIs) augment the physical space by coupling digital information to everyday objects and environments ▪ physical instantiation of GUI elements in TUI ▪ Key concepts ▪ interactive surfaces ▪ coupling of bits and atoms ▪ ambient media (inspired by Life Wire) Hiroshi Ishii and Brygg Ullmer, Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms, Proceedings of CHI 1997, ACM Conference on Human Factors in Computing Systems, Atlanta, USA, March 1997f Hiroshi Ishii Brygg Ullmer
  10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10

    May 2, 2022 Life Wire (1995) ▪ Life Wire (dangling string) designed by Natalie Jeremijenko while she was an artist in residence at Xerox PARC ▪ plastic cord attached to electric motor mounted on the ceiling ▪ motor connected to the local Ethernet and each passing network packet triggers a switch of the motor ▪ bits flowing through the network become tangible through motion, sound and touch Natalie Jeremijenko Life Wire, 1995
  11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11

    May 2, 2022 metaDESK ▪ Back-projected graphical surface with various tools ▪ phicons ▪ activeLENS and passiveLENS ▪ instruments ▪ Tangible Geospace ▪ application prototype Tangible Geospace on metaDESK Physical instantiation of GUI elements in TUI
  12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12

    May 2, 2022 Video: metaDESK
  13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13

    May 2, 2022 ambientROOM ▪ Complements the cogni tively-foreground inter- actions of the metaDESK with ambient media ▪ ambient light or shadows ▪ sound ▪ airflow ▪ Communicate information at the periphery of human perception ▪ investigate how the parallel background processing can be used to convey information via ambient media ▪ enable seamless transition between primary foreground task and background processing ambientROOM
  14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14

    May 2, 2022 Urp ▪ System supporting urban planning ▪ Integration of physical model with an interactive simulation ▪ tangible models of buildings throw a digital shadow on the interactive surface ▪ simulated wind shown as projected arrows on the table ▪ Various physical tools available to ▪ measure distance between two points or wind speed ▪ change material of building (e.g. glass walls) or daytime Urp system
  15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15

    May 2, 2022 ReacTIVision ▪ Open source toolkit for tangible multi-touch surfaces ▪ Fiducial markers and multi-touch finger tracking Kaltenbrunner and Bencina, 2007 Reactable
  16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16

    May 2, 2022 Video: Reactable
  17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17

    May 2, 2022 The Sand Noise Music Device ▪ Interactive art installation offering an intuitive and tactile method for control- ling and interacting with a generative electronic music system ▪ virtual objects move in the sand and obey the laws of gravity (e.g. speed up when flowing downhill) ▪ users can move the physical objects as well as change the topography of the sand landscape The sand noise music device
  18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18

    May 2, 2022 Video: The Sand Noise Music Device
  19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19

    May 2, 2022 ArtVis ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information ▪ new insights about large collections of data Bram Moerman
  20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20

    May 2, 2022 ArtVis ... Bram Moerman
  21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21

    May 2, 2022 ArtVis ... Bram Moerman
  22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22

    May 2, 2022 Sifteo Cubes (Siftables) ▪ Interactive gaming platform for ▪ spatial reasoning ▪ collaboration ▪ pattern recognition ▪ … ▪ Originated from Siftables ▪ Features ▪ 1.5" block with 128x128 colour TFT LCD ▪ 32 bit ARM CPU ▪ 3-axis accelerometer ▪ near-field object sensing technology (detect closeby Sifteo cubes) Sifteo Cubes
  23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23

    May 2, 2022 Video: Sifteo Cubes
  24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24

    May 2, 2022 ZeroN ▪ Anti-gravity interaction element that can be levitated and moved freely by a computer in 3D space ▪ explores how altering the fundamental rule of the physical world will transform interaction between humans and materials in the future ▪ Users can place or move the ZeroN in the mid-air 3D space in the same way they can place and interact with objects on surfaces ZeroN
  25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25

    May 2, 2022 Video: ZeroN
  26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26

    May 2, 2022 TRANSFORM TRANSFORM, MIT ▪ Dynamic shape display that can physically render 3D content ▪ tangible interaction with digital content - geospatial data - 3D modelling - … ▪ TRANSFORM display can also interact with the physical world around it ▪ remote users can be displayed physically ▪ Step towards MIT’s vision of Radical Atoms
  27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27

    May 2, 2022 Video: TRANSFORM
  28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28

    May 2, 2022 Radical Atoms (2012) ▪ Vision taking a leap beyond Tangible Bits ▪ assuming a hypothetical generation of materials that can change their form and appearance dynamically ▪ Radical Atoms is about ▪ a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) ▪ the future material that can transform its shape, conform to constraints and inform the users of its affordances ▪ a vision for the future of human-material interaction, in which all digital information has a physical manifestation so that we can interact directly with it ▪ a new Material User Interface (MUI) Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni and Jean-Baptiste Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions 19(1), January 2012
  29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29

    May 2, 2022 Radical Atoms Concept ▪ Transform ▪ interface can transform its shape to modify the model and reflect changes in the computational model ▪ Conform ▪ interface has to conform to some physical laws and user constraints (e.g. for safety) ▪ Inform ▪ user has to be informed about changing interface affordances
  30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30

    May 2, 2022 Interactions with Radical Atoms ▪ Direct touch and gestural interaction ▪ gestures coupled with direct touch create an interaction appropriate for Radical Atoms since users are able to rapidly reform dynamic materials at all scales ▪ Context-aware transformation ▪ context-aware transformations of the hand tool/interface - e.g. screwdriver adapting to the type of screw it is operating on ▪ Shape-memory clay: Perfect Red ▪ Perfect Red is a fictional material that can be sculpted like and responds according to rules inspired by CAD operations - e.g. if we split a piece in two even halves, then the operations performed on one part are mirrored on the other part
  31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31

    May 2, 2022 Vision-driven Design Research ▪ Quantum leaps in HCI rarely result from studies on user needs but from the passion and dreams of visionaries ▪ e.g. Douglas Engelbart ▪ Vision-driven research ▪ strong vision can last beyond our lifespan ▪ have to wait for enabling technologies but exploration of interaction design can already start!
  32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32

    May 2, 2022 Data Physicalisation ▪ Tangible user interfaces (TUIs) for interacting with digital data and services ▪ e.g. ArtVis ▪ Physical objects used for output ▪ often static encoding of digital information ▪ How can we achieve dynamic data physicalisation with dynamic affordances ▪ e.g. 3D visualisation with additional dimensions represented by physical variables such as temperature or texture ▪ exploration of big data sets ▪ tangible holograms as an experimental platform
  33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33

    May 2, 2022 Tangible Hologram (TangHo) Platform Timothy J. Curtin
  34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34

    May 2, 2022 Tangible Hologram (TangHo) Platform …
  35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35

    May 2, 2022 Dynamic Data Physicalisation ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform
  36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36

    May 2, 2022 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ replaceable sphere for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system will be body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin
  37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37

    May 2, 2022 TangHo Interactions
  38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38

    May 2, 2022 Data Physicalisation Framework ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Understanding the perceptual effectiveness ▪ e.g. what are the just-noticeable differences that a physical variable can convey? ▪ Framework for dynamic data physicalisation ▪ data processing with device-independent output - TextDIO: text for static or dynamic reports - PhysDIO: data points, data variables and physical variables ▪ text and visualisation drivers for representation on specific devices (e.g. TangHo) ▪ new data physicalisation design guidelines (as in InfoVis)
  39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39

    May 2, 2022 Data Physicalisation Framework …
  40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40

    May 2, 2022 Real-Time Cross-Media Data Exploration
  41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41

    May 2, 2022 Real-Time Cross-Media Data Exploration ...
  42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42

    May 2, 2022 Dynamic Data Physicalisation @ WISE ▪ Conceptual and software dataphys framework ▪ rapid prototyping ▪ drivers for physicalisations (e.g. TangHo) ▪ Dataphys grammar ▪ data-driven physicalisation ▪ exploration of large multi-dimensional datasets ▪ Dynamic dataphys design and interaction guidelines
  43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43

    May 2, 2022 Homework ▪ Read the following paper that is available on Canvas (papers/Ishii2012) ▪ H. Ishii, D. Lakatos, L. Bonanni and J.-B. Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, inter- actions, 19(1), January 2012
  44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44

    May 2, 2022 References ▪ G.W. Fitzmaurice, H. Ishii and W. Buxton, Bricks: Laying the Foundations for Graspable User Inter- faces, Proceedings of CHI 1995, ACM Conference on Human Factors in Computing Systems, Denver, USA, May 1995 ▪ https://dx.doi.org/10.1145/223904.223964 ▪ J.J. Gibson, The Ecological Approach to Visual Perception, Chapter 8: The Theory of Affordances, 1979, ISBN-13: 978-1848725782 ▪ https://cs.brown.edu/courses/cs137/readings/Gibson-AFF.pdf
  45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45

    May 2, 2022 References … ▪ H. Ishii and B. Ullmer, Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms, Proceedings of CHI 1997, ACM Conference on Human Factors in Computing Systems, Atlanta, USA, March 1997 ▪ https://dx.doi.org/10.1145/258549.258715 ▪ J. Underkoffler and H. Ishii, Urp: A Luminous-Tangible Workbench for Urban Planning and Design, Proceedings of CHI 1999, ACM Conference on Human Factors in Computing Systems, Pittsburgh, USA, May 1999 ▪ https://dx.doi.org/10.1145/302979.303114
  46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46

    May 2, 2022 References … ▪ S. Jordà, G. Geiger, M. Alonso and M. Kaltenbrunner, The reacTable: Exploring the Synergy Between Live Music Performance and Tabletop Tangible Interfaces, Proceedings of TEI 2007, Baton Rouge, USA, February 2007 ▪ https://dx.doi.org/10.1145/1226969.1226998 ▪ S. Follmer, D. Leithinger, A. Olwal, A. Hogge and H. Ishii, inFORM: Dynamic Physical Affordances and Constraints Through Shape and Object Actuation, Proceedings of UIST 2013, St Andrews, UK, October 2013 ▪ https://dx.doi.org/10.1145/2501988.2502032
  47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47

    May 2, 2022 References … ▪ H. Ishii, D. Lakatos, L. Bonanni and J.-B. Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions, 19(1), January 2012 ▪ https://dx.doi.org/10.1145/2065327.2065337 ▪ B. Dumas, B. Moerman, S. Trullemans and B. Signer, ArtVis: Combining Advanced Visualisation and Tangible Interaction for the Exploration, Analysis and Browsing of Digital Artwork Collections, Proceedings of AVI 2014, International Working Conference on Ad-vanced Visual Interfaces, Como, Italy, May 2014 ▪ https://beatsigner.com/publications/dumas_AVI2014.pdf
  48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48

    May 2, 2022 References … ▪ B. Signer and T.J. Curtin, Tangible Holograms: Towards Mobile Physical Augmentation of Virtual Objects, Technical Report WISE Lab, WISE-2017-01, March 2017 ▪ https://beatsigner.com/publications/signer_arXiv2017.pdf ▪ B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed K.A. Abdullah, Towards a Framework for Dynamic Data Phys- icalisation, Proceedings of the International Workshop Toward a Design Language for Data Physicalisation, Berlin, Germany, October 2018 ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf
  49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49

    May 2, 2022 References … ▪ Eva Hornecker, Tangible Interaction ▪ https://www.interaction-design.org/literature/book/the-glossary-of-human- computer-interaction/tangible-interaction ▪ Durrell Bishop’s Marble Answering Machine ▪ https://www.youtube.com/watch?v=RgVbXV1krgU ▪ The Norman Door ▪ https://www.youtube.com/watch?v=yY96hTb8WgI ▪ metaDESK ▪ https://www.youtube.com/watch?v=FsHHYK_UXkw ▪ ambientROOM ▪ https://www.youtube.com/watch?v=WO939s7FARQ
  50. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50

    May 2, 2022 References … ▪ Reactable ▪ https://www.youtube.com/watch?v=Mgy1S8qymx0&gl=BE ▪ The Sand Noise Music Device ▪ https://www.youtube.com/watch?v=VJgD-lEUPpo ▪ Siftables ▪ https://www.ted.com/talks/david_merrill_demos_siftables_the_smart_blocks ▪ ZeroN ▪ https://www.youtube.com/watch?v=-i2kJMJz7Wg ▪ TRANSFORM ▪ https://www.youtube.com/watch?v=lCARHatJQJA
  51. 2 December 2005 Next Lecture Virtual and Augmented Reality