Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interactive Tabletops and Surfaces - Lecture 7 - Next Generation User Interfaces (4018166FNR)

Interactive Tabletops and Surfaces - Lecture 7 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

Beat Signer
PRO

March 27, 2023
Tweet

More Decks by Beat Signer

Other Decks in Education

Transcript

  1. 2 December 2005
    Next Generation User Interfaces
    Interactive Tabletops and Surfaces
    Prof. Beat Signer
    Department of Computer Science
    Vrije Universiteit Brussel
    beatsigner.com

    View Slide

  2. Beat Signer - Department of Computer Science - [email protected] 2
    March 27, 2023
    Tabletops
    ▪ The term tabletop was introduced in 2001
    ▪ Tabletop interfaces are suitable for a number of
    specific activities
    ▪ multi-user (group) collaborative work
    ▪ shared horizontal workspace
    ▪ interaction with the table as well as objects placed on the table
    (affordances of the table)
    - relies on a user's mental model of traditional tables
    ▪ Tabletops support new forms of
    ▪ hand and gesture interaction
    ▪ tangible interaction
    ▪ interactive visualisations (WIMP is not good enough)

    View Slide

  3. Beat Signer - Department of Computer Science - [email protected] 3
    March 27, 2023
    Tabletops …
    ▪ Typical tabletop applications
    ▪ viewing photos or videos
    ▪ brainstorming
    ▪ advanced visualisation and data analysis
    ▪ what about productivity tasks?
    - word processing
    - email
    - …
    ▪ Current tabletop solutions seem to be used for dedicated
    tasks but not as all-purpose computing devices
    ▪ missing keyboard input, precise pointing and standard applications
    ▪ what about combining multi-touch with keyboard and mouse
    input?

    View Slide

  4. Beat Signer - Department of Computer Science - [email protected] 4
    March 27, 2023
    Tabletops …
    ▪ Ergonomic issues
    ▪ neck muscle strain or back problems when using a
    horizontal tabletop as primary input device
    ▪ what about tabletops with an adjustable angle?
    ▪ visibility and reachability of elements when working
    in larger groups

    View Slide

  5. Beat Signer - Department of Computer Science - [email protected] 5
    March 27, 2023
    Tabletop Research Domains
    ▪ Computer Supported
    Cooperative Work (CSCW)
    ▪ computer-mediated group collaboration
    and social interactions
    ▪ Human Computer
    Interaction (HCI)
    ▪ beyond the desktop metaphor
    ▪ individual user actions and performance
    ▪ Ubiquitous Computing (UbiComp)
    ▪ beyond traditional desktop computer displays
    ▪ Tangible User Interfaces (TEI)
    ▪ interaction with physical objects on a tabletop
    CSCW HCI
    UbiComp
    TEI

    View Slide

  6. Beat Signer - Department of Computer Science - [email protected] 6
    March 27, 2023
    Multi-User Tabletop Interfaces
    ▪ One of the core features of a tabletop interface is that
    there is enough room for multiple people to sit or stand
    around the table
    ▪ Users share the interface
    ▪ Studies have shown that users adopt territories on a
    tabletop for better coordination
    ▪ personal territories
    - reserved for personal use (generally in front of the user)
    ▪ group territories
    - space to perform main task activities (reachable by all users)
    ▪ storage territories
    - used to store task resources and non-task items (in the periphery)

    View Slide

  7. Beat Signer - Department of Computer Science - [email protected] 7
    March 27, 2023
    Multi-User Tabletop Interfaces …
    ▪ Some multi-user tabletop interfaces guidelines
    ▪ provide visibility and transparency of actions
    - users should see that actions are going on and which actions are performed
    - personal territory on laptop vs. personal territory on shared tabletop
    ▪ provide appropriate table space
    - users should not run out of space when not participating in the group activity
    Scott et al., 2004

    View Slide

  8. Beat Signer - Department of Computer Science - [email protected] 8
    March 27, 2023
    Multi-User Tabletop Interfaces …
    ▪ Some multi-user tabletop interfaces guidelines …
    ▪ provide functionality in the appropriate locality
    - e.g. reading/writing support in personal territory
    ▪ support casual grouping of items and tools in the workspace
    - e.g. group/organise resources in piles as reminder
    Scott et al., 2004

    View Slide

  9. Beat Signer - Department of Computer Science - [email protected] 9
    March 27, 2023
    Enabling Technologies
    ▪ Basic components of a tabletop system
    ▪ touch sensor technology
    - size that can be covered
    - real-time and multi-touch tracking
    - integration with display
    - planar or support for curved surfaces?
    ▪ display
    - projectors
    - LCD displays
    - OLED displays
    ▪ software
    - potentially events from multiple users and multiple fingers to be tracked
    - beyond traditional WIMP interfaces
    - rotation of display areas

    View Slide

  10. Beat Signer - Department of Computer Science - [email protected] 10
    March 27, 2023
    Resistive Touch Panels
    ▪ Typically two clear layers
    coated with transparent
    conductive substances
    ▪ insulating layer between the
    two conductive sheets
    ▪ Controller alternates
    between the layers
    ▪ driving electric current on
    one and measuring the
    current on the other
    - detection of horizontal and
    vertical position
    Schöning et al., 2008

    View Slide

  11. Beat Signer - Department of Computer Science - [email protected] 11
    March 27, 2023
    Resistive Touch Panels …
    ▪ Low-cost manufacturing
    ▪ Low power consumption compared to other approaches
    ▪ Input requires pressure on the outer layer
    ▪ can also be used with a stylus or when wearing gloves
    ▪ Reduced display quality due to the additional layers
    ▪ Typically used for mobile phones, PDAs,
    digital cameras, …

    View Slide

  12. Beat Signer - Department of Computer Science - [email protected] 12
    March 27, 2023
    Surface Capacitive Touch Panels
    ▪ Uniform transparent conduc-
    tive coating on glass panel
    ▪ Electrodes in each corner
    ▪ uniform electric field across
    the conductive layer
    ▪ Touch with finger (or other
    conductive object) results
    in transport of charge
    ▪ Better display quality than resistive panels
    ▪ High positional accuracy
    ▪ Difficult to detect multi-touch
    Schöning et al., 2008

    View Slide

  13. Beat Signer - Department of Computer Science - [email protected] 13
    March 27, 2023
    Projected Capacitive Touch Panels
    ▪ Sensor grid (electrodes)
    covered by front layer
    ▪ Enables accurate detection
    of multi-touch
    ▪ High positional accuracy
    ▪ Surface/projected ca-
    pacitive solutions only work
    with finger/special stylus
    ▪ Not suited for large panels
    ▪ slower transmission of electrical current
    ▪ Typically used in more recent mobile phones, …
    Schöning et al., 2008

    View Slide

  14. Beat Signer - Department of Computer Science - [email protected] 14
    March 27, 2023
    Surface Acoustic Wave (SAW)
    ▪ Glass panel with trans-
    mitting and receiving
    transducers as well as
    reflectors
    ▪ ultrasonic waves
    ▪ Soft material (e.g. finger)
    absorbs ultrasonic waves
    ▪ detected by transducers
    ▪ Excellent display quality since no layer on top
    ▪ Can support dual-touch
    ▪ Wide frame area necessary for transducers
    [http://www.dmccoltd.com/english/museum/touchscreens/technologies/AcousticWave.asp]

    View Slide

  15. Beat Signer - Department of Computer Science - [email protected] 15
    March 27, 2023
    Frustrated Total Internal Reflection (FTIR)
    ▪ Optical total internal
    reflection
    ▪ inner material must have
    a higher refractive index
    than outer material
    ▪ angle at the boundary
    has to be sufficiently small
    ▪ Infrared light is injected and reflected (leaving material)
    if a finger touches the surface
    ▪ computer vision algorithms used to compute the location
    ▪ Back projection can be used in combination with FTIR
    [http://www.teksol.in/2015/11/how-screen-touch-works.html]

    View Slide

  16. Beat Signer - Department of Computer Science - [email protected] 16
    March 27, 2023
    Diffused Illumination (DI)
    ▪ Infrared light placed be-
    hind the projection surface
    ▪ Depending on the diffuser,
    DI might also detect fingers
    and objects above the
    surface
    ▪ Easier tracking of physical objects which might be
    identified by their shape or fiducial markers

    View Slide

  17. Beat Signer - Department of Computer Science - [email protected] 17
    March 27, 2023
    DigitalDesk
    ▪ DigitalDesk developed by
    Pierre Wellner at Xerox
    EuroPARC
    ▪ camera-based tracking and
    projection
    ▪ "Instead of making the
    workstation more like a
    desk, make the desk more
    like a workstation"
    ▪ Many follow-up
    augmented desk projects
    Wellner, DigitalDesk, 1991

    View Slide

  18. Beat Signer - Department of Computer Science - [email protected] 18
    March 27, 2023
    DiamondTouch Table
    ▪ Developed at Mitsubishi
    Electric Research Labora-
    tories (MERL) in 2001
    ▪ Front-projected interactive
    display
    ▪ Can detect who is
    touching the surface
    ▪ capacitive coupling between
    the touch surface and
    receivers located in the chair
    of each user
    [http://www.guillaumeriviere.name/collection/tabletop.html]

    View Slide

  19. Beat Signer - Department of Computer Science - [email protected] 19
    March 27, 2023
    Video: DiamondTouch Table

    View Slide

  20. Beat Signer - Department of Computer Science - [email protected] 20
    March 27, 2023
    Jeff Han’s Multi-Touch Table
    ▪ Uses refined version of
    Frustrated Total Internal
    Reflection (FTIR)
    ▪ Simple and cheap
    implementation of a
    multi-touch surface
    presented in 2006

    View Slide

  21. Beat Signer - Department of Computer Science - [email protected] 21
    March 27, 2023
    Video: Jeff Han’s Multi-Touch Table

    View Slide

  22. Beat Signer - Department of Computer Science - [email protected] 22
    March 27, 2023
    BendDesk
    ▪ Seamlessly combines a
    vertical and a horizontal
    multi-touch surface
    ▪ Uses 2 projectors and
    3 cameras
    ▪ FTIR-based tracking
    Weiss et al., 2010

    View Slide

  23. Beat Signer - Department of Computer Science - [email protected] 23
    March 27, 2023
    Video: BendDesk

    View Slide

  24. Beat Signer - Department of Computer Science - [email protected] 24
    March 27, 2023
    Microsoft PixelSense (Samsung SUR40)
    ▪ Samsung produces the
    hardware and Microsoft
    the software
    ▪ 40 inch LED backlit LCD
    display (1920×1080)
    ▪ integrated PC and
    PixelSense technology
    ▪ Backlight with IR LEDs
    ▪ reflected by fingers and
    other objects
    - multi-touch and real objects
    ▪ pixels act as IR sensors
    [http://www.embeddedinsights.com/channels/wp-content/uploads/2011/03/110329-surface.png]

    View Slide

  25. Beat Signer - Department of Computer Science - [email protected] 25
    March 27, 2023
    Video: Microsoft PixelSense

    View Slide

  26. Beat Signer - Department of Computer Science - [email protected] 26
    March 27, 2023
    ReacTIVision
    ▪ Open source toolkit for
    tangible multi-touch
    surfaces
    ▪ Fiducial markers and
    multi-touch finger tracking
    Kaltenbrunner and Bencina, 2007
    Reactable

    View Slide

  27. Beat Signer - Department of Computer Science - [email protected] 27
    March 27, 2023
    Video: Reactable

    View Slide

  28. Beat Signer - Department of Computer Science - [email protected] 28
    March 27, 2023
    iTable Interactive Tabletop
    ▪ Interactive tabletop based
    on digital pen a paper
    technology
    ▪ developed by the GlobIS
    group at ETH Zurich
    ▪ table surface covered with
    Anoto pattern
    - e.g. pattern printed on paper and
    covered with glass
    ▪ top projection
    ▪ high precision pen tracking
    Pen-based Google Earth Browser
    Pen-based Drawing Tool

    View Slide

  29. Beat Signer - Department of Computer Science - [email protected] 29
    March 27, 2023
    Pen and Touch Interaction on Tabletops
    ▪ Pen and touch interaction
    for authoring and editing
    documents on tabletops
    ▪ simultaneous use of two
    modalities
    ▪ alternative to on-screen
    widgets (for experts)
    ▪ non-dominant hand postures
    define the pen mode
    Matulic and Norrie, 2013

    View Slide

  30. Beat Signer - Department of Computer Science - [email protected] 30
    March 27, 2023
    we-inspire Room
    ▪ Ideation technology
    ▪ based on Anoto's digital pen
    and paper technology
    ▪ Large collaborative
    interactive wall surface
    ▪ Various applications
    ▪ sketch, write, brainstorm
    and capture

    View Slide

  31. Beat Signer - Department of Computer Science - [email protected] 31
    March 27, 2023
    Video: we-inspire Room

    View Slide

  32. Beat Signer - Department of Computer Science - [email protected] 32
    March 27, 2023
    HP Sprout Pro
    ▪ Immersive computing
    platform by HP
    ▪ All-in-one PC
    ▪ capacitive tactile touchpad
    with top projection
    ▪ 3D scanning cameras
    ▪ creative work without mouse
    and keyboard …
    ▪ “blended reality”

    View Slide

  33. Beat Signer - Department of Computer Science - [email protected] 33
    March 27, 2023
    Video: HP Sprout Pro

    View Slide

  34. Beat Signer - Department of Computer Science - [email protected] 34
    March 27, 2023
    Microsoft Surface Studio 2
    ▪ All-in-one PC
    ▪ 28-inch 4.5K PixelSense
    display
    ▪ screen can be tilted to flat
    position (hinge design)
    ▪ New tools for creative
    process
    ▪ e.g. surface dial
    - control for different applications
    when placed on the screen
    - haptic feedback (menu options)

    View Slide

  35. Beat Signer - Department of Computer Science - [email protected] 35
    March 27, 2023
    Video: Microsoft Surface Studio 2

    View Slide

  36. Beat Signer - Department of Computer Science - [email protected] 36
    March 27, 2023
    OLED Technology
    ▪ Displays based on
    Organic Light Emitting
    Diodes (OLEDs)
    ▪ flexible/thinner than LCDs
    ▪ no background light
    ▪ 1000 times faster than LEDs
    ▪ can be printed and produced
    at less costs
    ▪ Combination of OLED
    displays with thin multi-
    touch devices
    ▪ ultimate tabletop system

    View Slide

  37. Beat Signer - Department of Computer Science - [email protected] 37
    March 27, 2023
    Windowless Plane
    ▪ Futuristic windowless
    plane design concept by
    Technicon Design
    ▪ Video stream from wing-
    mounted cameras
    ▪ Screen estate could also
    be used for
    ▪ showing movies
    ▪ video conferences
    ▪ …
    ▪ Plane needs less fuel

    View Slide

  38. Beat Signer - Department of Computer Science - [email protected] 38
    March 27, 2023
    Video: Windowless Plane

    View Slide

  39. Beat Signer - Department of Computer Science - [email protected] 39
    March 27, 2023
    AquaTop Display
    ▪ Interactive water surface
    that can for example be
    used in a bathroom
    ▪ gesture-based interaction
    ▪ Kinect in combination with
    top projection
    ▪ New possibilities to
    interact with a water
    surface
    ▪ e.g. poking fingers from
    beneath the water
    Takahashi et al., 2012

    View Slide

  40. Beat Signer - Department of Computer Science - [email protected] 40
    March 27, 2023
    Video: AquaTop Display

    View Slide

  41. Beat Signer - Department of Computer Science - [email protected] 41
    March 27, 2023
    Video: A Day Made of Glass

    View Slide

  42. Beat Signer - Department of Computer Science - [email protected] 42
    March 27, 2023
    Video: A Day Made of Glass 2

    View Slide

  43. Beat Signer - Department of Computer Science - [email protected] 43
    March 27, 2023
    Exercise 4
    ▪ Hands-on experience with WebXR

    View Slide

  44. Beat Signer - Department of Computer Science - [email protected] 44
    March 27, 2023
    References
    ▪ S.D. Scott, M. Sheelagh, T. Carpendale and
    K.M. Inkpen, Territoriality in Collaborative Tabletop
    Workspaces, Proceedings of CSCW 2004, International
    Conference on Computer Supported Cooperative Work,
    Chicago, USA, November 2004
    ▪ https://dx.doi.org/10.1145/1031607.1031655
    ▪ J. Schöning et al., Multi-Touch Surfaces: A Technical
    Guide, Technical Report TUM-I0833, 2008
    ▪ http://campar.in.tum.de/pub/schoening2008multitouch/schoening2008multit
    ouch.pdf
    ▪ Jeff Han’s Multi-Touch Table, 2006
    ▪ https://www.ted.com/talks/jeff_han_demos_his_breakthrough_touchscreen

    View Slide

  45. Beat Signer - Department of Computer Science - [email protected] 45
    March 27, 2023
    References …
    ▪ P. Wellner, The DigitalDesk Calculator: Tangi-
    ble Manipulation on a Desk Top Display, Proceedings of
    ACM UIST 1991, Symposium on User Interface Software
    and Technology, Hilton Head, USA, November 1991
    ▪ https://dx.doi.org/10.1145/120782.120785
    ▪ M. Weiss, S. Voelker, C. Sutter and J. Borchers,
    BendDesk: Dragging Across the Curve, Proceedings of
    ITS 2010, International Conference on Interactive
    Tabletops and Surfaces, Saarbrücken, Germany,
    November 2010
    ▪ https://dx.doi.org/10.1145/1936652.1936654

    View Slide

  46. Beat Signer - Department of Computer Science - [email protected] 46
    March 27, 2023
    References …
    ▪ M. Kaltenbrunner and R. Bencina,
    reacTIVision: A Computer-Vision Framework for Table-
    based Tangible Interaction, Proceedings of TEI 2007,
    International Conference on Tangible and Embedded
    Interaction, Baton Rouge, USA, February 2007
    ▪ https://dx.doi.org/10.1145/1226969.1226983
    ▪ F. Matulic and M.C. Norrie, Pen and Touch Gestural Environ-
    ment for Document Editing on Interactive Tabletops, Proceed-
    ings of ITS 2013, International Conference on Interactive
    Tabletops and Surfaces, St Andrews, UK, October 2013
    ▪ https://dx.doi.org/10.1145/2512349.2512802

    View Slide

  47. Beat Signer - Department of Computer Science - [email protected] 47
    March 27, 2023
    References …
    ▪ Y. Takahashi, Y. Matoba and H. Koike, Fluid
    Surface: Interactive Water Surface Display for Viewing
    Information in a Bathroom, Proceedings of ITS 2012,
    International Conference on Interactive Tabletops and
    Surfaces, Cambridge, USA, November 2012
    ▪ https://dx.doi.org/10.1145/2396636.2396687
    ▪ C. Müller-Tomfelde (Ed.), Tabletops: Horizontal
    Interactive Displays, Springer 2010
    ▪ https://dx.doi.org/10.1007/978-1-84996-113-4
    ▪ BendDesk Video
    ▪ https://www.youtube.com/watch?v=5VNTPwVvLzE

    View Slide

  48. Beat Signer - Department of Computer Science - [email protected] 48
    March 27, 2023
    References …
    ▪ Microsoft Pixelsense Video
    ▪ https://www.youtube.com/watch?v=58dsqozft3k
    ▪ HP Sprout Pro Video
    ▪ https://www.youtube.com/watch?v=axKbn0-WZrU
    ▪ Microsoft Surface Studio 2 Video
    ▪ https://www.youtube.com/watch?v=RmVAbB3M-4Y
    ▪ we-inspire Collaboration Technology Video
    ▪ https://www.youtube.com/watch?v=eLfpMlyt4BA
    ▪ AquaTop Display Video
    ▪ https://www.youtube.com/watch?v=fYJneaa2O8I
    ▪ Reactable Video
    ▪ https://www.youtube.com/watch?v=Mgy1S8qymx0

    View Slide

  49. Beat Signer - Department of Computer Science - [email protected] 49
    March 27, 2023
    References …
    ▪ A Day Made of Glass Video
    ▪ https://www.youtube.com/watch?v=6Cf7IL_eZ38
    ▪ A Day Made of Glass 2 Video
    ▪ https://www.youtube.com/watch?v=jZkHpNnXLB0
    ▪ Technicon Design Windowless Plane Video
    ▪ https://www.youtube.com/watch?v=lNo3Sj_ri78

    View Slide

  50. 2 December 2005
    Next Lecture
    Gesture-based Interaction

    View Slide