Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interaction modalities, technologies and tools for interactive art

Interaction modalities, technologies and tools for interactive art

Presentation about interaction modalities and technologies that can be used for interactive art projects, with emphasis on devices available at the School of Arts (Catholic University of Portugal), Porto.


Jorge C. S. Cardoso

October 03, 2012

More Decks by Jorge C. S. Cardoso

Other Decks in Education


  1. Interaction modalities, technologies, and tools for interactive art Jorge C.

    S. Cardoso http://jorgecardoso.eu
  2. Objectives • Overview of most common non-desktop interaction and computing paradigms

    • Examples of technologies and tools that can be used to implement those interactions o  Available tools at EArtes/UCP • Most prominent people/references in these areas • No code (for now)
  3. Topics Gestural interaction Tangible interaction Brain interaction Wearable computing Physical

    computing Digital public displays Mobile computing
  4. Gestural interaction Photo credits: Flickr user artmakesmesmile: http://www.flickr.com/photos/artmakesmesmile/169193698/

  5. Gestural interaction "A gesture is a motion of the body

    that contains information. Waving goodbye is a gesture. Pressing a key on a keyboard is not a gesture because the motion of a finger on its way to hitting a key is neither observed nor significant. All that matters is which key was pressed". -- Kurtenbach and Hulteen (1990), cited by Billinghurst & Buxton (2011)
  6. Gestural interaction Technologies/Tools (ordered by the complexity of gestures: simplest

    to more complex) •  Touch surfaces •  Computer vision techniques •  Wiimote •  Kinect •  MoCap
  7. Gestural interaction: Touch surfaces •  Mostly planar, rectangular, glass(y) surfaces

    •  Various technologies for detecting touch o  which have different limits for the number of simultaneous touches it can detect •  Gestures o  Fingers only (small devices) o  Hands/forearms (large(r) devices, depending on technology used) o  2d movements only •  Number of users o  one - personal devices (phones, tablets) o  few - large surfaces (tables, walls)
  8. Gestural interaction: Touch surfaces Tablets/phones •  Singing fingers o  Jay

    Silver o  https://vimeo.com/12010952 Single-person interaction, fingers
  9. Gestural interaction: Touch surfaces Medium/Large surfaces •  Touch-Wall @ EArtes

    o  Laser light plane (LLP) technique (see Sandler 2012) o  Multi-touch surface (in development) o  Vertical orientation o  One-two person interaction o  Fingers, hands
  10. Gestural interaction: Touch surfaces Medium/Large surfaces •  Bookstore @ EArtes

  11. Gestural interaction: Touch surfaces Medium/Large surfaces •  Interactive tables (Reactables)

    o  Diffused illumination technique o  Horizontal orientation o  Up to four people interacting o  Fingers, hands shapes o  Three available @ EArtes
  12. Gestural interaction: Touch surfaces Very large surfaces •  Massive Curved

    Multi-Touch Screen o  Reality touchscreen University of Groningen o  http://www.youtube.com/watch?v=AlWFtF06RFo Several-person interaction, fingers, hands
  13. Gestural interaction: computer vision Computer vision (CV) techniques (non depth-

    cameras) •  Various kinds of surfaces/scenes •  Gestures o  Varied, depends on positioning of camera §  fingers §  hands §  whole body o  mostly 2d, but can also infer distance o  not very precise §  Can be very precise with controlled conditions
  14. Gestural interaction: computer vision Video Place •  Myron Krueger • 

  15. Gestural interaction: computer vision Edge/Swap •  Tiago Dionisio, Rudolfo Quintas

    •  http://www.youtube.com/watch?v=OSKpXZjECtY •  http://www.youtube.com/watch?v=ryX4Vq2-IKE
  16. Gestural interaction: computer vision BES Interactive Video Wall •  YDreams

    •  http://www.youtube.com/watch?v=ijrZQbSOONQ
  17. Gestural interaction: computer vision Balance •  Jorge Coutinho •  https://vimeo.com/40506946

  18. Gestural interaction: computer vision Sketchsynth •  Billy Keyes •  http://vimeo.com/42053193

    •  http://golancourses.net/2012spring/05/13/billy-keyes- final-project-sketchsynth/
  19. Gestural interaction: computer vision All eyes on you •  Britzpetermann

    •  https://vimeo.com/33186969
  20. Gestural interaction: computer vision Knee deep •  Emily Gobeille &

    Theodore Watson •  https://vimeo.com/8805152
  21. Gestural interaction: computer vision Night lights •  YesYesNo •  http://vimeo.com/8525186

  22. Gestural interaction: computer vision aB •  Jorge Coutinho •  http://vimeo.com/29524291

  23. Gestural interaction: Wiimote Wiimote •  Nintendo game controller o  Wireless:

    uses bluetooth to connect to a computer o  Has various accelerometer sensors to sense movement o  IR sensor for pointing •  Gestures o  3d movement, rotation o  combined with the sensor bar, allows pointing gestures •  Various wiimotes available @ EArtes
  24. Gestural interaction: Wiimote Wiimote •  Wiimote hacks o  Johnny Chung

    Lee o  http://www.ted.com/talks/ johnny_lee_demos_wii_remote_hacks.html
  25. Gestural interaction: Wiimote Pizza breakout •  Bruno Santos

  26. Gestural interaction: Wiimote Wii Drum High Tutorial •  He Zhao

    •  http://vimeo.com/2406224
  27. Kinect •  XBox game controller o  Depth camera: senses the

    distance of each pixel to the camera (Borenstein 2012). •  Gestures o  Detects a "point cloud" - pixels and their distances to the camera o  With extra software it's possible to detect limbs (skeleton) positioning o  More accurate than simple cameras •  Two available @ EArtes Gestural interaction: Kinect
  28. Gestural interaction: Kinect Kinect Physics Tutorial for Processing •  Video

    showing the result of Processing code that uses Kinect •  http:// www.creativeapplications.net/ processing/kinect-physics- tutorial-for-processing/
  29. Gestural interaction: Kinect YScope medical interface •  YDreams •  Gestural

    interface for an operating room •  http://www.youtube.com/watch?v=91F6zErnCrs
  30. Gestural interaction: Kinect Control the Humanoid Robot by Kinect • 

    Using the skeleton detected by the Kinect to drive a robot •  ikaziso •  http://www.youtube.com/watch?v=w8BmgtMKFbY
  31. Gestural interaction: Kinect Unnamed soundsculpture •  Daniel Franke & Cedric

    Kiefer •  Using various Kinect to model a dancer and then process the data into visual effects •  https://vimeo.com/38505448 •  honorary mention from the Prix Ars Electronica
  32. Gestural interaction: Kinect Make the line dance •  "1024" • 

    Detection of skeleton + projection into human body •  https://vimeo.com/21308228
  33. Gestural interaction: Kinect Fragmentos de um olhar •  Pedro Serrano

  34. Gestural interaction: LeapMotion Leap Motion •  New controller for hand

    interactions •  https://leapmotion.com/
  35. Gestural interaction: MoCap MoCap - Motion Capture •  MoCap Room

    •  High-speed infrared cameras detect markers placed on actor's body •  Gestures o  Highly accurate gestures •  (Vicon MoCap available @ EArtes)
  36. Gestural interaction: MoCap T(ether) •  Tangible Media Group •  Interaction

    with virtual volumetric data •  http://vimeo.com/42173010
  37. Gestural interaction: MoCap Portrait of the ghost drummer •  Odaibe

    •  Visualization of the process of playing on a drum kit •  http://vimeo.com/34682556 •  Non-interactive
  38. Gestural interaction: computer vision Computer Vision Tools •  Processing o 

    BlobDetection o  JMyron •  Max/Msp o  Jitter o  jit.cv external •  Community Core Vision o  http://ccv.nuigroup.com/
  39. Gestural interaction: Wiimote Wiimote Tools •  Processing/Java o  Motej -

    http://motej.sourceforge.net/ o  Wigee - http://www.wiigee.org/ •  Max/Msp o  aka.wiiremote external - http://www.iamas.ac.jp/~aka/max/
  40. Gestural interaction: Kinect Kinect Tools •  http://openkinect.org/wiki/Main_Page •  Book "Making

    things see" (Borenstein 2012) •  Processing/Java o  OpenKinect - http://www.shiffman.net/p5/kinect/ o  SimpleOpenNI - http://code.google.com/p/simple-openni/ o  http://www.creativeapplications.net/processing/ kinect-physics-tutorial-for-processing/ •  Max/Msp o  jit.freenect.grab - http://jmpelletier.com/freenect/
  41. Gestural interaction: Touch surfaces Tools •  Web apps o  jQMultiTouch

    library §  http://dev.globis.ethz.ch/jqmultitouch/
  42. Gestural interaction References •  Billinghurst & Buxton (2011). Gesture Based

    Interaction, http://www.billbuxton.com/ input14.Gesture.pdf •  Borenstein, Greg (2012). Making Things See: 3D vision with Kinect, Processing, Arduino, and MakerBot, O'Reilly Media / Make •  Buxton, Bill (2012). Multi-Touch Systems that I Have Known and Loved, http:// www.billbuxton.com/multitouchOverview.html •  Kurtenbach, G. & Hulteen, E. (1990). Gestures in Human-Computer Communications. In B. Laurel (Ed.) The Art of Human Computer Interface Design. Addison-Wesley, 309-317, http:// www.billbuxton.com/inputManuscript.html •  Levin, Golan. Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers, http://www.flong.com/texts/essays/essay_cvad/ •  Sandler, Seth (2012). LASER LIGHT PLANE MULTITOUCH TECHNIQUE, http:// sethsandler.com/multitouch/llp/
  43. Tangible interaction Photo credits: Flickr user kedume: http://www.flickr.com/photos/kedume/746632510/sizes/l/in/photostream/

  44. Tangible interaction "[...] tangible interfaces give physical form to digital

    information, employing physical artifacts both as representations and controls for computational media. TUIs couple physical representations (e.g., spatially manipulable physical objects) with digital representations (e.g., graphics and audio), yielding user interfaces that are computationally mediated, but generally not identifiable as "computers" per se." - Ullmer & Ishii (2000)
  45. Tangible interaction Picture from Ullmer and Ishii (2000).

  46. Tangible interaction Tangible Media Group site •  http://tangible.media.mit.edu/projects/

  47. Tangible interaction Reactable •  Tangible musical instrument o  Users manipulate

    physical objects to create and control the sound o  Can be used as a general tangible framework •  http://www.youtube.com/watch?v=Mgy1S8qymx0
  48. Tangible interaction Little Boxes •  ECAL/Joelle Aeschlimann •  Music boxes

    for iPad •  https://vimeo.com/45704273
  49. Tangible interaction Sketch-a-TUI •  Alexander Wiethoff, et al. •  Set

    of printable objects for tangible prototypes •  http://vimeo.com/38793875 •  Paper o  http://tei-conf.org/12/Main/Sketchatui •  Printable objects o  http://project-premium.org/sketch-a-tui/index.html
  50. Tangible interaction Birds on Paper •  Chen-Wei Chiang, et al.

    •  Drawing music •  http://www.youtube.com/watch?v=qqkkNotSQMo •  Paper o  http://tei-conf.org/12/Main/Birdsonpaper
  51. Tangible interaction Chronotape •  Peter Bennett •  Tangible timeline with

    annotations •  https://vimeo.com/28191512 •  Paper o  http://tei-conf.org/12/Main/Chronotape
  52. Tangible interaction Splash Controllers •  Luc Geurts and Vero Vanden

    Abeele •  Uncareful Manipulation of Water •  Paper o  http://tei-conf.org/12/Main/Splashcontrollers
  53. Tangible interaction Skube •  Andrew Nip, Ruben van de Vleuten,

    Malthe Borch, and Andrew Spitz •  Tangible interface to Last.fm & Spotify Radio •  http://www.creativeapplications.net/maxmsp/skube- tangible-interface-to-last-fm-spotify-radio/
  54. Tangible interaction Faustine •  Carlos Caires & Jorge Cardoso • 

    Interactive video installation inspired by the short story "Morel's Invention" •  https://vimeo.com/19043202
  55. Tangible interaction The garden of time •  Carlos Caires &

    Jorge Cardoso •  Interactive video installation inspired by the short story "The Garden of Forking Paths" •  https://vimeo.com/24836769
  56. Tangible interaction Tools •  Reactable •  TUIO - Fiducial tracking

    o  http://www.tuio.org/?software o  Processing o  Max/MSP o  CCV •  Wiimotes •  Other off-the-shelf hardware/sensors
  57. Tangible interaction References •  G. W. Fitzmaurice, H. Ishii, and

    W. A. S. Buxton, “Bricks: Laying the Foundations for Graspable User Interfaces,” in Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’95, 1995, pp. 442–449. •  H. Ishii and B. Ullmer, “Tangible bits,” in Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’97, 1997, pp. 234–241. •  Ullmer, B. and Ishii, H. (2000). “Emerging frameworks for tangible user interfaces,” IBM Systems Journal, vol. 39, no. 3, pp. 915–931, Jul. 2000.
  58. Brain interfaces Photo credits: Flickr user Saad Faruque: http://www.flickr.com/photos/cblue98/7254347346/sizes/l/in/photostream/

  59. Brain interfaces Brain-Computer Interfaces (BCI) or Mind- Machine Interface (MMI)

    Using brain signals to interact with computers. •  Usually based on EEG readings •  Various commercial devices available •  http://en.wikipedia.org/wiki/Brain %E2%80%93computer_interface#Low-cost_BCI- based_Interfaces
  60. Brain interfaces Câmara Neuronal •  Adolfo Luxúria Canibal, João Martinho

    Moura, Miguel Pedro Guimarães •  http://projects.jmartinho.net/3486412/ Camara-Neuronal-Video-Teaser
  61. Brain interfaces Brainwave beats •  Collin Cunningham •  Midi notes

    from brain signals •  Uses the Neurosky Mindset device •  http://blog.makezine.com/2011/05/05/collins-lab- brainwave-beats/
  62. Brain interfaces Mansion Maze 3 BCI Game Trailer •  Eric

    Rosendale, Ian McCabe, Veronica Cole, Aaron DeChamplain, and Matt Stephan •  BCI Game •  http://www.youtube.com/watch?v=sdYAIrr9nDw
  63. Brain interfaces Mind Control of Lego NXT Telepresence Robot • 

    Mark Bruce •  Controling a lego robot using brain signals •  Uses EPOC BCI device •  https://www.youtube.com/watch?v=nX741DZw8l4
  64. Brain interfaces Tools •  Neurosky Mindset (available @ EArtes) • 

    EPOC neuroheadset - http://emotiv.com/emortal/cart/ •  Processing library o  Neurosky Mindset Processing §  http://jorgecardoso.eu/processing/ MindSetProcessing/ •  http://en.wikipedia.org/wiki/Brain %E2%80%93computer_interface#Low-cost_BCI- based_Interfaces
  65. Wearable computing Geordi La Forge, Star Trek The Next Generation

    character wearing the VISOR
  66. Wearable computing "Wearable computing is the study or practice of

    inventing, designing, building, or using miniature body-borne computational and sensory devices. Wearable computers may be worn under, over, or in clothing, or may also be themselves clothes" (Mann, 2012)
  67. Wearable computing Image from: http://www.eyetap.org/research/wearables/wearcomp/wearables.html Image from: http://www.interaction-design.org/

  68. Wearable computing Google Project Glass •  https://plus.google.com/+projectglass/posts •  https://www.youtube.com/watch? v=9c6W4CCU9M4

    (how will our world be in 5/10 years, when we all start wearing devices even more powerful than Glass?)
  69. Wearable computing XS Labs •  http://xslabs.net/

  70. Wearable computing Blazer 01.2 •  A light emitting bracelet • 

    Can be used to "draw" letters in the air •  Vincent leclerc •  http://www.uttermatter.com/blazer/012/index.htm#
  71. Wearable computing cidZiy •  Capacitive sensing textile surface •  xslabs

    •  http://www.youtube.com/watch?v=zH4J6Uaf88A
  72. Wearable computing Kukkia •  Kukkia flowers frame the face and

    slowly open and close over time •  xslabs •  http://www.youtube.com/watch?v=R5fd6AwGMhg
  73. Wearable computing See U in my Tee •  Inês Petiz

    •  T-shirts that react to the presence of other t-shirts •  http://wearable-technology.blogspot.pt/
  74. Wearable computing References •  Mann, Steve (2012): Wearable Computing. In:

    Soegaard, Mads and Dam, Rikke Friis (eds.). "Encyclopedia of Human-Computer Interaction". Aarhus, Denmark: The Interaction Design Foundation. Available online at http://www.interaction-design.org/encyclopedia/wearable_computing.html
  75. Physical computing

  76. Physical computing "Physical computing, in the broadest sense, means building

    interactive physical systems by the use of software and hardware that can sense and respond to the analog world. (...) In the broad sense, physical computing is a creative framework for understanding human beings' relationship to the digital world. In practical use, the term most often describes handmade art, design or DIY hobby projects that use sensors and microcontrollers to translate analog input to a software system, and/or control electro-mechanical devices such as motors, servos, lighting or other hardware." - (Physical computing 2012)
  77. Physical computing Physical computing book •  O'Sullivan & Igoe 2004

  78. Physical computing reaDIYmate •  Olivier Mével and Marc Chareyron • 

  79. Physical computing An Instrument for the Sonification of Everyday Things

    •  Dennis P Paul •  http:// www.creativeapplications.net/ processing/an-instrument-for-the- sonification-of-everyday-things/
  80. Physical computing Piano playing stairs •  http://www.youtube.com/watch? v=DuNWkKBokts

  81. Physical computing What the phonics •  Andrew Spitz & Momo

    Miyazaki •  https://vimeo.com/45747333
  82. Physical computing Barcode band •  Kang woon jin, Lee ha

    lim, Kim yong duk •  https://vimeo.com/41132461
  83. Physical computing Syndyn •  André Rangel and Anne-Kathrin Siegel • 

  84. Physical computing Little Bits •  Hardware components that can be

    assembled together •  http://market.littlebits.cc/ •  https://vimeo.com/45276780
  85. Physical computing MakeyMakey •  Circuit board that connects to a

    computer via USB and allows everyday objects to become controllers •  http://www.makeymakey.com/
  86. Physical computing Lego Mindstorms NXT •  Available @ EArtes • 

    Lego kit with o  motors o  distance sensor o  light sensor o  button o  lego parts...
  87. Physical computing Tools •  Arduino - http://www.arduino.cc/ •  Galago -

    http://outbreak.co/galago •  Digispark - http://digistump.com/ •  Raspberry pi - http://www.raspberrypi.org/ •  MakeyMakey - http://www.makeymakey.com/ •  LittleBits - http://littlebits.cc/ •  Lego NXT Mindstorms (available @EArtes) o  http://mindstorms.lego.com/en-us/Default.aspx
  88. Physical computing References •  Physical computing. (2012, July 21). In

    Wikipedia, The Free Encyclopedia. Retrieved 14:37, September 22, 2012, from http://en.wikipedia.org/w/index.php? title=Physical_computing&oldid=503476286 •  Dan O'Sullivan, Tom Igoe. (2004). Physical Computing: Sensing and Controlling the Physical World with Computers, Thomson Course Technology, Boston, Massachusetts,
  89. Digital public displays Photo credits: Flickr user bmann: http://www.flickr.com/photos/boris/5104900447/sizes/l/in/photostream/

  90. Digital public displays Digital displays located in public areas, allowing

    anyone to see/interact with them •  Traditionally, public displays have been used to broad information from one active source to millions of passive receivers •  Digital public displays can change this o  Turning receivers into active receivers o  Supporting dynamic locally relevant information o  Supporting user-generated content (many to many content distribution) •  Everyone can interact o  Everyone can see you interacting
  91. Digital public displays Advertising

  92. Digital public displays Honda Interactive Billboard using SMS and Bluetooth

    •  Users can "start" the car via SMS •  http://www.youtube.com/watch?v=OudaJGtwdQ8
  93. Digital public displays BBC Billboard •  Voting billboards, users can

    SMS their vote •  http://www.bbc.co.uk/blogs/theeditors/2007/06/ live_in_new_york_city.html
  94. Digital public displays Audio Jack Billboards •  Promoting a TV

    show "Big Love" •  Users can listen to people's thoughts by plugin headphones into the billboard •  http://www.trendhunter.com/trends/audio-jack-billboard- for-big-love
  95. Digital public displays Nokia's N90 interactive billboard •  YDreams • 

    The billboard snaps people's pictures and displays them on the billboard •  http://www.youtube.com/watch?v=UWgIswo3lV4
  96. Digital public displays Mini Billboard Interactive •  The billboard displays

    the names of the Mini car owners passing by •  http://www.nytimes.com/2007/01/29/business/media/ 29cooper.html
  97. Digital public displays Nikon D700 •  The billboard plays flashing

    cameras as people pass by. •  http://www.thecoolhunter.net/article/detail/1570/nikon- d700-guerrilla-style-billboard
  98. Digital public displays Ambient displays •  Information about their surroundings

    •  Information visualization, aesthetics, art
  99. Digital public displays Hello Wall •  (Prante et al. 2003)

    •  Can display various abstract information patterns •  Can display group activity
  100. Digital public displays Informative Art ambient display •  Skog et

    al. 2003 •  Display bus information (only the bus number 16 - two in each direction) o  Size of square: amount of time before bus leaves o  Color: Time to leave to the bus stop o  Position: right - buses to the city center; left - buses from the city center
  101. Digital public displays Information Percolator •  Heiner et al. 1999

    •  A water bubble display o  Can be used to display various types of ambient information •  http://www.cs.cmu.edu/~hudson/ video.html
  102. Digital public displays Entertainment/Waiting times

  103. Digital public displays Mobilenin •  Scheible, Jürgen and Ojala, Timo

    •  Paper o  http://www.mendeley.com/research/mobilenin-combining-multi-track- music-video-personal-mobile-phones-public-display-multi-user-interac o  Allows users to vote on the next video clip
  104. Digital public displays Public Youtube Player •  Allows users to

    have some control over the videos that are played •  at the EArtes' Bar display
  105. Digital public displays Everybody Votes •  Users can express their

    opinions regarding various questions and suggest their own questions •  at the EArtes' Bar display
  106. Digital public displays References Heiner, J. M., Hudson, S. E.,

    & Tanaka, K. (1999). The Information Percolator  : Ambient Information Display in a Decorative Object. Proceedings of the 12th annual ACM symposium on User interface software and technology - UIST ’99 (pp. 141–148). New York, New York, USA: ACM Press. doi:10.1145/320719.322595 Prante, T., Röcker, C., Streitz, N., Stenzel, R., Magerkurth, C., van Alphen, D., & Plewe, D. (2003). Hello.Wall - Beyond Ambient Displays. Video Track and Adjunct Proceedings of the 5th Intern. Conference on Ubiquitous Computing (UBICOMP’03). Seattle, Wash., USA. Skog, T., Ljungblad, S., & Holmquist, L. E. (n.d.). Between aesthetics and utility: designing ambient information visualizations. IEEE Symposium on Information Visualization 2003 (IEEE Cat. No. 03TH8714) (pp. 233–240). IEEE. doi:10.1109/INFVIS.2003.1249031
  107. Mobile computing

  108. Mobile computing "Mobile computing is human–computer interaction by which a

    computer is expected to be transported during normal usage. Mobile computing involves mobile communication, mobile hardware, and mobile software." - (Mobile computing 2012)
  109. Mobile computing X-Fetch •  Daniel Santos, João Rema •  City-wide

    game, peddy paper like •  Players have to discover a set of clues spread across the city (Porto) using a mobile device that plays videos pointing to next clues and that tracks players GPS locations •  http://jpn.c2com.up.pt/2005/10/29/ xfetch_quando_o_real_e_o_virtual_se_confundem.html
  110. Mobile computing •  GPS Drawing Jeremy Wood http://www.gpsdrawing.com/gallery.html

  111. Mobile computing Mobile mobile •  http://www.theophane.co.uk/mobile-mobile •  https://vimeo.com/8118881

  112. Mobile computing Pom •  Jorge Coutinho •  Jogo Pong estendido

    a seis jogadores •  https://vimeo.com/18579379
  113. Mobile computing References •  Mobile computing. (2012, September 8). In

    Wikipedia, The Free Encyclopedia. Retrieved 17:56, September 30, 2012, from http://en.wikipedia.org/w/index.php? title=Mobile_computing&oldid=511364202