Computer vision techniques for interactive art

Computer vision techniques for interactive art

Presentation about computer vision using Processing and Jitter for the Interactive Art PhD program at the School of Arts (http://artes.ucp.pt)

47984b654bec549462b585cad0761c1f?s=128

Jorge C. S. Cardoso

May 28, 2011
Tweet

Transcript

  1. Computer Vision Techniques for Interactive Art Using Processing and Max/Jitter

    Interactive Art Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 9/24/12 Jorge Cardoso 1
  2. Interactive Art Doctoral Program Topics •  Techniques –  Frame differencing

    (for motion detection) –  Color detection (where in the image, does a given color appear – can be used for tracking objects) –  Brightness detection (what is the position of the brightest pixels) –  Background subtraction (for object segmentation) –  Blob extraction (applied to the previous techniques to extract shape parameters and vectorizing silhuettes) –  Movement estimation (what direction are the objects moving) –  Face detection (is there a face in the image? where?) 9/24/12 Jorge Cardoso 2
  3. Interactive Art Doctoral Program How •  General overview of how

    those CV techniques work •  Examples of what can be accomplished with the techniques –  Vídeos –  Live demos •  Code examples of these techniques in Processing and Jitter •  Some – very simple(istic) – examples of how they can be used for interactive work 9/24/12 Jorge Cardoso 3
  4. Interactive Art Doctoral Program Computer Vision for Interaction •  Use

    a computer and camera to extract information about what people are doing and use that as an implicit or explicit form of interaction •  “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php? title=Computer_vision&oldid=285848177 9/24/12 Jorge Cardoso 4
  5. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 5 Image • 

    Computer vision are processes applied to sequences of images –  An image is just a sequence of number (usually arranged in a matrix form) •  Each number represents a colour 50 44 23 31 38 52 75 52 29 09 15 08 38 98 53 52 08 07 12 15 24 30 51 52 10 31 14 38 32 36 53 67 14 33 38 45 53 70 69 40 36 44 58 63 47 53 35 26 68 76 74 76 55 47 38 35 69 68 63 74 50 42 35 32
  6. Interactive Art Doctoral Program Image •  Each pixel (i.e., number

    in the matrix) is usually stored in RGB or ARGB format so –  It can be decomposed: 9/24/12 Jorge Cardoso 6 or
  7. Interactive Art Doctoral Program Frame Differencing •  If we subtract

    two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video •  Procedure: –  Go through the current frame image, pixel by pixel –  Calculate the distance between the color of the current frame’s pixel and the previous frame’s pixel –  If distance is smaller than a given threshold, assume no movement (black), otherwise assume movement (white) 9/24/12 Jorge Cardoso 7
  8. Interactive Art Doctoral Program Frame Differencing •  Color difference – 

    Q: How do we calculate the “distance” between two colors? –  A: Simple version: euclidean distance. •  Take a color as a point in 3D space (RGB -> XYZ) •  Calculate the distance between the two points –  Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2) –  Dist = Sqrt(dR*dR + dG*dG + dB*dB) –  A1: Slightly (perceptively) better version: weight each component differently: •  Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB) 9/24/12 Jorge Cardoso 8
  9. Interactive Art Doctoral Program Frame Differencing •  Q: What’s it

    good for? •  A: –  KALOETHOGRAPHIA (2010) •  Pedro Cascalheira •  http://artes.ucp.pt/blogs/index.php/vai/2011/02/14/alunos-2009-2010 –  aB (2011) •  Jorge Coutinho •  https://picasaweb.google.com/lh/photo/x9bGyxFsTLVFbkk9kD3xLA?feat=directlink –  WebCam Piano •  Memo Akten •  [video] 9/24/12 Jorge Cardoso 9
  10. Interactive Art Doctoral Program Frame Differencing - Code •  Processing

    Code –  FrameDifferencing –  FrameDifferencingWithMotionDetection –  FrameDifferencingForAirPiano •  Jitter Code –  FrameDifferencing.maxpat –  FrameDifferencingWithMotionDetection.maxpat 9/24/12 Jorge Cardoso 10
  11. Interactive Art Doctoral Program Color detection •  Isolate color regions

    in an image •  Procedure: –  Go through the image, pixel by pixel –  Calculate the distance between the color of the current pixel and the reference color –  If distance is smaller than a given threshold, keep the pixel 9/24/12 Jorge Cardoso 11
  12. Interactive Art Doctoral Program Color Detection •  Color Difference – 

    For a better approach than RGB distance, use HSB –  Give more weight to the Hue –  Dist = Sqrt(3*dH*dH + dS*dS + 2*dB*dB) 9/24/12 Jorge Cardoso 12
  13. Interactive Art Doctoral Program Color Detection •  Q: What’s it

    good for? •  A: –  Catch of the day [video] –  PlayDohAsInterface Keyboard [video] 9/24/12 Jorge Cardoso 13
  14. Interactive Art Doctoral Program Color Detection - Code •  Processing

    Code –  ColorDifference –  ColorDifferenceSequencer •  Jitter Code –  ColorDifferenceWithFindbounds.maxpat –  ColorDifference.maxpat 9/24/12 Jorge Cardoso 14
  15. Interactive Art Doctoral Program Brighteness detection •  Slight variation on

    color detection when we’re only interested in tracking bright pixels –  For example, to track a light •  Procedure: –  Same as color detection but extract the brightness of each pixel and keep only the highest 9/24/12 Jorge Cardoso 15
  16. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 16 Brighteness detection

    •  Q: How do we calculate the “brightness” of a color? •  A: –  In RGB: •  (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B) •  (physical/energy) Brightness = (R+G+B)/3 •  … –  In HSB: •  Brightness = B J –  Just use the brightness() function in whatever language…
  17. Interactive Art Doctoral Program Brightness detection •  Q: Why not

    just color detection? •  A: –  In some situations you don’t care about color, just the brightness (for example to detect a lantern) –  Brightness detection is more robust (i.e., less influenced by lighting conditions) 9/24/12 Jorge Cardoso 17
  18. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 18 Brighteness detection

    •  Q: What’s it good for? •  A: – Drawing With Light [video] – Burning the Sound [video]
  19. Interactive Art Doctoral Program Brighteness detection - Code •  Processing

    Code –  BrightnessDetection •  Jitter Code –  BrightnessDetection.maxpat 9/24/12 Jorge Cardoso 19
  20. Interactive Art Doctoral Program Background subtraction 9/24/12 Jorge Cardoso 20

    •  If we have a fixed camera, and a background image for reference, we can subtract the background from each frame to isolate new objects: This - This = This
  21. Interactive Art Doctoral Program Background subtraction •  Q: How do

    we “subtract” two images? •  A: Pixel by pixel •  Q: How do we “subtract” two pixels? •  A: Channel by channel (color component by color component, usually R, G and B) 9/24/12 Jorge Cardoso 21
  22. Interactive Art Doctoral Program Background subtraction •  Pixel subtraction alone

    doesn’t work very well due to noise or shadows in the image •  The resulting subtraction must be thresholded to eliminate noise 9/24/12 Jorge Cardoso 22
  23. Interactive Art Doctoral Program Background subtraction •  After subtracting and

    thresholding, we can binarize the image to get a mask 9/24/12 Jorge Cardoso 23
  24. Interactive Art Doctoral Program Background subtraction •  Q: What’s it

    good for? •  A: Isolating objects in a scene when you don’t have much control over the background. –  Background Subtraction Processing Demo [video] 9/24/12 Jorge Cardoso 24
  25. Interactive Art Doctoral Program Background subtraction - Code •  Processing

    code: –  BackgroundSubtraction •  Jitter code –  BackgroundSubraction.maxpat 9/24/12 Jorge Cardoso 25
  26. Interactive Art Doctoral Program Blob extraction •  All the previous

    techniques result in possibly many white areas distributed in the image 9/24/12 Jorge Cardoso 26 http://en.wikipedia.org/wiki/Connected_Component_Labeling
  27. Interactive Art Doctoral Program Blob extraction •  Blob extraction (blob

    detection, connected component labeling) identifies those regions and and allows the extractions of some features –  area, –  perimeter, –  orientation, –  bounding box, –  Silhouette 9/24/12 Jorge Cardoso 27
  28. Interactive Art Doctoral Program Blob extraction •  Procedure: –  See

    http://en.wikipedia.org/wiki/Connected_Component_Labeling 9/24/12 Jorge Cardoso 28
  29. Interactive Art Doctoral Program Blob extraction •  Q: What’s it

    good for? •  A: –  Loliness (2007) •  Telmo Rocha [video] –  Células (2009) •  Ana Lima [video] –  SilhouetteCollisions [video] 9/24/12 Jorge Cardoso 29
  30. Interactive Art Doctoral Program Blob extraction - Code 9/24/12 Jorge

    Cardoso 30 •  Processing libraries
  31. Interactive Art Doctoral Program Blob extraction - Code •  Processing

    Code (with Flob library) –  BlobExtractionFromFrameDifferencing –  BlobExtractionAfterColorDifference •  Processing Code (with Jmyron library) –  Em Mac Intel é preciso substituir o ficheiro libjmyron por este: http://www.jibberia.com/projects/ –  BlobExtractionAfterBackgroundSubtraction 9/24/12 Jorge Cardoso 31
  32. Interactive Art Doctoral Program Blob extraction - Code •  Jitter

    Code (com external cv.jit) –  BlobExtractionAfterBrightnessDetection 9/24/12 Jorge Cardoso 32
  33. Interactive Art Doctoral Program Motion estimation (aka Optical Flow) 9/24/12

    Jorge Cardoso 33 •  Estimate the movement vectors of blocks in a sequence of two images •  Procedure (naive) –  Divide both images into blocks –  For each block in image one, find the closest (more similar) block in image two –  http://en.wikipedia.org/wiki/Lucas %E2%80%93Kanade_method Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png
  34. Interactive Art Doctoral Program Motion estimation 9/24/12 Jorge Cardoso 34

    •  Q: What’s it good for? •  A: –  opticalflow-particles [video] –  opticalflow-navigation [video]
  35. Interactive Art Doctoral Program Motion estimation - Code •  Jitter

    Code –  OpticalFlow.maxpat 9/24/12 Jorge Cardoso 35
  36. Interactive Art Doctoral Program Face detection •  Not recognition! • 

    Needs a configuration file that determines what to detect (front faces, side faces, body, etc) •  Returns the position and size of detected faces 9/24/12 Jorge Cardoso 36
  37. Interactive Art Doctoral Program Face Detection •  Q: What’s it

    good for? •  A: –  Anatomias Urbanas (2008) •  Sara Henriques [video] –  Ex.001 (2010) •  Bernardo Santos •  [foto] https://picasaweb.google.com/lh/photo/_KCdCQGUeEw8lSzF8ZMPVw?feat=directlink 9/24/12 Jorge Cardoso 37
  38. Interactive Art Doctoral Program Face Detection - Code •  Processing

    (with OpenCV processing library) –  FaceDetection –  FaceDetectionFunnyEyes •  Jitter (with cv.jit) –  FaceDetectionWithASmiley.maxpat 9/24/12 Jorge Cardoso 38
  39. Interactive Art Doctoral Program Computer Vision Techniques for Interactive Art

    •  Not a recent area •  Myron Krueger experimented with some of these techniques in the 70s [video Myron Krueger] 9/24/12 Jorge Cardoso 39
  40. Interactive Art Doctoral Program The End 9/24/12 Jorge Cardoso 40

    http://slideshare.net/jorgecardoso (tag: ea-phd-ia-2011)