Computer vision techniques for interactive art

Presentation about computer vision using Processing and Jitter for the Interactive Art PhD program at the School of Arts (http://artes.ucp.pt)

1. Computer Vision Techniques for Interactive Art Using Processing and Max/Jitter Interactive Art Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 9/24/12 Jorge Cardoso 1
2. Interactive Art Doctoral Program Topics •  Techniques –  Frame differencing (for motion detection) –  Color detection (where in the image, does a given color appear – can be used for tracking objects) –  Brightness detection (what is the position of the brightest pixels) –  Background subtraction (for object segmentation) –  Blob extraction (applied to the previous techniques to extract shape parameters and vectorizing silhuettes) –  Movement estimation (what direction are the objects moving) –  Face detection (is there a face in the image? where?) 9/24/12 Jorge Cardoso 2
3. Interactive Art Doctoral Program How •  General overview of how those CV techniques work •  Examples of what can be accomplished with the techniques –  Vídeos –  Live demos •  Code examples of these techniques in Processing and Jitter •  Some – very simple(istic) – examples of how they can be used for interactive work 9/24/12 Jorge Cardoso 3
4. Interactive Art Doctoral Program Computer Vision for Interaction •  Use a computer and camera to extract information about what people are doing and use that as an implicit or explicit form of interaction •  “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php? title=Computer_vision&oldid=285848177 9/24/12 Jorge Cardoso 4
5. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 5 Image •  Computer vision are processes applied to sequences of images –  An image is just a sequence of number (usually arranged in a matrix form) •  Each number represents a colour 50 44 23 31 38 52 75 52 29 09 15 08 38 98 53 52 08 07 12 15 24 30 51 52 10 31 14 38 32 36 53 67 14 33 38 45 53 70 69 40 36 44 58 63 47 53 35 26 68 76 74 76 55 47 38 35 69 68 63 74 50 42 35 32
6. Interactive Art Doctoral Program Image •  Each pixel (i.e., number in the matrix) is usually stored in RGB or ARGB format so –  It can be decomposed: 9/24/12 Jorge Cardoso 6 or
7. Interactive Art Doctoral Program Frame Differencing •  If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video •  Procedure: –  Go through the current frame image, pixel by pixel –  Calculate the distance between the color of the current frame’s pixel and the previous frame’s pixel –  If distance is smaller than a given threshold, assume no movement (black), otherwise assume movement (white) 9/24/12 Jorge Cardoso 7
8. Interactive Art Doctoral Program Frame Differencing •  Color difference –  Q: How do we calculate the “distance” between two colors? –  A: Simple version: euclidean distance. •  Take a color as a point in 3D space (RGB -> XYZ) •  Calculate the distance between the two points –  Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2) –  Dist = Sqrt(dR*dR + dG*dG + dB*dB) –  A1: Slightly (perceptively) better version: weight each component differently: •  Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB) 9/24/12 Jorge Cardoso 8
9. Interactive Art Doctoral Program Frame Differencing •  Q: What’s it good for? •  A: –  KALOETHOGRAPHIA (2010) •  Pedro Cascalheira •  http://artes.ucp.pt/blogs/index.php/vai/2011/02/14/alunos-2009-2010 –  aB (2011) •  Jorge Coutinho •  https://picasaweb.google.com/lh/photo/x9bGyxFsTLVFbkk9kD3xLA?feat=directlink –  WebCam Piano •  Memo Akten •  [video] 9/24/12 Jorge Cardoso 9
10. Interactive Art Doctoral Program Frame Differencing - Code •  Processing Code –  FrameDifferencing –  FrameDifferencingWithMotionDetection –  FrameDifferencingForAirPiano •  Jitter Code –  FrameDifferencing.maxpat –  FrameDifferencingWithMotionDetection.maxpat 9/24/12 Jorge Cardoso 10
11. Interactive Art Doctoral Program Color detection •  Isolate color regions in an image •  Procedure: –  Go through the image, pixel by pixel –  Calculate the distance between the color of the current pixel and the reference color –  If distance is smaller than a given threshold, keep the pixel 9/24/12 Jorge Cardoso 11
12. Interactive Art Doctoral Program Color Detection •  Color Difference –  For a better approach than RGB distance, use HSB –  Give more weight to the Hue –  Dist = Sqrt(3*dH*dH + dS*dS + 2*dB*dB) 9/24/12 Jorge Cardoso 12
13. Interactive Art Doctoral Program Color Detection •  Q: What’s it good for? •  A: –  Catch of the day [video] –  PlayDohAsInterface Keyboard [video] 9/24/12 Jorge Cardoso 13
14. Interactive Art Doctoral Program Color Detection - Code •  Processing Code –  ColorDifference –  ColorDifferenceSequencer •  Jitter Code –  ColorDifferenceWithFindbounds.maxpat –  ColorDifference.maxpat 9/24/12 Jorge Cardoso 14
15. Interactive Art Doctoral Program Brighteness detection •  Slight variation on color detection when we’re only interested in tracking bright pixels –  For example, to track a light •  Procedure: –  Same as color detection but extract the brightness of each pixel and keep only the highest 9/24/12 Jorge Cardoso 15
16. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 16 Brighteness detection •  Q: How do we calculate the “brightness” of a color? •  A: –  In RGB: •  (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B) •  (physical/energy) Brightness = (R+G+B)/3 •  … –  In HSB: •  Brightness = B J –  Just use the brightness() function in whatever language…
17. Interactive Art Doctoral Program Brightness detection •  Q: Why not just color detection? •  A: –  In some situations you don’t care about color, just the brightness (for example to detect a lantern) –  Brightness detection is more robust (i.e., less influenced by lighting conditions) 9/24/12 Jorge Cardoso 17
18. Interactive Art Doctoral Program 9/24/12 Jorge Cardoso 18 Brighteness detection •  Q: What’s it good for? •  A: – Drawing With Light [video] – Burning the Sound [video]
19. Interactive Art Doctoral Program Brighteness detection - Code •  Processing Code –  BrightnessDetection •  Jitter Code –  BrightnessDetection.maxpat 9/24/12 Jorge Cardoso 19
20. Interactive Art Doctoral Program Background subtraction 9/24/12 Jorge Cardoso 20 •  If we have a fixed camera, and a background image for reference, we can subtract the background from each frame to isolate new objects: This - This = This
21. Interactive Art Doctoral Program Background subtraction •  Q: How do we “subtract” two images? •  A: Pixel by pixel •  Q: How do we “subtract” two pixels? •  A: Channel by channel (color component by color component, usually R, G and B) 9/24/12 Jorge Cardoso 21
22. Interactive Art Doctoral Program Background subtraction •  Pixel subtraction alone doesn’t work very well due to noise or shadows in the image •  The resulting subtraction must be thresholded to eliminate noise 9/24/12 Jorge Cardoso 22
23. Interactive Art Doctoral Program Background subtraction •  After subtracting and thresholding, we can binarize the image to get a mask 9/24/12 Jorge Cardoso 23
24. Interactive Art Doctoral Program Background subtraction •  Q: What’s it good for? •  A: Isolating objects in a scene when you don’t have much control over the background. –  Background Subtraction Processing Demo [video] 9/24/12 Jorge Cardoso 24
25. Interactive Art Doctoral Program Background subtraction - Code •  Processing code: –  BackgroundSubtraction •  Jitter code –  BackgroundSubraction.maxpat 9/24/12 Jorge Cardoso 25
26. Interactive Art Doctoral Program Blob extraction •  All the previous techniques result in possibly many white areas distributed in the image 9/24/12 Jorge Cardoso 26 http://en.wikipedia.org/wiki/Connected_Component_Labeling
27. Interactive Art Doctoral Program Blob extraction •  Blob extraction (blob detection, connected component labeling) identifies those regions and and allows the extractions of some features –  area, –  perimeter, –  orientation, –  bounding box, –  Silhouette 9/24/12 Jorge Cardoso 27
28. Interactive Art Doctoral Program Blob extraction •  Procedure: –  See http://en.wikipedia.org/wiki/Connected_Component_Labeling 9/24/12 Jorge Cardoso 28
29. Interactive Art Doctoral Program Blob extraction •  Q: What’s it good for? •  A: –  Loliness (2007) •  Telmo Rocha [video] –  Células (2009) •  Ana Lima [video] –  SilhouetteCollisions [video] 9/24/12 Jorge Cardoso 29
30. Interactive Art Doctoral Program Blob extraction - Code 9/24/12 Jorge Cardoso 30 •  Processing libraries
31. Interactive Art Doctoral Program Blob extraction - Code •  Processing Code (with Flob library) –  BlobExtractionFromFrameDifferencing –  BlobExtractionAfterColorDifference •  Processing Code (with Jmyron library) –  Em Mac Intel é preciso substituir o ficheiro libjmyron por este: http://www.jibberia.com/projects/ –  BlobExtractionAfterBackgroundSubtraction 9/24/12 Jorge Cardoso 31
32. Interactive Art Doctoral Program Blob extraction - Code •  Jitter Code (com external cv.jit) –  BlobExtractionAfterBrightnessDetection 9/24/12 Jorge Cardoso 32
33. Interactive Art Doctoral Program Motion estimation (aka Optical Flow) 9/24/12 Jorge Cardoso 33 •  Estimate the movement vectors of blocks in a sequence of two images •  Procedure (naive) –  Divide both images into blocks –  For each block in image one, find the closest (more similar) block in image two –  http://en.wikipedia.org/wiki/Lucas %E2%80%93Kanade_method Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png
34. Interactive Art Doctoral Program Motion estimation 9/24/12 Jorge Cardoso 34 •  Q: What’s it good for? •  A: –  opticalflow-particles [video] –  opticalflow-navigation [video]
35. Interactive Art Doctoral Program Motion estimation - Code •  Jitter Code –  OpticalFlow.maxpat 9/24/12 Jorge Cardoso 35
36. Interactive Art Doctoral Program Face detection •  Not recognition! •  Needs a configuration file that determines what to detect (front faces, side faces, body, etc) •  Returns the position and size of detected faces 9/24/12 Jorge Cardoso 36
37. Interactive Art Doctoral Program Face Detection •  Q: What’s it good for? •  A: –  Anatomias Urbanas (2008) •  Sara Henriques [video] –  Ex.001 (2010) •  Bernardo Santos •  [foto] https://picasaweb.google.com/lh/photo/_KCdCQGUeEw8lSzF8ZMPVw?feat=directlink 9/24/12 Jorge Cardoso 37
38. Interactive Art Doctoral Program Face Detection - Code •  Processing (with OpenCV processing library) –  FaceDetection –  FaceDetectionFunnyEyes •  Jitter (with cv.jit) –  FaceDetectionWithASmiley.maxpat 9/24/12 Jorge Cardoso 38
39. Interactive Art Doctoral Program Computer Vision Techniques for Interactive Art •  Not a recent area •  Myron Krueger experimented with some of these techniques in the 70s [video Myron Krueger] 9/24/12 Jorge Cardoso 39
40. Interactive Art Doctoral Program The End 9/24/12 Jorge Cardoso 40 http://slideshare.net/jorgecardoso (tag: ea-phd-ia-2011)