Image Processing on Canvas

Presentation at Sapo Codebits 2012

Transcript

1. Image processing on canvas Sérgio Pereira | sergio-pt@sapo.pt https://github.com/sergiompereira Sapo Codebits 2012
2. OVERVIEW 1. What can we do with image processing? 2. Why do it in the browser? : current applications. 3. How do we accomplish this? The canvas element and its rendering context. Geometric operations on an image : scale, crop and rotation. 4. Pixel manipulation The ImageData object 5. Filters 6. Pixel displacement (just a taste of it) 7. Optimization and performance 8. Libraries
3. 1. What can we do with image processing?
4. Image enhancement 1. What can we do with image processing?
5. Image enhancement 1. What can we do with image processing? • contrast • brightness • saturation and color adjustments • sharpness • noise removal • ...
6. Information extraction 1. What can we do with image processing?
7. Information extraction 1. What can we do with image processing?
8. Information extraction 1. What can we do with image processing? • brightness threshold • color value • color difference • edges • ...
9. Aesthetic or artistic effects 1. What can we do with image processing?
10. Aesthetic or artistic effects 1. What can we do with image processing?
11. Aesthetic or artistic effects 1. What can we do with image processing?
12. Aesthetic or artistic effects 1. What can we do with image processing? • greyscale • full contrast • color adjustments • monotone and duotone • posterization • ...
13. 2. Why do it in the browser? Current applications.
14. Gravatar profile picture editor gravatar.com 2. Why do it in the browser? : current applications.
15. Vimeo effects panel vimeo.com 2. Why do it in the browser? : current applications.
16. Google+ creative kit plus.google.com 2. Why do it in the browser? : current applications.
17. Imgur “aviary” editor imgur.com 2. Why do it in the browser? : current applications.
18. 3. How do we accomplish this?
19. Browser plugins 3. How do we accomplish this? • They might not be installed automatically • They might not be available for the user device
20. Browser plugins 3. How do we accomplish this? • They might not be installed automatically • They might not be available for the user device Server technologies • Too much load on the server • Client-server communication brakes the interaction flow
21. The html <canvas> 3. How do we accomplish this? “The canvas element provides scripts with a resolution-dependent bitmap canvas, which can be used for rendering graphs, game graphics, art, or other visual images on the fly.” http://www.w3.org/TR/html5/the-canvas-element.html
22. The html <canvas> 3. How do we accomplish this? • Web standard • Available (almost) everywhere • Lightweight • Allows detailed editing • Not semantic (like SVG) • Not very fast or high-performing
23. The CanvasRenderingContext2D 3. How do we accomplish this?
24. Drawing images: crop and scale 3. How do we accomplish this? context.drawImage(imageEl, sx, sy, sw, sh, dx, dy, dw, dh);
25. Drawing images: crop and scale 3. How do we accomplish this? context.drawImage(imageEl, sx, sy, sw, sh, dx, dy, dw, dh);
26. Drawing images: crop and scale 3. How do we accomplish this? context.drawImage(imageEl, sx, sy, sw, sh, dx, dy, dw, dh);
27. Drawing images: crop and scale 27 3. How do we accomplish this? context.drawImage(imageEl, sx, sy, sw, sh, dx, dy, dw, dh);
28. Drawing images: rotation 3. How do we accomplish this? Source image
29. Drawing images: rotation 3. How do we accomplish this? Source image
30. Drawing images: rotation 3. How do we accomplish this? Auxiliar canvas in memory (off screen)
31. Drawing images: rotation 3. How do we accomplish this? Auxiliar canvas in memory (off screen)
32. Drawing images: rotation 3. How do we accomplish this? Auxiliar canvas in memory (off screen)
33. Drawing images: rotation 3. How do we accomplish this? Auxiliar canvas in memory (off screen)
34. Drawing images: rotation 3. How do we accomplish this?
35. Drawing images: rotation 3. How do we accomplish this? /* [ cos, sin, Cx - cos*Cx - sin*Cy -sin, cos, Cy + sin*Cx - cos*Cy ] */ var cos = Math.cos(angle); var sin = Math.sin(angle); var Cx = rect.x + rect.width/2; var Cy = rect.y + rect.height/2;
36. Drawing images: rotation 3. How do we accomplish this? /* [ cos, sin, Cx - cos*Cx - sin*Cy -sin, cos, Cy + sin*Cx - cos*Cy ] */ var cos = Math.cos(angle); var sin = Math.sin(angle); var Cx = rect.x + rect.width/2; var Cy = rect.y + rect.height/2; aux_context.setTransform(cos, -sin, sin, cos, Cx-cos*Cx-sin*Cy, Cy+sin*Cx-cos*Cy);
37. Drawing images: rotation 3. How do we accomplish this? /* [ cos, sin, Cx - cos*Cx - sin*Cy -sin, cos, Cy + sin*Cx - cos*Cy ] */ var cos = Math.cos(angle); var sin = Math.sin(angle); var Cx = rect.x + rect.width/2; var Cy = rect.y + rect.height/2; aux_context.setTransform(cos, -sin, sin, cos, Cx-cos*Cx-sin*Cy, Cy+sin*Cx-cos*Cy); aux_context.drawImage(img,0,0);
38. Drawing images: rotation 3. How do we accomplish this? /* [ cos, sin, Cx - cos*Cx - sin*Cy -sin, cos, Cy + sin*Cx - cos*Cy ] */ var cos = Math.cos(angle); var sin = Math.sin(angle); var Cx = rect.x + rect.width/2; var Cy = rect.y + rect.height/2; aux_context.setTransform(cos, -sin, sin, cos, Cx-cos*Cx-sin*Cy, Cy+sin*Cx-cos*Cy); aux_context.drawImage(img,0,0); context.drawImage(aux_canvas, rect.x, rect.y, rect.width, rect.height, 0,0, canvas.width, canvas.height);
39. 4. Pixel manipulation
40. The ImageData object 4. Pixel manipulation var imagedata = context.getImageData(x, y, w, h);
41. The ImageData object 4. Pixel manipulation var imagedata = context.getImageData(x, y, w, h); var dataWidth = imagedata.width; var dataHeight = imagedata.height; var data = imagedata.data; //a one-dimensional array containing the data in RGBA order, as integers in the range 0 to 255.
42. The data array associated to a 10 by 10 pixels ImageData object 4. Pixel manipulation The ImageData data property
43. 4. Pixel manipulation Pixels and their colors
44. 4. Pixel manipulation Pixels and their colors
45. 4. Pixel manipulation Pixel order
46. 4. Pixel manipulation Get pixel color function getPixel(x,y){ var i = (y * width + x) * 4; return{ r:data[i], g:data[i+1], b:data[i+2], a:data[i+3] } }
47. 4. Pixel manipulation Set pixel color function setPixel(x,y, color){ var i = (y * width + x) * 4; data[i] = color.r; data[i+1] = color.g; data[i+2] = color.b; data[i+3] = color.a; } }
48. 4. Pixel manipulation Set pixel color var x,y,color; for(y=0; y<height; y++){ for(x=0; x<width; x++){ color = { r:x, g:255-Math.sqrt(x*x+y*y), b:y, a:255 }; setPixel(x,y,color); } } Canvas 255 x 255
49. 4. Pixel manipulation ImageData “buffer” var buffer = context.createImageData(width,height); var data = buffer.data; //process data context.putImageData(buffer, x,y);
50. 5. Filters
51. 5. Filters Point filters
52. 5. Filters Point filters : contrast
53. 5. Filters Point filters : contrast function contrast(amount, color){ return { r: ((color.r – 127) * amount) + 127, g: ((color.g – 127) * amount) + 127, b: ((color.b – 127) * amount) + 127, a: color.a }; } Push brighter and darker colors even farther away from the median
54. 5. Filters Point filters : greyscale
55. 5. Filters Point filters : greyscale var rlum = 0.2126; var glum = 0.7152; var blum = 0.0722; // rlum + glum + blum = 1 function greyscale(color){ var brightness= (rlum*color.r) + (glum*color.g) + (blum*color.b); return { r: brightness, g: brightness, b: brightness, a: color.a }; } Find the brightness of the color and apply it equally to the three color channels.
56. 5. Filters Point filters : duotone
57. 5. Filters Point filters : duotone function duotone(color, darkcolor, brightcolor){ var newcolor = greyscale(color); newcolor.r = normalize(newcolor.r, darkcolor.r, brightcolor.r); newcolor.g = normalize(newcolor.g, darkcolor.g, brightcolor.g); newcolor.b = normalize(newcolor.b, darkcolor.b, brightcolor.b); function normalize(value, outMin, outMax){ return value / 255 * (outMax - outMin) + outMin; } return newcolor; } A colorized greyscale
58. 5. Filters Convolution filters
59. 5. Filters Convolution filters
60. 5. Filters Convolution filters
61. 5. Filters Convolution filters • It must have an odd number of cells (it must have a central cell) • It is usually square • The greater the absolute weights around the center, the greater the effect. • The greater the central absolute weight, the lesser the effect. • In JavaScript, represent it as an Array, the weights ordered from left to right and top to down.
62. 5. Filters Convolution filters
63. 5. Filters Convolution filters
64. 5. Filters Convolution filters
65. 5. Filters Convolution filters For every pixel of the image, sum the products of the color value of the current pixel, or a neighbor of it, with the corresponding value of the filter matrix, and divide that sum by the sum of the weights of the matrix (if not 0).
66. 5. Filters Convolution filters : sharpen
67. 5. Filters Convolution filters : sharpen 0 -2 0 -2 16 -2 0 -2 0 0 0 0 0 1 0 0 0 0
68. 5. Filters Convolution filters : blur 0 1 0 1 9 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 Blur Motion (directional) blur
69. 5. Filters Convolution filters : emboss -2 -1 0 -1 2 1 0 1 2
70. 5. Filters Convolution filters Always use an empty ImageData as “buffer” Because we are reading neighbor pixels, we must not change the source.
71. 5. Filters Convolution filters var buffer = context.createImageData(width,height); var bufferData = buffer.data; var image = context.getImageData(0,0,width,height); Var imageData = image.data; Always use an empty ImageData as “buffer”
72. 5. Filters Convolution filters var buffer = context.createImageData(width,height); var bufferData = buffer.data; var image = context.getImageData(0,0,width,height); Var imageData = image.data; //loop through every pixel in imageData //loop through the matrix and compute the new pixel value //set the new value in bufferData context.putImageData(buffer, x,y); Always use an empty ImageData as “buffer”
73. 5. Filters Convolution filters Avoid large matrices 0 0 0 0 1 0 0 0 0 0 0 0 1 2 1 0 0 0 0 0 1 2 3 2 1 0 0 0 1 2 3 4 3 2 1 0 1 2 3 4 5 4 3 2 1 0 1 2 3 4 3 2 1 0 0 0 1 2 3 2 1 0 0 0 0 0 1 2 1 0 0 0 0 0 0 0 1 0 0 0 0 Because we have to work with nested loops. !
74. 6. Pixel displacement (just a taste of it)
75. 6. Pixel displacement Moving pixels and creating new ones
76. 6. Pixel displacement Moving pixels and creating new ones
77. 6. Pixel displacement Moving pixels and creating new ones • Nearest neighbor (pixel art) • Linear interpolation • Bicubic interpolation (best quality, lower performance) Requires interpolation algorithms:
78. Grab it canvas.addEventListener("click", saveImage); function saveImage(evt){ var imageData = canvas.toDataURL("image/png"); window.open(imageData,'userimage',...) imageel.src = imageData; }
79. 7. Optimization
80. 7. Optimization 1. Write on an empty ImageData (buffer) var bufferImage = context.createImageData(width, height); var bufferData = bufferImage.data;
81. 7. Optimization 2. If you are changing just a part of the image, define an enclosing rectangular area (x, y, w, h) and process just that area. var bufferImage = context.createImageData(w,h); var sourceImage = context.getImageData(x, y, w, h) Apply the end result with: context.putImageData(bufferImage,0,0,x,y,w,h)
82. 7. Optimization 3. You can apply multiple point filters on the same loop (but no convolution filters though) setPixel(x,y,constrast(brightness(greyscale(getPixel(x,y)))));
83. 7. Optimization 4. Avoid large convolution matrices if you don't need greater precision. These will do: 0 0 1 0 0 0 1 2 1 0 1 2 3 2 1 0 1 2 1 0 0 0 1 0 0 0 1 0 1 9 1 0 1 0 3 x 3 5 x 5
84. 7. Optimization 5. If you are animating effects, use requestAnimationFrame window.requestAnimationFrame = (function(){ return window.requestAnimationFrame || window.webkitRequestAnimationFrame || window.mozRequestAnimationFrame || window.oRequestAnimationFrame || window.msRequestAnimationFrame || function(callback, element){ window.setTimeout(callback, 1000 / 33); }; })();
85. 7. Optimization 5. If you are animating effects, use requestAnimationFrame window.requestAnimationFrame = (function(){ return window.requestAnimationFrame || window.webkitRequestAnimationFrame || window.mozRequestAnimationFrame || window.oRequestAnimationFrame || window.msRequestAnimationFrame || function(callback, element){ window.setTimeout(callback, 1000 / 33); }; })(); (function animloop(time){ requestAnimationFrame(animloop, domElement); //update your domElement now }());
86. 7. Optimization 6. If you are animating effects, webworkers //webworker.js (function(){ this.addEventListener('message', function(evt) { var source = evt.data.imgdata; var buffer = evt.data.buffer; //process postMessage({"buffer":buffer}); }, false); }());
87. 7. Optimization 6. If you are animating effects, webworkers //webworker.js (function(){ this.addEventListener('message', function(evt) { var source = evt.data.imgdata; var buffer = evt.data.buffer; //process postMessage({"buffer":buffer}); }, false); }()); //application.js filtersWorker = new Worker('js/webworker.js'); filtersWorker.onmessage = onWorkerResponse; function onWorkerResponse(evt){ context.putImageData(evt.data.buffer,0,0); }
88. 7. Optimization 6. If you are animating effects, webworkers //webworker.js (function(){ this.addEventListener('message', function(evt) { var source = evt.data.imgdata; var buffer = evt.data.buffer; //process postMessage({"buffer":buffer}); }, false); }()); //application.js filtersWorker = new Worker('js/webworker.js'); filtersWorker.onmessage = onWorkerResponse; function onWorkerResponse(evt){ context.putImageData(evt.data.buffer,0,0); } filtersWorker.postMessage({'imgdata':context.getImageData(0,0, w, h), 'buffer':context.createImageData(w, h)});
89. 8. Libraries
90. 8. Libraries Pixastic http://www.pixastic.com https://github.com/jseidelin/pixastic/
91. 8. Libraries Pixastic http://www.pixastic.com https://github.com/jseidelin/pixastic/ • A good set of point and convolution filters. • Supports applying the filter to just a rectangular area.
92. 8. Libraries Pixastic http://www.pixastic.com https://github.com/jseidelin/pixastic/ • A good set of point and convolution filters. • Supports applying the filter to just a rectangular area. Usage 1: Static call passing an image or canvas element, a filter name and options. A canvas will be returned.
93. 8. Libraries Pixastic http://www.pixastic.com https://github.com/jseidelin/pixastic/ • A good set of point and convolution filters. • Supports applying the filter to just a rectangular area. Usage 1: Static call passing an image or canvas element, a filter name and options. A canvas will be returned. Usage 2: Declare image or canvas elements with the class “pixastic” and the filter name and options. These elements will be replaced by a canvas.
94. 8. Libraries PaintbrushJS http://mezzoblue.github.com/PaintbrushJS/demo/ https://github.com/mezzoblue/PaintbrushJS
95. 8. Libraries PaintbrushJS http://mezzoblue.github.com/PaintbrushJS/demo/ https://github.com/mezzoblue/PaintbrushJS • A small set of filters. • Also replaces background images set with css on non image elements. • Not object oriented, nor namespaced.
96. 8. Libraries PaintbrushJS http://mezzoblue.github.com/PaintbrushJS/demo/ https://github.com/mezzoblue/PaintbrushJS • A small set of filters. • Also replaces background images set with css on non image elements. • Not object oriented, nor namespaced. Usage: Declare the elements to be processed with a class naming the filter. Provide the options in “data-” prefixed attribute.
97. 8. Libraries PaintbrushJS http://mezzoblue.github.com/PaintbrushJS/demo/ https://github.com/mezzoblue/PaintbrushJS • A small set of filters. • Also replaces background images set with css on non image elements. • Not object oriented, nor namespaced. Usage: Declare the elements to be processed with a class naming the filter. Provide the options in “data-” prefixed attribute. The element will be hidden, copied, the copy inserted in the same place and its source attribute will be filled with the bytestream output by the toDataUrl canvas method: <img src="data:image/png;base64,iVBORw0...
98. 8. Libraries FILTER.js https://github.com/foo123/FILTER.js
99. 8. Libraries FILTER.js https://github.com/foo123/FILTER.js • Reasonable support for point filters (applied as a color matrix) and good support for convolution filters, including motion blur. • Supports blend modes (mixing two images) and even pixel displacement based on the Adobe ActionScript version. • The fastest of these libraries.
100. 8. Libraries FILTER.js https://github.com/foo123/FILTER.js • Reasonable support for point filters (applied as a color matrix) and good support for convolution filters, including motion blur. • Supports blend modes (mixing two images) and even pixel displacement based on the Adobe ActionScript version. • The fastest of these libraries. Usage: Object oriented. Object instantiation using a namespace. Supports concatenation of methods by returning the object itself.
101. 8. Libraries Aviary http://www.aviary.com/features • Embeddable user interface component • Tools for enhancement • Effects • Customizable look and feel
102. • Crop, resize, rotate and save • Access pixels • “Paint” pixels • Change pixels based on their value • Change pixels based on their neighbors • Do it efficiently • Delegate the hard work to libraries To do next: have fun! Looking back
103. Presentation slides https://speakerdeck.com/sergiopereira/image-processing-on-canvas Demos source code https://github.com/sergiompereira/Codebits012-demos
104. Thank you! Questions? Sérgio Pereira sergio-pt@sapo.pt sergiompereira.com https://github.com/sergiompereira