Video edition with CSS & JavaScript

Transcript

1. Video Editing with CSS and JavaScript Eva Ferreira – Smashing TV
2. Hi! I’m Eva
3. Hi! I’m Eva • Front-end Developer by day • Protector of useless creativity at night • Mozilla Tech Speaker • Organizer of CSSConf Argentina @evaferreira92
4. Hi! I’m Maria Evangelina Ferreira Kuzminski • Front-end Developer by day • Protector of useless creativity at night • Mozilla Tech Speaker • Organizer of CSSConf Argentina @evaferreira92
5. @evaferreira92
6. Rotoscoping
7. Take on me – A-ha 1985
8. 21st Century Breakdown – Green Day 2009
9. Sheezus – Lily Allen 2014
10. Hard Times – Paramore 2017
11. Caught in the Middle – Paramore 2018
12. Rotoscoping 1. Record a video 2. Draw on top 15 to 25 images per second @evaferreira92
17. Animation on the web.
18. Traditional Web • Stop Motion  Rachel Nabors • Motion Graphics  Chris Gannon @evaferreira92
19. Can we go further?
20. Can we make rotoscoping on the web? No.
22. Interactive media Take Rotoscoping idea into the web.
23. Rotoscoping = Animation above video
24. With CSS or with JavaScript
25. The CSS Version 1. “Normal” video 2. “Animation” video @evaferreira92
26. https://vimeo.com/58933055
27. https://vimeo.com/49581518
28. Mix them
29. Blend them
30. Blend Modes
31. mix-blend-mode: luminosity;
32. Make it interactive
33. Add clip-path and filters
34. Interactive wow-ness Ok-ish browser support
35. https://caniuse.com/#feat=css-mixblendmode
36. Let’s try with JavaScript now
39. Chroma Key
40. Chroma Key • Green or blue background • Really good light @evaferreira92
41. No natural light
42. Chroma key with user’s camera
43. Chroma key in the browser 1. Take the user’s camera input 1. Put it in a video tag 2. Copy that into a canvas 3. Process canvas 1. Remove green or blue 2. Show background image or video @evaferreira92
44. 1. Take the user’s camera input
45. Take video input @evaferreira92 navigator.mediaDevices.getUserMedia({video: true}) .then() .catch();
46. Put it in a video tag @evaferreira92 <video id="video" controls></video>
47. Put it in a video tag @evaferreira92 .then(function (stream) { video.srcObject = stream; }) .catch(function (error) { console.log("Error:", error); }); navigator.mediaDevices.getUserMedia({video: true})
48. @evaferreira92
49. 2. Copy video into canvas
50. Create canvas @evaferreira92 <canvas id="canvas" class="canvas"></canvas> c1 = document.getElementById("canvas"); ctx1 = this.c1.getContext("2d");
51. Create the draw function @evaferreira92 function draw () { ctx1.drawImage(video, 0, 0, this.width, this.height); }
52. Update the canvas function time_callback () { if (video.paused || video.ended) { return; } draw(); // Update setTimeout(function () { this.time_callback(); }, 0); };
53. @evaferreira92
54. 3. Process the canvas Remove green or blue background
55. Pixel manipulation
56. What is a pixel?
57. @evaferreira92
58. @evaferreira92 Pixel
59. RGB Format
60. @evaferreira92 = 227 45 145
61. Alpha Channel @evaferreira92
62. Ask for pixel information (RGBa) • The red value • The green value • The blue value • The Alpha value @evaferreira92
63. Ask for pixel information (RGBa) • The red value • The green value • The blue value • The Alpha value  It will always return 255  … Unless we change it @evaferreira92
64. Making Chroma happen. 1. Find the green pixels 2. Remove them 1. Alpha value of 0 @evaferreira92
65. Find the green pixels
66. Ask for pixel data @evaferreira92 function findGreen () { // Ask for color data let frame = this.ctx1.getImageData(0, 0, this.width, this.height); ... }
67. @evaferreira92
68. @evaferreira92 1,228,800 640 x 480 x 4 =
69. @evaferreira92
70. @evaferreira92
71. @evaferreira92
72. @evaferreira92 Alpha value
73. @evaferreira92 R G B A R G B A R G B A R G B A R G B A
74. @evaferreira92 } rgba(55, 74, 86, 1)
75. Find the green pixels @evaferreira92 function findGreen () { ... let allData = frame.data.length / 4; for (let i = 0; i < allData; i++) { ... } }
76. @evaferreira92 // Save all the data of 1.2m pixels let allData = frame.data.length / 4; // Go through it and identify it for (let i = 0; i < allData; i++) { let r = frame.data[i * 4 + 0]; let g = frame.data[i * 4 + 1]; let b = frame.data[i * 4 + 2]; // Ask if the greatest value is green if (g > r && g > b) { // Remove opacity frame.data[i * 4 + 3] = 0; } }
77. Return the updated data @evaferreira92 function findGreen () { let frame = this.ctx1.getImageData(0, 0, this.width, this.height); let l = frame.data.length / 4; for (let i = 0; i < l; i++) { ... frame.data[i * 4 + 3] = 0; } this.ctx1.putImageData(frame, 0, 0); return; }
78. @evaferreira92
79. Improve it 1. Better light 2. Remove only if the green is > 100 @evaferreira92 if (g > r && g > b) { if (g > 100 ) { frame.data[i * 4 + 3] = 0; } }
81. @evaferreira92
82. Live demo
83. Invisibility Cloak
84. @evaferreira92
85. MOAR Pixel Manipulation
86. @evaferreira92
88. @evaferreira92
89. Copy content into canvases @evaferreira92 <canvas id="canvas1" class="canvas canvas1"></canvas> <canvas id="canvas2" class="canvas canvas2"></canvas> <canvas id="canvas3" class="canvas canvas3"></canvas>
90. Copy content into canvases @evaferreira92 function findColor (color) { let frame1 = this.ctx1.getImageData(0, 0, this.width, this.height); let frame2 = this.ctx1.getImageData(0, 0, this.width, this.height); let frame3 = this.ctx1.getImageData(0, 0, this.width, this.height); ... this.ctx1.putImageData(frame1, 0, 0); this.ctx2.putImageData(frame2, 0, 0); this.ctx3.putImageData(frame3, 0, 0); return; }
91. @evaferreira92
92. Turn what’s left into pink @evaferreira92 for (let i = 0; i < l; i++) { ... if (g > r && g > b) { frame1.data[i * 4 + 3] = 0; frame2.data[i * 4 + 3] = 0; frame3.data[i * 4 + 3] = 0; } else { frame2.data[i * 4 + 0] = 200; frame3.data[i * 4 + 0] = 200; frame2.data[i * 4 + 2] = 200; frame3.data[i * 4 + 2] = 200; } } 0 = Red 1 = Green 2 = Blue 3 = Alpha
93. @evaferreira92
94. @evaferreira92
95. @evaferreira92
96. @evaferreira92
97. Endless possibilities. Runs in your browser with JavaScript. No other software required.
98. Protect the useless. @evaferreira92
99. Thank you :) • Demos & links: https://github.com/evaferreira/smashing-tv • Green screen online: https://chroma-code.netlify.com/ • @evaferreira92