$30 off During Our Annual Pro Sale. View Details »

Video edition with CSS & JavaScript

Video edition with CSS & JavaScript

Eva Ferreira

June 19, 2019
Tweet

More Decks by Eva Ferreira

Other Decks in Programming

Transcript

  1. Video Editing with
    CSS and JavaScript
    Eva Ferreira – Smashing TV

    View Slide

  2. Hi! I’m Eva

    View Slide

  3. Hi! I’m Eva
    • Front-end Developer by day
    • Protector of useless creativity at night
    • Mozilla Tech Speaker
    • Organizer of CSSConf Argentina
    @evaferreira92

    View Slide

  4. Hi! I’m Maria Evangelina
    Ferreira Kuzminski
    • Front-end Developer by day
    • Protector of useless creativity at night
    • Mozilla Tech Speaker
    • Organizer of CSSConf Argentina
    @evaferreira92

    View Slide

  5. @evaferreira92

    View Slide

  6. Rotoscoping

    View Slide

  7. Take on me – A-ha
    1985

    View Slide

  8. 21st Century Breakdown – Green Day
    2009

    View Slide

  9. Sheezus – Lily Allen
    2014

    View Slide

  10. Hard Times – Paramore
    2017

    View Slide

  11. Caught in the Middle – Paramore
    2018

    View Slide

  12. Rotoscoping
    1. Record a video
    2. Draw on top
    15 to 25 images per second
    @evaferreira92

    View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. Animation
    on the web.

    View Slide

  18. Traditional Web
    • Stop Motion
     Rachel Nabors
    • Motion Graphics
     Chris Gannon
    @evaferreira92

    View Slide

  19. Can we go further?

    View Slide

  20. Can we make
    rotoscoping on the
    web?
    No.

    View Slide

  21. View Slide

  22. Interactive media
    Take Rotoscoping idea into the web.

    View Slide

  23. Rotoscoping
    =
    Animation above video

    View Slide

  24. With CSS or
    with JavaScript

    View Slide

  25. The CSS Version
    1. “Normal” video
    2. “Animation” video
    @evaferreira92

    View Slide

  26. https://vimeo.com/58933055

    View Slide

  27. https://vimeo.com/49581518

    View Slide

  28. Mix them

    View Slide

  29. Blend them

    View Slide

  30. Blend Modes

    View Slide

  31. mix-blend-mode: luminosity;

    View Slide

  32. Make it interactive

    View Slide

  33. Add clip-path and filters

    View Slide

  34. Interactive wow-ness
    Ok-ish browser support

    View Slide

  35. https://caniuse.com/#feat=css-mixblendmode

    View Slide

  36. Let’s try with
    JavaScript now

    View Slide

  37. View Slide

  38. View Slide

  39. Chroma Key

    View Slide

  40. Chroma Key
    • Green or blue background
    • Really good light
    @evaferreira92

    View Slide

  41. No natural light

    View Slide

  42. Chroma key
    with user’s camera

    View Slide

  43. Chroma key in the browser
    1. Take the user’s camera input
    1. Put it in a video tag
    2. Copy that into a canvas
    3. Process canvas
    1. Remove green or blue
    2. Show background image or video
    @evaferreira92

    View Slide

  44. 1.
    Take the user’s
    camera input

    View Slide

  45. Take video input
    @evaferreira92
    navigator.mediaDevices.getUserMedia({video: true})
    .then()
    .catch();

    View Slide

  46. Put it in a video tag
    @evaferreira92

    View Slide

  47. Put it in a video tag
    @evaferreira92
    .then(function (stream) {
    video.srcObject = stream;
    })
    .catch(function (error) {
    console.log("Error:", error);
    });
    navigator.mediaDevices.getUserMedia({video: true})

    View Slide

  48. @evaferreira92

    View Slide

  49. 2.
    Copy video
    into canvas

    View Slide

  50. Create canvas
    @evaferreira92

    c1 = document.getElementById("canvas");
    ctx1 = this.c1.getContext("2d");

    View Slide

  51. Create the draw function
    @evaferreira92
    function draw () {
    ctx1.drawImage(video, 0, 0, this.width, this.height);
    }

    View Slide

  52. Update the canvas
    function time_callback () {
    if (video.paused || video.ended) {
    return;
    }
    draw();
    // Update
    setTimeout(function () {
    this.time_callback();
    }, 0);
    };

    View Slide

  53. @evaferreira92

    View Slide

  54. 3.
    Process the canvas
    Remove green or blue background

    View Slide

  55. Pixel manipulation

    View Slide

  56. What is a pixel?

    View Slide

  57. @evaferreira92

    View Slide

  58. @evaferreira92
    Pixel

    View Slide

  59. RGB Format

    View Slide

  60. @evaferreira92
    =
    227 45 145

    View Slide

  61. Alpha Channel
    @evaferreira92

    View Slide

  62. Ask for pixel information (RGBa)
    • The red value
    • The green value
    • The blue value
    • The Alpha value
    @evaferreira92

    View Slide

  63. Ask for pixel information (RGBa)
    • The red value
    • The green value
    • The blue value
    • The Alpha value
     It will always return 255
     … Unless we change it
    @evaferreira92

    View Slide

  64. Making Chroma happen.
    1. Find the green pixels
    2. Remove them
    1. Alpha value of 0
    @evaferreira92

    View Slide

  65. Find the
    green pixels

    View Slide

  66. Ask for pixel data
    @evaferreira92
    function findGreen () {
    // Ask for color data
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);
    ...
    }

    View Slide

  67. @evaferreira92

    View Slide

  68. @evaferreira92
    1,228,800
    640 x 480 x 4
    =

    View Slide

  69. @evaferreira92

    View Slide

  70. @evaferreira92

    View Slide

  71. @evaferreira92

    View Slide

  72. @evaferreira92
    Alpha value

    View Slide

  73. @evaferreira92
    R
    G
    B
    A
    R
    G
    B
    A
    R
    G
    B
    A
    R
    G
    B
    A
    R
    G
    B
    A

    View Slide

  74. @evaferreira92
    } rgba(55, 74, 86, 1)

    View Slide

  75. Find the green pixels
    @evaferreira92
    function findGreen () {
    ...
    let allData = frame.data.length / 4;
    for (let i = 0; i < allData; i++) {
    ...
    }
    }

    View Slide

  76. @evaferreira92
    // Save all the data of 1.2m pixels
    let allData = frame.data.length / 4;
    // Go through it and identify it
    for (let i = 0; i < allData; i++) {
    let r = frame.data[i * 4 + 0];
    let g = frame.data[i * 4 + 1];
    let b = frame.data[i * 4 + 2];
    // Ask if the greatest value is green
    if (g > r && g > b) {
    // Remove opacity
    frame.data[i * 4 + 3] = 0;
    }
    }

    View Slide

  77. Return the updated data
    @evaferreira92
    function findGreen () {
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);
    let l = frame.data.length / 4;
    for (let i = 0; i < l; i++) {
    ...
    frame.data[i * 4 + 3] = 0;
    }
    this.ctx1.putImageData(frame, 0, 0);
    return;
    }

    View Slide

  78. @evaferreira92

    View Slide

  79. Improve it
    1. Better light
    2. Remove only if the green is > 100
    @evaferreira92
    if (g > r && g > b) {
    if (g > 100 ) {
    frame.data[i * 4 + 3] = 0;
    }
    }

    View Slide

  80. View Slide

  81. @evaferreira92

    View Slide

  82. Live demo

    View Slide

  83. Invisibility Cloak

    View Slide

  84. @evaferreira92

    View Slide

  85. MOAR
    Pixel Manipulation

    View Slide

  86. @evaferreira92

    View Slide

  87. View Slide

  88. @evaferreira92

    View Slide

  89. Copy content into canvases
    @evaferreira92



    View Slide

  90. Copy content into canvases
    @evaferreira92
    function findColor (color) {
    let frame1 = this.ctx1.getImageData(0, 0, this.width, this.height);
    let frame2 = this.ctx1.getImageData(0, 0, this.width, this.height);
    let frame3 = this.ctx1.getImageData(0, 0, this.width, this.height);
    ...
    this.ctx1.putImageData(frame1, 0, 0);
    this.ctx2.putImageData(frame2, 0, 0);
    this.ctx3.putImageData(frame3, 0, 0);
    return;
    }

    View Slide

  91. @evaferreira92

    View Slide

  92. Turn what’s left into pink
    @evaferreira92
    for (let i = 0; i < l; i++) {
    ...
    if (g > r && g > b) {
    frame1.data[i * 4 + 3] = 0;
    frame2.data[i * 4 + 3] = 0;
    frame3.data[i * 4 + 3] = 0;
    } else {
    frame2.data[i * 4 + 0] = 200;
    frame3.data[i * 4 + 0] = 200;
    frame2.data[i * 4 + 2] = 200;
    frame3.data[i * 4 + 2] = 200;
    }
    }
    0 = Red
    1 = Green
    2 = Blue
    3 = Alpha

    View Slide

  93. @evaferreira92

    View Slide

  94. @evaferreira92

    View Slide

  95. @evaferreira92

    View Slide

  96. @evaferreira92

    View Slide

  97. Endless
    possibilities.
    Runs in your browser with JavaScript.
    No other software required.

    View Slide

  98. Protect the useless.
    @evaferreira92

    View Slide

  99. Thank you :)
    • Demos & links: https://github.com/evaferreira/smashing-tv
    • Green screen online: https://chroma-code.netlify.com/
    • @evaferreira92

    View Slide