Save 37% off PRO during our Black Friday Sale! »

Everything they don’t tell you about video in the browser

8ec1383b240b5ba15ffb9743fceb3c0e?s=47 Phil Nash
October 27, 2018

Everything they don’t tell you about video in the browser

So, you want to make a video call in the browser? There are libraries available to handle the WebRTC for you, but what about everything else you want in that call? The browsers aren’t going to make that as easy for you I’m afraid.

This talk will be a deep dive into the features you want to see in a video call. We’ll start with a video chat and look into screen sharing, switching cameras, the web audio API and the exciting world of media constraints. You’ll leave this talk understanding the full promise of getUserMedia!

--

Links:

Screen capture in Chrome: https://www.twilio.com/blog/2017/10/screen-capture-in-google-chrome.html
Screen capture in Firefox: https://www.twilio.com/blog/2017/10/screen-capture-in-firefox.html
Screen capture in Edge: https://www.twilio.com/blog/2018/05/screen-capture-in-microsoft-edge.html
Add screen sharing to a Twilio Video application https://www.twilio.com/blog/2018/01/screen-sharing-twilio-video.html
Choosing cameras in JavaScript with the mediaDevices API: https://www.twilio.com/blog/2018/04/choosing-cameras-javascript-mediadevices-api.html
Switching cameras during a video chat with Twilio Video: https://www.twilio.com/blog/2018/06/switching-cameras-twilio-video-chat.html
Audio visualisation with the Web Audio API and React: https://www.twilio.com/blog/audio-visualisation-web-audio-api--react

8ec1383b240b5ba15ffb9743fceb3c0e?s=128

Phil Nash

October 27, 2018
Tweet

Transcript

  1. EVERYTHING THEY DON’T TELL YOU ABOUT VIDEO IN THE BROWSER

    @philnash
  2. Phil Nash @philnash http://philna.sh philnash@twilio.com

  3. WEB PLATFORM @philnash

  4. WEBRTC @philnash

  5. PEER TO PEER @philnash

  6. WebRTC Support ✅ ✅ ✅ ✅ ✅ @philnash

  7. SORT OF... @philnash

  8. THE ARCHITECTURE OF WEBRTC @philnash

  9. WEBRTC TODAY @philnash

  10. getUserMedia @philnash

  11. SELECTING CAMERAS @philnash

  12. THIS SHOULD BE EASY @philnash

  13. Selecting cameras navigator.mediaDevices.getUserMedia({ video: true }) 01. 02. 03. @philnash

  14. Selecting cameras navigator.mediaDevices.enumerateDevices() .then(devices => { // woo! }) 01.

    02. 03. 04. @philnash
  15. BUT... @philnash

  16. MediaConstraints @philnash

  17. Selecting cameras navigator.mediaDevices.getUserMedia({ video: ??? }) 01. 02. 03. @philnash

  18. facingMode @philnash

  19. Selecting cameras navigator.mediaDevices.getUserMedia({ video: { facingMode: ['user' | 'environment' |

    'left' | 'right'] } }) 01. 02. 03. 04. 05. @philnash
  20. deviceId @philnash

  21. Selecting cameras navigator.mediaDevices.enumerateDevices().then(devices => { navigator.mediaDevices.getUserMedia({ video: { deviceId: devices[0].deviceId

    } }) }); 01. 02. 03. 04. 05. 06. 07. @philnash
  22. BUT... @philnash

  23. Selecting cameras navigator.mediaDevices.enumerateDevices().then(devices => { navigator.mediaDevices.getUserMedia({ video: { deviceId: {

    exact: devices[0].deviceId } } }) }); 01. 02. 03. 04. 05. 06. 07. @philnash
  24. More constraints const constraints = { width: { min: 640,

    ideal: 1920, max: 1920 }, height: { min: 400, ideal: 1080 }, aspectRatio: 1.777777778, frameRate: { max: 30 }, facingMode: { exact: "user" } }; 01. 02. 03. 04. 05. 06. 07. @philnash
  25. OverconstrainedError @philnash

  26. SCREEN SHARING @philnash

  27. Screen capture support ✅ @philnash

  28. ✨getDisplayMedia✨ @philnash

  29. Screen capture navigator.getDisplayMedia() .then(stream => { // the stream is

    the display }); 01. 02. 03. 04. @philnash
  30. PLUGINS @philnash

  31. Screen capture - Chrome chrome.runtime.onMessageExternal.addListener((message, sender, sendResponse) =>; { const

    sources = message.sources; const tab = sender.tab; chrome.desktopCapture.chooseDesktopMedia(sources, tab, (streamId) => { if (!streamId) { sendResponse({ type: 'error', message: 'Failed to get stream ID' }); } else { sendResponse({ type: 'success', streamId: streamId }); } }); return true; }); 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. 13. 14. 15. 16. 17. 18. @philnash
  32. Screen capture - Chrome chrome.runtime.sendMessage(EXTENSION_ID, request, response => { if

    (response && response.type === 'success') { navigator.mediaDevices .getUserMedia({ video: { mandatory: { chromeMediaSource: 'desktop', chromeMediaSourceId: response.streamId } } }) } 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. @philnash
  33. NONSTANDARD BEHAVIOUR @philnash

  34. Screen capture - Firefox navigator.mediaDevices.getUserMedia({ video: { mediaSource: 'screen' }

    }).then(returnedStream => { // use the stream }); 01. 02. 03. 04. 05. 06. 07. @philnash
  35. Screen capture - Firefox function isFirefox() { var mediaSourceSupport =

    !!navigator.mediaDevices.getSupportedConstraints() .mediaSource; var matchData = navigator.userAgent.match(/Firefox\/(\d+)/); var firefoxVersion = 0; if (matchData && matchData[1]) { firefoxVersion = parseInt(matchData[1], 10); } return mediaSourceSupport && firefoxVersion >= 52; } 01. 02. 03. 04. 05. 06. 07. 08. 09. @philnash
  36. STANDARDS COMING SOON @philnash

  37. WEB AUDIO @philnash

  38. Web audio const ctx = new AudioContext(); const oscillator =

    ctx.createOscillator(); oscillator.frequency = 400; oscillator.connect(ctx.destination); oscillator.start() 01. 02. 03. 04. 05. @philnash
  39. Web audio const ctx = new AudioContext(); const analyser =

    ctx.createAnalyser(); 01. 02. @philnash
  40. WEB PLATFORM++ @philnash

  41. YOUR SKILLS++ @philnash

  42. EVERYTHING THEY DON’T TELL YOU ABOUT VIDEO IN THE BROWSER

    @philnash
  43. Thanks! @philnash http://philna.sh philnash@twilio.com