Everything they don’t tell you about video in the browser

So, you want to make a video call in the browser? There are libraries available to handle the WebRTC for you, but what about everything else you want in that call? The browsers aren’t going to make that as easy for you I’m afraid.

This talk will be a deep dive into the features you want to see in a video call. We’ll start with a video chat and look into screen sharing, switching cameras, the web audio API and the exciting world of media constraints. You’ll leave this talk understanding the full promise of getUserMedia!

--

Links:

Screen capture in Chrome: https://www.twilio.com/blog/2017/10/screen-capture-in-google-chrome.html
Screen capture in Firefox: https://www.twilio.com/blog/2017/10/screen-capture-in-firefox.html
Screen capture in Edge: https://www.twilio.com/blog/2018/05/screen-capture-in-microsoft-edge.html
Add screen sharing to a Twilio Video application https://www.twilio.com/blog/2018/01/screen-sharing-twilio-video.html
Choosing cameras in JavaScript with the mediaDevices API: https://www.twilio.com/blog/2018/04/choosing-cameras-javascript-mediadevices-api.html
Switching cameras during a video chat with Twilio Video: https://www.twilio.com/blog/2018/06/switching-cameras-twilio-video-chat.html
Audio visualisation with the Web Audio API and React: https://www.twilio.com/blog/audio-visualisation-web-audio-api--react

Transcript

1. EVERYTHING THEY DON’T TELL YOU ABOUT VIDEO IN THE BROWSER @philnash
2. Phil Nash @philnash http://philna.sh philnash@twilio.com
3. WEB PLATFORM @philnash
4. WEBRTC @philnash
5. PEER TO PEER @philnash
6. WebRTC Support ✅ ✅ ✅ ✅ ✅ @philnash
7. SORT OF... @philnash
8. THE ARCHITECTURE OF WEBRTC @philnash
9. WEBRTC TODAY @philnash
10. getUserMedia @philnash
11. SELECTING CAMERAS @philnash
12. THIS SHOULD BE EASY @philnash
13. Selecting cameras navigator.mediaDevices.getUserMedia({ video: true }) 01. 02. 03. @philnash
14. Selecting cameras navigator.mediaDevices.enumerateDevices() .then(devices => { // woo! }) 01. 02. 03. 04. @philnash
15. BUT... @philnash
16. MediaConstraints @philnash
17. Selecting cameras navigator.mediaDevices.getUserMedia({ video: ??? }) 01. 02. 03. @philnash
18. facingMode @philnash
19. Selecting cameras navigator.mediaDevices.getUserMedia({ video: { facingMode: ['user' | 'environment' | 'left' | 'right'] } }) 01. 02. 03. 04. 05. @philnash
20. deviceId @philnash
21. Selecting cameras navigator.mediaDevices.enumerateDevices().then(devices => { navigator.mediaDevices.getUserMedia({ video: { deviceId: devices[0].deviceId } }) }); 01. 02. 03. 04. 05. 06. 07. @philnash
22. BUT... @philnash
23. Selecting cameras navigator.mediaDevices.enumerateDevices().then(devices => { navigator.mediaDevices.getUserMedia({ video: { deviceId: { exact: devices[0].deviceId } } }) }); 01. 02. 03. 04. 05. 06. 07. @philnash
24. More constraints const constraints = { width: { min: 640, ideal: 1920, max: 1920 }, height: { min: 400, ideal: 1080 }, aspectRatio: 1.777777778, frameRate: { max: 30 }, facingMode: { exact: "user" } }; 01. 02. 03. 04. 05. 06. 07. @philnash
25. OverconstrainedError @philnash
26. SCREEN SHARING @philnash
27. Screen capture support ✅ @philnash
28. ✨getDisplayMedia✨ @philnash
29. Screen capture navigator.getDisplayMedia() .then(stream => { // the stream is the display }); 01. 02. 03. 04. @philnash
30. PLUGINS @philnash
31. Screen capture - Chrome chrome.runtime.onMessageExternal.addListener((message, sender, sendResponse) =>; { const sources = message.sources; const tab = sender.tab; chrome.desktopCapture.chooseDesktopMedia(sources, tab, (streamId) => { if (!streamId) { sendResponse({ type: 'error', message: 'Failed to get stream ID' }); } else { sendResponse({ type: 'success', streamId: streamId }); } }); return true; }); 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. 13. 14. 15. 16. 17. 18. @philnash
32. Screen capture - Chrome chrome.runtime.sendMessage(EXTENSION_ID, request, response => { if (response && response.type === 'success') { navigator.mediaDevices .getUserMedia({ video: { mandatory: { chromeMediaSource: 'desktop', chromeMediaSourceId: response.streamId } } }) } 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. @philnash
33. NONSTANDARD BEHAVIOUR @philnash
34. Screen capture - Firefox navigator.mediaDevices.getUserMedia({ video: { mediaSource: 'screen' } }).then(returnedStream => { // use the stream }); 01. 02. 03. 04. 05. 06. 07. @philnash
35. Screen capture - Firefox function isFirefox() { var mediaSourceSupport = !!navigator.mediaDevices.getSupportedConstraints() .mediaSource; var matchData = navigator.userAgent.match(/Firefox\/(\d+)/); var firefoxVersion = 0; if (matchData && matchData[1]) { firefoxVersion = parseInt(matchData[1], 10); } return mediaSourceSupport && firefoxVersion >= 52; } 01. 02. 03. 04. 05. 06. 07. 08. 09. @philnash
36. STANDARDS COMING SOON @philnash
37. WEB AUDIO @philnash
38. Web audio const ctx = new AudioContext(); const oscillator = ctx.createOscillator(); oscillator.frequency = 400; oscillator.connect(ctx.destination); oscillator.start() 01. 02. 03. 04. 05. @philnash
39. Web audio const ctx = new AudioContext(); const analyser = ctx.createAnalyser(); 01. 02. @philnash
40. WEB PLATFORM++ @philnash
41. YOUR SKILLS++ @philnash
42. EVERYTHING THEY DON’T TELL YOU ABOUT VIDEO IN THE BROWSER @philnash
43. Thanks! @philnash http://philna.sh philnash@twilio.com