You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

WebRTC is the technology that powers video calls in browsers. What you might not know are all the features WebRTC unlocks for the web. In this talk we'll take a journey through WebRTC, from the primitives and protocols to the projects that have been built.

You'll learn the basics of getting two peers connected in a video chat and move on to advanced topics like camera control, screen sharing, audio analysis, using the data channel and more. We might even be able to solve the "You're on mute!" problem that plagues every video chat.

When you understand WebRTC and the related APIs you'll be amazed with what you can build.

--

Links:

Choosing cameras in JavaScript: https://www.twilio.com/blog/choosing-cameras-javascript-mediadevices-api-html
MediaDevices API examples: https://github.com/philnash/mediadevices-camera-selection
Screen capture with getDisplayMedia: https://www.twilio.com/blog/screen-sharing-javascript-twilio-programmable-video

RTC Diagnostics library. Test bitrate, audio input, audio output and video input in the browser: https://github.com/twilio/rtc-diagnostics/

Transcript

1. YOU'RE ON MUTE! WEBRTC AND OUR LIVES ON SCREEN @philnash
2. Phil Nash @philnash https://philna.sh philnash@twilio.com
3. YOU'RE ON MUTE! WEBRTC AND OUR LIVES ON SCREEN @philnash
4. OUR LIVES ON SCREEN @philnash
5. WEBRTC VS @philnash
6. GREAT FOR WORK, FRIENDS OR FAMILY @philnash
7. NOT SO GREAT FOR ONE OFFS @philnash
8. NOT SO GREAT FOR SPECIALIST APPS @philnash
9. WEBRTC IS FOR ALL THE OTHER APPLICATIONS @philnash
10. REAL-TIME VIDEO, AUDIO AND DATA EXPERIENCES FOR YOUR USERS @philnash
11. Today 1. Basics of WebRTC 2. WebRTC Usage 3. Controlling the user experience 4. Tools and takeaways @philnash
12. BASICS OF WEBRTC @philnash
13. @philnash
14. Signalling @philnash
15. Peer to peer @philnash
16. SFU @philnash
17. ICE • ICE — Interactive Connectivity Establishment • NAT — Network Address Translation • STUN — Session Traversal Utilities for NAT • TURN — Traversal Using Relays around NAT @philnash
18. THE APIS @philnash
19. getUserMedia navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(stream => { // ... }); 01. 02. 03. 04. @philnash
20. RTCPeerConnection const pc = new RTCPeerConnection({ iceServers }); pc.addEventListener("icecandidate", (candidate) => { // send candidate to signalling }); 01. 02. 03. 04. @philnash
21. RTCPeerConnection pc.addStream(stream); pc.createOffer().then(offer => { pc.setLocalDescription(offer); // send offer to peer }); 01. 02. 03. 04. 05. @philnash
22. RTCPeerConnection pc.setRemoteDescription(offer); pc.createAnswer().then(answer => { pc.setLocalDescription(answer); // send answer to peer }); 01. 02. 03. 04. 05. @philnash
23. RTCPeerConnection pc.setRemoteDescription(answer); pc.addEventListener("track", event => { // Connected! // show remote video in UI }); 01. 02. 03. 04. 05. @philnash
24. RTCDataChannel const dataChannel = pc.createDataChannel("my-data-channel"); dataChannel.addEventListener("message", event => { console.log(event.data); }); 01. 02. 03. 04. @philnash
25. RTCDataChannel pc.addEventListener("datachannel", event => { event.channel.addEventListener("open", () => { event.channel.send("Hi!"); }); }); 01. 02. 03. 04. 05. @philnash
26. WEBRTC USAGE @philnash
27. @philnash
28. @philnash
29. IMPROVING THE USER EXPERIENCE @philnash
30. MEDIA @philnash
31. getUserMedia navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(stream => { // ... }); 01. 02. 03. 04. @philnash
32. getUserMedia - Permissions navigator.mediaDevices.getUserMedia({ audio: true, video: true }) .then(stream => { // success }) .catch(error => { // permission denied }); 01. 02. 03. 04. 05. 06. @philnash
33. getUserMedia - Media Constraints { audio: true, video: true } 01. 02. 03. 04. @philnash
34. Media Constraints - facingMode { audio: true, video: { facingMode: "user" // "environment", "right", "left" } } 01. 02. 03. 04. 05. 06. @philnash
35. Media Constraints - deviceId { audio: true, video: { deviceId: { exact: device.id } } } 01. 02. 03. 04. 05. 06. @philnash
36. enumerateDevices navigator.mediaDevices.enumerateDevices() .then(devices => { devices.forEach(device => { console.log(`${device.id}: ${device.label}`); }); }); 01. 02. 03. 04. 05. 06. @philnash
37. enumerateDevices - Permissions navigator.mediaDevices.enumerateDevices() .then(devices => { devices.forEach(device => { console.log(device.label); // ??? }); }); 01. 02. 03. 04. 05. 06. @philnash
38. Media Constraints - resolution { audio: true, video: { width: 1920, height: 1080 } } 01. 02. 03. 04. 05. 06. 07. @philnash
39. Media Constraints - resolution { video: { width: { min: 320, max: 1920, ideal: 1920 }, height: { min: 240, max: 1080, ideal: 1080 } } } 01. 02. 03. 04. 05. 06. @philnash
40. Media Constraints - resolution { video: { width: { exact: 1920 }, height: { exact: 1080 } } } // => 🚨 OverconstrainedError 01. 02. 03. 04. 05. 06. 07. @philnash
41. Media Constraints - torch 🔦😄 { video: { advanced: [{ torch: true }], } } 01. 02. 03. 04. 05. @philnash
42. Media Constraints - torch 🔦😄 { video: { advanced: [{ torch: false }], // doesn't work 🤨 } } 01. 02. 03. 04. 05. @philnash
43. Applying media constraints videoTrack.applyConstraints({ width: 1920, height: 1080 }); 01. 02. 03. 04. @philnash
44. Applying media constraints videoTrack.applyConstraints({ deviceId: { exact: device.id } }); ❌ Don't do this! ❌ 01. 02. 03. @philnash
45. Changing cameras videoTrack.stop(); navigator.mediaDevices.getUserMedia({ video: { deviceId: { exact: device.id } } }); 01. 02. 03. 04. 05. 06. @philnash
46. DEMO @philnash
47. RECOMMENDATION: ALWAYS PROVIDE A WAY TO CHECK AND CHANGE INPUTS @philnash
48. RECOMMENDATION: UNDERSTAND THE AVAILABLE MEDIA CONSTRAINTS @philnash
49. SCREEN SHARE @philnash
50. Screen share navigator.mediaDevices.getDisplayMedia() .then(stream => { // add stream track to peer connection }) 01. 02. 03. 04. @philnash
51. DEMO @philnash
52. RECOMMENDATION: LET PEOPLE SHARE SCREENS WITHOUT AN EXTENSION @philnash
53. WEB AUDIO API @philnash
54. Web Audio API Not part of WebRTC Vital part of displaying audio to the user @philnash
55. DEMO @philnash
56. YOU'RE ON MUTE! @philnash
57. DEMO @philnash
58. DATA CHANNEL @philnash
59. RTCDataChannel const dataChannel = pc.createDataChannel("my-data-channel"); dataChannel.addEventListener("message", event => { console.log(event.data); }); dataChannel.send("Hi!"); 01. 02. 03. 04. 05. @philnash
60. DEMO @philnash
61. USEFUL TOOLS @philnash
62. OPEN SOURCE DIAGNOSTICS @philnash
63. DEMO @philnash
64. RTC Diagnostics https://github.com/twilio/rtc-diagnostics/ @philnash
65. OUR LIVES ON SCREEN @philnash
66. WEBRTC IS EVERYWHERE @philnash
67. WEBRTC IS POWERFUL BECAUSE IT IS ON THE WEB @philnash
68. WE HAVE THE TOOLS TO BUILD GREAT WEBRTC EXPERIENCES @philnash
69. Thanks! @philnash https://philna.sh philnash@twilio.com