Upgrade to Pro — share decks privately, control downloads, hide ads and more …

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

WebRTC is the technology that powers video calls in browsers. What you might not know are all the features WebRTC unlocks for the web. In this talk we'll take a journey through WebRTC, from the primitives and protocols to the projects that have been built.

You'll learn the basics of getting two peers connected in a video chat and move on to advanced topics like camera control, screen sharing, audio analysis, using the data channel and more. We might even be able to solve the "You're on mute!" problem that plagues every video chat.

When you understand WebRTC and the related APIs you'll be amazed with what you can build.

--

Links:

Choosing cameras in JavaScript: https://www.twilio.com/blog/choosing-cameras-javascript-mediadevices-api-html
MediaDevices API examples: https://github.com/philnash/mediadevices-camera-selection
Screen capture with getDisplayMedia: https://www.twilio.com/blog/screen-sharing-javascript-twilio-programmable-video

RTC Diagnostics library. Test bitrate, audio input, audio output and video input in the browser: https://github.com/twilio/rtc-diagnostics/

8ec1383b240b5ba15ffb9743fceb3c0e?s=128

Phil Nash

April 27, 2021
Tweet

Transcript

  1. YOU'RE ON MUTE! WEBRTC AND OUR LIVES ON SCREEN @philnash

  2. Phil Nash @philnash https://philna.sh philnash@twilio.com

  3. YOU'RE ON MUTE! WEBRTC AND OUR LIVES ON SCREEN @philnash

  4. OUR LIVES ON SCREEN @philnash

  5. WEBRTC VS @philnash

  6. GREAT FOR WORK, FRIENDS OR FAMILY @philnash

  7. NOT SO GREAT FOR ONE OFFS @philnash

  8. NOT SO GREAT FOR SPECIALIST APPS @philnash

  9. WEBRTC IS FOR ALL THE OTHER APPLICATIONS @philnash

  10. REAL-TIME VIDEO, AUDIO AND DATA EXPERIENCES FOR YOUR USERS @philnash

  11. Today 1. Basics of WebRTC 2. WebRTC Usage 3. Controlling

    the user experience 4. Tools and takeaways @philnash
  12. BASICS OF WEBRTC @philnash

  13. @philnash

  14. Signalling @philnash

  15. Peer to peer @philnash

  16. SFU @philnash

  17. ICE • ICE — Interactive Connectivity Establishment • NAT —

    Network Address Translation • STUN — Session Traversal Utilities for NAT • TURN — Traversal Using Relays around NAT @philnash
  18. THE APIS @philnash

  19. getUserMedia navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(stream => { //

    ... }); 01. 02. 03. 04. @philnash
  20. RTCPeerConnection const pc = new RTCPeerConnection({ iceServers }); pc.addEventListener("icecandidate", (candidate)

    => { // send candidate to signalling }); 01. 02. 03. 04. @philnash
  21. RTCPeerConnection pc.addStream(stream); pc.createOffer().then(offer => { pc.setLocalDescription(offer); // send offer to

    peer }); 01. 02. 03. 04. 05. @philnash
  22. RTCPeerConnection pc.setRemoteDescription(offer); pc.createAnswer().then(answer => { pc.setLocalDescription(answer); // send answer to

    peer }); 01. 02. 03. 04. 05. @philnash
  23. RTCPeerConnection pc.setRemoteDescription(answer); pc.addEventListener("track", event => { // Connected! // show

    remote video in UI }); 01. 02. 03. 04. 05. @philnash
  24. RTCDataChannel const dataChannel = pc.createDataChannel("my-data-channel"); dataChannel.addEventListener("message", event => { console.log(event.data);

    }); 01. 02. 03. 04. @philnash
  25. RTCDataChannel pc.addEventListener("datachannel", event => { event.channel.addEventListener("open", () => { event.channel.send("Hi!");

    }); }); 01. 02. 03. 04. 05. @philnash
  26. WEBRTC USAGE @philnash

  27. @philnash

  28. @philnash

  29. IMPROVING THE USER EXPERIENCE @philnash

  30. MEDIA @philnash

  31. getUserMedia navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(stream => { //

    ... }); 01. 02. 03. 04. @philnash
  32. getUserMedia - Permissions navigator.mediaDevices.getUserMedia({ audio: true, video: true }) .then(stream

    => { // success }) .catch(error => { // permission denied }); 01. 02. 03. 04. 05. 06. @philnash
  33. getUserMedia - Media Constraints { audio: true, video: true }

    01. 02. 03. 04. @philnash
  34. Media Constraints - facingMode { audio: true, video: { facingMode:

    "user" // "environment", "right", "left" } } 01. 02. 03. 04. 05. 06. @philnash
  35. Media Constraints - deviceId { audio: true, video: { deviceId:

    { exact: device.id } } } 01. 02. 03. 04. 05. 06. @philnash
  36. enumerateDevices navigator.mediaDevices.enumerateDevices() .then(devices => { devices.forEach(device => { console.log(`${device.id}: ${device.label}`);

    }); }); 01. 02. 03. 04. 05. 06. @philnash
  37. enumerateDevices - Permissions navigator.mediaDevices.enumerateDevices() .then(devices => { devices.forEach(device => {

    console.log(device.label); // ??? }); }); 01. 02. 03. 04. 05. 06. @philnash
  38. Media Constraints - resolution { audio: true, video: { width:

    1920, height: 1080 } } 01. 02. 03. 04. 05. 06. 07. @philnash
  39. Media Constraints - resolution { video: { width: { min:

    320, max: 1920, ideal: 1920 }, height: { min: 240, max: 1080, ideal: 1080 } } } 01. 02. 03. 04. 05. 06. @philnash
  40. Media Constraints - resolution { video: { width: { exact:

    1920 }, height: { exact: 1080 } } } // => 🚨 OverconstrainedError 01. 02. 03. 04. 05. 06. 07. @philnash
  41. Media Constraints - torch 🔦😄 { video: { advanced: [{

    torch: true }], } } 01. 02. 03. 04. 05. @philnash
  42. Media Constraints - torch 🔦😄 { video: { advanced: [{

    torch: false }], // doesn't work 🤨 } } 01. 02. 03. 04. 05. @philnash
  43. Applying media constraints videoTrack.applyConstraints({ width: 1920, height: 1080 }); 01.

    02. 03. 04. @philnash
  44. Applying media constraints videoTrack.applyConstraints({ deviceId: { exact: device.id } });

    ❌ Don't do this! ❌ 01. 02. 03. @philnash
  45. Changing cameras videoTrack.stop(); navigator.mediaDevices.getUserMedia({ video: { deviceId: { exact: device.id

    } } }); 01. 02. 03. 04. 05. 06. @philnash
  46. DEMO @philnash

  47. RECOMMENDATION: ALWAYS PROVIDE A WAY TO CHECK AND CHANGE INPUTS

    @philnash
  48. RECOMMENDATION: UNDERSTAND THE AVAILABLE MEDIA CONSTRAINTS @philnash

  49. SCREEN SHARE @philnash

  50. Screen share navigator.mediaDevices.getDisplayMedia() .then(stream => { // add stream track

    to peer connection }) 01. 02. 03. 04. @philnash
  51. DEMO @philnash

  52. RECOMMENDATION: LET PEOPLE SHARE SCREENS WITHOUT AN EXTENSION @philnash

  53. WEB AUDIO API @philnash

  54. Web Audio API Not part of WebRTC Vital part of

    displaying audio to the user @philnash
  55. DEMO @philnash

  56. YOU'RE ON MUTE! @philnash

  57. DEMO @philnash

  58. DATA CHANNEL @philnash

  59. RTCDataChannel const dataChannel = pc.createDataChannel("my-data-channel"); dataChannel.addEventListener("message", event => { console.log(event.data);

    }); dataChannel.send("Hi!"); 01. 02. 03. 04. 05. @philnash
  60. DEMO @philnash

  61. USEFUL TOOLS @philnash

  62. OPEN SOURCE DIAGNOSTICS @philnash

  63. DEMO @philnash

  64. RTC Diagnostics https://github.com/twilio/rtc-diagnostics/ @philnash

  65. OUR LIVES ON SCREEN @philnash

  66. WEBRTC IS EVERYWHERE @philnash

  67. WEBRTC IS POWERFUL BECAUSE IT IS ON THE WEB @philnash

  68. WE HAVE THE TOOLS TO BUILD GREAT WEBRTC EXPERIENCES @philnash

  69. Thanks! @philnash https://philna.sh philnash@twilio.com