Let's build a video calling app with Web technologies and Firebase!

Slide 1

Slide 1 text

Software Developer, Newlogic Arnelle Balane @arnellebalane Let’s build a video calling app with Web technologies and Firebase

Slide 2

Slide 2 text

Arnelle Balane Software Developer at Newlogic Google Developers Expert for Web Technologies I write about Web stuff on my blog, arnellebalane.com @arnellebalane

Slide 3

Slide 3 text

WebRTC

Slide 4

Slide 4 text

Part 1 of 2 Concepts

Slide 5

Slide 5 text

1. Get streaming audio and video data 2. Determine network information, such as IP addresses and ports to use 3. Determine information about media capabilities, such as resolutions and available media codecs Video calling apps need to:

Slide 6

Slide 6 text

4. Facilitate communications to initiate or close connections, and exchange network and media information 5. Stream audio and video data Video calling apps need to…

Slide 7

Slide 7 text

Represents a stream of audio and/or video content https://developer.mozilla.org/en-US/docs/Web/API/MediaStream Concepts MediaStream API

Slide 8

Slide 8 text

● media input devices, like the user’s camera or microphone We can obtain media streams from: navigator.mediaDevices.getUserMedia();

Slide 9

Slide 9 text

● a display device ● can be used to share a user’s screen We can obtain media streams from: navigator.mediaDevices.getDisplayMedia();

Slide 10

Slide 10 text

● an HTML canvas element ● can be used to add e.g. a whiteboard feature in the video calling app We can obtain media streams from: canvasElement.captureStream();

Slide 11

Slide 11 text

const mediaStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true, });

Slide 12

Slide 12 text

Concepts RTCPeerConnection API Represents a connection between two peers https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection

Slide 13

Slide 13 text

const peerConnection = new RTCPeerConnection(); local peer remote peer your browser your friend’s browser direct peer-to-peer connection

Slide 14

Slide 14 text

Add media stream to the peer connection mediaStream.getTracks().forEach((track) => { peerConnection.addTrack(track, mediaStream); });

Slide 15

Slide 15 text

Concepts Interactive Connectivity Establishment (ICE) Negotiate the best way for the peers to connect

Slide 16

Slide 16 text

Image from https://web.dev/webrtc-basics/

Slide 17

Slide 17 text

● Returns a peer’s public IP address and port STUN (Session Traversal Utilities for NAT) ● Relays data through an intermediary server in case direct connection is not available TURN (Traversal Using Relay NAT)

Slide 18

Slide 18 text

● The process of determining IP addresses and ports that can be used to establish a connection “finding candidates” ● Each potential option for establishing a peer-to-peer connection ICE candidates

Slide 19

Slide 19 text

Define ICE options for peer connection const peerConnection = new RTCPeerConnection({ iceServers: [{ urls: ['stun:stun.l.google.com:19302'], }], });

Slide 20

Slide 20 text

Concepts Session Description Protocol (SDP) Describes a peer’s network and media information

Slide 21

Slide 21 text

v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Concepts Signaling Facilitate communications between peers (to exchange SDP, initiate and end calls, etc.)

Slide 24

Slide 24 text

1. Fetch API + Server-Sent Events 2. WebSockets 3. BroadcastChannel API Signaling options

Slide 25

Slide 25 text

Initialize the signaling mechanism const signaling = new BroadcastChannel('signaling');

Slide 26

Slide 26 text

(1) So you want to call your friend… you create an “offer” // YOU const offer = await peerConnection.createOffer(); await peerConnection.setLocalDescription(offer); // to be continued in the next slide...

Slide 27

Slide 27 text

(2) Send your “offer” to your friend through signaling // YOU signaling.postMessage({ type: 'offer', payload: offer.toJSON(), });

Slide 28

Slide 28 text

(3) Your friend receives offer through signaling // YOUR FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'offer') { // to be continued in the next step... } };

Slide 29

Slide 29 text

(4) Your friend sets your offer as their remote peer // YOUR FRIEND const peerConnection = new RTCPeerConnection(); await peerConnection.setRemoteDescription(payload); // payload contains your offer // to be continued in the next step...

Slide 30

Slide 30 text

(5) Your friend creates an “answer” and sets it as their local peer // YOUR FRIEND const answer = await peerConnection.createAnswer(); await peerConnection.setLocalDescription(answer); // to be continued in the next step...

Slide 31

Slide 31 text

(6) Your friend sends their answer to you through signaling // YOUR FRIEND signaling.postMessage({ type: 'answer', payload: answer.toJSON(), });

Slide 32

Slide 32 text

(7) You receive the “answer” and set it as your remote peer // YOU signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'answer') { await peerConnection.setRemoteDescription(payload); // payload is your friend's answer } }

Slide 33

Slide 33 text

🎉 ✨🎉 Connection established!

Slide 34

Slide 34 text

Concepts Events Just a couple more things to handle…

Slide 35

Slide 35 text

onicecandidate when a new ICE candidate is determined // YOU peerConnection.onicecandidate = (event) => { signaling.postMessage({ type: 'candidate', payload: event.candidate.toJSON(), }); };

Slide 36

Slide 36 text

onicecandidate when a new ICE candidate is determined // YOUR FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'candidate') { await peerConnection.addIceCandidate(payload); // payload is your ice candidate } };

Slide 37

Slide 37 text

ontrack when our friend’s media stream arrives to us // YOU peerConnection.ontrack = (event) => { displayMediaStream(event.streams[0]); // event.streams are our friend's media streams };

Slide 38

Slide 38 text

Link to full code 👇 https://github.com/arnellebalane/video-conference -app/tree/webrtc 🎉 ✨🎉 Demo time!

Slide 39

Slide 39 text

Part 2 of 2 Using Firebase

Slide 40

Slide 40 text

1. Allow many calls to happen at the same time 2. Allow multiple call participants 3. Allow anyone online to join a call Changes we’ll make

Slide 41

Slide 41 text

1. Create a Firebase project, enable Firestore 2. Install Firebase CLI 3. Initialize Firebase in our project 4. Change signaling mechanism to use Firestore Using Firebase

Slide 42

Slide 42 text

Database structure collections and documents calls :call_id participants :participant_id peers :peer_id candidates :candidate_id

Slide 43

Slide 43 text

Import what we need We’ll use Firebase JS SDK v9 (modular) import { initializeApp } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-app.js'; import { getFirestore, collection, doc, addDoc, getDoc, setDoc, deleteDoc, getDocs, onSnapshot, } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-firestore.js';

Slide 44

Slide 44 text

Initialize app initializeApp({ apiKey: 'your-api-key', authDomain: 'project-id.firebaseapp.com', projectId: 'project-id', storageBucket: 'project-id.appspot.com', messagingSenderId: '...', appId: 'your-app-id', }); const db = getFirestore();

Slide 45

Slide 45 text

Start a call https://call-app.com?call_id=any-call-id // Firestore calls :any-call-id participants :first-participant

Slide 46

Slide 46 text

Start a call const params = new URLSearchParams(location.search); const callId = params.get('call_id'); const participantsRef = collection( db, 'calls', callId, 'participants' ); const participantRef = await addDoc(participantsRef, {});

Slide 47

Slide 47 text

Start a call

Slide 48

Slide 48 text

Join a call https://call-app.com?call_id=any-call-id // Firestore calls :any-call-id participants :first-participant :second-participant

Slide 49

Slide 49 text

Establish P2P connections // Firestore :first-participant peers :second-participant - offer

Slide 50

Slide 50 text

Establish P2P connections // Firestore :first-participant peers :second-participant - offer :second-participant peers :first-participant - answer

Slide 51

Slide 51 text

Establish P2P connections const participants = await getDocs(participantsRef); for (const participantDoc of participants.docs) { if (participantDoc.id !== participantRef.id) { const offer = await offerPeerConnection(participantDoc.id); await setDoc( doc(participantDoc.ref, 'peers', participantRef.id), { offer } ); } }

Slide 52

Slide 52 text

Establish P2P connections

Slide 53

Slide 53 text

Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.offer) { const answer = await answerPeerConnection(change.doc.id, data.offer); await setDoc( doc(participantsRef, change.doc.id, 'peers', participantRef.id), { answer } ); } } } });

Slide 54

Slide 54 text

Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.answer) { await completePeerConnection(change.doc.id, data.answer); } } } });

Slide 55

Slide 55 text

Establish P2P connections

Slide 56

Slide 56 text

Share ICE candidates // Firestore :first-participant candidates :candidate-id - peer_id - candidate :candidate-id - peer_id - candidate

Slide 57

Slide 57 text

Share ICE candidates peerConnection.onicecandidate = async ({ candidate }) => { await addDoc( collection(participantsRef, change.doc.id, 'candidates'), { candidate, peerId: participantRef.id, } ); };

Slide 58

Slide 58 text

Share ICE candidates const candidatesRef = collection(participantRef, 'candidates'); onSnapshot(candidatesRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const { peerId, candidate } = change.doc.data(); await receiveRemoteIceCandidate(peerId, candidate); } } });

Slide 59

Slide 59 text

Share ICE candidates

Slide 60

Slide 60 text

Leave a call const $leaveButton = document.querySelector('.leave-call'); $leaveButton.addEventListener('click', async () => { await deleteDoc(participantRef); window.close(); })

Slide 61

Slide 61 text

Leave a call onSnapshot(participantsRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'removed') { disconnectPeerConnection(change.doc.id); } } });

Slide 62

Slide 62 text

Link to full code 👇 https://github.com/arnellebalane/video-conference -app/tree/main 🎉 ✨🎉 Demo time!

Slide 63

Slide 63 text

1. Use Firebase Local Emulator for Firestore 2. chrome://webrtc-internals Additional tips

Slide 64

Slide 64 text

Shameless plugs 😂 https://dev.to/arnellebalane/lets-build-a-video -calling-app-part-one-2ggl (Part Two is still in progress, maybe next week)

Slide 65

Slide 65 text

Shameless plugs 😂 Repository: https://github.com/arnellebalane /video-conference-app Learn more 👇 https://hacktoberfest.com

Slide 66

Slide 66 text

🤔🤔🤔 Any questions?