Let's build a video streaming app using Web technologies

Slide 1

Slide 1 text

GDE, Web Technologies Arnelle Balane @arnellebalane Let’s build a video streaming app using Web technologies

Slide 2

Slide 2 text

Google Developers Expert for Web Technologies I write about Web stuff on my blog, arnellebalane.com Arnelle Balane @arnellebalane

Slide 3

Slide 3 text

VIDEO STREAMING APP

Slide 4

Slide 4 text

Part 1 of 3 Concepts

Slide 5

Slide 5 text

Simple video delivery we have a video file uploaded and stored in some server or cloud storage played in the browser using tag

Slide 6

Slide 6 text

Simple video delivery

Slide 7

Slide 7 text

Simple video delivery Sorry, your browser doesn't support HTML video.

Slide 8

Slide 8 text

✅ Very easy to setup ❌ Little control over how the video is buffered ❌ Large videos are not ideal for mobile devices Simple video delivery

Slide 9

Slide 9 text

Third party embeds we have a video file uploaded to third-party platforms like Youtube, Vimeo, etc. embedded into our pages using tag

Slide 10

Slide 10 text

Third party embeds

Slide 11

Slide 11 text

✅ Very easy to use ✅ No need to host the video files ourselves ✅ Efficient loading of video across devices ❌ Little control over the embedded content Third party embeds

Slide 12

Slide 12 text

or we can do it 🙌 ourselves 🙌

Slide 13

Slide 13 text

✅ Full control on the content delivery and video player interface ❌ Our app has to do a lot Our own video streaming app

Slide 14

Slide 14 text

Our own video streaming app we have a video file processed and stored in our custom video pipeline played through our custom video player

Slide 15

Slide 15 text

Video processing original video file

Slide 16

Slide 16 text

Video processing video demux separate video and audio streams audio

Slide 17

Slide 17 text

Video processing transcode create multiple representations of the video stream 480p 720p 1080p

Slide 18

Slide 18 text

Video processing we can also add alternate audio streams, e.g. for dubbed content 480p 720p 1080p English Tagalog French

Slide 19

Slide 19 text

segment break the video and audio streams into small chunks Video processing

Slide 20

Slide 20 text

package create a manifest file with info about all the representations and segments Video processing manifest.mpd

Slide 21

Slide 21 text

Video playback src https://example.com/sample.mp4

Slide 22

Slide 22 text

Video playback src MediaSource MediaSource on MDN: https://developer.mozilla.org/en-US/docs/Web/API/MediaSource

Slide 23

Slide 23 text

Video playback src SourceBuffer SourceBuffer on MDN: https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer

Slide 24

Slide 24 text

Video playback src app loads manifest file to determine the names of the video and audio segments manifest.mpd

Slide 25

Slide 25 text

Video playback src app loads manifest file to determine the names of the video and audio segments manifest.mpd

Slide 26

Slide 26 text

Video playback src app populates the source buffer with data from the video/audio segments manifest.mpd

Slide 27

Slide 27 text

🎉 An international standard for adaptive bitrate streaming 🎉 Builds on top of existing HTTP infrastructure DASH Dynamic Adaptive Streaming over HTTP

Slide 28

Slide 28 text

Part 2 of 3 Process Media Content

Slide 29

Slide 29 text

👉 ffmpeg - a suite of libraries for handling multimedia files and streams Tools we’re going to use brew install ffmpeg

Slide 30

Slide 30 text

👉 MP4box - multimedia packaging library under the gpac multimedia framework Tools we’re going to use brew install gpac

Slide 31

Slide 31 text

Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv': Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Demux audio stream ffmpeg -i sample.mkv -map 0:1 # gets the audio track by index (index 0 is the video) -c:a aac # audio encoder -b:a 640k # audio bitrate -ar 48000 # sampling rate (48 kHz) -ac 2 # audio channels (2 = stereo) -vn # exclude video track outputs/audio.m4a

Slide 34

Slide 34 text

Demux and transcode video stream ffmpeg -i sample.mkv -c:v libx264 # video encoder -r 24 # frame rate -x264opts 'keyint=48:min-keyint=48:no-scenecut' -vf scale=-2:480 # resize video to 480p -b:v 1050k # video bitrate -maxrate 1050k # video max bitrate -bufsize 2100k # buffer size -movflags faststart -profile:v main -preset ultrafast -an # exclude audio track outputs/video-480p-1050k.mp4

Slide 35

Slide 35 text

Example bitrates and resolutions Bitrate (kbps) Resolution (px) 235 320 x 240 375 384 x 288 560 512 x 384 1050 640 x 480 1750 720 x 480 3000 1280 x 720 5800 1920 x 1080 repeat previous command to transcode video into each desired bitrate and resolution

Slide 36

Slide 36 text

After demux and transcode ~/dev/video-streaming-app 🐈 t . ├── outputs │ ├── audio.m4a │ ├── video-1080p-5800k.mp4 │ ├── video-480p-1050k.mp4 │ └── video-720p-3000k.mp4 └── sample.mkv

Slide 37

Slide 37 text

Segment and package MP4Box -dash 4000 # break videos into 4-second segments -frag 4000 -rap -segment-name 'segment-$RepresentationID$-' -fps 24 # frame rate outputs/video-480p-1050k.mp4#video:id=480p outputs/video-720p-3000k.mp4#video:id=720p outputs/video-1080p-5800k.mp4#video:id=1080p outputs/audio.m4a#audio:id=English:role=main -out dash/manifest.mpd

Slide 38

Slide 38 text

Final DASH files ~/dev/video-streaming-app 🐈 ls dash manifest.mpd segment-720p-.mp4 segment-1080p-.mp4 segment-720p-1.m4s segment-1080p-1.m4s segment-720p-10.m4s segment-1080p-10.m4s segment-720p-100.m4s segment-1080p-100.m4s segment-720p-101.m4s segment-1080p-101.m4s segment-720p-102.m4s segment-1080p-102.m4s segment-720p-103.m4s

Slide 39

Slide 39 text

Put it in the cloud! ☁ DASH files

Slide 40

Slide 40 text

👉 ffmpeg is pre-installed! cloud.google.com/functions/docs/reference/system-packages 👉 Be mindful of quotas and limits firebase.google.com/docs/firestore/quotas 👉 Other caveats When using Firebase

Slide 41

Slide 41 text

Google App Engine Put it in the cloud! ☁ DASH files Google Cloud Storage

Slide 42

Slide 42 text

🎉 ✨🎉 Demo time!

Slide 43

Slide 43 text

Part 3 of 3 Playback Media Content

Slide 44

Slide 44 text

1⃣ Download and parse DASH manifest 2⃣ Initialize video element and JS objects 3⃣ Download and add segments to source buffer 4⃣ Switch between available video qualities What we’re going to do

Slide 45

Slide 45 text

Download manifest file const manifestUrl = 'dash/manifest.mpd'; const manifestResponse = await fetch(manifestUrl); const manifestString = await manifestResponse.text(); Fetch on MDN: https://developer.mozilla.org/en-US/docs/Web/API/fetch 1⃣

Slide 46

Slide 46 text

Parse manifest file const parser = new DOMParser(); const manifest = parser.parseFromString( manifestString, 'text/xml' ); DOMParser on MDN: https://developer.mozilla.org/en-US/docs/Web/API/DOMParser

Slide 47

Slide 47 text

Parse manifest file manifest.querySelector('MPD') .getAttribute('maxSegmentDuration'); manifest.querySelectorAll('Representation');

Slide 48

Slide 48 text

We need to extract these values… let segmentTmpl; // ☝ 'dash/segment-$RepresentationID$-$Number$.m4s' let initTmpl; // 'dash/segment-$RepresentationID$-.mp4' let mimeType; // 'video/mp4; codecs="avc1.424034"' let videoDuration; // 888.053 seconds let segmentDuration; // 4.011 seconds let segmentMax; // 222 let representations; // ['480p', '720p', '1080p']

Slide 49

Slide 49 text

… and initialize these let representation = representations[0]; // '480p' let segmentNumber = 1; let video = document.querySelector('video'); let mediaSource; let sourceBuffer;

Slide 50

Slide 50 text

Initialize video and JS objects mediaSource = new MediaSource(); mediaSource.addEventListener('sourceopen', () => { sourceBuffer = mediaSource.addSourceBuffer(mimeType); }); video.src = URL.createObjectURL(mediaSource); 2⃣

Slide 51

Slide 51 text

Download initialization segment const initUrl = initTmpl .replace('$RepresentationID$', representation); const initResponse = await fetch(initUrl); const initBuffer = await initResponse.arrayBuffer(); sourceBuffer.appendBuffer(initBuffer);

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

Download individual segments const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); 3⃣

Slide 54

Slide 54 text

Download individual segments async function loadNextSegment() { const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); } 3⃣

Slide 55

Slide 55 text

Video buffering strategies // Load everything at once, until final segment while (notYetAtFinalSegment()) { await loadNextSegment(); }

Slide 56

Slide 56 text

Video buffering strategies // Load next segment only when almost running out of data video.addEventListener('timeupdate', async () => { if (almostRunningOutOfBufferedData()) { await loadNextSegment(); } });

Slide 57

Slide 57 text

Video buffering strategies // Load correct segment when seeking the video video.addEventListener('seeking', async () => { segmentNumber = computeSegmentNumberFromTime( video.currentTime ); await loadNextSegment(); });

Slide 58

Slide 58 text

Switch between video qualities

480p 720p 1080p

4⃣

Slide 59

Slide 59 text

Switch between video qualities controls.addEventListener('click', async (event) => { if (event.target.closest('button')) { representation = event.target.dataset.quality; await loadNextSegment(); } }); 4⃣

Slide 60

Slide 60 text

Switch between video qualities 4⃣

Slide 61

Slide 61 text

🎉 ✨🎉 Demo time!

Slide 62

Slide 62 text

we don’t need to do it all ourselves 🙌 🙌 🙌

Slide 63

Slide 63 text

Use existing services and libraries

Slide 64

Slide 64 text

Link to full code 👇 https://github.com/arnellebalane/video-streaming-app Thanks for watching! Any questions? 🤔