How video tech works, on the web, 2023

20230723, NYC Code and Coffee 2.70 @ General Assembly

Transcript

1. How video tech works? 📹 📼 📺 On the web 🌐
2. Too bright! 👀
3. Disclaimer: The opinions expressed in this presentation and on the following slides are solely those of the presenter. 👋 About speaker Artur Parkhisenko Sr. Software engineer at Vimeo Working in the Video Playback Experience team with OTT apps using Universal Windows Application (Xbox), Tizen Smart TV (Samsung), Google Cast (Chromecast) and different video players.
4. A modern approach to learning? https://openai.com/chatgpt Then, is this the last slide? …
5. What is video? Photo by Swapnil Sharma
6. What is video? https://en.wikipedia.org/wiki/The_Horse_in_Motion Have you ever had/made a flipbook? See reference to that in a movie: Nope 🛸 2022 ‧ Horror/Sci-fi Recommendation to visit: Museum of the Moving Image https://movingimage.us/ 🦒🍦⛳ aka gif
7. Is it like rocket science? no? Why Video is awesome HARD part 2 https://www.mux.com/blog/why-video-is-awesome 😉 Video is HARD part 1 https://www.mux.com/blog/video-is-too-hard
8. What is widely known about media content? ???
9. Let’s take a look at the TV or Monitor product name? ⏸ 👀 Sony 55 Inch 4K Ultra HD TV X85K Series: LED Smart Google TV Dolby Vision HDR and Native 120HZ Refresh Rate KD55X85K- 2022 Model w/HT-G700: 3.1CH Dolby Atmos/:X Soundbar with Bluetooth Tech TUF Gaming VG27AQ HDR Gaming Monitor – 27 inch QHD (2560x1440), IPS, 165Hz*, Extreme Low Motion Blur Sync, NVIDIA G-SYNC Compatible, Adaptive-Sync, 1ms (MPRT), HDR10 Widely known media info 📺
10. Let’s take a look at the TV or Monitor product name? Sony 55 Inch 4K Ultra HD TV X85K Series: LED Smart Google TV Dolby Vision HDR and Native 120HZ Refresh Rate KD55X85K- 2022 Model w/HT-G700: 3.1CH Dolby Atmos/:X Soundbar with Bluetooth Tech TUF Gaming VG27AQ HDR Gaming Monitor – 27 inch QHD (2560x1440), IPS, 165Hz*, Extreme Low Motion Blur Sync, NVIDIA G-SYNC Compatible, Adaptive-Sync, 1ms (MPRT), HDR10 In short: HD, Full HD, QHD, UHD, HDR, Dolby, 4K, 30/60/120 FPS Widely known media info 📺
11. General Complete name : /Users/a/blog/html5-video-tag/testsrc-720p-24fps.mp4 Format : MPEG-4 Format profile : Base Media Codec ID : isom (isom/iso2/avc1/mp41) File size : 74.2 KiB Duration : 7 s 459 ms Overall bit rate : 81.5 kb/s Writing application : Lavf58.29.100 Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : High 4:4:4 Predictive@L3.1 Format settings : CABAC / 4 Ref Frames Format settings, CABAC : Yes Format settings, Reference frames : 4 frames Codec ID : avc1 Codec ID/Info : Advanced Video Coding Duration : 7 s 459 ms Bit rate : 78.3 kb/s Width : 1 280 pixels Height : 720 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 24.000 FPS Chroma subsampling : 4:4:4 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.004 Stream size : 71.3 KiB (96%) Writing library : x264 core 159 r2999 296494a Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=4 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=24 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00 Codec configuration box : avcC Media file info example https://www.w3.org/2007/08/video/report
12. Now, what’s on the Web? ???
13. <video controls src="SMPTE-color-bars.mp4" /> HTML5 media element video
14. Most of the videos you watch, aren’t used like that why? Progressive download and adaptive streaming, and many other reasons ;)
15. Playback Demo Demo UI and adaptive streaming using DevTools for this asset: https://shaka-player-demo.appspot.com/demo/#audiolang=en;textlang=en;uilang=en ;asset=https://storage.googleapis.com/shaka-demo-assets/angel-one-hls/hls.m3u8 ;panel=CUSTOM%20CONTENT;build=debug_compiled
16. HLS, Playlist relationships Example video: (51*4) + 2 = 206s = 3.43min A movie: 1h video, 3600 seconds 3600s / 6s = 600 segments only for one quality.
17. General architecture https://github.com/leandromoreira/digital_video_introduction#general-architecture HTTP Live Streaming example Examples: Smartphone, Console, OBS studio, ... https://developer.apple.com/documentation/http-live-streaming/understanding-the-http-live-streaming-architecture
18. • Media Source Extensions (MSE) • Codecs (H264 AVC, H265 HEVC*, AV1*, VVC*, VP8/9*) • Media Containers (TS, MP4*, WebM*) • Delivery (HLS, DASH*) • FFmpeg (our toolbelt) • chrome://media-internals/ * optional for beginners, but even the minimum isn’t easy, potentially includes Bits and Bytes, Numeral systems: Bin, Dec, Hex, Encoding: ASCII, UTF-8, UTF-16, Endianness, ArrayBuffers… and many more, so Alice, just be aware that’s a very deep rabbit hole. Some terms: • AVC - Advanced Video Coding • DASH - Dynamic Adaptive Streaming over HTTP • HEVC - High Efficiency Video Coding • HLS - HTTP Live Streaming • TS - MPEG Transport Stream Web video tech minimum 🤯 More here: https://awesome.video/
19. FFmpeg FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. A complete, cross-platform solution to record, convert and stream audio and video. FFmpeg tools are: ffmpeg, ffplay, ffprobe
20. A typical day for video tech eng (Apps) Describe example of what Apps engineer does (supp. multiple platforms and backend part) TV and Devices (sticks/boxes) things - Hardware & Source specifics: input, output - Playback issues/features - Environment / OS / Tech SDK Video Player things - Bugs - Features Never-ending learning…
21. • How Video Works https://howvideo.works/ • Digital Video Introduction https://github.com/leandromoreira/digital_video_introduction • Awesome Video https://awesome.video/ • HTML5 Video Events and API https://www.w3.org/2010/05/video/mediaevents.html • A Digital Media Primer For Geeks (Monty Montgomery @ xiph.org) https://youtu.be/FG9jemV1T7I • D/A and A/D | Digital Show and Tell (Monty Montgomery @ xiph.org) https://youtu.be/cIQ9IXSUzuM • Video Glossary https://bitmovin.com/video-glossary/ • Podcast Demuxed https://www.heavybit.com/library/podcasts/demuxed/ • Podcast We Are Netflix http://weare.netflix.net/ • Video is HARD part 1 https://www.mux.com/blog/video-is-too-hard • Video is HARD part 2 https://www.mux.com/blog/why-video-is-awesome • WebRTC technology https://webrtc.org/ • Dolby https://dolby.io/ • How Codecs Work https://vimeo.com/104554788 • Video could be Analyzed and remastered (upscale) with AI https://youtu.be/AwmvwTopbas?t=235 • Complexity of supporting for different devices and platforms • A lot of features like: Analytics, Live (and viewers counter), Thumbnails, Filters, Masks • Tools like MediaInfo, VLC media player, Video editors • Backend part related to the transcoding • Media Source Extensions with Jeff Tapper https://youtu.be/33kDks00Pr8 • Video onboarding https://github.com/theREDspace/video-onboarding • ABR Part one, two, three • Video File formats https://www.mux.com/blog/a-beginners-guide-to-video-file-formats • HTML5 Video https://ronallo.com/blog/html5-video-everything-i-needed-to-know/ • HTML5 Video and audio tutorial https://www.bogotobogo.com/HTML5/HTML5_Tutorial_Audio_Video.php • Video onboarding https://github.com/theREDspace/video-onboarding • HTML5 MSE https://www.smashingmagazine.com/2016/04/html5-media-source-extensions-bringing-production-video-web/ • WASM booth https://mtharrison.github.io/wasmbooth/ • Indian-head Test Pattern https://en.wikipedia.org/wiki/Indian-head_test_pattern • Big Buck Bunny https://peach.blender.org/ What’s missing on those slides
22. Community The conference & community for video devs. https://www.demuxed.com/ Video tech meetup in NYC, (there’s a lot of them across the world) https://www.meetup.com/Video-Tech-NYC/ Video Dev Slack (jobs channel is there) https://www.video-dev.org/
23. Where to work Amazon Prime Video, Bitmovin, Brightcove, Discovery, Disney, Facebook, FuboTV, Hulu, Instagram, JWPlayer, Kick, LinkedIn, Mux, NBC Universal, Netflix, New York Times, Peacock, Snapchat, Trovo, Twitch, Vimeo, YouTube and many others… p.s. ping me to add your company to the list
24. for await (let question of questions) { console.log(answer(question)); } console.log("Thanks"); Artur Parkhisenko July 2023