Things Developers Believe About Video Files (Proven Wrong by User Uploads) - Derek Buitenhuis

Think you know all there is to know about different video formats, codecs, and containers? Your users are here to prove you wrong! Let's take a quick dive the insanity of petabytes of user uploads on a top video website.

Presented by Derek Buitenhuis at Demuxed 2017

Transcript

1. Things Developers Believe About Video Files (Proven Wrong by User Uploads) derekb@vimeo.com @daemon404 Derek Buitenhuis 5 October 2017 San Francisco, USA
2. /whois derekb 1 5 October 2017 • Senior Video Encoding Engineer at Vimeo • Distributed chunk transcoder • On-the-fly image service (before it was cool) • HLS/DASH on-the-fly packager • Open Source Multimedia Troll^WDeveloper • FFmpeg, L-SMASH, FFMS2, x264, etc. • Daemon404 @ FreeNode • VideoLAN Non-Profit Association Member
3. Everything’s Great Until Users Show Up 2 5 October 2017
4. Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4)
5. Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above
6. Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest
7. Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest • Fixing the worst files has lead to a much more robust all around product
8. The “Obvious” Ones 3 5 October 2017
9. The “Obvious” Ones 3 5 October 2017
10. The “Obvious” Ones 3 5 October 2017
11. The “Obvious” Ones 3 5 October 2017 • That properties cannot change between frames
12. Things Change… 3.5 5 October 2017 • Pixel packing format • Color family • Color primaries • Transfer function • Color matrix • Resolution • Frame rate (more on this later!) • Color range • Codec • Sampling rate • Aspect ratio • Bit depth • Whether it’s interlaced or not • Whether it’s telecined or not • Field order • Chroma location • Number of audio channels • Channel layout • Audio sample format • About 10 billion more!
13. The “Obvious” Ones 3 5 October 2017 • That properties cannot change between frames
14. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate)
15. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?)
16. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV
17. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles
18. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp
19. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp • That discontinuous timestamps are invalid • OGG, MPEG-TS
20. 4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom Yes, Timestamps Are Pain, Have Some More
21. 4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows Yes, Timestamps Are Pain, Have Some More
22. 4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska Yes, Timestamps Are Pain, Have Some More
23. 4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska • That these will match the timestamp deltas Yes, Timestamps Are Pain, Have Some More
24. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017
25. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh
26. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters
27. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames
28. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1
29. Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1 • That you only need to play one track of a kind at once • Multiple default-on MOV tracks
30. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved
31. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all
32. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order
33. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties
34. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both
35. Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both • That the color properties will even map to some existing standard • MOV’s embedded ICC profile in the colr atom
36. “multimedia is basically neverending pain” 7 5 October 2017 — wm4, mpv-player lead
37. 7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom “multimedia is basically neverending pain” — wm4, mpv-player lead
38. 7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV “multimedia is basically neverending pain” — wm4, mpv-player lead
39. 7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV • That you can easily downmix or play a stereo track on a mono speaker • Phase cancellation is great fun “multimedia is basically neverending pain” — wm4, mpv-player lead
40. 8 5 October 2017 • Non-bit-exact IDCT in MPEG-2 and MPEG-4 “multimedia is basically neverending pain” — wm4, mpv-player lead
41. 8 5 October 2017 • Everything is MP3: “multimedia is basically neverending pain” — wm4, mpv-player lead
42. Tools, Sources, etc. 8 5 October 2017 • ffprobe: http://git.videolan.org/?p=ffmpeg.git • boxdumper: https://github.com/l-smash/l-smash • bxd: https://github.com/dwbuiten/bxd • Beyond Compare is great as well • ffms2: https://github.com/FFMS/ffms2/ • dvbsnoop: http://dvbsnoop.sourceforge.net/ • JPEG/MP3 sample epic bug report: • https://github.com/mpv-player/mpv/issues/3973 • Source of IDCT Images: • https://guru.multimedia.cx/the-mpeg124-and-h26123-idct
43. 9 5 October 2017 Thanks! Questions (or )?