Things Developers Believe About Video Files (Proven Wrong by User Uploads) derekb@vimeo.com @daemon404 Derek Buitenhuis 5 October 2017 San Francisco, USA

/whois derekb 1 5 October 2017 • Senior Video Encoding Engineer at Vimeo • Distributed chunk transcoder • On-the-fly image service (before it was cool) • HLS/DASH on-the-fly packager • Open Source Multimedia Troll^WDeveloper • FFmpeg, L-SMASH, FFMS2, x264, etc. • Daemon404 @ FreeNode • VideoLAN Non-Profit Association Member

Everything’s Great Until Users Show Up 2 5 October 2017

Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4)

Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above

Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest

Everything’s Great Until Users Show Up 2 5 October 2017 • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest • Fixing the worst files has lead to a much more robust all around product

The “Obvious” Ones 3 5 October 2017

The “Obvious” Ones 3 5 October 2017

The “Obvious” Ones 3 5 October 2017

The “Obvious” Ones 3 5 October 2017 • That properties cannot change between frames

Things Change… 3.5 5 October 2017 • Pixel packing format • Color family • Color primaries • Transfer function • Color matrix • Resolution • Frame rate (more on this later!) • Color range • Codec • Sampling rate • Aspect ratio • Bit depth • Whether it’s interlaced or not • Whether it’s telecined or not • Field order • Chroma location • Number of audio channels • Channel layout • Audio sample format • About 10 billion more!

The “Obvious” Ones 3 5 October 2017 • That properties cannot change between frames

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate)

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?)

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp

The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 • That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp • That discontinuous timestamps are invalid • OGG, MPEG-TS

4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom Yes, Timestamps Are Pain, Have Some More

4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows Yes, Timestamps Are Pain, Have Some More

4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska Yes, Timestamps Are Pain, Have Some More

4 5 October 2017 • That you shouldn’t display negative timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska • That these will match the timestamp deltas Yes, Timestamps Are Pain, Have Some More

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1

Less Obvious: That Was a Fun Day of Debugging 5 5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1 • That you only need to play one track of a kind at once • Multiple default-on MOV tracks

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both

Less Obvious: That Was a Fun Week of Debugging 6 5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both • That the color properties will even map to some existing standard • MOV’s embedded ICC profile in the colr atom

“multimedia is basically neverending pain” 7 5 October 2017 — wm4, mpv-player lead

7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom “multimedia is basically neverending pain” — wm4, mpv-player lead

7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV “multimedia is basically neverending pain” — wm4, mpv-player lead

7 5 October 2017 • That file actually has an end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV • That you can easily downmix or play a stereo track on a mono speaker • Phase cancellation is great fun “multimedia is basically neverending pain” — wm4, mpv-player lead

8 5 October 2017 • Non-bit-exact IDCT in MPEG-2 and MPEG-4 “multimedia is basically neverending pain” — wm4, mpv-player lead

8 5 October 2017 • Everything is MP3: “multimedia is basically neverending pain” — wm4, mpv-player lead

Tools, Sources, etc. 8 5 October 2017 • ffprobe: http://git.videolan.org/?p=ffmpeg.git • boxdumper: https://github.com/l-smash/l-smash • bxd: https://github.com/dwbuiten/bxd • Beyond Compare is great as well • ffms2: https://github.com/FFMS/ffms2/ • dvbsnoop: http://dvbsnoop.sourceforge.net/ • JPEG/MP3 sample epic bug report: • https://github.com/mpv-player/mpv/issues/3973 • Source of IDCT Images: • https://guru.multimedia.cx/the-mpeg124-and-h26123-idct

9 5 October 2017 Thanks! Questions (or )?