Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Things Developers Believe About Video Files (Proven Wrong by User Uploads) - Derek Buitenhuis

1c4a54c9cd8bc148ec055734a15c6492?s=47 Demuxed
October 05, 2017

Things Developers Believe About Video Files (Proven Wrong by User Uploads) - Derek Buitenhuis

Think you know all there is to know about different video formats, codecs, and containers? Your users are here to prove you wrong! Let's take a quick dive the insanity of petabytes of user uploads on a top video website.

Presented by Derek Buitenhuis at Demuxed 2017

1c4a54c9cd8bc148ec055734a15c6492?s=128

Demuxed

October 05, 2017
Tweet

Transcript

  1. Things Developers Believe About Video Files (Proven Wrong by User

    Uploads) derekb@vimeo.com @daemon404 Derek Buitenhuis 5 October 2017 San Francisco, USA
  2. /whois derekb 1 5 October 2017 • Senior Video Encoding

    Engineer at Vimeo • Distributed chunk transcoder • On-the-fly image service (before it was cool) • HLS/DASH on-the-fly packager • Open Source Multimedia Troll^WDeveloper • FFmpeg, L-SMASH, FFMS2, x264, etc. • Daemon404 @ FreeNode • VideoLAN Non-Profit Association Member
  3. Everything’s Great Until Users Show Up 2 5 October 2017

  4. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4)
  5. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above
  6. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest
  7. Everything’s Great Until Users Show Up 2 5 October 2017

    • Most users are well behaved (H.264 + AAC in MOV/MP4) • Others… • In the last 90 or so days: • 119 Video Codecs • 73 Audio Codecs • 39 Container • N Combinations of the above • Observation: Users with the worst files tend to complain the loudest • Fixing the worst files has lead to a much more robust all around product
  8. The “Obvious” Ones 3 5 October 2017

  9. The “Obvious” Ones 3 5 October 2017

  10. The “Obvious” Ones 3 5 October 2017

  11. The “Obvious” Ones 3 5 October 2017 • That properties

    cannot change between frames
  12. Things Change… 3.5 5 October 2017 • Pixel packing format

    • Color family • Color primaries • Transfer function • Color matrix • Resolution • Frame rate (more on this later!) • Color range • Codec • Sampling rate • Aspect ratio • Bit depth • Whether it’s interlaced or not • Whether it’s telecined or not • Field order • Chroma location • Number of audio channels • Channel layout • Audio sample format • About 10 billion more!
  13. The “Obvious” Ones 3 5 October 2017 • That properties

    cannot change between frames
  14. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate)
  15. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?)
  16. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV
  17. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles
  18. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp
  19. The “Obvious” Ones: Mostly Timestamps 3 5 October 2017 •

    That properties cannot change between frames • That you’ll have a frame rate at all (variable framerate) • That non-constant timestamps mean you don’t have a framerate • Oscillating timestamps due to use of 90kHz time base • Insane android “constant” frame rate based off camera timings (?) • That even when you do have a constant frame rate, that you play at that rate • mvhd rate and edts media_rate in ISOBMFF / MOV • That you don’t need to do rate conversion • Example: sub-1fps slideshow videos → bad seeking, bad subtitles • That every frame will have a timestamp • That discontinuous timestamps are invalid • OGG, MPEG-TS
  20. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom Yes, Timestamps Are Pain, Have Some More
  21. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows Yes, Timestamps Are Pain, Have Some More
  22. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska Yes, Timestamps Are Pain, Have Some More
  23. 4 5 October 2017 • That you shouldn’t display negative

    timestamped samples • AllF entry in udta atom • That you don’t need sample durations • Duration of the last frame + sub-1fps slideshows • That you’ll have sample durations • Looking at you, Matroska • That these will match the timestamp deltas Yes, Timestamps Are Pain, Have Some More
  24. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017
  25. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh
  26. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters
  27. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames
  28. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1
  29. Less Obvious: That Was a Fun Day of Debugging 5

    5 October 2017 • That you can seek in the input without pre-indexing every packet • IPPPPPPP • Periodic Intra Refresh • That coded samples match what should be displayed • Edit lists, ordered chapters • That you only need to display what’s in the file • Audio silence gap-filling, post-stream black frames • That you should display every packet/sample • VP8, VP9, AV1 • That you only need to play one track of a kind at once • Multiple default-on MOV tracks
  30. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved
  31. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all
  32. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order
  33. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties
  34. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both
  35. Less Obvious: That Was a Fun Week of Debugging 6

    5 October 2017 • That packets will be well interleaved • That packets will even be interleaved at all • That packets will even be vaguely in-order • That you should use the bitstream color properties • That you should use the container color properties • That you shouldn’t use both • That the color properties will even map to some existing standard • MOV’s embedded ICC profile in the colr atom
  36. “multimedia is basically neverending pain” 7 5 October 2017 —

    wm4, mpv-player lead
  37. 7 5 October 2017 • That file actually has an

    end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom “multimedia is basically neverending pain” — wm4, mpv-player lead
  38. 7 5 October 2017 • That file actually has an

    end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV “multimedia is basically neverending pain” — wm4, mpv-player lead
  39. 7 5 October 2017 • That file actually has an

    end • Segment linking or dref atom loop (A → B → A → B → …) • LOOP entry in udta atom • That the video track is even a video “codec” • PowerPoint-in-MOV, FIRE, VNC commands, RAR in MKV • That you can easily downmix or play a stereo track on a mono speaker • Phase cancellation is great fun “multimedia is basically neverending pain” — wm4, mpv-player lead
  40. 8 5 October 2017 • Non-bit-exact IDCT in MPEG-2 and

    MPEG-4 “multimedia is basically neverending pain” — wm4, mpv-player lead
  41. 8 5 October 2017 • Everything is MP3: “multimedia is

    basically neverending pain” — wm4, mpv-player lead
  42. Tools, Sources, etc. 8 5 October 2017 • ffprobe: http://git.videolan.org/?p=ffmpeg.git

    • boxdumper: https://github.com/l-smash/l-smash • bxd: https://github.com/dwbuiten/bxd • Beyond Compare is great as well • ffms2: https://github.com/FFMS/ffms2/ • dvbsnoop: http://dvbsnoop.sourceforge.net/ • JPEG/MP3 sample epic bug report: • https://github.com/mpv-player/mpv/issues/3973 • Source of IDCT Images: • https://guru.multimedia.cx/the-mpeg124-and-h26123-idct
  43. 9 5 October 2017 Thanks! Questions (or )?