Building a Closed Captions Ecosystem at The New York Times

Building a Closed Captions Ecosystem at The New York Times

Talk about how we are building closed captions support for The New York Times video library at the Video Tech Meetup in New York.

5dc4abe3dc14c25021ff2295fc380d73?s=128

Flávio Ribeiro

July 19, 2017
Tweet

Transcript

  1. Building a Closed Captions Ecosystem at The New York Times

    Said Ketchman & Flavio Ribeiro
  2. Said Ketchman Engineering Manager, Video http://github.com/saidketchman http://twitter.com/introvertedeng http://introvertedengineer.com

  3. Flavio Ribeiro Lead Engineer, Video http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com

  4. An Appeal for Closed Captions 1

  5. An Appeal for Closed Captions Why Standardized Captions? • Following

    Web Standards • Complying with Accessibility Standards • Enabling Muted Autoplay • Avoiding 360 Videos Burn-in • Enhancing Site SEO • Internationalization (translations) Our Required Approach • Integration with “Media Factory” • Avoiding Vendor Dependency • Providing Multiple Generation Options • Allowing Robust Editability of Captions • Fully Integrated with NYTimes’ CMS • Ease of Use for Video Editors/Journalists
  6. First, Some Context 2

  7. Media Factory Transcoding API Acquisition API Storage Distribution API Media

    Factory Client Media Factory API Database CDN
  8. A New Approach for Video Delivery (OTFP) 3

  9. Media Factory Transcoding API Acquisition API Storage Distribution API Media

    Factory Client Media Factory API Database CDN + CDN GCS
  10. Vendor Agnostic Approach for Greatest Flexibility 4

  11. Vendor Agnostic Caption Generation • Caption Generation should not be

    coupled to a single vendor • Easily switch between vendors via request parameters • Similar approach as our Transcoding API Closed Captions API 3Play Google Speech API Amazon Lex Generation Client Abstraction etc...
  12. Adding Closed Captions Generation to Our Encoding Workflow 5

  13. Media Factory w/ OFTP Transcoding API Acquisition API Storage Distribution

    API Media Factory Client Media Factory API Database CDN + CDN GCS Closed Captions API
  14. Not Done Yet: Editing Matters Too 6

  15. Editing Captions on the Front-end • Captions API shouldn’t care

    what’s used for editing • Saved edits should be treated as new versions of a caption Closed Captions API Editor Client Captions JS Client Request Caption by uid Pass Caption Data Pass New Caption Submit New Caption Version
  16. Playback Matters Too 7

  17. WebVTT Files on HLS #EXTM3U #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",LANGUAGE="en",NAME="English",AUTOSELECT=YES,DEFAULT=YES,URI="index-f6.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3778274,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f1-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=365226,RESOLUTION=426x240,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT

    ITLES="subs0" index-f2-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=967515,RESOLUTION=640x360,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT ITLES="subs0" index-f3-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1568184,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f4-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2573061,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f5-v1-a1.m3u8
  18. WebVTT Files on HLS • HLS.js • Apple Safari •

    Google ExoPlayer • Apple AVPlayer
  19. Adding to All Our Players • Playback on Desktop, Mobile,

    iOS native and Android native • “Safe Zones” to avoid visual collisions
  20. Future 8

  21. What We’re Thinking About Next • Ingest Captions ◦ Facebook

    ◦ YouTube • Use Closed Captions to help Search ◦ Content discoverability • Use Closed Captions as Metadata ◦ Highlights Detection ◦ Recommendations Systems • Internationalization
  22. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com

  23. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com

  24. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com

  25. Thank you! /saidketchman { , } /flavioribeiro /introvertedeng