Building a Closed Captions Ecosystem at The New York Times

Talk about how we are building closed captions support for The New York Times video library at the Video Tech Meetup in New York.

Transcript

1. Building a Closed Captions Ecosystem at The New York Times Said Ketchman & Flavio Ribeiro
2. Said Ketchman Engineering Manager, Video http://github.com/saidketchman http://twitter.com/introvertedeng http://introvertedengineer.com
3. Flavio Ribeiro Lead Engineer, Video http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com
4. An Appeal for Closed Captions 1
5. An Appeal for Closed Captions Why Standardized Captions? • Following Web Standards • Complying with Accessibility Standards • Enabling Muted Autoplay • Avoiding 360 Videos Burn-in • Enhancing Site SEO • Internationalization (translations) Our Required Approach • Integration with “Media Factory” • Avoiding Vendor Dependency • Providing Multiple Generation Options • Allowing Robust Editability of Captions • Fully Integrated with NYTimes’ CMS • Ease of Use for Video Editors/Journalists
6. First, Some Context 2
7. Media Factory Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN
8. A New Approach for Video Delivery (OTFP) 3
9. Media Factory Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN + CDN GCS
10. Vendor Agnostic Approach for Greatest Flexibility 4
11. Vendor Agnostic Caption Generation • Caption Generation should not be coupled to a single vendor • Easily switch between vendors via request parameters • Similar approach as our Transcoding API Closed Captions API 3Play Google Speech API Amazon Lex Generation Client Abstraction etc...
12. Adding Closed Captions Generation to Our Encoding Workflow 5
13. Media Factory w/ OFTP Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN + CDN GCS Closed Captions API
14. Not Done Yet: Editing Matters Too 6
15. Editing Captions on the Front-end • Captions API shouldn’t care what’s used for editing • Saved edits should be treated as new versions of a caption Closed Captions API Editor Client Captions JS Client Request Caption by uid Pass Caption Data Pass New Caption Submit New Caption Version
16. Playback Matters Too 7
17. WebVTT Files on HLS #EXTM3U #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",LANGUAGE="en",NAME="English",AUTOSELECT=YES,DEFAULT=YES,URI="index-f6.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3778274,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f1-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=365226,RESOLUTION=426x240,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT ITLES="subs0" index-f2-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=967515,RESOLUTION=640x360,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT ITLES="subs0" index-f3-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1568184,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f4-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2573061,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f5-v1-a1.m3u8
18. WebVTT Files on HLS • HLS.js • Apple Safari • Google ExoPlayer • Apple AVPlayer
19. Adding to All Our Players • Playback on Desktop, Mobile, iOS native and Android native • “Safe Zones” to avoid visual collisions
20. Future 8
21. What We’re Thinking About Next • Ingest Captions ◦ Facebook ◦ YouTube • Use Closed Captions to help Search ◦ Content discoverability • Use Closed Captions as Metadata ◦ Highlights Detection ◦ Recommendations Systems • Internationalization
22. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
23. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
24. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
25. Thank you! /saidketchman { , } /flavioribeiro /introvertedeng