Building a Closed Captions Ecosystem at The New York Times Said Ketchman & Flavio Ribeiro 

Said Ketchman Engineering Manager, Video http://github.com/saidketchman http://twitter.com/introvertedeng http://introvertedengineer.com 

Flavio Ribeiro Lead Engineer, Video http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com 

An Appeal for Closed Captions 1 

An Appeal for Closed Captions Why Standardized Captions? ● Following Web Standards ● Complying with Accessibility Standards ● Enabling Muted Autoplay ● Avoiding 360 Videos Burn-in ● Enhancing Site SEO ● Internationalization (translations) Our Required Approach ● Integration with “Media Factory” ● Avoiding Vendor Dependency ● Providing Multiple Generation Options ● Allowing Robust Editability of Captions ● Fully Integrated with NYTimes’ CMS ● Ease of Use for Video Editors/Journalists 

First, Some Context 2 

Media Factory Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN 

A New Approach for Video Delivery (OTFP) 3 

Media Factory Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN + CDN GCS 

Vendor Agnostic Approach for Greatest Flexibility 4 

Vendor Agnostic Caption Generation ● Caption Generation should not be coupled to a single vendor ● Easily switch between vendors via request parameters ● Similar approach as our Transcoding API Closed Captions API 3Play Google Speech API Amazon Lex Generation Client Abstraction etc... 

Adding Closed Captions Generation to Our Encoding Workflow 5 

Media Factory w/ OFTP Transcoding API Acquisition API Storage Distribution API Media Factory Client Media Factory API Database CDN + CDN GCS Closed Captions API 

Not Done Yet: Editing Matters Too 6 

Editing Captions on the Front-end ● Captions API shouldn’t care what’s used for editing ● Saved edits should be treated as new versions of a caption Closed Captions API Editor Client Captions JS Client Request Caption by uid Pass Caption Data Pass New Caption Submit New Caption Version 

Playback Matters Too 7 

WebVTT Files on HLS #EXTM3U #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",LANGUAGE="en",NAME="English",AUTOSELECT=YES,DEFAULT=YES,URI="index-f6.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3778274,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f1-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=365226,RESOLUTION=426x240,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT ITLES="subs0" index-f2-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=967515,RESOLUTION=640x360,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBT ITLES="subs0" index-f3-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1568184,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f4-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2573061,RESOLUTION=854x480,FRAME-RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUB TITLES="subs0" index-f5-v1-a1.m3u8 

WebVTT Files on HLS ● HLS.js ● Apple Safari ● Google ExoPlayer ● Apple AVPlayer 

Adding to All Our Players ● Playback on Desktop, Mobile, iOS native and Android native ● “Safe Zones” to avoid visual collisions 

Future 8 

What We’re Thinking About Next ● Ingest Captions ○ Facebook ○ YouTube ● Use Closed Captions to help Search ○ Content discoverability ● Use Closed Captions as Metadata ○ Highlights Detection ○ Recommendations Systems ● Internationalization 

We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com 

Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com 

Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com 

Thank you! /saidketchman { , } /flavioribeiro /introvertedeng