Building a Video Encoding Pipeline at The New York Times

These slides were presented at the Streaming Media West conference in 2016. This talk is also a reference to the blog post "Using Microservices to Encode and Publish Videos at The New York Times" at The New York Times Open blog.

- Streaming Media West 2016: http://streamingmedia.com/Conferences/West2016/
- Open Blog: http://open.blogs.nytimes.com/2016/11/01/using-microservices-to-encode-and-publish-videos-at-the-new-york-times/

Transcript

1. Building a Video Encoding Pipeline at The New York Times Streaming Media West, 2016
2. Maxwell Dayvson Da Silva Sr. Director of Engineering http://github.com/dayvson http://twitter.com/dayvson http://dayvson.me
3. Flávio Ribeiro Sr. Software Engineer http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com
4. First, a little bit of history
5. The Architecture of the old system Video CMS (StoryLog) NYTimes CMS (Scoop) Internal Encoding Cluster Internal Storage CDN
6. The flaws
7. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs are required to publish a video.
8. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. 4. Unstable transcoding (proprietary software) 5. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4
9. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Production ID
10. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 5. 43380_1_saudi-documentary_wg_720p.mp4 Version
11. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Slug
12. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. 5. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Resolution
13. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
14. The Flaws 1. Filenames 2. Several watch folders 3. U 4. Hard to scale (cluster in-house)
15. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
16. The Flaws
17. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
18. Number of videos published per year.
19. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs to publish a video
20. The challenges
21. The Challenges 1. Fast encoding. News Clips are very time sensitive 2. Scalability 3. Reliability 4. Company-wide video transcoding solution 5. High-quality transcoding ◦ 360°, Drone footage, Live Action
22. The solution
24. The Video Pipeline Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
25. Media Factory Client Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API Database CDN
26. Media Factory Client 1. EcmaScript 6 Library 2. Upload Files from Browser to S3 ◦ http://github.com/TTLabs/EvaporateJS 3. Trigger Jobs on Media Factory API 4. Used by other teams
27. Acquisition API Distribution API Acquisition API Transcoding API Storage Media Factory Client Database Media Factory API CDN
28. Acquisition API 1. Back-end integration to upload files ◦ External partners 2. Plan to support resumable uploads
29. Database Media Factory Client Storage Media Factory API Distribution API CDN Acquisition API Transcoding API Media Factory API
30. Media Factory API 1. All NYTimes business logics ◦ Interact with our internal APIs ◦ Define which presets will be used based on job type (Standard, 360) ◦ http://github.com/nytimes/video-presets
31. Transcoding API Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
32. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification
33. Transcoding API - Job Example { "provider": "elastictranscoder", "source": "s3://somebucket/folder/my_video.mov", "streamingParams": { "playlistFileName": "hls/playlist.m3u8", "segmentDuration": 5, "protocol": "hls" } "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ] }
34. Transcoding API - Preset Example "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "profile": "Main", "profileLevel": "3.1", "rateControl": "VBR", "video": { "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }
35. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification 5. http://nyti.ms/mediafactory
36. The Distribution API Database Media Factory API Distribution API Acquisition API Media Factory Client Transcoding API Storage CDN
37. Distribution API 1. Fast file transfer leveraging Aspera protocol 2. Plan to support distribution to our partners ◦ Youtube, Facebook, other partners 3. Plan to support to multiple CDNs
38. Just a recap! Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
39. Future 1. Video Quality ◦ Classify into groups ◦ VMAF, Reingest on the Media Factory API 2. Open Source ◦ Distribution API ◦ Acquisition API 3. Snickers (http://github.com/snickers/snickers)
40. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
41. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
42. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
43. Thank you! { , } /flavioribeiro { , } /dayvson