Improving the Video Delivery at The New York Times

This talk presents the improvements we made on the on-demand stack and live streaming platform at The New York Times.

1. Improving the Video Delivery at Streaming Media West, 2017.
2. Flavio Ribeiro Lead Engineer
 { , } /flavioribeiro http://flavioribeiro.com
3. Flavio Ribeiro Lead Engineer
 { , } /flavioribeiro http://flavioribeiro.com Maxwell Da Silva Executive Director of Engineering
 { , } /dayvson http://dayvson.me
4. agenda
5. agenda • recap of 2016’s SMWest
6. agenda • recap of 2016’s SMWest • on-demand stack improvements
7. agenda • recap of 2016’s SMWest • on-demand stack improvements • live streaming stack improvements
8. agenda • recap of 2016’s SMWest • on-demand stack improvements • live streaming stack improvements • what’s coming next
9. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API CDN Storage / Swap Area recap
10. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API CDN Storage / Swap Area recap
13. some numbers
14. some numbers • > 20k encoding/publishing jobs
15. some numbers • > 20k encoding/publishing jobs • ~ 12 different outputs each (H264/MP4, HLS levels, VP8/WebM)
16. some numbers • > 20k encoding/publishing jobs • ~ 12 different outputs each (H264/MP4, HLS levels, VP8/WebM) • 4 external pull requests on the transcoding API
17. some numbers • > 20k encoding/publishing jobs • ~ 12 different outputs each (H264/MP4, HLS levels, VP8/WebM) • 4 external pull requests on the transcoding API • 6 encoding services supported
18. on-demand stack improvements
20. cdn migration: our video library
21. • migrate assets to Google Cloud Storage 
 cdn migration: our video library
22. • migrate assets to Google Cloud Storage except HLS • generate adaptive formats on-the-fly 
 cdn migration: our video library
23. • migrate assets to Google Cloud Storage except HLS • generate adaptive formats on-the-fly • less storage (~40%) • more formats delivered (MPEG-Dash for free) • faster transcoding process • help on closed captions support
 cdn migration: our video library
24. cdn migration: our video library
25. cdn migration: our video library GCS H264/MP4 files
26. cdn migration: our video library GCS Helper GCS H264/MP4 files
27. cdn migration: our video library GCS Helper + GCS H264/MP4 files
28. cdn migration: our video library GCS Helper + Cache GCS H264/MP4 files
29. cdn migration: our video library Before and after. Less storage, more formats delivered.
32. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API CDN Storage / Swap Area pipeline integration
33. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API Storage / Swap Area pipeline integration
34. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API Storage / Swap Area pipeline integration GCS GCS Helper + Cache
35. closed captions
36. closed captions
37. closed captions • accessibility
38. closed captions • accessibility • follow web standards
39. closed captions • accessibility • follow web standards • muted autoplay
40. closed captions • accessibility • follow web standards • muted autoplay • first step for internationalization
41. closed captions • accessibility • follow web standards • muted autoplay • first step for internationalization • integrate with the pipeline
42. closed captions • accessibility • follow web standards • muted autoplay • first step for internationalization • integrate with the pipeline • avoid vendor dependency
43. closed captions • accessibility • follow web standards • muted autoplay • first step for internationalization • integrate with the pipeline • avoid vendor dependency • multiple generation options
44. closed captions • accessibility • follow web standards • muted autoplay • first step for internationalization • integrate with the pipeline • avoid vendor dependency • multiple generation options • Integrated with our CMS
45. closed captions Captions API 3Play Google Speech API Amazon Lex Generation Client Abstraction etc...
46. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API Storage / Swap Area GCS GCS Helper + Cache pipeline integration
47. Media Factory Client Acquisition API Media Factory API Transcoding API Database Distribution API Storage / Swap Area GCS GCS Helper + Cache Captions API pipeline integration
48. FOTO DO CMS
49. FOTO DO CMS
50. #EXTM3U #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP- ID="subs0",LANGUAGE="en",NAME="English",AUTOSELECT=YES,DEFAULT=YES,URI="index-f6.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3778274,RESOLUTION=854x480,FRAME- RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBTITLES="subs0" index-f1-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=365226,RESOLUTION=426x240,FRAME- RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBTITLES="subs0" index-f2-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=967515,RESOLUTION=640x360,FRAME- RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBTITLES="subs0" index-f3-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1568184,RESOLUTION=854x480,FRAME- RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBTITLES="subs0" index-f4-v1-a1.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2573061,RESOLUTION=854x480,FRAME- RATE=29.970,CODECS="avc1.4d401f,mp4a.40.2",SUBTITLES="subs0" index-f5-v1-a1.m3u8 segmented WebVTT on HLS
51. player integration
52. • hls.js player integration
53. • hls.js • Google ExoPlayer player integration
54. • hls.js • Google ExoPlayer • Apple AVPlayer player integration
55. • hls.js • Google ExoPlayer • Apple AVPlayer • Apple Safari player integration
56. player integration
57. on-demand stack improvements recap!
58. on-demand stack improvements recap! • we are shutting down our datacenters
59. on-demand stack improvements recap! • we are shutting down our datacenters • serving adaptive formats on-the-fly (less storage, more formats, speed up encoding)
60. on-demand stack improvements recap! • we are shutting down our datacenters • serving adaptive formats on-the-fly (less storage, more formats, speed up encoding) • a/b tests to avoid hurting playback experience
61. on-demand stack improvements recap! • we are shutting down our datacenters • serving adaptive formats on-the-fly (less storage, more formats, speed up encoding) • a/b tests to avoid hurting playback experience • closed captions API is just a wrapper for vendors
62. on-demand stack improvements recap! • we are shutting down our datacenters • serving adaptive formats on-the-fly (less storage, more formats, speed up encoding) • a/b tests to avoid hurting playback experience • closed captions API is just a wrapper for vendors • on-the-fly packager takes care of serving the captions too
63. on-demand stack improvements recap! • we are shutting down our datacenters • serving adaptive formats on-the-fly (less storage, more formats, speed up encoding) • a/b tests to avoid hurting playback experience • closed captions API is just a wrapper for vendors • on-the-fly packager takes care of serving the captions too • WebVTT on HLS for the win!
64. live streaming stack improvements
65. some background
66. some background • we had a basic support for live
67. some background • we had a basic support for live • newsroom expressed interest in doing more live streaming events
68. some background • we had a basic support for live • newsroom expressed interest in doing more live streaming events • Presidential Election debates and correlated events
69. live streaming stack @ 2016
70. live streaming stack @ 2016
71. live streaming stack @ 2016
72. live streaming stack @ 2016
73. problem 1: delivery
74. problem 1: delivery • tons of HTTP PUT requests through open internet
75. problem 1: delivery • tons of HTTP PUT requests through open internet • our deal with the CDN was coming to an end
76. live streaming stack @ 2016
77. our new live streaming stack
78. our new live streaming stack
79. our new live streaming stack
80. our new live streaming stack
81. our new live streaming stack
82. problem 2: management
83. problem 2: management • manual setup of the live events
84. problem 2: management • manual setup of the live events • no clear/automated way for live-to-VoD
85. problem 2: management • manual setup of the live events • no clear/automated way for live-to-VoD • no clear/automated way to ingest the stream on partners
86. our new live streaming stack
87. our new live streaming stack
88. our new live streaming stack
92. what’s next?
93. what’s next
94. what’s next • thumbnails service
95. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS
96. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS • A/B test: variable segments size for faster start
97. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS • A/B test: variable segments size for faster start • A/B test: motion covers for click-to-play engagement
98. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS • A/B test: variable segments size for faster start • A/B test: motion covers for click-to-play engagement • closed captions feeding recommendations / search
99. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS • A/B test: variable segments size for faster start • A/B test: motion covers for click-to-play engagement • closed captions feeding recommendations / search • closed captions integration with partners
100. what’s next • thumbnails service • A/B test: fragmented MP4’s versus MPEG-TS • A/B test: variable segments size for faster start • A/B test: motion covers for click-to-play engagement • closed captions feeding recommendations / search • closed captions integration with partners • DVR for live streaming
101. + thumbnails service Application nginx-thumbextractor Google Cloud Storage
102. wanna see more?
103. http://github.com/nytimes/video-transcoding-api Transcoding API http://github.com/nytimes/video-captions-api Captions API http://github.com/nytimes/gcs-helper GCS Helper http://github.com/nytimes/elemental-live-client http://github.com/nytimes/nginx-vod-module-docker +
104. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
105. Stay updated open.nytimes.com
106. Connect with us on Slack! http://video-dev.org
107. thank you! { , } /flavioribeiro
108. • https://speakerdeck.com/flavioribeiro/building-a-closed-captions-ecosystem-at-the-new-york-times • http://blog.flavioribeiro.com/using-open-source-to-create-a-video-thumbnails-service/ • https://open.nytimes.com/improving-our-video-experience-part-one-our-on-demand-video-platform-cf818e03353d • https://medium.com/ellation-tech/just-in-time-vod-packaging-in-an-aws-environment-with-role-based-credentials-4e33202a4450 • https://media.netflix.com/en/company-blog/the-power-of-a-picture more links