Engineering a Live Streaming Workflow for Super Bowl 53 at CBS

Presentation about how we built the video processing and delivery pipelines for Super Bowl 53. Presented at the Boston Video Technology Meetup in August, 2019.

https://www.meetup.com/BOS-Video-Technology/events/262652682/

1. Engineering a Live Streaming Workflow for Super Bowl 53 at CBS
2. Flavio Ribeiro Director of Engineering, Video Processing Team
3. AGENDA • Signal Acquisition & Encoding ◦ Bitrate Ladder ◦ Output Locking • Advertisement System • Delivery • Monitoring • Testing • Game Day
4. MISSION: PROMOTE A GOOD EXPERIENCE FOR OUR USERS THROUGHOUT THE GAME
5. SIGNAL ACQUISITION & ENCODING
6. SIGNAL ACQUISITION & ENCODING
7. SIGNAL ACQUISITION & ENCODING ← Master Encoder @ NYBC
8. SIGNAL ACQUISITION & ENCODING ← TIMECODE EMBEDDER FRAME TIMECODE EMBEDDER CARD ➞
9. BITRATE LADDER • 400x224, 29.97 @ 250 Kbps (QVBR, Quality Level: 7) • 512x288, 29.97 @ 500 Kbps (QVBR, Quality Level: 7) • 640x360, 29.97 @ 900 Kbps (QVBR, Quality Level: 7) • 960x540, 29.97 @ 1.8 Mbps (QVBR, Quality Level: 7) • 1280x720, 29.97 @ 3 Mbps (QVBR, Quality Level: 7) • 1280x720, 59.94 @ 6 Mbps (QVBR, Quality Level, 8)
10. QVBR
11. OUTPUT LOCKING More about it: https://d2908q01vomqb2.cloudfront.net/fb644351560d8296fe6da332236b1f8d61b2828a/2019/02/25/AWS_Product-Documentation_OUTPUT-LOCKING.pdf • Same output medatada across encoders in the pool ◦ PTS/DTS • Encoders talk via Multicast • Distributed Consensus ◦ No master node • Beware of Ad Breaks and SCTE/ID3/API integrations
12. ADVERTISEMENT SYSTEM
13. DELIVERY
14. OUTCOME
15. OUTCOME
16. MONITORING • Mux.com
17. MONITORING • Mux.com • TouchStream
18. MONITORING • Mux.com • TouchStream • SumoLogic
19. MONITORING • Mux.com • TouchStream • SumoLogic • Manifest Monitor
20. MONITORING • Mux.com • TouchStream • SumoLogic • Manifest Monitor • MultiScreen Viewer ◦ Clappr + Clappr RTMP + Nginx
21. MONITORING • Mux.com • TouchStream • SumoLogic • Manifest Monitor • MultiScreen Viewer ◦ Clappr + Clappr RTMP + Nginx • Slack Alerts
22. TESTING • Streaming in Production prior to Game Day • Identifying Potential Points of Failure ◦ Simulate Chaos • Runbooks / Playbooks
23. TESTING • Streaming in Production prior to Game Day • Identifying Potential Points of Failure ◦ Simulate Chaos • Runbooks / Playbooks
24. TESTING • Streaming in Production prior to Game Day • Identifying Potential Points of Failure ◦ Simulate Chaos • Runbooks / Playbooks
25. TESTING • Streaming in Production prior to Game Day • Identifying Potential Points of Failure ◦ Simulate Chaos • Runbooks / Playbooks • Tooling for Quickly Making Changes ◦ Must be well tested
26. GAME DAY NY - ENCODING SF - DELIVERY FL - SITE
28. GAME DAY Streaming coverage of Super Bowl LIII set viewership records, (...). Across 7.5 million unique devices, (...) viewers consumed more than 560 million total minutes of live game coverage, up +19% from last year. The average minute audience of 2.6 million viewers during the game window is up +31% year-over-year. Reference: https://www.cbspressexpress.com/cbs-sports/releases/view?id=51928
29. WE ARE HIRING!
30. Thanks! @flavioribeiro