Top 5 Things I've Messed Up in Live Streaming

June 24, 2014 - SF Video Technology Meetup. Live streaming can be difficult and challenging, yet extremely rewarding. In the video gaming world, it has become the most powerful way of reaching your audience. It combines traditional web technologies with even more traditional broadcast video stacks in a "it's going live now" environment. In this presentation Lee Chen, head of product at Fastly, goes over the top 5 things that have lost him sleep, made his eyebrows go up, and sometimes even made him fail -- what he did to correct it.

Transcript

1. Top 5 things I’ve F’d up (In Live Streaming)
2. About me •  GotFrag.com •  Major League Gaming •  Product Manager for Streaming Media at Fastly •  Head of Product at Fastly
3. What’s this all about? •  Live streaming (not live linear) •  The (comedic) errors: 5. Forest, Trees. The big picture. 4. Latency sucks. 3. Failover. Not Fall Over. 2. Redundancy is not redundant. 1. Death by a thousand cuts. Measure it!
4. Some assumptions •  What: –  Live streaming is hard –  It shouldn’t be. –  But it is. –  It’s real time. –  So it’s hard. •  Why: –  “Analog” broadcast to “Digital” medium –  Every step of the signal path can break it –  And then there’s the Internet.
8. #5: The whole picture is important •  Analog to Digital •  Wires to Code •  Complexity •  Troubleshooting and QA is therefore hard •  Studio != Touring Show
9. Live Streaming (Analog Side)
10. Live Streaming (Digital side)
11. #5: What got F’d. •  Audio out of sync •  Root cause was on the switcher <> audio board •  Keyframes >1s on Encoder compounded problem
12. #4: Spoilers suck •  Time Behind Live (TBL) is critical •  Broadcast standard is <15s TBL •  In other words, latency sucks. – Latency = Buffering, jitters – Latency = Spoilers – Latency = WTF just happened.
13. Latency Sources (Digital side) Source   Ideal  Avg  Latency   Encoding   <100ms   Uplink   IP  =  <100ms,  Sat  =  <1500ms   Ingest   <10ms   Transcoding   50  -­‐  500ms   CDN   150  –  1500  ms   Last  Mile:  Player  /  Browser  /  User   Internet   ?????   Total:   3610ms  (3.6sec)  +  Last  mile  
14. #4 What got F’d •  RTMP delivery sucks on a CDN –  5min into broadcast = 5sec TBL –  30min into broadcast = 30sec TBL –  8hrs into broadcast = 2min TBL •  Trolls hate spoilers. So do legit fans. •  Separate stack from normal delivery, older hardware •  HTTP streaming is the answer (HLS!!)
15. #3: Fail over. Not Fall over. •  Duh.
16. #3: What got F’d •  Just cause the vendor says so, doesn’t mean it _IS_ so. •  Test it. •  No really, TEST IT. •  Can you fail back and forth? •  Does it cause disconnects at player?
17. #2: Redundancy is not redundant •  Analog – Matrixed encoders – Multiple uplinks – Back up routers •  Digital – Cloud transcoding – Multi-CDN
18. #2: What got F’d •  Production Budget – $10k/hr for sat truck •  Upstream ISP link from venue died •  Fix: Microwave pt to pt to the Luxor.
19. #1: Real time feedback. Flexible solutions Digital feedback: •  Analytics. –  Disconnects –  Buffering –  Edge usage –  Load on systems –  Geo distro –  etc Analog feedback •  Leverage the audience –  Twitter •  Watch the stream yourself •  Filter the noise Flexible partners and solutions
20. #1: What got F’d •  Tendency is to have digital side isolated from the analog side. •  Don’t. Analog thinks about the room. Digital thinks about the “Internets” Put the two together.
21. Thank you www.fastly.com