nginx.conf 15 - globo.com's live video platform for fifa world cup 14

In this talk, we will describe globo.com's live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .
Nginx is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.

this was presented by leandro moreira and juarez bochi at nginx.conf in 09/23/15

Transcript

1. Globo.com’s Live Video Platform for FIFA World Cup 14 09/23/15 #nginx #nginxconf
2. an amazing teamwork Leandro Moreira other teams Daniel Martins Flávio Ribeiro Juarez Bochi Hugo Roque Robson Soares Rafael Borsani Bernardo Camilo Lucas Mundim Juan Almeida more people
4. streaming rights for Brazil only
5. Agenda (4+ years of work) 2010 Architecture for FIFA World Cup 10 2012 Mobile support 2013 HLS adoption 2014 DVR 2014 Waiting room 2014 FIFA World Cup results 2015 Recap & Next steps
6. architecture in 2010 #nginx #nginxconf
7. {2010} in the beginning encoder h264+aac rtmp tcp h264+aac rtmp tcp flash player firewall 1935 1º 2º 3º 4º RTMP
8. {2010} good part small delay (2-5s)
9. {2010} issue :: instrumentation tail -f kill -9 ps aux
10. {2010} issue :: scalability $ $ $ $ $
11. {2010} issue :: scalability
12. {2010} some data from FIFA10 • ~= 285k simult. users * • max bitrate 800 Kbps *peak
13. mobile support #nginx #nginxconf
14. {2012} and then multi-devices... hls h264+aac rtmp tcp h264+aac rtmp tcp h264+aac rtmp tcp hls
15. {2012} how does HLS work? seg1.ts seg2.ts seg3.ts seg4.ts 240p.m3u8 480p.m3u8 720p.m3u8 seg1.ts seg2.ts seg3.ts seg4.ts seg1.ts seg2.ts seg3.ts seg4.ts 1 - GET /variant.m3u8 (text) 2 - GET /720p.m3u8 (text) 240p.m3u8 variant.m3u8 480p.m3u8 720p.m3u8 3 - GET /seg3.ts (video) 4 - GET /seg4.ts (video) 5 - GET /720p.m3u8 (text) 6 - GET /seg5.ts (video) 7 - GET /seg6.ts (video) 8 - ... http http seg3.ts seg4.ts seg5.ts seg6.ts seg3.ts seg4.ts seg5.ts seg6.ts seg3.ts seg4.ts seg5.ts seg6.ts
16. {2012} first hls solution encoder h264+aac rtmp tcp h264+aac http tcp flash player 2º 3º 5º 4º android ios 1º HTTP
17. {2012} generation and distribution rtmp video playlists files location ~ /hls { root "/vol/hls/"; } dest = "/vol/hls"; http BE FE segmenter
18. {2012} first solution architecture BE FE LB SE LB LB auth & cache mostly cache hls generation upstream upstream http
19. HLS adoption #nginx #nginxconf
20. {2012} one protocol to rule them all
21. {2013} buffering hls x rtmp 6x less buffering with HLS
22. {2013} instrumentation Graphite • status code • request t. • cache hit • cpu usage • memory • buffer rate • bitrate time-series data graph api http enqueue dequeue metric x t snmp metric x t agents
23. {2013} sauron Graphite
24. {2014} caching configuration don’t forget to: proxy_cache_use_stale updating 011011100 110011101 101001011 011100111 1000 video0.ts video0.ts expired UPDATING FE BE proxy_cache_lock
25. {2014} OS fine tuning
26. {2014} OS fine tuning 19Gpbs maxconn: 70k, cpu affinity 15Gpbs maxconn: 40k, cpu affinity 12Gpbs maxconn: 25k, irqbalance packet loss 4Gbps, no fine tuning *each machine has 2 NIC bonded (10GB each one)
27. {2014} what we’ve got here is Scalability port:80 UX Instrumentation 2G 4G Wifi 1080p 720p 320p Caching
28. {2014} farm size FE: 80 nodes 24 cores 64 GB 1TB 10 Gbps 10 Gbps BE: 6 nodes SE: 2 nodes
29. DVR #nginx #nginxconf
30. FIFA 2014 World Cup Brasil
31. {2014} what is DVR for hls? /720p.m3u8 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts #EXTINF:5, seg2.ts #EXTINF:5, seg3.ts /720p.m3u8 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts #EXTINF:5, seg2.ts #EXTINF:5, seg3.ts #EXTINF:5, seg4.ts #EXTINF:5, seg5.ts #EXTINF:5, seg6.ts #EXTINF:5, seg7.ts #EXTINF:5, seg8.ts #EXTINF:5, seg9.ts #EXTINF:5, seg10.ts
32. ingest {2014} reviewing current solution frontend encoder h264+aac rtmp tcp h264+aac http tcp flash player 2º 3º 5º 4º android ios 1º
33. {2014} dvr challenges :: failover didn’t have a failover option for DVR #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:10 #EXTINF:5, seg10.ts #EXTINF:5, seg11.ts #EXTINF:5, seg12.ts #EXTINF:5, seg13.ts #EXTINF:5, seg14.ts #EXTINF:5, seg15.ts #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5, seg1.ts
34. {2014} dvr challenges :: storage 2h * 60m * 60s / 5s * 4.5mb ~= 6.4GB } 1.6MB 1.1MB 0.8MB 0.5MB 0.3MB 0.1MB 4.5MB per 5s 2 games + 4 other streams ~= 38.4GB 02:00:00 00:00:00
35. {2014} redis as a datastore ingest embed-script: fetch and serve hls frontend daemon: mon file change and save hls
36. {2014} Brazil’s general election too many simultaneous live streams
37. {2014} from redis to cassandra ingest embed-script: fetch and serve hls frontend daemon: mon file change and save hls
38. waiting room #nginx #nginxconf
39. {2014} multihoming (anycast) ISPx ISPy ISPk SP BGP announcement 186.192.81.0/24 ISPx ISPy ISPk RJ 186.192.81.2 BGP announcement 186.192.81.0/24 PoP PoP
40. {2014} user’s bandwidth ISP X ISP Y 20Gbps 5Gbps IXP ISP K 80Gbps 8Gbps
42. FIFA World Cup 2014 results #nginx #nginxconf
43. FIFA 2014 World Cup results
44. FIFA 2014 World Cup Brazil data
45. • > 500k simult. users (peak 1 game) • ~= 125.000 video req/s (peak 1 game) • > 640 Gbps (peak 1 game) FIFA 2014 World Cup Brazil data
46. FIFA 2014 World Cup Brazil data • ~= 1600 watched years (sum all) • > 40M video views (sum all) • ~= avg 6 Gbps, max 19 Gbps
47. • ~= cpu usage max 10% (per node, FE) • ~= avg bitrate 480 Kbps (crowded) • ~= avg bitrate 1864 Kbps (less crowded) FIFA 2014 World Cup Brazil data
48. Recap and next steps #nginx #nginxconf
49. • video streaming: ◦ delivery and caching ◦ geolocation (geofencing) ◦ authorization / authentication • microservices ◦ playlist generation ◦ waiting room ◦ streams per account locking system Nginx usage
50. is amazing Lua was made in Brazil :) TDD with busted easier to develop performance near C (lang)
51. OSS from this history m3u8 parser cassandra driver clappr evostream api live_thumb nginx-audio-track-module BemTV iprange
52. summary and future 2010 FIFA 2012 BBB 2014 FIFA Elections • hls • http • flash • rtmp • hls • http • dvr 2016 Olympic Games • dash ??? • ingest ??? • 4k ???
53. Questions? THANK YOU! leandromoreira.com.br jbochi.me