Netflix Recommendations Infrastructure - Spark Summit 2018

Transcript

1. #ML7SAIS Elliot Chow, Netflix Nitin Sharma, Netflix Near Real-Time Recommendations - Spark Streaming #ML7SAIS
2. #ML7SAIS Agenda • Recommendations @ Netflix • The Need for Near Real Time • Use Cases • Common Infrastructure • Scaling Challenges
3. #ML7SAIS Recommendations at Netflix • Personalize the Netflix experience for each member ◦ Goal: Quickly help members find content they’d like to watch ◦ Risk: Member may lose interest and abandon the service ◦ Challenge: Recommending at scale
4. #ML7SAIS Scale @ Netflix • 125M+ active members • 190 countries • 450B+ unique events/day • 700+ Kafka topics
5. #ML7SAIS • Data stored in Hive/S3 • Batch ETLs using Spark/Hive • Table partitioning by day or hour • Job scheduling by both CRON or data availability • Latency often is on the order of days Typical Data Pipelines @ Netflix
6. #ML7SAIS • Dynamic catalog • Growing member base • Time sensitivity ◦ Content popularity changes ◦ Member interests evolve The Need for Near Real Time (NRT)
7. #ML7SAIS • Increasing amount of data ◦ Process data as soon as possible to keep latencies low ◦ Minimize amount of data to reprocess in case of failure • Multi-Armed Bandits Adoption The Need for Near Real Time (NRT)
8. #ML7SAIS Use Cases • Video Insights • ML Pipelines for Recommendations
9. #ML7SAIS NRT for Video Insights
10. #ML7SAIS Video Insights • New title launches • Early reads on title performance • Slice by various dimensions
11. #ML7SAIS Video Insights Service
12. #ML7SAIS Video Insights Service
13. #ML7SAIS Video Insights Service
14. #ML7SAIS Video Insights Service
15. #ML7SAIS Video Insights Service Client
16. #ML7SAIS Video Insights - State • Counts maintained in Spark • Custom state management ◦ Based on mapWithState implementation ◦ Easier to re-use the function f in batch mode ◦ Lower-level control over state management ◦ scanByKey alternative for keyed state input.scan(initRDD)((currentRDD, batchRDD) => f(currentRDD, batchRDD))
17. #ML7SAIS NRT for Recommendations
18. #ML7SAIS Billboard Recommendations • Recommend a relevant title to each member • Right time • Respond quickly to member feedback
19. #ML7SAIS Artwork Personalization 19 • Personalized Image • Visual Evidence • Quickly adapting - Title launches, member tastes • Rapid learning - Cold start
20. #ML7SAIS Traditional Recommendations Millions of Play related Signals Training pipelines Models Precompute/Live Compute Rankings DataStore/ Online caches ETL Layers Few days
21. #ML7SAIS NRT Recommendations Millions of Play related Signals Models Precompute/Live Compute Rankings DataStore/Online caches Training pipelines Streaming layer NRT
22. #ML7SAIS Required Data • Impressions, Plays, etc. • Attribution • Explore/Exploit Metadata
23. #ML7SAIS Attribution Infrastructure Stream Processing Batch Processing Query API
24. #ML7SAIS Stream Processing - Zoomed in Impression Cache Video Metadata Play Enrich
25. #ML7SAIS Batch Processing - Zoomed in Impression Play Experimentation Data Windowed Impression Windowed Play Dedupe Join Video Metadata
26. #ML7SAIS Common Infrastructure
27. #ML7SAIS Netflix Spark Stack • Jenkins • Spinnaker • ASG • Runners: Marathon, Meson • Resource Manager: Mesos • Storage: HDFS, S3, EFS • Multi-Region
28. #ML7SAIS Multi Region Challenges • Geo routing • Impression in one region; play in another • Streaming - Multi Region • Batch Reduce/Merge - One region
29. #ML7SAIS Can it withstand Chaos? • Chaos is a design principle • Instance Failovers => Region Failovers • Transparent to the consumers • Over provisioned • Long term - Autoscale
30. #ML7SAIS When Things Go South • What if something doesn’t look right? ◦ Stream Processing is stuck ◦ Driver/Executor failures ◦ Intermittent issues with external dependencies • Metrics - Spark metrics to Atlas (similar to RRDTool + Graphite) • Getting paged at 2 am - Not fun :) ! • Need for auto-remediation - less operational overhead
31. #ML7SAIS Auto Remediation Infrastructure SQS Auto Remediation De-queue Triggers Metadata
32. #ML7SAIS • Scalability Performance tuning – Micro batch interval – Memory Tuning – Parallelism/Shuffle tradeoff • Data quality issues – Low latency vs data quality Streaming Challenges