ML Frameworks @ Groupon

Transcript

1. ML Frameworks @ Groupon Andrew Lisy and Laura Hamilton November 2, 2017
2. Machine Learning @ Groupon Supply Intelligence - There are millions of merchants we could call at any time to get onto our platform...how do we pick the best ones? Fraud prevention - Fighting the bad guys, in real time Discovery and personalization - Andrew eats a lot of pizza. Laura prefers tacos. Image recognition - Identify the best user-generated images with neural networks Logistics - Get ahead of order rush by sending extra inventory to the warehouse in advance of big demand Image: Groupon.com
3. Challenge: Groupon Scale - One million merchants worked with to date - 49.1 million active customers - 164 million downloads of the mobile app - One billion Groupons sold - $20 billion saved by consumers - Tens of billions of user actions per month - Decisions made in fractions of a second
4. Let’s Talk About Supply Tens of millions of merchants around the world! Thousands of sales and operations people How do we delight our customers with the best possible deals? Image: Flickr
5. We Can Predict the Best Merchants! First….what does best mean? Now, what can we use to predict best? • Location • What do they offer? • Buzz • Past performance • Merchant size
6. So...What Did We Do? Objective Predict potential $$ performance of every merchant that could run on Groupon How? • Look at all the data that we have about every merchant in our system • Clean it up! • Test it with several machine learning models (Train/Validate/Test) • Pick the one where the scoring ($ prediction) most closely matches actual performance How Would We Use It? • Sort leads top to bottom by expected performance • Sales then goes after the most valuable Image: Pixbay
7. We Proved the MVP...now what? The tightly coupled ETLs that fed the ML model to predict the best merchant worked! Great! There’s ton of value here Now, how do we build a truly production system that we can monitor, measure, and iterate on? Image: XKCD
8. What Could Go Wrong? (2015 edition) • ETLs! (Extract, Transform, Load) • Tables built on tables built on tables, glued together with bash scripts and cron jobs • Tightly coupled? You bet. But it worked! (most of the time) ...but most of the time is way worse than all of the time Image: Wikimedia Commons
9. Two Big Challenges Clean Data in Production How do we untangle the ETLs into separate features that we can monitor and quality-check independently? Swappable and Testable Models How do we allow Data Scientists to test new versions of the model without rebuilding everything from scratch? Think like engineers! Separate the concerns, unite them with clean interfaces!
10. Challenge: Loud & Silent Failures The challenge: - Subtle changes in a single data field can seriously impact model performance - Nuances in your data set can look fine to tests, but fail in the real world The solution: - Treat data as a first-class citizen -- data source failures = production failures - Anomaly detection, e.g. monitoring number of null features or mean of a certain feature - Avoid monolithic ETLs with catastrophic failure scenarios - Keep production ML model in a state data scientists can easily understand - Monitor constantly
11. Challenge: Embedded Models The challenge: - It’s hard to test ML models deeply embedded in code - Data Scientists have to throw the model over the wall to engineers to reimplement The solution: - Genericize a system that takes a model object and the required input data, and returns a score - Wire these together in a simple config! No coding required - Put some goodies into the platform! - Map the scoring problem across huge clusters - Automatically measure accuracy and drift over time - “Built-in” alerting Image: Public Domain Pictures
12. Solutions @ Groupon - QED QED is “Quantum Engineered Data”— it’s an ETL management platform that reads for any source and has built in cleaning, error correction, and anomaly detection - Clean data is preserved and made available as a “feature catalog” - Handles failures smartly—can we fall back to yesterday? Do we fail the entire process? - Plugs into any source of truth—streams, warehouse tables, JSON endpoints Image: Pixabay
13. Solutions @ Groupon - Flux We built a generic, extensible machine learning platform called Flux. Flux is the “Rosetta Stone” between data scientists and engineers - Data scientists work primarily in R - Python is the “glue” that connects R and Java - Flux models written in Java and Clojure for stability and speed - Run on Groupon’s large Hadoop cluster Image: Wikimedia Commons
14. Utopia? Not yet...but a huge step that has dramatically simplified machine learning, feature productionalization, and model deployment. Lots of really exciting next steps: • Lambda vs Kappa architectures • More lengthy workflow support • In-system testing Image: Pixabay
15. Questions?