Building Machine Learning Products at Scale

Lessons learned deploying machine learning models to 49 million customers at Groupon.

Transcript

1. BUILDING MACHINE LEARNING PRODUCTS AT SCALE LESSONS LEARNED DEPLOYING MACHINE LEARNING MODELS TO 49 MILLION CUSTOMERS Laura Hamilton Group Product Manager at Groupon Predictive Analytics Summit, Chicago October 30, 2018
2. CHALLENGE: GROUPON SCALE - 15 countries and 500+ markets - 49.3 million active customers globally - Nearly 1.5 billion Groupons sold - More than $30 billion saved by consumers - 183 million app downloads - Top 6 ranked app of all time ...and we need to make decisions on what deals to show the customer in fractions of a second!
3. ML APPLICATIONS Discovery and personalization—Provide personalized deal recommendations based on your previous purchase and browse history. Supply Intelligence—There are millions of merchants we could call at any time to get onto our platform. How do we pick the best ones? Fraud prevention—Fighting the bad guys in real time. Logistics—Reduce the time it takes customers to receive their Groupon Goods by sending extra inventory to our warehouse in advance of forecasted demand. Image recognition—Typing is so 2017. Take a picture of your credit card and we’ll do the rest.
4. AGENDA • 8 challenges Groupon has faced deploying machine-learning algorithms at scale • How Groupon has solved those problems with 3 ML frameworks Image credit: XKCD
5. LESSON #1: COUPLING • Tightly coupled ETLs (Extract, Transform, Load) • Tables built on tables built on tables • Glued together with bash scripts and cron jobs • Difficult to monitor • Frequent failures Image credit: XKCD
6. LESSON #2: LOUD & SILENT FAILURES • Subtle changes in a single data field can seriously impact model performance • Nuances in your data set can look fine to tests, but fail in the real world Image credit: Giphy
7. QED: Quantum Engineered Data An ETL-management platform that reads any source. Built-in cleaning, error correction, and anomaly detection. Image: Unsplash SOLUTION #1: QED
8. LESSON #3: BLACK BOXES • Data scientists have to throw the model over the wall to engineers to implement • ML models look like magical black boxes to most engineers • Tracing a problem to the model behavior is not straightforward Image credit: Unsplash
9. LESSON #4: SLOW INNOVATION • To make a change, data scientists need to rebuild everything from scratch • Optimizing the hyperparameters requires coding and a deployment • It can be difficult to anticipate the impact of a model change in order to gauge the risk of said change, which can paralyze the model lifecycle Image credit: Unsplash
10. The rosetta stone between data scientists and engineers R => Python => Java & Clojure (on Hadoop Cluster) SOLUTION #2: FLUX Image: Wikimedia.org
11. LESSON #7: LOCALIZATION • Vertical mix • Deal density • Propensity to travel • Merchandising strategy • Consumer trends Image credit: Unsplash
12. LESSON #8: EXPERIMENTATION IS HARD • Old Naive Bayes model was hand-crafted and hand-tuned • No great way to handle continuous data such as distance • Fundamental changes to algorithm require code changes • No standard way to incorporate new features and data sources • No standard way to evaluate impact and accuracy of new data sources • Localization is time- and labor-intensive Image credit: Unsplash
13. SOLUTION: YODA How it works: • Data processing, storage, and access with data engineering platform • Offline learning with sparkling water • Online evaluation with ElasticSearch plugin using a Flux library Benefits: • Clear separation of concerns • Automagical localization • Continuous experimentation • Continuous variables! • Reusable by multiple teams Not pictured: Yoda
14. Questions?