Google I/O Extended 2019 presentation

Content recommendation engine slides

Transcript

1. Productionalize Personalized Content Recommendation Engine & Full Machine Learning Pipeline Hafeez Nazri & Kim Ming Teh Data Science & Analytics Media Prima Digital
2. Overview Business Goal 1 Data & Technology 2 Data Science 3 Impact 4
3. Understand The Business
4. WHO ARE YOUR USERS? WHAT DO THEY REALLY LOOK FOR?
5. User Behavior Shift
6. Customer Journey, Touchpoint & Decision
7. Customer 360 Opportunity Obtain a complete view of digital users through the creation of a Customer 360 system Data & Analytics Combining and aggregating data from multiple sources – Gigya / CRM, Lotame, GA and external – and matching a single person across all digital touch points Outcome Better understanding of customer behaviour; able to deliver personalized experiences to our users - through highly-targeted ads for instance - to drive marketing and sales growth
8. Enables Micro-Segmentation of Users • Users with 75% scroll depth +super interested in SPORTS +med interest in BUSINESS +low interest in CRIME • Mothers who are football fans • Outdoor adventurers with income above RM150,000 per year
9. Understand over 14+ MILLIONS user behavior & better serve them with relevant content and services Opportunities
10. Personalized User Experience
11. Content Recommendation Opportunity Content recommendation system in our digital assets to improve user engagement by recommending relevant content Data & Analytics Data ingested from GA and article database…and using an AI technique called collaborative filtering – recommendation based on similarity to other users Outcome • Savings of US$250,000 per year from replacing third-party Cxense • Users who click on recommended items show more engagement, have 3x higher pages/session and stay 2x longer compared to the average user
13. Identify Your Challenges & Risks Pre-development Short timeline - We were given 1 month to deliver and deploy this project! WTH? 1 Resources - small team ( 2 DE and 1 DS) 2 Integration & Technology - widget using Drupal - anonymous users? WTF? - cheapskate? easiest yet accurate & efficient? 3
14. • capture, store, process real-time and big data • create data-driven application Data & Technology
16. Customer360 Hi-Level Design
17. User, Item & Interaction Data
19. 1 2,3,4 5 Process 1. ETL 2. Article Mapping 3. Tensorflow Training 4. Output Mapping 5. API Output Cache High Level System Data Architecture
20. What Is Big Query? • GCP’s Enterprise Data Warehouse for Analytics • Petabyte scale storage and analysis • Real-time analytics on streaming data • Fully-managed and serverless • Support standard SQL, JDBC & ODBC • Power of Big Query : having petabyte scale of data that is fully managed and serverless • CHEAP!
21. ETL Process
22. Recommendation Mapping Process
23. API Output
24. Request: { “ClientId” : 1000003707.1519054 } Response object: { "nid": 417373, "title": "Najib spends three hours with AMLA Unit police", “click_url” : To-be-confirmed the method to track the click event “rec_type” : user-based ( Optional) "url": "https://www.nst.com.my/news/nation/2018/10/417373/najib-spends-three-hours-amla-unit-police", "field_article_images": [ { "url": "https://assets.nst.com.my/images/articles/najibcharged_1538549306.jpg", "caption": "File picture of former prime minister Datuk Seri Najib Razak when he was charged with several counts under the Anti-Money Laundering and Terrorism Financing Act 2001 on Aug 8. Najib was called in for questioning by police today in connection with investigations into 1Malaysia Development Bhd (1MDB)" } Recommendation API Output
25. • Get Real - Data is not always came in “perfect” structures as your wishlist or tutorial! • Technology / Technical Gap - in your organisation i.e. legacy systems & integration • Performance - Architecture & pipelines to solve performance i.e. increment of concurrent users • You need knowledge in software infrastructures, development pipeline & deployment! - Be Creative Technical Risk & Challenge
26. • type of recommender system and algorithm • model development & evaluation Recommender System & Algorithm
27. Trending (Most read today) Personalized Recommender (Recommended for you) Content-Based Recommender (Related stories) Type of Recommendation System
28. User, Item & Interaction Data
29. User-based Collaborative Filtering
30. • Based on observed user behavior, and not require user profile or pre-label • Users who interact with items in a similar manner will share preferences WHY User-based Collaborative Filtering?
31. MODEL DEVELOPMENT
32. Prepare Input Data & Initiate Model
33. Model Training & Hyperparameter Tuning
34. Generate Recommendation For Each User
35. Business & user engagement metrics • average pages/session • average time/session • bounce rate • Click through Rate (CTR) Recommender system metrics • RMSE (root mean square error) Model Evaluation & Measurement
36. Model Evaluation - Recommender System Metric
37. https://cloud.google.com/solutions/machine-learning/recommend ation-system-tensorflow-overview Reference & Resource Link
38. • Higher user engagement and retention • Cost saving (replacing existing vendor solution) Impact & Benefit
39. Cost saving (replacing legacy system) Higher User Engagement Impact & Benefit ~RM1 MILLION
40. • Get Buy-in Early - Identify all your stakeholders and involve them since project initiation ( Avoid i build first, they will come later) • Goal & Impact Oriented - Understand what matter to the organization & identify high impact and low-hanging fruit use case • Be Agile - Start small scale fast! Build first prototype, get feedback and run iteration Tips in Deploying Your ML Product
41. QUESTION ?
42. THANK YOU! We’re Hiring! hafeez.nazri@mediaprima.com.my kim.meng@mediaprima.com.my