Improving Customer Satisfaction with Automated Monitoring and Anomaly Detection (Madhumita Mantri, StarTree & Leon Graveland, JustEat Takeaway) RTA Summit '23

In today's fast-paced business environment, it is essential to have real-time visibility into customers' experiences and quickly identify any issues that may arise.

By automating the monitoring of critical business metrics such as customer satisfaction we can quickly detect anomalies that may indicate a problem and take action to resolve it.

Identifying anomalies in the measurement funnel orders for a food delivery app can help drive customer satisfaction by allowing for the quick detection and resolution of any issues that may negatively impact the customer experience.

For example, if there is a sudden drop in orders at a certain stage of the funnel, it could indicate a problem with the ordering process, such as technical difficulties or unclear instructions. By identifying and fixing these anomalies, the food delivery app can ensure that customers are able to complete their orders smoothly and efficiently, leading to a higher level of satisfaction.

Additionally, regularly monitoring the measurement funnel can help identify trends and areas for improvement, allowing the app to continuously optimize the customer experience.

Additionally, by using anomaly detection algorithms, we can identify patterns and trends that may not be immediately obvious, providing valuable insights into customer behavior and preferences.

In this session, we will share practical examples and best practices for implementing automated monitoring and anomaly detection to improve customer satisfaction and drive business success using StarTree ThirdEye.

Transcript

1. Improving Customer Satisfaction Automated Monitoring and Anomaly Detection Techniques Madhumita Mantri Product Lead StarTree Leon Graveland Data Engineer Just Eat Takeaway.com
2. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. “Customer satisfaction is not a goal, it's the outcome of consistently delivering a memorable experience.” - Forbes. 2
3. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Agenda 01 Introduction 02 Roadblocks to “Food delivery apps” customer satisfaction 03 What is an Anomaly Event and why it is important? 04 What it takes to spot anomalies and why to automate? 05 Overview of StarTree ThirdEye 06 Automated Anomaly Detection using StarTree ThirdEye 07 Just Eat Takeaway.com Case Study 08 Demo 09 Conclusion
4. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Introduction Improved and seamless experience is crucial for food delivery applications. • Ensures customer satisfaction • Increased usage of the application, and customer retention
5. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Roadblocks to “Food delivery apps” customer satisfaction • Technical issues • Product experience • Demand vs supply • Connectivity issues in the app with external networks
6. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. What is an Anomaly Event? Unusual patterns or outliers which is not obvious. Examples • Unexpected spike or dip in # of food deliveries • Gradual change in customer behavior using food delivery app • Drop in engagement and usage due to increased wait time because of food delivery driver’s unavailability
7. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Why anomalous events are important for Customer Satisfaction? • Spot real issues in real-time - minimize the impact • Spot positive opportunities - drive business growth
8. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. What it takes to identify anomalous events? Monitor Critical Metrics Business Ops “Any idea why there is a drop in food delivery orders”? Investigation Anomaly detection and Root-Cause Analysis Time Time to book Order bookings 10:00 15% -3% 11:00 14% -9% 12:00 20% -7% 13:00 15% -5% 14:00 2% -6% 15:00 13% -7%
9. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Why to automate anomaly detection? Troubleshooting data issues vs data team size
10. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Automated Anomaly Detection Find impact (Insights) Why went wrong? Identify anomalies (asap) What went wrong? Monitor KPIs (Key performance indicators)
11. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye An advanced and efficient “real-time” automated anomaly detection product that fast tracks problem solving by unlocking actionable insights. Keeps up with the increasing complexity of data to help businesses with a direct impact on revenue margins. Interactive root-cause analysis Fast-track problem solving Why did it go wrong and what can we do? (Dimension drills, correlated events) Applied science and smart monitoring Detect anomalies with better accuracy & context What went wrong? Connect to data (real-time, batch) What do you want to monitor? Onboard metrics/KPIs APIs • Write to ThirdEye • Read from ThirdEye • Custom Apps
12. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye - Augmenting Data Teams to do more with less! Using ThirdEye Data Engineers & Data Scientists Empowered to do more with less
13. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Developer Productivity Developer Productivity Out of the box with StarTree ThirdEye _ StarTree Cloud Data Manager + Pinot (scale, low latency and point and click) What it takes to productionize anomaly detection solution VS How StarTree ThirdEye is helping?
14. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye (Before/After) Pain • No real-time actionable insights • False alarms/false positives • Operations complexity • Doesn't scale • Long time to detect & resolve issues • Collaboration Gain • Real-time detection + actionable insights • Accurate anomalies • Ease of ops (point & click + easy to scale) • Data informed decision making ◦ Spot growth opportunities ◦ Real-time insights into top contributors
15. Improving Customer Satisfaction Automated Monitoring and Anomaly Detection Techniques Leon Graveland Data Engineer Just Eat Takeaway.com Case Study
16. Source: https://www.justeattakeaway.com/ as at 30 June 2022 Active Consumers 94m Restaurants 680k Countries 20 GTV €28bn Orders 1.0bn 16 | RTASummit - Just Eat Takeaway.com Case Study
17. 17 | RTASummit - Just Eat Takeaway.com Case Study
18. We’re monitoring the two main drivers of customer satisfaction 18 | RTASummit - Just Eat Takeaway.com Case Study • Conversion rate • Forward rate • # transactions Online Experience Logistics • Place-to-Delivery • Closings
19. Drops in conversion funnel steps are a great indicator of anomalous behaviour 19 | RTASummit - Just Eat Takeaway.com Case Study Online Experience forward-rate conversion-rate # transactions home restaurants menu check-out transaction
20. A customer is mostly concerned with fast delivery 20 | RTASummit - Just Eat Takeaway.com Case Study Logistics Place Order Customer places an order. Delivery of food Order is delivered at the customer Restaurant Status Based on capacity restaurant can close restaurant Restaurants Prepares Meal Bon Appetit Time from place to delivery % Closings
21. Three steps for making impact with anomaly detection 21 | RTASummit - Just Eat Takeaway.com Case Study Implementing the anomaly detection process SETUP Iterate until we’re happy with the anomalies produced by the model TUNE Profit from the detected anomalies by taking swift action LEVERAGE
22. Don’t start detecting anomalies without a plan 22 | RTASummit - Just Eat Takeaway.com Case Study SETUP Implementing the anomaly detection process Define metrics that impact customer satisfaction > forward-rate > conversion-rate > # transactions Ingest event stream data > from Amazon Kinesis > into Apache Pinot > using Datamanager Implement anomaly detection model > exp. Smoothing > regression model
23. Tuning your model is crucial for successful implementation 23 | RTASummit - Just Eat Takeaway.com Case Study TUNE Iterate until we’re happy with the anomalies produced by the model Set granularity level > 15 min time interval > per country Visualize in one graph > actual vs. predicted > upper- and lower bound Investigate found anomalies > root cause > fix or insight
24. Management of anomalies is key for success 24 | RTASummit - Just Eat Takeaway.com Case Study LEVERAGE Profit from the detected anomalies by taking swift action Notify Subscription Groups > relevant people > no alert fatigue Set and store Anomaly Status > no duplicate work > track performance Perform a Root Cause Analysis > heatmap > events
25. Edit page reference 25 | Month Year | Title of project • Real-world examples of businesses that have successfully used automated monitoring and anomaly detection to improve customer satisfaction • JET: ◦ For real-time app/web events to track if measurement/tagging is working as expected. (Note: JET team currently only monitor the funnel step 'transaction', but we do plan for other events (funnel steps) to track. They don't have dimensions so we currently aren't using RCA) ◦ if there is a sudden drop in orders at a certain stage of the funnel, it could indicate a problem with the ordering process, such as technical difficulties or unclear instructions (Ex:outage examples: payment service provider down, DDOS attack, new software releases). By identifying and fixing these anomalies, the food delivery app can ensure that customers are able to complete their orders smoothly and efficiently, leading to a higher level of satisfaction. ◦ Additionally, regularly monitoring the measurement funnel can help identify trends and areas for improvement, allowing the app to continuously optimize the customer experience. ◦ We can use the following steps for the demo script ▪ To identify anomalies in the measurement funnel orders for a food delivery app, you can follow these steps: • Define the measurement funnel: The count of events (spikes/drops) help Identify the changes in key stages of the ordering process, such as: add item to basket, go to checkout, list restaurants, open restaurant menu. • Collect and analyze data: Collect data on the number of customers at each stage of the funnel and analyze it to identify any unusual patterns or deviations from the expected results. • Use statistical models: Use statistical models, such as regression analysis or time series analysis, to detect outliers or unusual patterns in the data. (include challenges and solutions for detecting anomalies on highly seasonal data on a small time aggregation (15min)) • Determine granularity: ◦ define whether you’d like to receive outliers per dimension value. For example per country or state. ◦ Define the latency and time monitoring interval with respect to time. We picked 15m time interval with 1 minute latency. • Visualize the data: Use data visualization tools, such as charts or graphs, to help visualize the data and identify any anomalies more easily. • Investigate the anomalies: Once anomalies have been identified, it's important to investigate the root cause of the issue and take corrective action to improve the customer experience.
26. 26 | RTASummit - Just Eat Takeaway.com Case Study Investigate Visualize Set Granularity 15 min per country Upper/lower bound Root Cause Analysis Fix or Share
27. 27 | RTASummit - Just Eat Takeaway.com Case Study Demo
29. 29 | RTASummit - Just Eat Takeaway.com Case Study Wrap-up and summarize Case Study
30. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Product Announcements
31. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Product (Current vs Next) In Beta • Dimension level monitoring • Real-time monitoring (15 mins ~ 1 hr) • Anomaly filters (configurable) • Advanced algorithms (StarTree ETS, StarTree Matrix Profile) • Interactive point & click user interface (monitoring, notifications & “root-cause analysis”) • Data mutability support 2023~2024 (Coming soon!) • ThirdEye Ginnie - recommend top contributors to monitor • Automated algorithm recommendations • Data Drift detection • Root Cause Analysis 2.0 • Enterprise readiness ◦ RBAC ◦ Security ◦ Scale ◦ Operational efficiency improvements
32. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree Product Launches - COMING SOON!!!
33. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Conclusion StarTree ThirdEye overview, uniqueness and how it is built, concepts optimized for implementing automated monitoring and anomaly detection in consumer products • Future meet-ups join our community! • Join StarTree Community Slack (Get QR code) • Sign up for free trial or StarTree Community Edition (does not require credit card) • Talk to us! #thirdeye Download StarTree Community Edition! Join StarTree Community! Sign-up 30 days free Self-serve SaaS Trial!
34. Appendix
35. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye (Out of box algorithms) Basic anomaly detection • Threshold • Mean Variance • Percentage Change • Absolute Change Advanced anomaly detection • Exponential time smoothing (ETS) • MSTL (multiple patterns) • Matrix Profile • Remote HTTP
36. ThirdEye Architecture Modular architecture High level building blocks
37. ThirdEye Community (free) vs Enterprise editions (free trial no credit card) Features Community Enterprise All the UI/UX (self-serve features) Yes Yes Swagger UI to access ThirdEye APIs Yes Yes Data & KPI Onboarding Data sources connectivity (manual) Yes Yes Data sources (auto connect Apache Pinot with advanced UI and ease of use) No Yes Pluggable data sources (Apache Pinot) Yes Yes Pluggable data sources (Amazon DynamoDB, BigQuery) No (write your own) Yes (Future roadmap) Dataset onboarding Yes (manual) Yes (Automated and complete data reload, future roadmap) Onboard metrics from Pinot to ThirdEye Yes Yes Features Community Enterprise Alerts & Notifications Alert configuration (JSON code) Yes Yes Alert configuration using alert templates (low code) (Basic detection algorithms) Yes Yes Alert configuration using alert templates (low code) (Advanced detection algorithms) No Yes Auto onboard pre-configured alert templates with existing detection techniques (no code UI) No Yes Manage alert severity No Yes (Future roadmap) Dimension exploration (Dimension level anomaly detection) No Yes Anomaly Detection Algorithms StarTree ETS (Advanced Holtwinters) statistical model, StarTree Matrix, Top contributors recommender (Identify what to monitor)) No Yes Features Community Enterprise Anomaly detection (Holiday prediction model) No Yes Pluggable detectors: Thirdeye prophet plugin, Thirdeye + tensorflow/torch serve) No (write your own) Yes (beta) HTTP API Detector (bring your own model and algos) No Yes Missing data identifier No Yes (Future roadmap) Anomalies (Basic Chart/Graph) Yes Yes Advanced anomaly filters for improved accuracy No Yes Root Cause Analysis RCA/Anomaly Analysis (Simple algorithms) Yes Yes RCA/Anomaly Analysis (Complex algorithms) No Yes Automated data insights and plotting anomalies in the reports No Yes (Future roadmap)