Slide 1

Slide 1 text

Improving Customer Satisfaction Automated Monitoring and Anomaly Detection Techniques Madhumita Mantri Product Lead StarTree Leon Graveland Data Engineer Just Eat Takeaway.com

Slide 2

Slide 2 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. “Customer satisfaction is not a goal, it's the outcome of consistently delivering a memorable experience.” - Forbes. 2

Slide 3

Slide 3 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Agenda 01 Introduction 02 Roadblocks to “Food delivery apps” customer satisfaction 03 What is an Anomaly Event and why it is important? 04 What it takes to spot anomalies and why to automate? 05 Overview of StarTree ThirdEye 06 Automated Anomaly Detection using StarTree ThirdEye 07 Just Eat Takeaway.com Case Study 08 Demo 09 Conclusion

Slide 4

Slide 4 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Introduction Improved and seamless experience is crucial for food delivery applications. ● Ensures customer satisfaction ● Increased usage of the application, and customer retention

Slide 5

Slide 5 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Roadblocks to “Food delivery apps” customer satisfaction ● Technical issues ● Product experience ● Demand vs supply ● Connectivity issues in the app with external networks

Slide 6

Slide 6 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. What is an Anomaly Event? Unusual patterns or outliers which is not obvious. Examples ● Unexpected spike or dip in # of food deliveries ● Gradual change in customer behavior using food delivery app ● Drop in engagement and usage due to increased wait time because of food delivery driver’s unavailability

Slide 7

Slide 7 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Why anomalous events are important for Customer Satisfaction? ● Spot real issues in real-time - minimize the impact ● Spot positive opportunities - drive business growth

Slide 8

Slide 8 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. What it takes to identify anomalous events? Monitor Critical Metrics Business Ops “Any idea why there is a drop in food delivery orders”? Investigation Anomaly detection and Root-Cause Analysis Time Time to book Order bookings 10:00 15% -3% 11:00 14% -9% 12:00 20% -7% 13:00 15% -5% 14:00 2% -6% 15:00 13% -7%

Slide 9

Slide 9 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Why to automate anomaly detection? Troubleshooting data issues vs data team size

Slide 10

Slide 10 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Automated Anomaly Detection Find impact (Insights) Why went wrong? Identify anomalies (asap) What went wrong? Monitor KPIs (Key performance indicators)

Slide 11

Slide 11 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye An advanced and efficient “real-time” automated anomaly detection product that fast tracks problem solving by unlocking actionable insights. Keeps up with the increasing complexity of data to help businesses with a direct impact on revenue margins. Interactive root-cause analysis Fast-track problem solving Why did it go wrong and what can we do? (Dimension drills, correlated events) Applied science and smart monitoring Detect anomalies with better accuracy & context What went wrong? Connect to data (real-time, batch) What do you want to monitor? Onboard metrics/KPIs APIs • Write to ThirdEye • Read from ThirdEye • Custom Apps

Slide 12

Slide 12 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye - Augmenting Data Teams to do more with less! Using ThirdEye Data Engineers & Data Scientists Empowered to do more with less

Slide 13

Slide 13 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Developer Productivity Developer Productivity Out of the box with StarTree ThirdEye _ StarTree Cloud Data Manager + Pinot (scale, low latency and point and click) What it takes to productionize anomaly detection solution VS How StarTree ThirdEye is helping?

Slide 14

Slide 14 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye (Before/After) Pain ● No real-time actionable insights ● False alarms/false positives ● Operations complexity ● Doesn't scale ● Long time to detect & resolve issues ● Collaboration Gain ● Real-time detection + actionable insights ● Accurate anomalies ● Ease of ops (point & click + easy to scale) ● Data informed decision making ○ Spot growth opportunities ○ Real-time insights into top contributors

Slide 15

Slide 15 text

Improving Customer Satisfaction Automated Monitoring and Anomaly Detection Techniques Leon Graveland Data Engineer Just Eat Takeaway.com Case Study

Slide 16

Slide 16 text

Source: https://www.justeattakeaway.com/ as at 30 June 2022 Active Consumers 94m Restaurants 680k Countries 20 GTV €28bn Orders 1.0bn 16 | RTASummit - Just Eat Takeaway.com Case Study

Slide 17

Slide 17 text

17 | RTASummit - Just Eat Takeaway.com Case Study

Slide 18

Slide 18 text

We’re monitoring the two main drivers of customer satisfaction 18 | RTASummit - Just Eat Takeaway.com Case Study ● Conversion rate ● Forward rate ● # transactions Online Experience Logistics ● Place-to-Delivery ● Closings

Slide 19

Slide 19 text

Drops in conversion funnel steps are a great indicator of anomalous behaviour 19 | RTASummit - Just Eat Takeaway.com Case Study Online Experience forward-rate conversion-rate # transactions home restaurants menu check-out transaction

Slide 20

Slide 20 text

A customer is mostly concerned with fast delivery 20 | RTASummit - Just Eat Takeaway.com Case Study Logistics Place Order Customer places an order. Delivery of food Order is delivered at the customer Restaurant Status Based on capacity restaurant can close restaurant Restaurants Prepares Meal Bon Appetit Time from place to delivery % Closings

Slide 21

Slide 21 text

Three steps for making impact with anomaly detection 21 | RTASummit - Just Eat Takeaway.com Case Study Implementing the anomaly detection process SETUP Iterate until we’re happy with the anomalies produced by the model TUNE Profit from the detected anomalies by taking swift action LEVERAGE

Slide 22

Slide 22 text

Don’t start detecting anomalies without a plan 22 | RTASummit - Just Eat Takeaway.com Case Study SETUP Implementing the anomaly detection process Define metrics that impact customer satisfaction > forward-rate > conversion-rate > # transactions Ingest event stream data > from Amazon Kinesis > into Apache Pinot > using Datamanager Implement anomaly detection model > exp. Smoothing > regression model

Slide 23

Slide 23 text

Tuning your model is crucial for successful implementation 23 | RTASummit - Just Eat Takeaway.com Case Study TUNE Iterate until we’re happy with the anomalies produced by the model Set granularity level > 15 min time interval > per country Visualize in one graph > actual vs. predicted > upper- and lower bound Investigate found anomalies > root cause > fix or insight

Slide 24

Slide 24 text

Management of anomalies is key for success 24 | RTASummit - Just Eat Takeaway.com Case Study LEVERAGE Profit from the detected anomalies by taking swift action Notify Subscription Groups > relevant people > no alert fatigue Set and store Anomaly Status > no duplicate work > track performance Perform a Root Cause Analysis > heatmap > events

Slide 25

Slide 25 text

Edit page reference 25 | Month Year | Title of project ● Real-world examples of businesses that have successfully used automated monitoring and anomaly detection to improve customer satisfaction ● JET: ○ For real-time app/web events to track if measurement/tagging is working as expected. (Note: JET team currently only monitor the funnel step 'transaction', but we do plan for other events (funnel steps) to track. They don't have dimensions so we currently aren't using RCA) ○ if there is a sudden drop in orders at a certain stage of the funnel, it could indicate a problem with the ordering process, such as technical difficulties or unclear instructions (Ex:outage examples: payment service provider down, DDOS attack, new software releases). By identifying and fixing these anomalies, the food delivery app can ensure that customers are able to complete their orders smoothly and efficiently, leading to a higher level of satisfaction. ○ Additionally, regularly monitoring the measurement funnel can help identify trends and areas for improvement, allowing the app to continuously optimize the customer experience. ○ We can use the following steps for the demo script ■ To identify anomalies in the measurement funnel orders for a food delivery app, you can follow these steps: ● Define the measurement funnel: The count of events (spikes/drops) help Identify the changes in key stages of the ordering process, such as: add item to basket, go to checkout, list restaurants, open restaurant menu. ● Collect and analyze data: Collect data on the number of customers at each stage of the funnel and analyze it to identify any unusual patterns or deviations from the expected results. ● Use statistical models: Use statistical models, such as regression analysis or time series analysis, to detect outliers or unusual patterns in the data. (include challenges and solutions for detecting anomalies on highly seasonal data on a small time aggregation (15min)) ● Determine granularity: ○ define whether you’d like to receive outliers per dimension value. For example per country or state. ○ Define the latency and time monitoring interval with respect to time. We picked 15m time interval with 1 minute latency. ● Visualize the data: Use data visualization tools, such as charts or graphs, to help visualize the data and identify any anomalies more easily. ● Investigate the anomalies: Once anomalies have been identified, it's important to investigate the root cause of the issue and take corrective action to improve the customer experience.

Slide 26

Slide 26 text

26 | RTASummit - Just Eat Takeaway.com Case Study Investigate Visualize Set Granularity 15 min per country Upper/lower bound Root Cause Analysis Fix or Share

Slide 27

Slide 27 text

27 | RTASummit - Just Eat Takeaway.com Case Study Demo

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

29 | RTASummit - Just Eat Takeaway.com Case Study Wrap-up and summarize Case Study

Slide 30

Slide 30 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Product Announcements

Slide 31

Slide 31 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye Product (Current vs Next) In Beta ● Dimension level monitoring ● Real-time monitoring (15 mins ~ 1 hr) ● Anomaly filters (configurable) ● Advanced algorithms (StarTree ETS, StarTree Matrix Profile) ● Interactive point & click user interface (monitoring, notifications & “root-cause analysis”) ● Data mutability support 2023~2024 (Coming soon!) ● ThirdEye Ginnie - recommend top contributors to monitor ● Automated algorithm recommendations ● Data Drift detection ● Root Cause Analysis 2.0 ● Enterprise readiness ○ RBAC ○ Security ○ Scale ○ Operational efficiency improvements

Slide 32

Slide 32 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree Product Launches - COMING SOON!!!

Slide 33

Slide 33 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. Conclusion StarTree ThirdEye overview, uniqueness and how it is built, concepts optimized for implementing automated monitoring and anomaly detection in consumer products ● Future meet-ups join our community! ● Join StarTree Community Slack (Get QR code) ● Sign up for free trial or StarTree Community Edition (does not require credit card) ● Talk to us! #thirdeye Download StarTree Community Edition! Join StarTree Community! Sign-up 30 days free Self-serve SaaS Trial!

Slide 34

Slide 34 text

Appendix

Slide 35

Slide 35 text

CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc. StarTree ThirdEye (Out of box algorithms) Basic anomaly detection ● Threshold ● Mean Variance ● Percentage Change ● Absolute Change Advanced anomaly detection ● Exponential time smoothing (ETS) ● MSTL (multiple patterns) ● Matrix Profile ● Remote HTTP

Slide 36

Slide 36 text

ThirdEye Architecture Modular architecture High level building blocks

Slide 37

Slide 37 text

ThirdEye Community (free) vs Enterprise editions (free trial no credit card) Features Community Enterprise All the UI/UX (self-serve features) Yes Yes Swagger UI to access ThirdEye APIs Yes Yes Data & KPI Onboarding Data sources connectivity (manual) Yes Yes Data sources (auto connect Apache Pinot with advanced UI and ease of use) No Yes Pluggable data sources (Apache Pinot) Yes Yes Pluggable data sources (Amazon DynamoDB, BigQuery) No (write your own) Yes (Future roadmap) Dataset onboarding Yes (manual) Yes (Automated and complete data reload, future roadmap) Onboard metrics from Pinot to ThirdEye Yes Yes Features Community Enterprise Alerts & Notifications Alert configuration (JSON code) Yes Yes Alert configuration using alert templates (low code) (Basic detection algorithms) Yes Yes Alert configuration using alert templates (low code) (Advanced detection algorithms) No Yes Auto onboard pre-configured alert templates with existing detection techniques (no code UI) No Yes Manage alert severity No Yes (Future roadmap) Dimension exploration (Dimension level anomaly detection) No Yes Anomaly Detection Algorithms StarTree ETS (Advanced Holtwinters) statistical model, StarTree Matrix, Top contributors recommender (Identify what to monitor)) No Yes Features Community Enterprise Anomaly detection (Holiday prediction model) No Yes Pluggable detectors: Thirdeye prophet plugin, Thirdeye + tensorflow/torch serve) No (write your own) Yes (beta) HTTP API Detector (bring your own model and algos) No Yes Missing data identifier No Yes (Future roadmap) Anomalies (Basic Chart/Graph) Yes Yes Advanced anomaly filters for improved accuracy No Yes Root Cause Analysis RCA/Anomaly Analysis (Simple algorithms) Yes Yes RCA/Anomaly Analysis (Complex algorithms) No Yes Automated data insights and plotting anomalies in the reports No Yes (Future roadmap)