Improving Customer Satisfaction with Automated Monitoring and Anomaly Detection (Madhumita Mantri, StarTree & Leon Graveland, JustEat Takeaway) RTA Summit '23

In today's fast-paced business environment, it is essential to have real-time visibility into customers' experiences and quickly identify any issues that may arise.

By automating the monitoring of critical business metrics such as customer satisfaction we can quickly detect anomalies that may indicate a problem and take action to resolve it.

Identifying anomalies in the measurement funnel orders for a food delivery app can help drive customer satisfaction by allowing for the quick detection and resolution of any issues that may negatively impact the customer experience.

For example, if there is a sudden drop in orders at a certain stage of the funnel, it could indicate a problem with the ordering process, such as technical difficulties or unclear instructions. By identifying and fixing these anomalies, the food delivery app can ensure that customers are able to complete their orders smoothly and efficiently, leading to a higher level of satisfaction.

Additionally, regularly monitoring the measurement funnel can help identify trends and areas for improvement, allowing the app to continuously optimize the customer experience.

Additionally, by using anomaly detection algorithms, we can identify patterns and trends that may not be immediately obvious, providing valuable insights into customer behavior and preferences.

In this session, we will share practical examples and best practices for implementing automated monitoring and anomaly detection to improve customer satisfaction and drive business success using StarTree ThirdEye.

Transcript

1. Improving Customer Satisfaction
   Automated Monitoring and Anomaly Detection Techniques
   Madhumita Mantri
   Product Lead
   StarTree
   Leon Graveland
   Data Engineer
   Just Eat Takeaway.com
   

   View Slide

2. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   “Customer satisfaction is not a goal, it's the
   outcome of consistently delivering a
   memorable experience.” - Forbes.
   2
   

   View Slide

3. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   Agenda
   01 Introduction
   02 Roadblocks to “Food delivery apps” customer satisfaction
   03 What is an Anomaly Event and why it is important?
   04 What it takes to spot anomalies and why to automate?
   05 Overview of StarTree ThirdEye
   06 Automated Anomaly Detection using StarTree ThirdEye
   07 Just Eat Takeaway.com Case Study
   08 Demo
   09 Conclusion
   

   View Slide

4. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   Introduction
   Improved and seamless experience is crucial for food
   delivery applications.
   ● Ensures customer satisfaction
   ● Increased usage of the application, and customer
   retention
   

   View Slide

5. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   Roadblocks to
   “Food delivery apps”
   customer satisfaction
   ● Technical issues
   ● Product experience
   ● Demand vs supply
   ● Connectivity issues in the app with external
   networks
   

   View Slide

6. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   What is an Anomaly Event?
   Unusual patterns or outliers which is not obvious.
   Examples
   ● Unexpected spike or dip in # of food deliveries
   ● Gradual change in customer behavior using food delivery app
   ● Drop in engagement and usage due to increased wait time because of food delivery
   driver’s unavailability
   

   View Slide

7. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   Why anomalous events
   are important for
   Customer Satisfaction?
   ● Spot real issues in real-time - minimize
   the impact
   ● Spot positive opportunities - drive
   business growth
   

   View Slide

8. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   What it takes to identify anomalous events?
   Monitor Critical Metrics
   Business Ops
   “Any idea why there is a drop in food delivery orders”?
   Investigation
   Anomaly detection and Root-Cause Analysis
   Time Time to book Order bookings
   10:00 15% -3%
   11:00 14% -9%
   12:00 20% -7%
   13:00 15% -5%
   14:00 2% -6%
   15:00 13% -7%
   

   View Slide

9. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
   Why to automate anomaly detection?
   Troubleshooting data issues vs data team size
   

   View Slide

10. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    Automated Anomaly Detection
    Find impact
    (Insights)
    Why went wrong?
    Identify anomalies (asap)
    What went wrong?
    Monitor KPIs
    (Key performance
    indicators)
    

    View Slide

11. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye
    An advanced and efficient “real-time” automated anomaly detection product that fast tracks problem
    solving by unlocking actionable insights. Keeps up with the increasing complexity of data to help
    businesses with a direct impact on revenue margins.
    Interactive root-cause
    analysis
    Fast-track problem solving
    Why did it go wrong and what can
    we do?
    (Dimension drills, correlated events)
    Applied science and smart
    monitoring
    Detect anomalies with
    better accuracy & context
    What went wrong?
    Connect to data
    (real-time, batch)
    What do you want to monitor?
    Onboard metrics/KPIs
    APIs
    • Write to ThirdEye • Read from ThirdEye • Custom Apps
    

    View Slide

12. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye - Augmenting Data
    Teams to do more with less!
    Using ThirdEye
    Data Engineers & Data Scientists
    Empowered to do more with less
    

    View Slide

13. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye Developer Productivity
    Developer Productivity
    Out of the box with StarTree
    ThirdEye _ StarTree Cloud
    Data Manager +
    Pinot (scale, low
    latency and point
    and click)
    What it takes to productionize
    anomaly detection solution
    VS
    How StarTree ThirdEye is helping?
    

    View Slide

14. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye (Before/After)
    Pain
    ● No real-time actionable insights
    ● False alarms/false positives
    ● Operations complexity
    ● Doesn't scale
    ● Long time to detect & resolve
    issues
    ● Collaboration
    Gain
    ● Real-time detection + actionable
    insights
    ● Accurate anomalies
    ● Ease of ops (point & click + easy to
    scale)
    ● Data informed decision making
    ○ Spot growth opportunities
    ○ Real-time insights into top
    contributors
    

    View Slide

15. Improving Customer Satisfaction
    Automated Monitoring and Anomaly
    Detection Techniques
    Leon Graveland
    Data Engineer
    Just Eat Takeaway.com
    Case Study
    

    View Slide

16. Source: https://www.justeattakeaway.com/ as at 30 June 2022
    Active
    Consumers
    94m
    Restaurants
    680k
    Countries
    20
    GTV
    €28bn
    Orders
    1.0bn
    16 | RTASummit - Just Eat Takeaway.com Case Study
    

    View Slide

17. 17 | RTASummit - Just Eat Takeaway.com Case Study
    

    View Slide

18. We’re monitoring the two main drivers of customer
    satisfaction
    18 | RTASummit - Just Eat Takeaway.com Case Study
    ● Conversion rate
    ● Forward rate
    ● # transactions
    Online Experience Logistics
    ● Place-to-Delivery
    ● Closings
    

    View Slide

19. Drops in conversion funnel steps are a great indicator
    of anomalous behaviour
    19 | RTASummit - Just Eat Takeaway.com Case Study
    Online Experience
    forward-rate
    conversion-rate
    # transactions
    home restaurants menu check-out transaction
    

    View Slide

20. A customer is mostly concerned with fast delivery
    20 | RTASummit - Just Eat Takeaway.com Case Study
    Logistics
    Place Order
    Customer places an order.
    Delivery of food
    Order is delivered at the
    customer
    Restaurant Status
    Based on capacity restaurant
    can close restaurant
    Restaurants
    Prepares Meal
    Bon Appetit
    Time from place to delivery
    % Closings
    

    View Slide

21. Three steps for making impact with anomaly detection
    21 | RTASummit - Just Eat Takeaway.com Case Study
    Implementing the
    anomaly detection
    process
    SETUP
    Iterate until we’re happy
    with the anomalies
    produced by the model
    TUNE
    Profit from the detected
    anomalies by taking
    swift action
    LEVERAGE
    

    View Slide

22. Don’t start detecting anomalies without a plan
    22 | RTASummit - Just Eat Takeaway.com Case Study
    SETUP
    Implementing the anomaly
    detection process
    Define metrics that
    impact customer
    satisfaction
    > forward-rate
    > conversion-rate
    > # transactions
    Ingest event stream
    data
    > from Amazon Kinesis
    > into Apache Pinot
    > using Datamanager
    Implement anomaly
    detection model
    > exp. Smoothing
    > regression model
    

    View Slide

23. Tuning your model is crucial for successful implementation
    23 | RTASummit - Just Eat Takeaway.com Case Study
    TUNE
    Iterate until we’re happy with
    the anomalies produced by the
    model
    Set granularity level
    > 15 min time interval
    > per country
    Visualize in one
    graph
    > actual vs. predicted
    > upper- and lower
    bound
    Investigate found
    anomalies
    > root cause
    > fix or insight
    

    View Slide

24. Management of anomalies is key for success
    24 | RTASummit - Just Eat Takeaway.com Case Study
    LEVERAGE
    Profit from the detected
    anomalies by taking swift
    action
    Notify Subscription
    Groups
    > relevant people
    > no alert fatigue
    Set and store
    Anomaly Status
    > no duplicate work
    > track performance
    Perform a Root
    Cause Analysis
    > heatmap
    > events
    

    View Slide

25. Edit page reference
    25 | Month Year | Title of project
    ● Real-world examples of businesses that have successfully used automated monitoring and anomaly detection to improve customer satisfaction
    ● JET:
    ○ For real-time app/web events to track if measurement/tagging is working as expected. (Note: JET team currently only monitor the funnel step
    'transaction', but we do plan for other events (funnel steps) to track. They don't have dimensions so we currently aren't using RCA)
    ○ if there is a sudden drop in orders at a certain stage of the funnel, it could indicate a problem with the ordering process, such as technical difficulties or
    unclear instructions (Ex:outage examples: payment service provider down, DDOS attack, new software releases). By identifying and fixing these anomalies,
    the food delivery app can ensure that customers are able to complete their orders smoothly and efficiently, leading to a higher level of satisfaction.
    ○ Additionally, regularly monitoring the measurement funnel can help identify trends and areas for improvement, allowing the app to continuously
    optimize the customer experience.
    ○ We can use the following steps for the demo script
    ■ To identify anomalies in the measurement funnel orders for a food delivery app, you can follow these steps:
    ● Define the measurement funnel: The count of events (spikes/drops) help Identify the changes in key stages of the ordering process,
    such as: add item to basket, go to checkout, list restaurants, open restaurant menu.
    ● Collect and analyze data: Collect data on the number of customers at each stage of the funnel and analyze it to identify any unusual
    patterns or deviations from the expected results.
    ● Use statistical models: Use statistical models, such as regression analysis or time series analysis, to detect outliers or unusual
    patterns in the data. (include challenges and solutions for detecting anomalies on highly seasonal data on a small time aggregation
    (15min))
    ● Determine granularity:
    ○ define whether you’d like to receive outliers per dimension value. For example per country or state.
    ○ Define the latency and time monitoring interval with respect to time. We picked 15m time interval with 1 minute latency.
    ● Visualize the data: Use data visualization tools, such as charts or graphs, to help visualize the data and identify any anomalies more
    easily.
    ● Investigate the anomalies: Once anomalies have been identified, it's important to investigate the root cause of the issue and take
    corrective action to improve the customer experience.
    

    View Slide

26. 26 | RTASummit - Just Eat Takeaway.com Case Study
    Investigate
    Visualize
    Set Granularity
    15 min
    per country
    Upper/lower bound
    Root Cause
    Analysis
    Fix or Share
    

    View Slide

27. 27 | RTASummit - Just Eat Takeaway.com Case Study
    Demo
    

    View Slide

28. View Slide

29. 29 | RTASummit - Just Eat Takeaway.com Case Study
    Wrap-up and summarize Case Study
    

    View Slide

30. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye
    Product Announcements
    

    View Slide

31. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye Product (Current vs Next)
    In Beta
    ● Dimension level monitoring
    ● Real-time monitoring (15 mins ~ 1 hr)
    ● Anomaly filters (configurable)
    ● Advanced algorithms (StarTree ETS,
    StarTree Matrix Profile)
    ● Interactive point & click user interface
    (monitoring, notifications & “root-cause
    analysis”)
    ● Data mutability support
    2023~2024 (Coming soon!)
    ● ThirdEye Ginnie - recommend top
    contributors to monitor
    ● Automated algorithm recommendations
    ● Data Drift detection
    ● Root Cause Analysis 2.0
    ● Enterprise readiness
    ○ RBAC
    ○ Security
    ○ Scale
    ○ Operational efficiency
    improvements
    

    View Slide

32. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree Product Launches - COMING SOON!!!
    

    View Slide

33. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    Conclusion
    StarTree ThirdEye overview, uniqueness and how it is built, concepts optimized for implementing
    automated monitoring and anomaly detection in consumer products
    ● Future meet-ups join our community!
    ● Join StarTree Community Slack (Get QR code)
    ● Sign up for free trial or StarTree Community Edition (does not require credit card)
    ● Talk to us! #thirdeye
    Download StarTree Community
    Edition!
    Join StarTree Community!
    Sign-up 30 days free
    Self-serve SaaS Trial!
    

    View Slide

34. Appendix
    

    View Slide

35. CONFIDENTIAL. Do not duplicate or distribute without consent of StarTree Inc.
    StarTree ThirdEye (Out of box algorithms)
    Basic anomaly detection
    ● Threshold
    ● Mean Variance
    ● Percentage Change
    ● Absolute Change
    Advanced anomaly detection
    ● Exponential time smoothing (ETS)
    ● MSTL (multiple patterns)
    ● Matrix Profile
    ● Remote HTTP
    

    View Slide

36. ThirdEye Architecture
    Modular architecture High level building blocks
    

    View Slide

37. ThirdEye Community (free) vs Enterprise editions (free trial no credit card)
    Features Community Enterprise
    All the UI/UX (self-serve
    features)
    Yes Yes
    Swagger UI to access
    ThirdEye APIs
    Yes Yes
    Data & KPI Onboarding
    Data sources connectivity
    (manual)
    Yes Yes
    Data sources (auto connect
    Apache Pinot with
    advanced UI and ease of
    use)
    No Yes
    Pluggable data sources
    (Apache Pinot)
    Yes Yes
    Pluggable data sources
    (Amazon DynamoDB,
    BigQuery)
    No (write your own) Yes (Future roadmap)
    Dataset onboarding Yes (manual) Yes (Automated and
    complete data reload,
    future roadmap)
    Onboard metrics from
    Pinot to ThirdEye
    Yes Yes
    Features Community Enterprise
    Alerts & Notifications
    Alert configuration (JSON
    code)
    Yes Yes
    Alert configuration using
    alert templates (low code)
    (Basic detection algorithms)
    Yes Yes
    Alert configuration using
    alert templates (low code)
    (Advanced detection
    algorithms)
    No Yes
    Auto onboard pre-configured
    alert templates with existing
    detection techniques (no
    code UI)
    No Yes
    Manage alert severity No Yes (Future roadmap)
    Dimension exploration
    (Dimension level anomaly
    detection)
    No Yes
    Anomaly Detection Algorithms
    StarTree ETS (Advanced
    Holtwinters) statistical
    model, StarTree Matrix, Top
    contributors recommender
    (Identify what to monitor))
    No Yes
    Features Community Enterprise
    Anomaly detection (Holiday
    prediction model)
    No Yes
    Pluggable detectors:
    Thirdeye prophet plugin,
    Thirdeye + tensorflow/torch
    serve)
    No (write your own) Yes (beta)
    HTTP API Detector (bring
    your own model and algos)
    No Yes
    Missing data identifier No Yes (Future roadmap)
    Anomalies (Basic
    Chart/Graph)
    Yes Yes
    Advanced anomaly filters
    for improved accuracy
    No Yes
    Root Cause Analysis
    RCA/Anomaly Analysis
    (Simple algorithms)
    Yes Yes
    RCA/Anomaly Analysis
    (Complex algorithms)
    No Yes
    Automated data insights
    and plotting anomalies in
    the reports
    No Yes (Future roadmap)
    

    View Slide