Share
Embed
Demae-can Machine Learning Pipeline and Continuous Product Improvement
Hiroo Takaku (Demae-can / Machine Learning Team / Technical Product Manager)
https://tech-verse.me/ja/sessions/118
https://tech-verse.me/en/sessions/118
https://tech-verse.me/ko/sessions/118
Tech-Verse2022
November 17, 2022
Transcript
131 slides
-
Overview Machine learning pipeline • Automate ML process for continuous
improvement and stable operation • Build ML pipeline to connect each process -
Overview Machine learning pipeline • Automate ML process for continuous
improvement and stable operation • Build ML pipeline to connect each process Continuous product improvement • ML components affect each other and unexpected things occur • Identify problems and improve the products with data -
Target audience Data scientist ML engineer Software engineer and all
people who are interested in ML -
Agenda - What service Demae-can provides - How ML components
work - What ML pipeline we builds - Continuous improvement of ML products -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Order (user) -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Order (user) Item preparation (merchant) -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Delivery offer Order (user) Item preparation (merchant) -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Delivery offer Item pick up (driver) Order (user) Item preparation (merchant) -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) -
Demae-can is a multi-sided platform Driver User Merchant / Shop
Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) • Foods • Daily necessities • Medicine -
User problem • Tons of foods, drinks and other delivery
items to choose from • Difficult to find new options Solution • Suggest items users don't know about yet Recommendations -
User demand • Want to know when deliveries will arrive
Solution • Show the expected arrival time Delivery time prediction -
Driver demand • Want to know where driver can receive
lots of delivery offers Solution • Show areas with expected high demand and few drivers Order forecast -
Driver demand • Want to increase delivery offers & income
• Want to pick up items for delivery and nearby shops • Don’t want to wait for orders to be ready Solution • Provide globally-efficient order-driver allocation Delivery offer (Order dispatch) -
You see the details of ML components on LINE DEVELOPER
DAY 2021 YouTube https://www.youtube.com/watch?v=LN38JM2Oo1c logmi https://logmi.jp/tech/articles/325404 The details of ML components -
Item preparation time prediction order arrive at shop items ready
delivery time arrive at destination -
Item preparation time prediction order arrive at shop items ready
item preparation time delivery time arrive at destination -
Item preparation time prediction order arrive at shop items ready
item preparation time delivery time shop arrival time arrive at destination -
Item preparation time prediction order arrive at shop items ready
item preparation time travel time from shop to destination delivery time shop arrival time arrive at destination -
Item preparation time prediction order arrive at shop items ready
arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c -
Item preparation time prediction order arrive at shop items ready
item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction arrive at destination -
Item preparation time prediction Model • lightGBM (daily update) Real
time feature • Item pick up time (≒item preparation time) in the last n minutes -
Item preparation time prediction Model • lightGBM (daily update) Real
time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes -
Item preparation time prediction Model • lightGBM (daily update) Real
time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes Offline feature • shop info / genre info etc. (daily update) -
Use ML models that created with latest data Use real
time features that created with latest data Summary Use offline features that created with latest data -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement -
ML process or lifecycle Model verification Model training Data ETL
Data collection Monitoring Deployment Automation & pipeline orchestration -
ML process or lifecycle Model verification Model training Data ETL
Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster -
ML process or lifecycle Model verification Model training Data ETL
Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster • Use Airflow (workflow), ArgoCD (CD), MLflow (ML lifecycle management) -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Data collection Data ETL -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Model training Model verification Deployment Data collection Data ETL -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL -
Real time features (Redis) Offline features (Redis) Data collection &
Data ETL ML real time system Prediction server Demae-can server ML batch system -
Real time features (Redis) Offline features (Redis) Data collection &
Data ETL ML real time system Prediction server Demae-can server ML batch system -
Real time features (Redis) Offline features (Redis) Data collection &
Data ETL ML real time system Prediction server Demae-can server ML batch system -
Real time features (Redis) Offline features (Redis) Data collection &
Data ETL ML real time system Prediction server Demae-can server ML batch system -
Real time features (Redis) Offline features (Redis) Data collection &
Data ETL ML real time system Prediction server Demae-can server ML batch system -
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)
Data collection & Data ETL ML real time system Prediction server Demae-can server Hadoop cluster ML batch system -
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)
Data collection & Data ETL ML real time system ETL Prediction server Demae-can server Hadoop cluster ML batch system -
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)
Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system -
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)
Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka) -
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)
Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Streaming ETL server Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka) -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Streaming data pick up items at shop Driver User Merchant / Shop -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 Streaming data pick up items at shop Driver User Merchant / Shop Order Order offer -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 Streaming data pick up items at shop Driver User Merchant / Shop Shop accept -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 Streaming data pick up items at shop Driver User Merchant / Shop Delivery offer -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 Streaming data pick up items at shop Driver User Merchant / Shop Driver accept -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at shop -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 Streaming data pick up items at shop Driver User Merchant / Shop Pick up items at shop (driver) -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at destination -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) Streaming data pick up items at shop Streaming ETL -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data pick up items at shop Streaming ETL -
order arrive at shop arrive at destination delivery offer shop
accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data sync • Time-series data is useful for analysis • Insert to Hadoop cluster (DWH) Streaming data pick up items at shop Streaming ETL -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Data collection Model verification Deployment Data ETL -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model
update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Real time features (Redis) Offline features (Redis) Hadoop cluster -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model
update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Real time features (Redis) Offline features (Redis) Hadoop cluster -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model
update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster -
MLFlow allows us to check when we created models and
deployed them Daily model update -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming
ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming
ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming
ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming
ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming
ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
Create real time features & offline features with latest data
Record all the data related with prediction Create ML models with latest data Summary Monitoring Model training Data collection Model verification Deployment Data ETL -
Contents delivery time (prediction data) delivery time (actual data) •
This graph shows delivery time (prediction data and actual data) on a certain day -
Contents delivery time (prediction data) delivery time (actual data) •
There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This graph shows delivery time (prediction data and actual data) on a certain day -
Contents delivery time (prediction data) delivery time (actual data) •
There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This issue had a negative effect on drivers and users • We had to identify and handle this issue • This graph shows delivery time (prediction data and actual data) on a certain day -
The feedback loop occurred Item preparation time prediction Delivery offer
• ML components (delivery offer & item preparation time prediction) affected each other -
The feedback loop occurred Item preparation time prediction Delivery offer
• ML components (delivery offer & item preparation time prediction) affected each other Item pick up time (actual data) • Item pick up time (actual data) increased -
The feedback loop occurred Item preparation time prediction Delivery offer
• ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased Item pick up time (actual data) -
The feedback loop occurred Item preparation time prediction Delivery offer
• ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased • This feedback loop occurred between ML components Item pick up time (actual data) -
Item preparation time prediction order arrive at shop items ready
arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction -
order arrive at shop arrive at destination delivery offer shop
accept driver accept pick up items at shop Item preparation time prediction • Use mean item pick up time in the last n minutes as a real time feature • When mean item pick up time increases, the item preparation prediction time increases Item pick up time -
driver merchant order arrive at shop arrive at destination delivery
offer shop accept driver accept pick up items at shop Delivery offer • Delivery offer timing depends on item preparation time and travel time -
driver merchant order arrive at shop arrive at destination delivery
offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time -
driver merchant order arrive at shop arrive at destination delivery
offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time • Deliver offer component uses item preparation prediction time • When the item preparation prediction time increases, delivery offer time delays -
What occurred ? Item preparation time prediction Delivery offer 1.
Item pick up time (actual data) increased for some reason Item pick up time (actual data) -
What occurred ? Item preparation time prediction Delivery offer 1.
Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased Item preparation prediction time Item pick up time (actual data) -
What occurred ? Item preparation time prediction Delivery offer 1.
Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed Item preparation prediction time Item pick up time (actual data) -
What occurred ? Item preparation time prediction Delivery offer 1.
Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased Item preparation prediction time Item pick up time (actual data) -
What occurred ? Item preparation time prediction Delivery offer 1.
Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased 5. This feedback loop occurred, especially item preparation prediction time increased Item preparation prediction time Item pick up time (actual data) -
ML components affected each other We had to get the
clue of this feedback loop The difficulty to identify this problem -
ML components affected each other The clue was the delay
of delivery offer timing We had to get the clue of this feedback loop The difficulty to identify this problem -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server The data
that we used to identify this problem Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
1. Divide delivery prediction time The process to identify this
problem Contents Delivery time prediction Actual delivery time -
1. Divide delivery prediction time & find the increase of
item preparation prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time -
1. Divide delivery prediction time & find the increase of
item preparation prediction time 2. Check the model and features The process to identify this problem Contents Delivery time prediction Actual delivery time -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Check the
model and features Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster -
Check the model and features • Feature 12 is mean
pick up time (real time feature / actual data) Check the feature importance Mean item pick up time (real time feature) -
Check the model and features • Feature 12 is mean
pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Mean item pick up time (actual data) Mean item pick up time (real time feature) -
Check the model and features • Feature 12 is mean
pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature) -
Check the model and features • Feature 12 is mean
pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature) -
1. Divide delivery prediction time & find the increase of
item preparation prediction time 2. Check the model and features & find the increase of item pick up time 3. Divide item pick up time The process to identify this problem Contents Delivery time prediction Actual delivery time -
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Divide item
pick up time Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system -
order arrive at destination delivery offer shop accept driver accept
arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop -
order arrive at destination delivery offer shop accept driver accept
Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Order Order offer Shop accept -
order arrive at destination delivery offer shop accept driver accept
Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Delivery offer Shop accept -
order arrive at destination delivery offer shop accept driver accept
Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Delivery offer -
order arrive at destination delivery offer shop accept driver accept
Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Arrive at shop Pick up items at shop (driver) -
Contents Time from order to shop accept (actual data) Time
from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 -
Contents Time from order to shop accept (actual data) Time
from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 -
driver merchant order arrive at shop arrive at destination delivery
offer shop accept driver accept pick up items at shop Pick up time is 5 min item preparation time is 15 min Delivery offer Wait for delivery offer • When the item preparation prediction time increases, delivery offer time delays -
This feedback loop occurred Item preparation time prediction Delivery offer
Prepatation prediction time Item pick up time (actual data) -
order arrive at shop arrive at destination delivery offer shop
accept driver accept pick up items at shop Item preparation time prediction • Decided not to use mean item pick up time in the last n minutes as a real time feature Item pick up time -
ML components are affected by many things that happened on
Demae-can Summary -
ML components are affected by many things that happened on
Demae-can Monitor & analyze the data and improve ML products Summary -
ML components are affected by many things that happened on
Demae-can Have the perspective that overall system works well Monitor & analyze the data and improve ML components Summary