Overview Machine learning pipeline • Automate ML process for continuous improvement and stable operation • Build ML pipeline to connect each process Continuous product improvement • ML components affect each other and unexpected things occur • Identify problems and improve the products with data
Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Order (user) Item preparation (merchant)
Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant)
Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) • Foods • Daily necessities • Medicine
User problem • Tons of foods, drinks and other delivery items to choose from • Difficult to find new options Solution • Suggest items users don't know about yet Recommendations
Driver demand • Want to know where driver can receive lots of delivery offers Solution • Show areas with expected high demand and few drivers Order forecast
Driver demand • Want to increase delivery offers & income • Want to pick up items for delivery and nearby shops • Don’t want to wait for orders to be ready Solution • Provide globally-efficient order-driver allocation Delivery offer (Order dispatch)
You see the details of ML components on LINE DEVELOPER DAY 2021 YouTube https://www.youtube.com/watch?v=LN38JM2Oo1c logmi https://logmi.jp/tech/articles/325404 The details of ML components
Item preparation time prediction order arrive at shop items ready item preparation time travel time from shop to destination delivery time shop arrival time arrive at destination
Item preparation time prediction order arrive at shop items ready arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c
Item preparation time prediction order arrive at shop items ready item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction arrive at destination
Item preparation time prediction Model • lightGBM (daily update) Real time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes
Item preparation time prediction Model • lightGBM (daily update) Real time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes Offline feature • shop info / genre info etc. (daily update)
Use ML models that created with latest data Use real time features that created with latest data Summary Use offline features that created with latest data
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
ML process or lifecycle Model verification Model training Data ETL Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster
ML process or lifecycle Model verification Model training Data ETL Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster • Use Airflow (workflow), ArgoCD (CD), MLflow (ML lifecycle management)
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Data collection Data ETL
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Model training Model verification Deployment Data collection Data ETL
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system Prediction server Demae-can server Hadoop cluster ML batch system
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system ETL Prediction server Demae-can server Hadoop cluster ML batch system
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Streaming ETL server Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 Streaming data pick up items at shop Driver User Merchant / Shop Order Order offer
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 Streaming data pick up items at shop Driver User Merchant / Shop Shop accept
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 Streaming data pick up items at shop Driver User Merchant / Shop Delivery offer
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 Streaming data pick up items at shop Driver User Merchant / Shop Driver accept
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at shop
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 Streaming data pick up items at shop Driver User Merchant / Shop Pick up items at shop (driver)
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at destination
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) Streaming data pick up items at shop Streaming ETL
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data pick up items at shop Streaming ETL
order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data sync • Time-series data is useful for analysis • Insert to Hadoop cluster (DWH) Streaming data pick up items at shop Streaming ETL
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Data collection Model verification Deployment Data ETL
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Real time features (Redis) Offline features (Redis) Hadoop cluster
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Real time features (Redis) Offline features (Redis) Hadoop cluster
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Summary Monitoring Model training Data collection Model verification Deployment Data ETL
Contents delivery time (prediction data) delivery time (actual data) • This graph shows delivery time (prediction data and actual data) on a certain day
Contents delivery time (prediction data) delivery time (actual data) • There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This graph shows delivery time (prediction data and actual data) on a certain day
Contents delivery time (prediction data) delivery time (actual data) • There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This issue had a negative effect on drivers and users • We had to identify and handle this issue • This graph shows delivery time (prediction data and actual data) on a certain day
The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other
The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item pick up time (actual data) • Item pick up time (actual data) increased
The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased Item pick up time (actual data)
The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased • This feedback loop occurred between ML components Item pick up time (actual data)
Item preparation time prediction order arrive at shop items ready arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction
order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Item preparation time prediction • Use mean item pick up time in the last n minutes as a real time feature • When mean item pick up time increases, the item preparation prediction time increases Item pick up time
driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Delivery offer • Delivery offer timing depends on item preparation time and travel time
driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time
driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time • Deliver offer component uses item preparation prediction time • When the item preparation prediction time increases, delivery offer time delays
What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason Item pick up time (actual data)
What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed Item preparation prediction time Item pick up time (actual data)
What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased Item preparation prediction time Item pick up time (actual data)
What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased 5. This feedback loop occurred, especially item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
ML components affected each other The clue was the delay of delivery offer timing We had to get the clue of this feedback loop The difficulty to identify this problem
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server The data that we used to identify this problem Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
1. Divide delivery prediction time & find the increase of item preparation prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time
1. Divide delivery prediction time & find the increase of item preparation prediction time 2. Check the model and features The process to identify this problem Contents Delivery time prediction Actual delivery time
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Check the model and features Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the feature importance Mean item pick up time (real time feature)
Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
1. Divide delivery prediction time & find the increase of item preparation prediction time 2. Check the model and features & find the increase of item pick up time 3. Divide item pick up time The process to identify this problem Contents Delivery time prediction Actual delivery time
Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Divide item pick up time Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
order arrive at destination delivery offer shop accept driver accept arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop
order arrive at destination delivery offer shop accept driver accept Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Order Order offer Shop accept
order arrive at destination delivery offer shop accept driver accept Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Delivery offer Shop accept
order arrive at destination delivery offer shop accept driver accept Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Delivery offer
order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Arrive at shop Pick up items at shop (driver)
Contents Time from order to shop accept (actual data) Time from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
Contents Time from order to shop accept (actual data) Time from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Pick up time is 5 min item preparation time is 15 min Delivery offer Wait for delivery offer • When the item preparation prediction time increases, delivery offer time delays
order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Item preparation time prediction • Decided not to use mean item pick up time in the last n minutes as a real time feature Item pick up time
ML components are affected by many things that happened on Demae-can Have the perspective that overall system works well Monitor & analyze the data and improve ML components Summary
techverse_2022
yoshimi0227
nishiuma
niftycorp
biwashi
aoshun7
knsg16
yoshiitaka
s2terminal
minorun365
kazamori
shibuiwilliam
sinsoku
jakevdp
bcantrill
orderedlist
eileencodes
hannesfritz
zenorocha
philhawksworth
lara
bkeepers
kneath
jcasabona
marktimemedia