Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Demae-can Machine Learning Pipeline and Continuous Product Improvement

Demae-can Machine Learning Pipeline and Continuous Product Improvement

Hiroo Takaku (Demae-can / Machine Learning Team / Technical Product Manager)

https://tech-verse.me/ja/sessions/118
https://tech-verse.me/en/sessions/118
https://tech-verse.me/ko/sessions/118

Tech-Verse2022

November 17, 2022
Tweet

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. Overview Machine learning pipeline • Automate ML process for continuous

    improvement and stable operation • Build ML pipeline to connect each process
  2. Overview Machine learning pipeline • Automate ML process for continuous

    improvement and stable operation • Build ML pipeline to connect each process Continuous product improvement • ML components affect each other and unexpected things occur • Identify problems and improve the products with data
  3. Agenda - What service Demae-can provides - How ML components

    work - What ML pipeline we builds - Continuous improvement of ML products
  4. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Order (user) Item preparation (merchant)
  5. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Order (user) Item preparation (merchant)
  6. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Order (user) Item preparation (merchant)
  7. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant)
  8. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) • Foods • Daily necessities • Medicine
  9. User problem • Tons of foods, drinks and other delivery

    items to choose from • Difficult to find new options Solution • Suggest items users don't know about yet Recommendations
  10. User demand • Want to know when deliveries will arrive

    Solution • Show the expected arrival time Delivery time prediction
  11. Driver demand • Want to know where driver can receive

    lots of delivery offers Solution • Show areas with expected high demand and few drivers Order forecast
  12. Driver demand • Want to increase delivery offers & income

    • Want to pick up items for delivery and nearby shops • Don’t want to wait for orders to be ready Solution • Provide globally-efficient order-driver allocation Delivery offer (Order dispatch)
  13. You see the details of ML components on LINE DEVELOPER

    DAY 2021 YouTube https://www.youtube.com/watch?v=LN38JM2Oo1c logmi https://logmi.jp/tech/articles/325404 The details of ML components
  14. Item preparation time prediction order arrive at shop items ready

    item preparation time delivery time arrive at destination
  15. Item preparation time prediction order arrive at shop items ready

    item preparation time delivery time shop arrival time arrive at destination
  16. Item preparation time prediction order arrive at shop items ready

    item preparation time travel time from shop to destination delivery time shop arrival time arrive at destination
  17. Item preparation time prediction order arrive at shop items ready

    arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c
  18. Item preparation time prediction order arrive at shop items ready

    item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction arrive at destination
  19. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes
  20. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes
  21. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes Offline feature • shop info / genre info etc. (daily update)
  22. Use ML models that created with latest data Use real

    time features that created with latest data Summary Use offline features that created with latest data
  23. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
  24. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration
  25. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster
  26. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster • Use Airflow (workflow), ArgoCD (CD), MLflow (ML lifecycle management)
  27. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
  28. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Data collection Data ETL
  29. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Model training Model verification Deployment Data collection Data ETL
  30. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  31. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  32. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  33. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  34. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  35. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  36. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  37. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system Prediction server Demae-can server Hadoop cluster ML batch system
  38. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system ETL Prediction server Demae-can server Hadoop cluster ML batch system
  39. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system
  40. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
  41. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Streaming ETL server Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
  42. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Streaming data pick up items at shop Driver User Merchant / Shop
  43. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 Streaming data pick up items at shop Driver User Merchant / Shop Order Order offer
  44. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 Streaming data pick up items at shop Driver User Merchant / Shop Shop accept
  45. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 Streaming data pick up items at shop Driver User Merchant / Shop Delivery offer
  46. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 Streaming data pick up items at shop Driver User Merchant / Shop Driver accept
  47. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at shop
  48. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 Streaming data pick up items at shop Driver User Merchant / Shop Pick up items at shop (driver)
  49. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at destination
  50. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) Streaming data pick up items at shop Streaming ETL
  51. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data pick up items at shop Streaming ETL
  52. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data sync • Time-series data is useful for analysis • Insert to Hadoop cluster (DWH) Streaming data pick up items at shop Streaming ETL
  53. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Data collection Model verification Deployment Data ETL
  54. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Real time features (Redis) Offline features (Redis) Hadoop cluster
  55. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Real time features (Redis) Offline features (Redis) Hadoop cluster
  56. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
  57. MLFlow allows us to check when we created models and

    deployed them Daily model update
  58. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  59. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  60. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  61. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  62. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  63. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  64. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Summary Monitoring Model training Data collection Model verification Deployment Data ETL
  65. Contents delivery time (prediction data) delivery time (actual data) •

    This graph shows delivery time (prediction data and actual data) on a certain day
  66. Contents delivery time (prediction data) delivery time (actual data) •

    There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This graph shows delivery time (prediction data and actual data) on a certain day
  67. Contents delivery time (prediction data) delivery time (actual data) •

    There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This issue had a negative effect on drivers and users • We had to identify and handle this issue • This graph shows delivery time (prediction data and actual data) on a certain day
  68. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other
  69. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item pick up time (actual data) • Item pick up time (actual data) increased
  70. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased Item pick up time (actual data)
  71. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased • This feedback loop occurred between ML components Item pick up time (actual data)
  72. Item preparation time prediction order arrive at shop items ready

    arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction
  73. order arrive at shop arrive at destination delivery offer shop

    accept driver accept pick up items at shop Item preparation time prediction • Use mean item pick up time in the last n minutes as a real time feature • When mean item pick up time increases, the item preparation prediction time increases Item pick up time
  74. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Delivery offer • Delivery offer timing depends on item preparation time and travel time
  75. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time
  76. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time • Deliver offer component uses item preparation prediction time • When the item preparation prediction time increases, delivery offer time delays
  77. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason Item pick up time (actual data)
  78. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
  79. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed Item preparation prediction time Item pick up time (actual data)
  80. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased Item preparation prediction time Item pick up time (actual data)
  81. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased 5. This feedback loop occurred, especially item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
  82. ML components affected each other We had to get the

    clue of this feedback loop The difficulty to identify this problem
  83. ML components affected each other The clue was the delay

    of delivery offer timing We had to get the clue of this feedback loop The difficulty to identify this problem
  84. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server The data

    that we used to identify this problem Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  85. 1. Divide delivery prediction time The process to identify this

    problem Contents Delivery time prediction Actual delivery time
  86. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time
  87. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time 2. Check the model and features The process to identify this problem Contents Delivery time prediction Actual delivery time
  88. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Check the

    model and features Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
  89. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the feature importance Mean item pick up time (real time feature)
  90. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  91. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  92. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  93. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time 2. Check the model and features & find the increase of item pick up time 3. Divide item pick up time The process to identify this problem Contents Delivery time prediction Actual delivery time
  94. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Divide item

    pick up time Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  95. order arrive at destination delivery offer shop accept driver accept

    arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop
  96. order arrive at destination delivery offer shop accept driver accept

    Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Order Order offer Shop accept
  97. order arrive at destination delivery offer shop accept driver accept

    Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Delivery offer Shop accept
  98. order arrive at destination delivery offer shop accept driver accept

    Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Delivery offer
  99. order arrive at destination delivery offer shop accept driver accept

    Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Arrive at shop Pick up items at shop (driver)
  100. Contents Time from order to shop accept (actual data) Time

    from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
  101. Contents Time from order to shop accept (actual data) Time

    from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
  102. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Pick up time is 5 min item preparation time is 15 min Delivery offer Wait for delivery offer • When the item preparation prediction time increases, delivery offer time delays
  103. This feedback loop occurred Item preparation time prediction Delivery offer

    Prepatation prediction time Item pick up time (actual data)
  104. order arrive at shop arrive at destination delivery offer shop

    accept driver accept pick up items at shop Item preparation time prediction • Decided not to use mean item pick up time in the last n minutes as a real time feature Item pick up time
  105. ML components are affected by many things that happened on

    Demae-can Monitor & analyze the data and improve ML products Summary
  106. ML components are affected by many things that happened on

    Demae-can Have the perspective that overall system works well Monitor & analyze the data and improve ML components Summary