Demae-can Machine Learning Pipeline and Continuous Product Improvement

Hiroo Takaku (Demae-can / Machine Learning Team / Technical Product Manager)

https://tech-verse.me/ja/sessions/118
https://tech-verse.me/en/sessions/118
https://tech-verse.me/ko/sessions/118

Transcript

2. Overview Machine learning pipeline • Automate ML process for continuous improvement and stable operation • Build ML pipeline to connect each process
3. Overview Machine learning pipeline • Automate ML process for continuous improvement and stable operation • Build ML pipeline to connect each process Continuous product improvement • ML components affect each other and unexpected things occur • Identify problems and improve the products with data
4. Target audience Data scientist ML engineer Software engineer
5. Target audience Data scientist ML engineer Software engineer and all people who are interested in ML
6. Agenda - What service Demae-can provides - How ML components work - What ML pipeline we builds - Continuous improvement of ML products
7. What service Demae-can provides
8. Demae-can is a multi-sided platform User
9. Demae-can is a multi-sided platform Driver User
10. Demae-can is a multi-sided platform User Merchant / Shop Driver
11. Demae-can is a multi-sided platform Driver User Merchant / Shop Order (user)
12. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Order (user)
13. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Order (user) Item preparation (merchant)
14. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Order (user) Item preparation (merchant)
15. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Order (user) Item preparation (merchant)
16. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant)
17. Demae-can is a multi-sided platform Driver User Merchant / Shop Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) • Foods • Daily necessities • Medicine
18. What ML components we build
19. User problem • Tons of foods, drinks and other delivery items to choose from • Difficult to find new options Solution • Suggest items users don't know about yet Recommendations
20. User demand • Want to know when deliveries will arrive Solution • Show the expected arrival time Delivery time prediction
21. Driver demand • Want to know where driver can receive lots of delivery offers Solution • Show areas with expected high demand and few drivers Order forecast
22. Driver demand • Want to increase delivery offers & income • Want to pick up items for delivery and nearby shops • Don’t want to wait for orders to be ready Solution • Provide globally-efficient order-driver allocation Delivery offer (Order dispatch)
23. How ML components work
24. You see the details of ML components on LINE DEVELOPER DAY 2021 YouTube https://www.youtube.com/watch?v=LN38JM2Oo1c logmi https://logmi.jp/tech/articles/325404 The details of ML components
25. Item preparation time prediction (Delivery time prediction)
26. Item preparation time prediction order arrive at shop items ready delivery time arrive at destination
27. Item preparation time prediction order arrive at shop items ready item preparation time delivery time arrive at destination
28. Item preparation time prediction order arrive at shop items ready item preparation time delivery time shop arrival time arrive at destination
29. Item preparation time prediction order arrive at shop items ready item preparation time travel time from shop to destination delivery time shop arrival time arrive at destination
30. Item preparation time prediction order arrive at shop items ready arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c
31. Item preparation time prediction order arrive at shop items ready item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction arrive at destination
32. Item preparation time prediction Model • lightGBM (daily update)
33. Item preparation time prediction Model • lightGBM (daily update) Real time feature • Item pick up time (≒item preparation time) in the last n minutes
34. Item preparation time prediction Model • lightGBM (daily update) Real time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes
35. Item preparation time prediction Model • lightGBM (daily update) Real time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes Offline feature • shop info / genre info etc. (daily update)
36. Use ML models that created with latest data Use real time features that created with latest data Summary Use offline features that created with latest data
37. What ML pipeline Demae-can builds
38. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
39. ML process or lifecycle Model verification Model training Data ETL Data collection Monitoring Deployment Automation & pipeline orchestration
40. ML process or lifecycle Model verification Model training Data ETL Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster
41. ML process or lifecycle Model verification Model training Data ETL Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster • Use Airflow (workflow), ArgoCD (CD), MLflow (ML lifecycle management)
42. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
43. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Data collection Data ETL
44. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Model training Model verification Deployment Data collection Data ETL
45. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
46. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
47. Real time features (Redis) Offline features (Redis) Data collection & Data ETL ML real time system Prediction server Demae-can server ML batch system
48. Real time features (Redis) Offline features (Redis) Data collection & Data ETL ML real time system Prediction server Demae-can server ML batch system
49. Real time features (Redis) Offline features (Redis) Data collection & Data ETL ML real time system Prediction server Demae-can server ML batch system
50. Real time features (Redis) Offline features (Redis) Data collection & Data ETL ML real time system Prediction server Demae-can server ML batch system
51. Real time features (Redis) Offline features (Redis) Data collection & Data ETL ML real time system Prediction server Demae-can server ML batch system
52. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system Prediction server Demae-can server Hadoop cluster ML batch system
53. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system ETL Prediction server Demae-can server Hadoop cluster ML batch system
54. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system
55. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
56. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB) Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Streaming ETL server Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
57. order arrive at shop arrive at destination delivery offer shop accept driver accept Streaming data pick up items at shop Driver User Merchant / Shop
58. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 Streaming data pick up items at shop Driver User Merchant / Shop Order Order offer
59. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 Streaming data pick up items at shop Driver User Merchant / Shop Shop accept
60. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 Streaming data pick up items at shop Driver User Merchant / Shop Delivery offer
61. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 Streaming data pick up items at shop Driver User Merchant / Shop Driver accept
62. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at shop
63. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 Streaming data pick up items at shop Driver User Merchant / Shop Pick up items at shop (driver)
64. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at destination
65. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) Streaming data pick up items at shop Streaming ETL
66. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data pick up items at shop Streaming ETL
67. order arrive at shop arrive at destination delivery offer shop accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data sync • Time-series data is useful for analysis • Insert to Hadoop cluster (DWH) Streaming data pick up items at shop Streaming ETL
68. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Data collection Model verification Deployment Data ETL
69. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Real time features (Redis) Offline features (Redis) Hadoop cluster
70. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Real time features (Redis) Offline features (Redis) Hadoop cluster
71. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
72. MLFlow allows us to check when we created models and deployed them Daily model update
73. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
74. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
75. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
76. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
77. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
78. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
79. Create real time features & offline features with latest data Record all the data related with prediction Create ML models with latest data Summary Monitoring Model training Data collection Model verification Deployment Data ETL
80. Continuous improvement of ML products
81. Contents delivery time (prediction data) delivery time (actual data) • This graph shows delivery time (prediction data and actual data) on a certain day
82. Contents delivery time (prediction data) delivery time (actual data) • There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This graph shows delivery time (prediction data and actual data) on a certain day
83. Contents delivery time (prediction data) delivery time (actual data) • There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This issue had a negative effect on drivers and users • We had to identify and handle this issue • This graph shows delivery time (prediction data and actual data) on a certain day
84. The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other
85. The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item pick up time (actual data) • Item pick up time (actual data) increased
86. The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased Item pick up time (actual data)
87. The feedback loop occurred Item preparation time prediction Delivery offer • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased • This feedback loop occurred between ML components Item pick up time (actual data)
88. Item preparation time prediction order arrive at shop items ready arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction
89. order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Item preparation time prediction • Use mean item pick up time in the last n minutes as a real time feature • When mean item pick up time increases, the item preparation prediction time increases Item pick up time
90. driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Delivery offer • Delivery offer timing depends on item preparation time and travel time
91. driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time
92. driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time • Deliver offer component uses item preparation prediction time • When the item preparation prediction time increases, delivery offer time delays
93. What occurred ? Item preparation time prediction Delivery offer
94. What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason Item pick up time (actual data)
95. What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
96. What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed Item preparation prediction time Item pick up time (actual data)
97. What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased Item preparation prediction time Item pick up time (actual data)
98. What occurred ? Item preparation time prediction Delivery offer 1. Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased 5. This feedback loop occurred, especially item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
99. How did we identify this problem ?
100. ML components affected each other The difficulty to identify this problem
101. ML components affected each other We had to get the clue of this feedback loop The difficulty to identify this problem
102. ML components affected each other The clue was the delay of delivery offer timing We had to get the clue of this feedback loop The difficulty to identify this problem
103. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server The data that we used to identify this problem Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
104. 1. Divide delivery prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time
105. 1. Divide delivery prediction time & find the increase of item preparation prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time
106. 1. Divide delivery prediction time & find the increase of item preparation prediction time 2. Check the model and features The process to identify this problem Contents Delivery time prediction Actual delivery time
107. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Check the model and features Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
108. Check the model and features Check the feature importance
109. Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the feature importance Mean item pick up time (real time feature)
110. Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
111. Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
112. Check the model and features • Feature 12 is mean pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
113. 1. Divide delivery prediction time & find the increase of item preparation prediction time 2. Check the model and features & find the increase of item pick up time 3. Divide item pick up time The process to identify this problem Contents Delivery time prediction Actual delivery time
114. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Divide item pick up time Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
115. order arrive at destination delivery offer shop accept driver accept arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop
116. order arrive at destination delivery offer shop accept driver accept Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Order Order offer Shop accept
117. order arrive at destination delivery offer shop accept driver accept Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Delivery offer Shop accept
118. order arrive at destination delivery offer shop accept driver accept Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Delivery offer
119. order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Arrive at shop Pick up items at shop (driver)
120. Contents Time from order to shop accept (actual data) Time from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
121. Contents Time from order to shop accept (actual data) Time from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
122. driver merchant order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Pick up time is 5 min item preparation time is 15 min Delivery offer Wait for delivery offer • When the item preparation prediction time increases, delivery offer time delays
123. This feedback loop occurred Item preparation time prediction Delivery offer Prepatation prediction time Item pick up time (actual data)
124. How did we handle this problem ?
125. order arrive at shop arrive at destination delivery offer shop accept driver accept pick up items at shop Item preparation time prediction • Decided not to use mean item pick up time in the last n minutes as a real time feature Item pick up time
126. ML components are affected by many things that happened on Demae-can Summary
127. ML components are affected by many things that happened on Demae-can Monitor & analyze the data and improve ML products Summary
128. ML components are affected by many things that happened on Demae-can Have the perspective that overall system works well Monitor & analyze the data and improve ML components Summary
129. Lastly
130. What time is it ? Demae-can time
131. Thank you