$30 off During Our Annual Pro Sale. View Details »

Demae-can Machine Learning Pipeline and Continuous Product Improvement

Demae-can Machine Learning Pipeline and Continuous Product Improvement

Hiroo Takaku (Demae-can / Machine Learning Team / Technical Product Manager)

https://tech-verse.me/ja/sessions/118
https://tech-verse.me/en/sessions/118
https://tech-verse.me/ko/sessions/118

Tech-Verse2022
PRO

November 17, 2022
Tweet

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. None
  2. Overview Machine learning pipeline • Automate ML process for continuous

    improvement and stable operation • Build ML pipeline to connect each process
  3. Overview Machine learning pipeline • Automate ML process for continuous

    improvement and stable operation • Build ML pipeline to connect each process Continuous product improvement • ML components affect each other and unexpected things occur • Identify problems and improve the products with data
  4. Target audience Data scientist ML engineer Software engineer

  5. Target audience Data scientist ML engineer Software engineer and all

    people who are interested in ML
  6. Agenda - What service Demae-can provides - How ML components

    work - What ML pipeline we builds - Continuous improvement of ML products
  7. What service Demae-can provides

  8. Demae-can is a multi-sided platform User

  9. Demae-can is a multi-sided platform Driver User

  10. Demae-can is a multi-sided platform User Merchant / Shop Driver

  11. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order (user)
  12. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Order (user)
  13. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Order (user) Item preparation (merchant)
  14. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Order (user) Item preparation (merchant)
  15. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Order (user) Item preparation (merchant)
  16. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant)
  17. Demae-can is a multi-sided platform Driver User Merchant / Shop

    Order offer Delivery offer Item pick up (driver) Delivery (driver) Order (user) Item preparation (merchant) • Foods • Daily necessities • Medicine
  18. What ML components we build

  19. User problem • Tons of foods, drinks and other delivery

    items to choose from • Difficult to find new options Solution • Suggest items users don't know about yet Recommendations
  20. User demand • Want to know when deliveries will arrive

    Solution • Show the expected arrival time Delivery time prediction
  21. Driver demand • Want to know where driver can receive

    lots of delivery offers Solution • Show areas with expected high demand and few drivers Order forecast
  22. Driver demand • Want to increase delivery offers & income

    • Want to pick up items for delivery and nearby shops • Don’t want to wait for orders to be ready Solution • Provide globally-efficient order-driver allocation Delivery offer (Order dispatch)
  23. How ML components work

  24. You see the details of ML components on LINE DEVELOPER

    DAY 2021 YouTube https://www.youtube.com/watch?v=LN38JM2Oo1c logmi https://logmi.jp/tech/articles/325404 The details of ML components
  25. Item preparation time prediction (Delivery time prediction)

  26. Item preparation time prediction order arrive at shop items ready

    delivery time arrive at destination
  27. Item preparation time prediction order arrive at shop items ready

    item preparation time delivery time arrive at destination
  28. Item preparation time prediction order arrive at shop items ready

    item preparation time delivery time shop arrival time arrive at destination
  29. Item preparation time prediction order arrive at shop items ready

    item preparation time travel time from shop to destination delivery time shop arrival time arrive at destination
  30. Item preparation time prediction order arrive at shop items ready

    arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c
  31. Item preparation time prediction order arrive at shop items ready

    item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction arrive at destination
  32. Item preparation time prediction Model • lightGBM (daily update)

  33. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes
  34. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes
  35. Item preparation time prediction Model • lightGBM (daily update) Real

    time feature • Item pick up time (≒item preparation time) in the last n minutes • Ratio of (number of drivers) / (number of orders) in the last n minutes Offline feature • shop info / genre info etc. (daily update)
  36. Use ML models that created with latest data Use real

    time features that created with latest data Summary Use offline features that created with latest data
  37. What ML pipeline Demae-can builds

  38. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
  39. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration
  40. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster
  41. ML process or lifecycle Model verification Model training Data ETL

    Data collection Monitoring Deployment Automation & pipeline orchestration • Don’t use public cloud service • Use self-managed k8s cluster, Hadoop cluster (DWH), Kafka cluster • Use Airflow (workflow), ArgoCD (CD), MLflow (ML lifecycle management)
  42. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement
  43. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Data collection Data ETL
  44. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Model training Model verification Deployment Data collection Data ETL
  45. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  46. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  47. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  48. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  49. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  50. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  51. Real time features (Redis) Offline features (Redis) Data collection &

    Data ETL ML real time system Prediction server Demae-can server ML batch system
  52. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system Prediction server Demae-can server Hadoop cluster ML batch system
  53. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system ETL Prediction server Demae-can server Hadoop cluster ML batch system
  54. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system
  55. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
  56. Real time features (Redis) Offline features (Redis) Demae-can DB (RDB)

    Demae-can server Data collection & Data ETL ML real time system Feature loader ETL Streaming ETL server Prediction server Demae-can server Hadoop cluster ML batch system Data streaming (Kafka)
  57. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Streaming data pick up items at shop Driver User Merchant / Shop
  58. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 Streaming data pick up items at shop Driver User Merchant / Shop Order Order offer
  59. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 Streaming data pick up items at shop Driver User Merchant / Shop Shop accept
  60. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 Streaming data pick up items at shop Driver User Merchant / Shop Delivery offer
  61. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 Streaming data pick up items at shop Driver User Merchant / Shop Driver accept
  62. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at shop
  63. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 Streaming data pick up items at shop Driver User Merchant / Shop Pick up items at shop (driver)
  64. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Streaming data pick up items at shop Driver User Merchant / Shop Arrive at destination
  65. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) Streaming data pick up items at shop Streaming ETL
  66. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data pick up items at shop Streaming ETL
  67. order arrive at shop arrive at destination delivery offer shop

    accept driver accept Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Ex. The mean time to pick up items in the last 15 minutes in a certain area • Window aggregation (15 minutes) • Extraction (area) Streaming data sync • Time-series data is useful for analysis • Insert to Hadoop cluster (DWH) Streaming data pick up items at shop Streaming ETL
  68. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Data collection Model verification Deployment Data ETL
  69. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Real time features (Redis) Offline features (Redis) Hadoop cluster
  70. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Real time features (Redis) Offline features (Redis) Hadoop cluster
  71. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Daily model

    update Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
  72. MLFlow allows us to check when we created models and

    deployed them Daily model update
  73. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Demae-can ML components requirement Monitoring Model training Model verification Deployment Data collection Data ETL
  74. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  75. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  76. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  77. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  78. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Monitoring Streaming

    ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  79. Create real time features & offline features with latest data

    Record all the data related with prediction Create ML models with latest data Summary Monitoring Model training Data collection Model verification Deployment Data ETL
  80. Continuous improvement of ML products

  81. Contents delivery time (prediction data) delivery time (actual data) •

    This graph shows delivery time (prediction data and actual data) on a certain day
  82. Contents delivery time (prediction data) delivery time (actual data) •

    There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This graph shows delivery time (prediction data and actual data) on a certain day
  83. Contents delivery time (prediction data) delivery time (actual data) •

    There was an issue occurring several times per month that caused prediction data to be much longer than actual data • This issue had a negative effect on drivers and users • We had to identify and handle this issue • This graph shows delivery time (prediction data and actual data) on a certain day
  84. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other
  85. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item pick up time (actual data) • Item pick up time (actual data) increased
  86. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased Item pick up time (actual data)
  87. The feedback loop occurred Item preparation time prediction Delivery offer

    • ML components (delivery offer & item preparation time prediction) affected each other Item preparation prediction time • Item pick up time (actual data) increased • Item preparation prediction time increased • This feedback loop occurred between ML components Item pick up time (actual data)
  88. Item preparation time prediction order arrive at shop items ready

    arrive at destination item preparation time (a) travel time from shop to destination (c) delivery time (d) shop arrival time (b) d = max(a, b) + c Prediction
  89. order arrive at shop arrive at destination delivery offer shop

    accept driver accept pick up items at shop Item preparation time prediction • Use mean item pick up time in the last n minutes as a real time feature • When mean item pick up time increases, the item preparation prediction time increases Item pick up time
  90. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Delivery offer • Delivery offer timing depends on item preparation time and travel time
  91. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time
  92. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Travel time is 5 min Item preparation time is 15 min Delivery offer Delivery offer timing delays • Delivery offer timing depends on item preparation time and travel time • Deliver offer component uses item preparation prediction time • When the item preparation prediction time increases, delivery offer time delays
  93. What occurred ? Item preparation time prediction Delivery offer

  94. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason Item pick up time (actual data)
  95. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
  96. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed Item preparation prediction time Item pick up time (actual data)
  97. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased Item preparation prediction time Item pick up time (actual data)
  98. What occurred ? Item preparation time prediction Delivery offer 1.

    Item pick up time (actual data) increased for some reason 2. Item preparation prediction time increased 3. Delivery offer timing delayed 4. Item pick up time (actual data) increased 5. This feedback loop occurred, especially item preparation prediction time increased Item preparation prediction time Item pick up time (actual data)
  99. How did we identify this problem ?

  100. ML components affected each other The difficulty to identify this

    problem
  101. ML components affected each other We had to get the

    clue of this feedback loop The difficulty to identify this problem
  102. ML components affected each other The clue was the delay

    of delivery offer timing We had to get the clue of this feedback loop The difficulty to identify this problem
  103. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server The data

    that we used to identify this problem Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  104. 1. Divide delivery prediction time The process to identify this

    problem Contents Delivery time prediction Actual delivery time
  105. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time The process to identify this problem Contents Delivery time prediction Actual delivery time
  106. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time 2. Check the model and features The process to identify this problem Contents Delivery time prediction Actual delivery time
  107. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Check the

    model and features Streaming ETL server Model training Prediction server ML real time system ML batch system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster
  108. Check the model and features Check the feature importance

  109. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the feature importance Mean item pick up time (real time feature)
  110. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  111. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  112. Check the model and features • Feature 12 is mean

    pick up time (real time feature / actual data) Check the mean pick up time (real time feature / actual data) • Compared to item preparation time (prediction data) and both values increased Check the feature importance Contents Item preparation time (prediction data) Contents Mean item pick up time (actual data) Mean item pick up time (real time feature)
  113. 1. Divide delivery prediction time & find the increase of

    item preparation prediction time 2. Check the model and features & find the increase of item pick up time 3. Divide item pick up time The process to identify this problem Contents Delivery time prediction Actual delivery time
  114. Data streaming (Kafka) Demae-can DB (RDB) Demae-can server Divide item

    pick up time Streaming ETL server Model training Prediction server ML real time system Feature loader ETL Demae-can server Model validation Model loader Real time features (Redis) Offline features (Redis) Hadoop cluster ML batch system
  115. order arrive at destination delivery offer shop accept driver accept

    arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop
  116. order arrive at destination delivery offer shop accept driver accept

    Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Order Order offer Shop accept
  117. order arrive at destination delivery offer shop accept driver accept

    Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Delivery offer Shop accept
  118. order arrive at destination delivery offer shop accept driver accept

    Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Delivery offer
  119. order arrive at destination delivery offer shop accept driver accept

    Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00 Driver User Merchant / Shop Driver accept Arrive at shop Pick up items at shop (driver)
  120. Contents Time from order to shop accept (actual data) Time

    from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
  121. Contents Time from order to shop accept (actual data) Time

    from shop accept to delivery offer (actual data) Time from delivery offer to driver accept (actual data) Time from driver accept to item pick up time(actual data) order arrive at destination delivery offer shop accept driver accept Time from driver accept to item pick up time (actual data) Time from delivery offer to driver accept (actual data) Time from shop accept to delivery offer (actual data) Time from order to shop accept (actual data) arrive at shop pick up items at shop Order id Geo id Status Time 1 3 order 14:00:00 1 3 shop accept 14:01:00 1 3 delivery offer 14:03:00 1 3 driver accept 14:05:00 1 3 arrive at shop 14:15:00 1 3 pick up items at shop 14:16:00 1 3 arrive at destination 14:30:00
  122. driver merchant order arrive at shop arrive at destination delivery

    offer shop accept driver accept pick up items at shop Pick up time is 5 min item preparation time is 15 min Delivery offer Wait for delivery offer • When the item preparation prediction time increases, delivery offer time delays
  123. This feedback loop occurred Item preparation time prediction Delivery offer

    Prepatation prediction time Item pick up time (actual data)
  124. How did we handle this problem ?

  125. order arrive at shop arrive at destination delivery offer shop

    accept driver accept pick up items at shop Item preparation time prediction • Decided not to use mean item pick up time in the last n minutes as a real time feature Item pick up time
  126. ML components are affected by many things that happened on

    Demae-can Summary
  127. ML components are affected by many things that happened on

    Demae-can Monitor & analyze the data and improve ML products Summary
  128. ML components are affected by many things that happened on

    Demae-can Have the perspective that overall system works well Monitor & analyze the data and improve ML components Summary
  129. Lastly

  130. What time is it ? Demae-can time

  131. Thank you