$30 off During Our Annual Pro Sale. View Details »

Upgrading the Food Delivery Service Using Machine Learning

Upgrading the Food Delivery Service Using Machine Learning

LINE DEVDAY 2021
PRO

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. None
  2. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  3. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  4. What Demaecan is - Currently undergoing major system update -

    Food delivery service, launched in October, 2000. - Now with (as of August 2021) • merchants: 84 thousand • active users : 7.3 million • yearly GMV : 163 billion JPY
  5. How Demaecan works ”Sharing delivery service” Demaecan user merchant merchant

    driver
  6. How Demaecan works ”Sharing delivery service” Demaecan driver order user

    merchant
  7. How Demaecan works ”Sharing delivery service” offer/assign orders order driver

    order user merchant Demaecan
  8. How Demaecan works ”Sharing delivery service” offer/assign orders pick up

    order driver order user merchant deliver Demaecan
  9. How Demaecan works ”Sharing delivery service” offer/assign orders pick up

    order driver order user merchant deliver Demaecan - Develop 5 machine learning components. - We will give their overview in the next section. - Delve into some of them in subsequent sections.
  10. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  11. Recommendation offer/assign orders pick up order driver order user merchant

    deliver Demaecan - Many merchants, difficult to explore - Task : to provide users with personalized recommendation of merchants task definition
  12. Recommendation challenge and approach - To balance accuracy with serendipity

    and diversity ⇒ Use multiple metrics (e.g. NDCG, entropy, …) - To reduce the bias of food genres within a recommendation list ⇒ Employ two-phase model
  13. Delivery time prediction task definition - Want to know the

    time it takes for an order to arrive (for different stages). offer/assign orders pick up order driver order user merchant deliver Demaecan
  14. Delivery time prediction task definition order arrive at shop delivery

    complete food ready time delivery time
  15. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time time delivery time
  16. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time shop arrival time time delivery time
  17. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
  18. Delivery time prediction task definition - delivery time = max(shop

    arrival time, preparation time) + travel time from shop to destination - Machine learning task : to predict (1) shop arrival time and (2) preparation time order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
  19. Delivery time prediction - The time largely depends on current

    situation, e.g. how busy the shop is, the number of active drivers, etc. ⇒ Use online (real time) feature to make the model aware of current situation. - No ground truth data available(preparation time) ⇒ subsequent section challenge and approach
  20. Order forecast task definition offer/assign orders pick up order driver

    user merchant deliver Demaecan - Want to wait orders at places with high demand for drivers - Task : to predict the number of orders for different areas and times order
  21. Order forecast - Data sparsity (many areas) ⇒ Train a

    single model on data from all the areas - Affected by sudden events (e.g. sudden rainfall) ⇒ Utilize online (real time) feature challenge and approach (dummy data) Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL.
  22. Order dispatch offer/assign orders pick up order driver user merchant

    deliver Demaecan - Task : to give order-driver allocation, which is globally efficient task definition order
  23. Order dispatch - Optimal assignment problem that allows multiple assignment

    Challenge challenge and approach
  24. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach
  25. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
  26. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
  27. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. Approach - Simulator with actual store, user, order, driver information
  28. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge - Simulator with actual store, user, order, driver information - Utilization of reinforcement learning Approach %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. challenge and approach
  29. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order Overview of ML components
  30. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order recommendation Overview of ML components
  31. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction recommendation Overview of ML components
  32. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
  33. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components order dispatch
  34. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order order dispatch preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
  35. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  36. Task overview - Task : providing personalized recommendation of shops

    offer/assign orders pick up order driver order user merchant deliver Demaecan
  37. - Shop overall diversity • Fairness between merchants • Avoid

    overlap with ranking - Genre individual diversity • Show shops of diverse genre within a recommendation list • Prevent users from being bored. The difficulties of user recommendation in food delivery service - NOTE : trade-off between accuracy and diversity
  38. The approach to shop overall diversity - Ground truth sampling

    for evaluation • Sampling weight : inversely proportional to popularity • Reduces the popularity bias in evaluation - Positive/Negative sampling in the training phase • Tune trade-off between accuracy and diversity • Do hyper-parameter search with optuna (https://optuna.org) c.f. masala in LINE Developer Day 2020 ( https://linedevday.linecorp.com/2020/en/sessions/9641 )
  39. Evaluating shop overall diversity - Metrics : nDCG(accuracy) and entropy

    (shop overall diversity) - Blue dots are good candidates for online test
  40. The approach to genre individual diversity - Embedding model :

    accurate, but low genre diversity (e.g.) GNN - Co-occurrence recommendation model : high genre diversity, but less accurate (e.g.) PMI model Apply Two-stage model 1st stage : PMI model 2nd stage : GNN
  41. Evaluating genre individual diversity - Apply Intra List Similarity as

    the indicator of genre individual diversity Model nDCG@20 Entropy@20 Intra List Similarity@20 GNN 0.017 14.92 15.58 PMI 0.015 14.16 14.16 Two-stage (PMI and GNN) 0.019 16.15 14.17
  42. Summary recommendation - Task : providing personalized recommendation of shops

    - Challenge 1: shop overall diversity ⇒ weighted sampling for reducing popularity bias - Challenge 2: genre individual diversity ⇒ two-stage model
  43. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  44. Task overview order arrive at shop delivery complete food ready

    preparation time travel time from shop to destination time delivery time Task definition shop arrival time
  45. Task overview order arrive at shop delivery complete food ready

    preparation time travel time from shop to destination time delivery time - Task : Predict how long it will take from order to finish preparing the food - preparation time includes time it takes for the shop to confirm the order, cook the food, pack the food, etc. Task definition shop arrival time
  46. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  47. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  48. offer/assign orders pick up driver order user merchant delivery Demaecan

    order No precise ground truth problem Problem description
  49. offer/assign orders pick up driver order user merchant delivery Demaecan

    order No precise ground truth problem Problem description - No data on when the preparation is finished - Available timestamps: - order_tm : when the order is placed - pickup_tm : when the driver received the items at the merchant - preparation time ≒ pickup_tm - order_tm
  50. No precise ground truth problem Data cleansing - Sometimes drivers

    do not tap buttons at proper time. → model learns wrong targets
  51. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order
  52. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  53. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  54. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  55. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  56. pick up user Demaecan merchant delivery offer order No precise

    ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets
  57. pick up user Demaecan merchant delivery offer order No precise

    ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets - Filter those orders based on the driver’s GPS and shop location
  58. Reliable Driver waited at shop → preparation was done at

    pickup tm Unreliable Driver did not wait at shop → preparation may be done earlier No precise ground truth problem Distinguishing pickup time logs arrive at shop and pick up food ready time
  59. No precise ground truth problem Custom objective function : time

    of pickup : minimum preparation time : predicted value
  60. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  61. count of shops grouped by # of orders # of

    order heatmap(dummy data) Data sparsity Problem description Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL. Shinjuku Ikebukuro Shibuya
  62. Why LightGBM? tree-based It treat bias of area/shop well if

    they are in the features light-weight What’s LightGBM? An implementation of gradient boosting decision tree compared with deep learning model works well on relatively small data Data sparsity Using LightGBM
  63. Ordinary Target Encoding Smoothed Target Encoding Data sparsity Smoothed target

    encoding
  64. pre-trained Use pre-trained efficient-net to extract embedding … … …

    + LightGBM Data sparsity Item image feature
  65. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  66. Online (real-time) features - target encoding in last n minutes

    - target encoding for orders that was received in the last n minutes - aggregate for 3 categories: global, shop, area - driver to order ratio - ratio of (number of drivers) / (number of orders) in last n minutes To indirectly obtain real-time information, following two real-time features were created
  67. lower is better Online (real-time) features

  68. Summary Preparation time prediction - No exact time when the

    preparation was finished ⇒ cleansing using GPS data, custom loss function - Data sparsity depending on the shop, area ⇒ boosting model, smoothed target encoding, image embedding feature - Preparation time strongly depends on real-time situations ⇒ real-time feature order food ready time
  69. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  70. Conclusion and future works Subtitle - We are tackling diverse

    challenges with various techniques, including meticulous feature engineering, online processing … etc. - Future work : improving accuracy, performance monitoring, developing other ML components ... etc - Demaecan is now undergoing a major system update. We are trying to utilize machine learning to improve the user experience for everyone who uses the service.