Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Upgrading the Food Delivery Service Using Machine Learning

Upgrading the Food Delivery Service Using Machine Learning

LINE DEVDAY 2021

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  2. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  3. What Demaecan is - Currently undergoing major system update -

    Food delivery service, launched in October, 2000. - Now with (as of August 2021) • merchants: 84 thousand • active users : 7.3 million • yearly GMV : 163 billion JPY
  4. How Demaecan works ”Sharing delivery service” offer/assign orders pick up

    order driver order user merchant deliver Demaecan
  5. How Demaecan works ”Sharing delivery service” offer/assign orders pick up

    order driver order user merchant deliver Demaecan - Develop 5 machine learning components. - We will give their overview in the next section. - Delve into some of them in subsequent sections.
  6. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  7. Recommendation offer/assign orders pick up order driver order user merchant

    deliver Demaecan - Many merchants, difficult to explore - Task : to provide users with personalized recommendation of merchants task definition
  8. Recommendation challenge and approach - To balance accuracy with serendipity

    and diversity ⇒ Use multiple metrics (e.g. NDCG, entropy, …) - To reduce the bias of food genres within a recommendation list ⇒ Employ two-phase model
  9. Delivery time prediction task definition - Want to know the

    time it takes for an order to arrive (for different stages). offer/assign orders pick up order driver order user merchant deliver Demaecan
  10. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time time delivery time
  11. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time shop arrival time time delivery time
  12. Delivery time prediction task definition order arrive at shop delivery

    complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
  13. Delivery time prediction task definition - delivery time = max(shop

    arrival time, preparation time) + travel time from shop to destination - Machine learning task : to predict (1) shop arrival time and (2) preparation time order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
  14. Delivery time prediction - The time largely depends on current

    situation, e.g. how busy the shop is, the number of active drivers, etc. ⇒ Use online (real time) feature to make the model aware of current situation. - No ground truth data available(preparation time) ⇒ subsequent section challenge and approach
  15. Order forecast task definition offer/assign orders pick up order driver

    user merchant deliver Demaecan - Want to wait orders at places with high demand for drivers - Task : to predict the number of orders for different areas and times order
  16. Order forecast - Data sparsity (many areas) ⇒ Train a

    single model on data from all the areas - Affected by sudden events (e.g. sudden rainfall) ⇒ Utilize online (real time) feature challenge and approach (dummy data) Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL.
  17. Order dispatch offer/assign orders pick up order driver user merchant

    deliver Demaecan - Task : to give order-driver allocation, which is globally efficient task definition order
  18. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach
  19. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
  20. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
  21. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. Approach - Simulator with actual store, user, order, driver information
  22. Order dispatch - Optimal assignment problem that allows multiple assignment

    - Preparation time at shop matters - Near real time Challenge - Simulator with actual store, user, order, driver information - Utilization of reinforcement learning Approach %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. challenge and approach
  23. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order recommendation Overview of ML components
  24. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction recommendation Overview of ML components
  25. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
  26. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components order dispatch
  27. Summary offer/assign orders pick up order driver user merchant deliver

    Demaecan order order dispatch preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
  28. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  29. Task overview - Task : providing personalized recommendation of shops

    offer/assign orders pick up order driver order user merchant deliver Demaecan
  30. - Shop overall diversity • Fairness between merchants • Avoid

    overlap with ranking - Genre individual diversity • Show shops of diverse genre within a recommendation list • Prevent users from being bored. The difficulties of user recommendation in food delivery service - NOTE : trade-off between accuracy and diversity
  31. The approach to shop overall diversity - Ground truth sampling

    for evaluation • Sampling weight : inversely proportional to popularity • Reduces the popularity bias in evaluation - Positive/Negative sampling in the training phase • Tune trade-off between accuracy and diversity • Do hyper-parameter search with optuna (https://optuna.org) c.f. masala in LINE Developer Day 2020 ( https://linedevday.linecorp.com/2020/en/sessions/9641 )
  32. Evaluating shop overall diversity - Metrics : nDCG(accuracy) and entropy

    (shop overall diversity) - Blue dots are good candidates for online test
  33. The approach to genre individual diversity - Embedding model :

    accurate, but low genre diversity (e.g.) GNN - Co-occurrence recommendation model : high genre diversity, but less accurate (e.g.) PMI model Apply Two-stage model 1st stage : PMI model 2nd stage : GNN
  34. Evaluating genre individual diversity - Apply Intra List Similarity as

    the indicator of genre individual diversity Model nDCG@20 Entropy@20 Intra List Similarity@20 GNN 0.017 14.92 15.58 PMI 0.015 14.16 14.16 Two-stage (PMI and GNN) 0.019 16.15 14.17
  35. Summary recommendation - Task : providing personalized recommendation of shops

    - Challenge 1: shop overall diversity ⇒ weighted sampling for reducing popularity bias - Challenge 2: genre individual diversity ⇒ two-stage model
  36. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  37. Task overview order arrive at shop delivery complete food ready

    preparation time travel time from shop to destination time delivery time Task definition shop arrival time
  38. Task overview order arrive at shop delivery complete food ready

    preparation time travel time from shop to destination time delivery time - Task : Predict how long it will take from order to finish preparing the food - preparation time includes time it takes for the shop to confirm the order, cook the food, pack the food, etc. Task definition shop arrival time
  39. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  40. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  41. offer/assign orders pick up driver order user merchant delivery Demaecan

    order No precise ground truth problem Problem description
  42. offer/assign orders pick up driver order user merchant delivery Demaecan

    order No precise ground truth problem Problem description - No data on when the preparation is finished - Available timestamps: - order_tm : when the order is placed - pickup_tm : when the driver received the items at the merchant - preparation time ≒ pickup_tm - order_tm
  43. No precise ground truth problem Data cleansing - Sometimes drivers

    do not tap buttons at proper time. → model learns wrong targets
  44. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order
  45. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  46. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  47. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  48. user Demaecan merchant No precise ground truth problem Data cleansing

    - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
  49. pick up user Demaecan merchant delivery offer order No precise

    ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets
  50. pick up user Demaecan merchant delivery offer order No precise

    ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets - Filter those orders based on the driver’s GPS and shop location
  51. Reliable Driver waited at shop → preparation was done at

    pickup tm Unreliable Driver did not wait at shop → preparation may be done earlier No precise ground truth problem Distinguishing pickup time logs arrive at shop and pick up food ready time
  52. No precise ground truth problem Custom objective function : time

    of pickup : minimum preparation time : predicted value
  53. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  54. count of shops grouped by # of orders # of

    order heatmap(dummy data) Data sparsity Problem description Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL. Shinjuku Ikebukuro Shibuya
  55. Why LightGBM? tree-based It treat bias of area/shop well if

    they are in the features light-weight What’s LightGBM? An implementation of gradient boosting decision tree compared with deep learning model works well on relatively small data Data sparsity Using LightGBM
  56. Task overview Challenge and approach - There is no exact

    time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
  57. Online (real-time) features - target encoding in last n minutes

    - target encoding for orders that was received in the last n minutes - aggregate for 3 categories: global, shop, area - driver to order ratio - ratio of (number of drivers) / (number of orders) in last n minutes To indirectly obtain real-time information, following two real-time features were created
  58. Summary Preparation time prediction - No exact time when the

    preparation was finished ⇒ cleansing using GPS data, custom loss function - Data sparsity depending on the shop, area ⇒ boosting model, smoothed target encoding, image embedding feature - Preparation time strongly depends on real-time situations ⇒ real-time feature order food ready time
  59. Agenda - Background - Overview of Demaecan ML components -

    Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
  60. Conclusion and future works Subtitle - We are tackling diverse

    challenges with various techniques, including meticulous feature engineering, online processing … etc. - Future work : improving accuracy, performance monitoring, developing other ML components ... etc - Demaecan is now undergoing a major system update. We are trying to utilize machine learning to improve the user experience for everyone who uses the service.