Upgrading the Food Delivery Service Using Machine Learning

Hongbo Zeng
LINE / Machine Learning Solution 1 Team / Machine Learning Engineer

https://linedevday.linecorp.com/2021/ja/sessions/82
https://linedevday.linecorp.com/2021/en/sessions/82
https://linedevday.linecorp.com/2021/ko/sessions/82

Transcript

2. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
3. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
4. What Demaecan is - Currently undergoing major system update - Food delivery service, launched in October, 2000. - Now with (as of August 2021) • merchants: 84 thousand • active users : 7.3 million • yearly GMV : 163 billion JPY
5. How Demaecan works ”Sharing delivery service” Demaecan user merchant merchant driver
6. How Demaecan works ”Sharing delivery service” Demaecan driver order user merchant
7. How Demaecan works ”Sharing delivery service” offer/assign orders order driver order user merchant Demaecan
8. How Demaecan works ”Sharing delivery service” offer/assign orders pick up order driver order user merchant deliver Demaecan
9. How Demaecan works ”Sharing delivery service” offer/assign orders pick up order driver order user merchant deliver Demaecan - Develop 5 machine learning components. - We will give their overview in the next section. - Delve into some of them in subsequent sections.
10. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
11. Recommendation offer/assign orders pick up order driver order user merchant deliver Demaecan - Many merchants, difficult to explore - Task : to provide users with personalized recommendation of merchants task definition
12. Recommendation challenge and approach - To balance accuracy with serendipity and diversity ⇒ Use multiple metrics (e.g. NDCG, entropy, …) - To reduce the bias of food genres within a recommendation list ⇒ Employ two-phase model
13. Delivery time prediction task definition - Want to know the time it takes for an order to arrive (for different stages). offer/assign orders pick up order driver order user merchant deliver Demaecan
14. Delivery time prediction task definition order arrive at shop delivery complete food ready time delivery time
15. Delivery time prediction task definition order arrive at shop delivery complete food ready preparation time time delivery time
16. Delivery time prediction task definition order arrive at shop delivery complete food ready preparation time shop arrival time time delivery time
17. Delivery time prediction task definition order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
18. Delivery time prediction task definition - delivery time = max(shop arrival time, preparation time) + travel time from shop to destination - Machine learning task : to predict (1) shop arrival time and (2) preparation time order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time shop arrival time
19. Delivery time prediction - The time largely depends on current situation, e.g. how busy the shop is, the number of active drivers, etc. ⇒ Use online (real time) feature to make the model aware of current situation. - No ground truth data available(preparation time) ⇒ subsequent section challenge and approach
20. Order forecast task definition offer/assign orders pick up order driver user merchant deliver Demaecan - Want to wait orders at places with high demand for drivers - Task : to predict the number of orders for different areas and times order
21. Order forecast - Data sparsity (many areas) ⇒ Train a single model on data from all the areas - Affected by sudden events (e.g. sudden rainfall) ⇒ Utilize online (real time) feature challenge and approach (dummy data) Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL.
22. Order dispatch offer/assign orders pick up order driver user merchant deliver Demaecan - Task : to give order-driver allocation, which is globally efficient task definition order
23. Order dispatch - Optimal assignment problem that allows multiple assignment Challenge challenge and approach
24. Order dispatch - Optimal assignment problem that allows multiple assignment - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach
25. Order dispatch - Optimal assignment problem that allows multiple assignment - Preparation time at shop matters Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
26. Order dispatch - Optimal assignment problem that allows multiple assignment - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned.
27. Order dispatch - Optimal assignment problem that allows multiple assignment - Preparation time at shop matters - Near real time Challenge %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order challenge and approach driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. Approach - Simulator with actual store, user, order, driver information
28. Order dispatch - Optimal assignment problem that allows multiple assignment - Preparation time at shop matters - Near real time Challenge - Simulator with actual store, user, order, driver information - Utilization of reinforcement learning Approach %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A Order B Order C order driver assignment %SJWFS  4IPQ " 6TFS " 6TFS # 4IPQ # 4IPQ $ %SJWFS  6TFS $ Order A and B are assigned. Order C is assigned. challenge and approach
29. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order Overview of ML components
30. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order recommendation Overview of ML components
31. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order preparation time prediction shop arrival time prediction recommendation Overview of ML components
32. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
33. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components order dispatch
34. Summary offer/assign orders pick up order driver user merchant deliver Demaecan order order dispatch preparation time prediction shop arrival time prediction order forecast recommendation Overview of ML components
35. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
36. Task overview - Task : providing personalized recommendation of shops offer/assign orders pick up order driver order user merchant deliver Demaecan
37. - Shop overall diversity • Fairness between merchants • Avoid overlap with ranking - Genre individual diversity • Show shops of diverse genre within a recommendation list • Prevent users from being bored. The difficulties of user recommendation in food delivery service - NOTE : trade-off between accuracy and diversity
38. The approach to shop overall diversity - Ground truth sampling for evaluation • Sampling weight : inversely proportional to popularity • Reduces the popularity bias in evaluation - Positive/Negative sampling in the training phase • Tune trade-off between accuracy and diversity • Do hyper-parameter search with optuna (https://optuna.org) c.f. masala in LINE Developer Day 2020 ( https://linedevday.linecorp.com/2020/en/sessions/9641 )
39. Evaluating shop overall diversity - Metrics : nDCG(accuracy) and entropy (shop overall diversity) - Blue dots are good candidates for online test
40. The approach to genre individual diversity - Embedding model : accurate, but low genre diversity (e.g.) GNN - Co-occurrence recommendation model : high genre diversity, but less accurate (e.g.) PMI model Apply Two-stage model 1st stage : PMI model 2nd stage : GNN
41. Evaluating genre individual diversity - Apply Intra List Similarity as the indicator of genre individual diversity Model nDCG@20 Entropy@20 Intra List Similarity@20 GNN 0.017 14.92 15.58 PMI 0.015 14.16 14.16 Two-stage (PMI and GNN) 0.019 16.15 14.17
42. Summary recommendation - Task : providing personalized recommendation of shops - Challenge 1: shop overall diversity ⇒ weighted sampling for reducing popularity bias - Challenge 2: genre individual diversity ⇒ two-stage model
43. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
44. Task overview order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time Task definition shop arrival time
45. Task overview order arrive at shop delivery complete food ready preparation time travel time from shop to destination time delivery time - Task : Predict how long it will take from order to finish preparing the food - preparation time includes time it takes for the shop to confirm the order, cook the food, pack the food, etc. Task definition shop arrival time
46. Task overview Challenge and approach - There is no exact time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
47. Task overview Challenge and approach - There is no exact time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
48. offer/assign orders pick up driver order user merchant delivery Demaecan order No precise ground truth problem Problem description
49. offer/assign orders pick up driver order user merchant delivery Demaecan order No precise ground truth problem Problem description - No data on when the preparation is finished - Available timestamps: - order_tm : when the order is placed - pickup_tm : when the driver received the items at the merchant - preparation time ≒ pickup_tm - order_tm
50. No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets
51. user Demaecan merchant No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order
52. user Demaecan merchant No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
53. user Demaecan merchant No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
54. user Demaecan merchant No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
55. user Demaecan merchant No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets offer order pick up delivery
56. pick up user Demaecan merchant delivery offer order No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets
57. pick up user Demaecan merchant delivery offer order No precise ground truth problem Data cleansing - Sometimes drivers do not tap buttons at proper time. → model learns wrong targets - Filter those orders based on the driver’s GPS and shop location
58. Reliable Driver waited at shop → preparation was done at pickup tm Unreliable Driver did not wait at shop → preparation may be done earlier No precise ground truth problem Distinguishing pickup time logs arrive at shop and pick up food ready time
59. No precise ground truth problem Custom objective function : time of pickup : minimum preparation time : predicted value
60. Task overview Challenge and approach - There is no exact time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
61. count of shops grouped by # of orders # of order heatmap(dummy data) Data sparsity Problem description Modified from Map tiles by Stamen Design, under CC BY 3.0. Data by © OpenStreetMap, under ODbL. Shinjuku Ikebukuro Shibuya
62. Why LightGBM? tree-based It treat bias of area/shop well if they are in the features light-weight What’s LightGBM? An implementation of gradient boosting decision tree compared with deep learning model works well on relatively small data Data sparsity Using LightGBM
63. Ordinary Target Encoding Smoothed Target Encoding Data sparsity Smoothed target encoding
64. pre-trained Use pre-trained efficient-net to extract embedding … … … + LightGBM Data sparsity Item image feature
65. Task overview Challenge and approach - There is no exact time when the preparation was completed. - There is data sparsity depending on the shop, area - The time is unstable due to the (offline) crowding level of the shop, the proficiency of the shop staff, etc…
66. Online (real-time) features - target encoding in last n minutes - target encoding for orders that was received in the last n minutes - aggregate for 3 categories: global, shop, area - driver to order ratio - ratio of (number of drivers) / (number of orders) in last n minutes To indirectly obtain real-time information, following two real-time features were created
67. lower is better Online (real-time) features
68. Summary Preparation time prediction - No exact time when the preparation was finished ⇒ cleansing using GPS data, custom loss function - Data sparsity depending on the shop, area ⇒ boosting model, smoothed target encoding, image embedding feature - Preparation time strongly depends on real-time situations ⇒ real-time feature order food ready time
69. Agenda - Background - Overview of Demaecan ML components - Case study 1 : recommendation - Case study 2 : preparation time prediction - Conclusion and future works
70. Conclusion and future works Subtitle - We are tackling diverse challenges with various techniques, including meticulous feature engineering, online processing … etc. - Future work : improving accuracy, performance monitoring, developing other ML components ... etc - Demaecan is now undergoing a major system update. We are trying to utilize machine learning to improve the user experience for everyone who uses the service.