Modern gold mining: Leveraging Deep Learning to predict GMV for 100k+ shops

Shopify houses more than 600,000 merchants all around the world. Shopify Capital aims to provide merchants with accessible cash to help them grow their businesses in a timely manner. With our high volume of data we are able to provide better funding options for our merchants by looking into many features to predict how well they will perform, thus providing them with affordable and coherent cash advances.

In this talk we will present the challenges faced along the way with technical information about how we explore and fine tune our models: from a fully isolated gradient boosting tree model to an attentional decoupled deep learning model sending predictions over Kafka. We also explore how we model and organize the our scalable ML pipelines at Shopify.

Transcript

1. Modern Gold Mining How we use Deep Learning to predict sales of 100k+ shops
2. Hi, I’m Breno! • Data Scientist @Shopify • M.Sc. in ML • Web Dev for many years • Beanie collector (15 so far) • @brenolf_ • @brenolf • //breno.io
4. Currency helped scaling human production.
7. Tech shaped the way we interact with products.
9. 175 800k+ Shopify in numbers countries merchants $1.5B over 2018 BFCM $100B total sales
10. Starting a dream is not always easy.
11. Shopify Capital provides merchants timely access to affordable funding to help them grow.
12. What’s the problem we’re solving?
13. US$0 US$2,250 US$4,500 US$6,750 US$9,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales Monthly Sales Offer
14. O ct N ov D ec Jan This is an offer
15. - Google’s Rules of Machine Learning “Rule #1: Don’t be afraid to launch a product without machine learning.”
18. US$0 US$2,250 US$4,500 US$6,750 US$9,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales
19. How do we solve it linearly, again?
20. How do we solve it linearly, again? y ∼ mx + b
21. How do we solve it linearly, again? y ∼ mx + b E(y | X) = mx + b
22. How do we solve it linearly, again? y ∼ mx + b E(y | X) = mx + b RSS = n ∑ i= 1 (y i − (mx i + b))2
23. US$0 US$4,000 US$8,000 US$12,000 US$16,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales
24. US$0 US$4,000 US$8,000 US$12,000 US$16,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales }
25. US$0 US$75,000 US$150,000 US$225,000 US$300,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales
26. US$0 US$75,000 US$150,000 US$225,000 US$300,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales
27. We need confidence levels
28. We need confidence levels y ∼ mx + b
29. We need confidence levels y ∼ mx + b Q (y | X) (τ) = in f{y : F (y | X) (y) ≥ τ}
30. We need confidence levels y ∼ mx + b Q (y | X) (τ) = in f{y : F (y | X) (y) ≥ τ} QL = n ∑ i= 1 (y i − (mx i + b)) ⋅ (τ − I (yi −(mxi + b)< 0) )
31. We need confidence levels y ∼ mx + b Q (y | X) (τ) = in f{y : F (y | X) (y) ≥ τ} QL = n ∑ i= 1 (y i − (mx i + b)) ⋅ (τ − I (yi −(mxi + b)< 0) )
32. US$0 US$77,500 US$155,000 US$232,500 US$310,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales 10% 90%
33. Inference + Business Logic = Capital
34. How did we implement it?
36. Record-Based Data Databases
37. Record-Based Data Databases Data Acquisition
38. Record-Based Data Data Integration Databases Starscream Data Acquisition
39. Record-Based Data Data Integration Databases Starscream Data Acquisition
40. Record-Based Data Data Integration Delivery Databases Starscream Data Acquisition
41. Record-Based Data Data Integration Delivery Analytics Databases Starscream Data Acquisition
42. Feature Base Slice Feature Slice #1 Feature Slice #2 Features Set Targets Test/Train Split
43. Feature Base Slice Feature Slice #1 Feature Slice #2 Features Set Targets Test/Train Split Isolating shops’ histories
44. Model Definition Training Backtesting Inference Metrics Feature Base Slice Feature Slice #1 Feature Slice #2 Features Set Targets Test/Train Split Isolating shops’ histories
45. First Iteration • Implemented in R • Quantile Random Forest • List of Offers manually generated and sent to Bourgeois
46. First Iteration • Implemented in R • Quantile Random Forest • List of Offers manually generated and sent to Bourgeois
47. Moving into Starscream
48. Moving into Starscream
49. Moving into Starscream HTTP
50. Moving into Starscream HTTP
51. Moving into Starscream HTTP
52. Moving into Starscream
53. Moving into Starscream
54. How intricate was it? • 21 Models per confidence level • Pickled individually • 2000 trees, 8 levels deep • 42 containers with 2 cores • 19Gb Python RAM + 8Gb Java • +50Gb driver heap • Feature set has ~440M rows • using ~35 features
55. Into the Neural Nets
56. US$0 US$2,250 US$4,500 US$6,750 US$9,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales Monthly Sales Offer
57. US$0 US$2,250 US$4,500 US$6,750 US$9,000 Apr M ay Jun Jul Aug Sep O ct N ov D ec Jan Cumulative Sales Monthly Sales Offer
58. How did we move from Quantile Forest? • Set product goals to achieve • Can we reduce the fast payers? • Can we make better offers? • XGBoost analysis • Started to play with monthly prediction with simpler neural net • All in Jupyter Notebooks • Moved into a different structure for weekly prediction
59. Setup for the problem • We want to get the fluctuations • We have a series of sequential features • GMV, orders, support data, etc. • We have static features • Cumulatives, admin information, etc. • We want to predict their cumulative GMV in the future Sounds a lot like sequence to sequence
60. - Deep Learning, Goodfellow et. al., 2016 “Much as a convolutional network […] is specialized for processing […] an image, a recurrent network […] is specialized for processing a sequence of values […].”
61. Feature Base Slice Feature Slice #1 Feature Slice #2 Features Set Targets Test/Train Split Train tfrecords Test tfrecords
62. Model Definition Training Backtesting Inference Metrics Feature Base Slice Feature Slice #1 Feature Slice #2 Features Set Targets Test/Train Split Train tfrecords Test tfrecords
63. Features 992.1 46.2 … 45 67.1 125.5 … 90.1 … N0 NT … Numeric “abc” “edf” … “lak” “oqj” “xyz” … “uyw” … E0 ET … 0.5 -0.5 … 1.2 -1 3.4 … 5 … 1.3 4.5 … -5 5.2 1.2 … 1 … ⊕ … ⊕ Non-Numeric
64. Static Features N0 NT … 992.1 46.2 … 45 67.1 125.5 … 90.1 … Numeric E0 ET … “abc” “edf” … “lak” “oqj” “xyz” … “uyw” … 0.5 -0.5 … 1.2 -1 3.4 … 5 … 1.3 4.5 … -5 5.2 1.2 … 1 … ⊕ … ⊕ Non-Numeric
65. Static Features N0 NT … Numeric E0 ET … Non-Numeric
66. N0 NT … Numeric E0 ET … Non-Numeric Static Features Dense S
67. N0 NT … E0 ET … Dimensionality Reduction Convolution N0 N(T/q) … Reshape E0 E(T-q) … Eq ET ⊕ ⊕
69. Encoder N0 NT … E0 ET ⊕ ⊕
70. Encoder N0 NT … E0 ET ⊕ ⊕ X0 XT …
71. Encoder N0 NT … E0 ET ⊕ ⊕ GRU GRU X0 XT … … … GRU GRU
72. Encoder N0 NT … E0 ET ⊕ ⊕ GRU GRU X0 XT … … … GRU GRU H0 HT
73. Encoder N0 NT … E0 ET ⊕ ⊕ GRU GRU X0 XT … … … GRU GRU H0 HT S Dense C
74. Decoder XT C
75. Decoder GRU XT C
76. Decoder GRU Dense O1 XT C
77. Decoder GRU Dense O1 XT C O2T-1 … … GRU OT Dense
78. Decoder GRU … GRU
79. Decoder GRU … GRU C H’1 H’T …
80. Decoder GRU … GRU C Dense Dense P1 PT H’1 H’T …
81. Decoder GRU1 GRU2
82. Decoder GRU1 GRU2 H0 H1 H2 H3 HT … Memory * The same effect could also be achieved by using a sufficiently large RNN trained for long enough — Cho et al. (2014), Sutskever et. Al (2014)
83. November January February March October
84. November January February March October
85. November January February March October
86. November January February March October
87. November January February March October
88. November January February March October 30 metrics!
89. How intricate is it now? • 1 model trained x 4 confidence levels • 2 Layers for Encoder/Decoder • 256 units • 40% dropout • Trained on 8 NVIDIA Tesla K80 • Feature set has ~440M rows • using ~35 features
90. +15% +17% How much better was it? Duration Offer Amounts -30% Fast Payers ~0% Loss Rate
91. Thanks! @brenolf_ //breno.io bit.ly/qcon-shopify