Mercari-1st-place-solution

https://mercari.connpass.com/event/84299/

Transcript

1. Mercari 1st place solution Konstantin Lopuhin & Pawel Jankiewicz 2018-05-09
2. About us
3. Summary I 3 di↵erent datasets I 4 models per dataset I Sparse feed-forward neural network I Data processing: diversity, merging text fields, custom vectorizers I Libraries: scikit-learn, Tensorflow, MXNet
4. Data preprocessing: Declarative vs Imperative Imperative D vect = CountVectorizer() A vect.fit(X) A mat = vect.transform(X) D rf = RandomForestRegressor() A rf.fit(mat, y) Declarative D model = make_pipeline( D CountVectorizer(), D RandomForestRegressor() D ) A model.fit(X, y) D = declaration, A = action
5. ”It’s pipelines all the way down”
6. Preprocessing I Text preprocessing - stemming I Bag of words - 1,2-grams (with/without Tf-Idf) I One hot encoding for categorical columns I Bag of character 3-grams I Joining name, brand name and description into a single field I NumericalVectorizer - vectorizing words using preceding numbers
7. Why ensemble?
8. Why 3 datasets?
9. Our progress
10. Workhorse model: sparse MLP (feedforward neural network)
11. Why MLP? I Fast to train: can a↵ord hidden size 256 instead of 32–64 for RNN or Conv1D. I Captures interactions between text and categorical features. I Huge variance gives a strong ensemble with a single model type.
12. Training Adam, double batch size after each epoch, overfit
13. Training Adam, double batch size after each epoch, overfit, profit!
14. Tricks I Huber loss I Regression via. classification I Cheap feature binarization
15. Huber Loss
16. Regression via Classification
17. Cheap feature binarization TF-IDF features ) Binary features
18. Sparse MLP Implementation I TensorFlow: tf.sparse tensor dense matmul I MXNet: RowSparseNDArray , sparse updates! I Keras: keras.Input(sparse=True) I Any framework: via embedding
19. Optimization: One Model per Core
20. Optimization: Memory I TensorFlow: threading, use per session threads I MXNet: multiprocessing, memory e cient data loader
21. Ensembling via Lasso 5% local validation, 1% on Kaggle. Very good LB correlation.
22. Didn’t Work I Grid Search I Skip Connections I Mixture of Experts I Factorization Machines I Fitting residuals
23. Code Golf: 0.3875 CV in 75 LOC, 1900 s I Sparse MLP in Keras I Train 4 models on 4 cores I Custom preprocessing
27. Feature Engineering
28. The Model
29. Main di↵erences of our approach I One model kind, 3 datasets I Train 12 models I Sparse MLP model I Early merge: almost all good ideas created after merging https://github.com/pjankiewicz/mercari-solution
30. Questions?
31. First Layer Hidden Size Hidden size Score (delta) 128 0.3757 (+0.0024) 256 0.3733 (+0.0000) 384 0.3728 ( 0.0005)
32. Binariezed Features, Classification Setup Score (delta) default 0.3733 (+0.0000) no binary 0.3740 (+0.0007) no clf 0.3742 (+0.0009) no both 0.3748 (+0.0015)