Reproducibility and Selection Bias in Learning: when just cross validation is not enough

Reproducibility - the ability to recompute results — and replicability— the chances other experimenters will achieve a consistent result[1]- are among the main important beliefs of the scientific method.

Surprisingly, these two aspects are often underestimated or not even considered when setting up scientific experimental pipelines. In this, one of the main threat to replicability is the selection bias, that is the error in choosing the individuals or groups to take part in a study. Selection bias may come in different flavours: the selection of the population of samples in the dataset (sample bias); the selection of features used by the learning models, particularly sensible in case of high dimensionality; the selection of hyper parameter best performing on specific dataset(s). If not properly considered, the selection bias may strongly affect the validity of derived conclusions, as well as the reliability of the learning model.

In this talk I will provide a solid introduction to the topics of reproducibility and selection bias, with examples taken from the biomedical research, in which reliability is paramount.

From a more technological perspective, to date the scientific Python ecosystem still misses tools to consolidate the experimental pipelines in in research, that can be used together with Machine and Deep learning frameworks (e.g. sklearn and keras). In this talk, I will present reproducible-learn, a new Python frameworks for reproducible research to be used for machine and deep learning.

During the talk, the main features of the framework will be presented, along with several examples, technical insights and implementation choices to be discussed with the audience.

The talk is intended for intermediate PyData researchers and practitioners. Basic prior knowledge of the main Machine Learning concepts is assumed for the first part of the talk. On the other hand, good proficiency with the Python language and with scientific python libraries (e.g. numpy, sklearn) are required for the second part.

– 1 Reproducible research can still be wrong: Adopting a prevention approach by Jeffrey T. Leek, and Roger D. Peng

2 Dictionary of Cancer Terms -> “selection bias”

Transcript

1. Reproducibility and 
 Selection Bias 
 in Machine Learning When just Cross Validation is not enough! Valerio Maggio @leriomaggio
2. Reproducibility Selection Bias
3. Reproducibility Selection Bias
4. Reproducibility the ability to recompute results and 
 replicability (i.e. the chances other experimenters will achieve a consistent results) 
 are among the main important beliefs of the 
 scientific method
5. 1. Recompute
6. Tools
7. The scientific 
 paper is obsolete
8. 2. Replicate
9. Replicability Pillars Random Seeds !!!
10. Replicability Pillars Random Seeds !!!
11. Replicability Pillars Random Seeds !!!
12. cuDNN
13. Reproducibility & 
 Machine Learning ?
14. Reproducibility as 
 a measure of Confidence for Machine Learning Models
15. Reproducibility Selection Bias
16. Check Supervised Learning Cross Validation
17. Input data Features Machine Learning Algorithm Output label Labels Predictive Model New input data Features Supervised Learning
18. Building ML Models keep error as low as possible. Two major sources of error are bias and variance. If we managed to reduce these two, then we could build more accurate models.
19. bit.ly/ml-data-junk
20. How we diagnose 
 bias and variance ? What actions should be take, in case?
21. Bias-Variance Generalisation
25. from sklearn.cross_validation import train_test_split X_tr, X_ts, y_tr, y_ts = train_test_split(X, y, train_size=0.3, 
 random_state=42)
26. Bias-Variance feature(s) target f X y f is almost completely unknown
27. Bias-Variance feature(s) target f’ X y Model try to estimate
28. Bias-Variance f’ Model TS1 Training Sets Estimates Y1 f’ TS1 Y1 f’ TS1 Y1
29. The amount by which the model varies as we change training data is Variance
30. Bias-Variance f’ Model TS1 Training Sets Estimates Y1 f’ TS1 Y1 f’ TS1 Y1
31. The bias reflects the amount of assumptions we do on the model
33. Bias-Variance Trade-off
36. Learning Curve
37. Learning Curve
38. Learning Curve
39. from sklearn.model_selection import learning_curve train_sizes, train_scores, validation_scores = learning_curve(estimator, X, y, train_sizes=0.8, cv=5, scoring='accuracy')
40. Cross Validation
41. from sklearn.model_selection import KFold
42. from sklearn.model_selection import ShuffleSplit
43. from sklearn.model_selection import StratifiedKFold
44. Replicable CV
45. Reproducibility Selection Bias
46. Selection Bias Selection Bias is the selection of data in such a way that proper randomisation is not achieved the sample obtained is not representative of the population selection bias not considered => conclusions not accurate
47. Nested CV 5 x 3 CV - Hyper Parameter Search 10 x 5 CV - Selection Bias
48. Nested CV todo list ✓ Initialise and set random seeds ✓ Be sure to create NEW models at each run ✓ Calculate the Error (metric) at each run ✓ Average and Get the Confidence Intervals ✓ Save your models and checkpoints
49. Boilerplate Code Model Independent Easily tuneable to be used w/ ML | DL Feature Normalisation ?? Feature Selection??
50. Reproducible Learn Introducing leriomaggio/reproducible-learn
52. Settings
53. DL Settings
54. DAP Test
55. Template Method
56. Runners
57. TODO List • Improve Parallelisation • Integrate DB Analytics • frameworks agnostic for DL (Backend) • Integrate Multiple Metrics
58. PyData Challenge AI for Precision Medicine http://contest.pycon.it
59. Valerio Maggio @leriomaggio Thanks a lot for you 
 kind attention