Bookkeeping for Machine Learning

Transcript

1. 1 BOOKKEEPING IN MACHINE LEARNING JAIDEV DESHPANDE, SENIOR DATA SCIENTIST /@jaidevd
2. 2 DEEP LEARNING EFFORT
3. DEEP LEARNING HAS BECOME CHEAPER* *NOT CHEAP ENOUGH FOR REPETITIVE MISTAKES THE BIG IDEA
4. “FAIL FAST. EITHER DO THE RIGHT THING OR STOP.” “WHY DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT?” JIM GRAY (1985)
5. 5 WHAT’S SUCCESS OR FAILURE IN DEEP LEARNING? • Neural networks should learn • Neural networks should learn well • Neural networks should generalize well
6. WHAT KEEPS A NETWORK FROM DOING ALL THAT?
7. 7 THE TYPICAL CYCLE
8. 8 THE TYPICAL CYCLE
9. 9 THE TYPICAL CYCLE
10. THE BIAS-VARIANCE TRADEOFF THE INHERENT PROPERTY OF LEAST-SQUARES ESTIMATION THAT PLAGUES EVERY SUPERVISED LEARNING TASK
11. INTRODUCING TEST DRIVEN DEEPLEARNING DEVELOPMENT
12. 12 SANITY CHECKS CAN BE APPLIED TO: • The training data ◦ Well conditioned problem ◦ Redundancy or bias / variance in datasets ◦ Stratified train/dev splits • Training heuristics ◦ Reasonable optimizers with learning rates ◦ Batch sizes ◦ Shuffled training samples • The model architecture ◦ Number of layers ◦ Layer arrangement and ordering • Layers ◦ Weight initializations ◦ Activations ◦ Regularization • Losses & metrics ◦ Consistency across epochs ◦ Avoidable bias Don’t bother training if too many of these checks fail!
13. 13 BOOKKEEPING FOR DEEP LEARNING EXPERIMENTS • Don’t repeat yourself • Keep track of metrics across experiments • Efficient hyperparameter grid search • Searching for models • Transfer Learning - you might already have a model for a given problem, only trained on a different dataset
14. INTRODUCING KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS HTTPS://GITHUB.COM/JAIDEVD/KEPLER
15. 15 KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS • Like an IDE, organize models into projects • Each project has experiments • An experiment consists of a single training / validation session, contains ◦ Configuration of the model used ◦ Basic statistics on the training / validation data like number of features, samples, etc ◦ Error curves ◦ Metadata like datetime, path to the related files, etc • A “check” system (like PEP8) which warns the user at different stages of the project about various inconsistencies in the model • A simple search engine for models • A grid search manager - DRY when doing hyperparameter search • A verbose logger that logs all events in all projects
16. 16 SOME OF THE CHECKS IN KEPLER
17. 17 $ cat model.py import keras from sklearn.datasets import load_digits digits = load_digits() X = digits['data'] y = keras.utils.to_categorical(digits['target']) model = keras.models.Sequential([ keras.layers.Dense(32, input_shape=(64,)), keras.layers.Activation('sigmoid'), keras.layers.Dense(10), keras.layers.Activation('sigmoid') ]) model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD()) $ kepler init Welcome to Kepler! $ ipython >>> from model import * >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) KEPLER EXAMPLES
18. 18 KEPLER EXAMPLES /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K102: Training data is not shuffled. This may slow training down. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K301: Training samples are correlated. There may be redundancy in the data. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K302: There might be duplicate training samples. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K303: Training data not normalized. warnings.warn(self.code + ": " + self.msg) Epoch 1/5 1797/1797 [==============================] - 0s 148us/step - loss: 2.3471 Epoch 2/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.2209 Epoch 3/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.1414 Epoch 4/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0842 Epoch 5/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0339
19. 19 KEPLER EXAMPLES $ ipython >>> from model import * >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) ... There are 1 models similar to this one. Would you like to see their graphs? [Y/n] : y Enter location for saving graphs [~/.kepler/models/tf_logs]: >>> Graphs written to /Users/jaidevd/.kepler/models/tf_logs Please point Tensorboard to /Users/jaidevd/.kepler/models/tf_logs Continue training? [Y/n] : n >>> exit() $ tensorboard --logdir ~/.kepler/models/tf_logs TensorBoard 1.12.0 at http://localhost:6006 (Press CTRL+C to quit)
20. 20 KEPLER EXAMPLES
21. THE KEPLER PHILOSOPHY 1. RIGOROUSLY INSPECT MODELS AND DETECT INTERNAL INCONSISTENCIES 2. LOG EVERYTHING 3. DON’T REPEAT YOURSELF
22. 22 TECH DETAILS • Fully written in Python • So far, supports only Keras with a Tensorflow backend • Models are guaranteed remain untouched • Well tested but not well documented :-( • Beta release scheduled for Q2 2019 • Bug reports & pull requests welcome!
23. 23 THANK YOU! /@jaidevd /@jaidevd deshpande.jaidev@gmail.com