Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bookkeeping for Machine Learning

Bookkeeping for Machine Learning

28289f966d95dfc0af5da839e41047a6?s=128

Jaidev Deshpande

April 06, 2019
Tweet

More Decks by Jaidev Deshpande

Other Decks in Technology

Transcript

  1. 1 BOOKKEEPING IN MACHINE LEARNING JAIDEV DESHPANDE, SENIOR DATA SCIENTIST

    /@jaidevd
  2. 2 DEEP LEARNING EFFORT

  3. DEEP LEARNING HAS BECOME CHEAPER* *NOT CHEAP ENOUGH FOR REPETITIVE

    MISTAKES THE BIG IDEA
  4. “FAIL FAST. EITHER DO THE RIGHT THING OR STOP.” “WHY

    DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT?” JIM GRAY (1985)
  5. 5 WHAT’S SUCCESS OR FAILURE IN DEEP LEARNING? • Neural

    networks should learn • Neural networks should learn well • Neural networks should generalize well
  6. WHAT KEEPS A NETWORK FROM DOING ALL THAT?

  7. 7 THE TYPICAL CYCLE

  8. 8 THE TYPICAL CYCLE

  9. 9 THE TYPICAL CYCLE

  10. THE BIAS-VARIANCE TRADEOFF THE INHERENT PROPERTY OF LEAST-SQUARES ESTIMATION THAT

    PLAGUES EVERY SUPERVISED LEARNING TASK
  11. INTRODUCING TEST DRIVEN DEEPLEARNING DEVELOPMENT

  12. 12 SANITY CHECKS CAN BE APPLIED TO: • The training

    data ◦ Well conditioned problem ◦ Redundancy or bias / variance in datasets ◦ Stratified train/dev splits • Training heuristics ◦ Reasonable optimizers with learning rates ◦ Batch sizes ◦ Shuffled training samples • The model architecture ◦ Number of layers ◦ Layer arrangement and ordering • Layers ◦ Weight initializations ◦ Activations ◦ Regularization • Losses & metrics ◦ Consistency across epochs ◦ Avoidable bias Don’t bother training if too many of these checks fail!
  13. 13 BOOKKEEPING FOR DEEP LEARNING EXPERIMENTS • Don’t repeat yourself

    • Keep track of metrics across experiments • Efficient hyperparameter grid search • Searching for models • Transfer Learning - you might already have a model for a given problem, only trained on a different dataset
  14. INTRODUCING KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS

    HTTPS://GITHUB.COM/JAIDEVD/KEPLER
  15. 15 KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS

    • Like an IDE, organize models into projects • Each project has experiments • An experiment consists of a single training / validation session, contains ◦ Configuration of the model used ◦ Basic statistics on the training / validation data like number of features, samples, etc ◦ Error curves ◦ Metadata like datetime, path to the related files, etc • A “check” system (like PEP8) which warns the user at different stages of the project about various inconsistencies in the model • A simple search engine for models • A grid search manager - DRY when doing hyperparameter search • A verbose logger that logs all events in all projects
  16. 16 SOME OF THE CHECKS IN KEPLER

  17. 17 $ cat model.py import keras from sklearn.datasets import load_digits

    digits = load_digits() X = digits['data'] y = keras.utils.to_categorical(digits['target']) model = keras.models.Sequential([ keras.layers.Dense(32, input_shape=(64,)), keras.layers.Activation('sigmoid'), keras.layers.Dense(10), keras.layers.Activation('sigmoid') ]) model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD()) $ kepler init Welcome to Kepler! $ ipython >>> from model import * >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) KEPLER EXAMPLES
  18. 18 KEPLER EXAMPLES /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K102: Training data is not

    shuffled. This may slow training down. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K301: Training samples are correlated. There may be redundancy in the data. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K302: There might be duplicate training samples. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K303: Training data not normalized. warnings.warn(self.code + ": " + self.msg) Epoch 1/5 1797/1797 [==============================] - 0s 148us/step - loss: 2.3471 Epoch 2/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.2209 Epoch 3/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.1414 Epoch 4/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0842 Epoch 5/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0339
  19. 19 KEPLER EXAMPLES $ ipython >>> from model import *

    >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) ... There are 1 models similar to this one. Would you like to see their graphs? [Y/n] : y Enter location for saving graphs [~/.kepler/models/tf_logs]: >>> Graphs written to /Users/jaidevd/.kepler/models/tf_logs Please point Tensorboard to /Users/jaidevd/.kepler/models/tf_logs Continue training? [Y/n] : n >>> exit() $ tensorboard --logdir ~/.kepler/models/tf_logs TensorBoard 1.12.0 at http://localhost:6006 (Press CTRL+C to quit)
  20. 20 KEPLER EXAMPLES

  21. THE KEPLER PHILOSOPHY 1. RIGOROUSLY INSPECT MODELS AND DETECT INTERNAL

    INCONSISTENCIES 2. LOG EVERYTHING 3. DON’T REPEAT YOURSELF
  22. 22 TECH DETAILS • Fully written in Python • So

    far, supports only Keras with a Tensorflow backend • Models are guaranteed remain untouched • Well tested but not well documented :-( • Beta release scheduled for Q2 2019 • Bug reports & pull requests welcome!
  23. 23 THANK YOU! /@jaidevd /@jaidevd deshpande.jaidev@gmail.com