Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automating AI Workflows

Automating AI Workflows

28289f966d95dfc0af5da839e41047a6?s=128

Jaidev Deshpande

May 21, 2019
Tweet

More Decks by Jaidev Deshpande

Other Decks in Technology

Transcript

  1. 1 AUTOMATING WORKFLOWS IN AI PROJECTS JAIDEV DESHPANDE, SENIOR DATA

    SCIENTIST /@jaidevd
  2. DEEP LEARNING HAS BECOME CHEAPER* *NOT CHEAP ENOUGH FOR REPETITIVE

    MISTAKES THE BIG IDEA
  3. “FAIL FAST. EITHER DO THE RIGHT THING OR STOP.” “WHY

    DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT?” JIM GRAY (1985)
  4. 4 WHAT’S SUCCESS OR FAILURE IN DEEP LEARNING? • Neural

    networks should learn • Neural networks should learn well • Neural networks should generalize well
  5. WHAT KEEPS A NETWORK FROM DOING ALL THAT? • The

    Fundamental Theorem of Statistical Learning: or the inherent inseparability of data • The Bias-Variance tradeoff: The inherent property of least squares systems that plagues all supervised learning tasks • No Free Lunch Theorem: All models are, on average, equally bad!
  6. 6 THE TYPICAL CYCLE

  7. 7 THE TYPICAL CYCLE

  8. 8 THE TYPICAL CYCLE

  9. INTRODUCING TEST DRIVEN DEEPLEARNING DEVELOPMENT

  10. 10 SANITY CHECKS CAN BE APPLIED TO: • The training

    data ◦ Well conditioned problem ◦ Redundancy or bias / variance in datasets ◦ Stratified train/dev splits • Training heuristics ◦ Reasonable optimizers with learning rates ◦ Batch sizes ◦ Shuffled training samples • The model architecture ◦ Number of layers ◦ Layer arrangement and ordering • Layers ◦ Weight initializations ◦ Activations ◦ Regularization • Losses & metrics ◦ Consistency across epochs ◦ Avoidable bias Don’t bother training if too many of these checks fail!
  11. 11 BOOKKEEPING FOR DEEP LEARNING EXPERIMENTS • Don’t repeat yourself

    • Keep track of metrics across experiments • Efficient hyperparameter grid search • Searching for models • Transfer Learning - you might already have a model for a given problem, only trained on a different dataset
  12. INTRODUCING KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS

    HTTPS://GITHUB.COM/JAIDEVD/KEPLER
  13. 13 DEEP LEARNING EFFORT

  14. 14 KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS

    • Like an IDE, organize models into projects • Each project has experiments • An experiment consists of a single training / validation session, contains ◦ Configuration of the model used ◦ Basic statistics on the training / validation data like number of features, samples, etc ◦ Error curves ◦ Metadata like datetime, path to the related files, etc • A “check” system (like PEP8) which warns the user at different stages of the project about various inconsistencies in the model • A simple search engine for models • A grid search manager - DRY when doing hyperparameter search • A verbose logger that logs all events in all projects
  15. 15 SOME OF THE CHECKS IN KEPLER

  16. 16 $ cat model.py import keras from sklearn.datasets import load_digits

    digits = load_digits() X = digits['data'] y = keras.utils.to_categorical(digits['target']) model = keras.models.Sequential([ keras.layers.Dense(32, input_shape=(64,)), keras.layers.Activation('sigmoid'), keras.layers.Dense(10), keras.layers.Activation('sigmoid') ]) model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD()) $ kepler init Welcome to Kepler! $ ipython >>> from model import * >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) KEPLER EXAMPLES
  17. 17 KEPLER EXAMPLES /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K102: Training data is not

    shuffled. This may slow training down. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K301: Training samples are correlated. There may be redundancy in the data. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K302: There might be duplicate training samples. warnings.warn(self.code + ": " + self.msg) /Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K303: Training data not normalized. warnings.warn(self.code + ": " + self.msg) Epoch 1/5 1797/1797 [==============================] - 0s 148us/step - loss: 2.3471 Epoch 2/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.2209 Epoch 3/5 1797/1797 [==============================] - 0s 35us/step - loss: 2.1414 Epoch 4/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0842 Epoch 5/5 1797/1797 [==============================] - 0s 37us/step - loss: 2.0339
  18. 18 KEPLER EXAMPLES $ ipython >>> from model import *

    >>> from kepler import ModelInspector >>> with ModelInspector(model=model) as mi: ... mi.fit(X, y, epochs=5) ... There are 1 models similar to this one. Would you like to see their graphs? [Y/n] : y Enter location for saving graphs [~/.kepler/models/tf_logs]: >>> Graphs written to /Users/jaidevd/.kepler/models/tf_logs Please point Tensorboard to /Users/jaidevd/.kepler/models/tf_logs Continue training? [Y/n] : n >>> exit() $ tensorboard --logdir ~/.kepler/models/tf_logs TensorBoard 1.12.0 at http://localhost:6006 (Press CTRL+C to quit)
  19. 19 KEPLER EXAMPLES

  20. THE KEPLER PHILOSOPHY 1. RIGOROUSLY INSPECT MODELS AND DETECT INTERNAL

    INCONSISTENCIES 2. LOG EVERYTHING 3. DON’T REPEAT YOURSELF
  21. 21 TECH DETAILS • Fully written in Python • So

    far, supports only Keras with a Tensorflow backend • Models are guaranteed remain untouched • Well tested but not well documented :-( • Beta release scheduled for Q2 2019 • Bug reports & pull requests welcome!
  22. 22 THANK YOU! /@jaidevd /@jaidevd deshpande.jaidev@gmail.com