Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning with Python & TensorFlow

Ian Lewis
September 26, 2016

Deep Learning with Python & TensorFlow

Python has lots of scientific, data analysis, and machine learning libraries. But there are many problems. Which do you use? How do they compare to each other? How can you use a model that has been trained in your production application?

TensorFlow is a new Open Source framework created at Google for building Deep Learning applications. Tensorflow allows you to construct easy to understand data flow graphs which form a mathematical and logical pipeline. Creating data flow graphs allow easier visualization of complicated algorithms as well as running the training operations over multiple hardware GPUs.

Tensorflow data flow graphs and operations are written in Python. In this talk I will discuss how you can use TensorFlow to create Deep Learning applications. I will discuss how it compares to other Python machine learning libraries like Theano or Chainer. Finally, I will discuss how trained TensorFlow models could be deployed into a production system using TensorFlow Serve.

Ian Lewis

September 26, 2016
Tweet

More Decks by Ian Lewis

Other Decks in Technology

Transcript

  1. Deep Learning with Python &
    TensorFlow
    PyCon JP 2016
    #pyconjp #pyconjp_201

    View full-size slide

  2. Confidential & Proprietary
    Google Cloud Platform 2
    Ian Lewis
    Developer Advocate - Google Cloud Platform
    Tokyo, Japan
    +Ian Lewis
    @IanMLewis

    View full-size slide

  3. Confidential & Proprietary
    Google Cloud Platform 3

    View full-size slide

  4. Confidential & Proprietary
    Google Cloud Platform 4

    View full-size slide

  5. Confidential & Proprietary
    Google Cloud Platform 5
    Deep Learning 101

    View full-size slide

  6. Confidential & Proprietary
    Google Cloud Platform 6
    Neural Networks
    can find a way to
    solve the problem
    How do you classify
    these data points?

    View full-size slide

  7. Google Cloud Platform 7
    ["cat"]
    Input Hidden Output(label)
    pixels( )

    View full-size slide

  8. Confidential & Proprietary
    Google Cloud Platform 8

    View full-size slide

  9. Google Cloud Platform 9
    (x,y,z,?,?,?,?,...)

    View full-size slide

  10. Google Cloud Platform 10
    v[x] => vector

    View full-size slide

  11. Google Cloud Platform 11
    m[x][y][z] => matrix

    View full-size slide

  12. Google Cloud Platform 12
    t[x][y][z][?][?]... => tensor

    View full-size slide

  13. Google Cloud Platform 13

    View full-size slide

  14. Confidential & Proprietary
    Google Cloud Platform 14

    View full-size slide

  15. Confidential & Proprietary
    Google Cloud Platform 15
    Breakthroughs

    View full-size slide

  16. Google Cloud Platform 16
    The Inception model (GoogLeNet, 2015)

    View full-size slide

  17. From: Andrew Ng

    View full-size slide

  18. DNN = a large matrix ops
    a few GPUs >> CPU
    (but it still takes hours/days to train)
    a supercomputer >> a few GPUs
    (but you don't have a supercomputer)
    You need Distributed Training

    View full-size slide

  19. Confidential & Proprietary
    Google Cloud Platform 21
    TensorFlow

    View full-size slide

  20. 22
    Google's open source library for
    machine intelligence
    tensorflow.org launched in Nov 2015
    The second generation
    Used by many production ML projects
    What is Tensorflow?

    View full-size slide

  21. 23
    Operates over tensors: n-dimensional arrays
    Using a flow graph: data flow computation framework
    TensorFlow
    ● Flexible, intuitive construction
    ● automatic differentiation
    ● Support for threads, queues, and asynchronous
    computation; distributed runtime
    ● Train on CPUs, GPUs
    ● Run wherever you like

    View full-size slide

  22. Google Cloud Platform 24
    Core TensorFlow data structures and concepts...
    - Graph: A TensorFlow computation, represented as a
    dataflow graph.
    - collection of ops that may be executed together as a
    group
    - Operation: a graph node that performs computation on
    tensors
    - Tensor: a handle to one of the outputs of an Operation
    - provides a means of computing the value in a
    TensorFlow Session.

    View full-size slide

  23. Google Cloud Platform 25
    Core TensorFlow data structures and concepts
    - Constants
    - Placeholders: must be fed with data on execution
    - Variables: a modifiable tensor that lives in TensorFlow's
    graph of interacting operations.
    - Session: encapsulates the environment in which
    Operation objects are executed, and Tensor objects are
    evaluated.

    View full-size slide

  24. Google Cloud Platform 26
    Category
    Element-wise math ops
    Array ops
    Matrix ops
    Stateful ops
    NN building blocks
    Checkpointing ops
    Queue & synch ops
    Control flow ops
    Operations
    Examples
    Add, Sub, Mul, Div, Exp, Log, Greater, Less…
    Concat, Slice, Split, Constant, Rank, Shape…
    MatMul, MatrixInverse, MatrixDeterminant…
    Variable, Assign, AssignAdd...
    SoftMax, Sigmoid, ReLU, Convolution2D…
    Save, Restore
    Enqueue, Dequeue, MutexAcquire…
    Merge, Switch, Enter, Leave...

    View full-size slide

  25. Google Cloud Platform 27

    View full-size slide

  26. Distributed Training with TensorFlow

    View full-size slide

  27. ● CPU/GPU scheduling
    ● Communications
    ○ Local, RPC, RDMA
    ○ 32/16/8 bit quantization
    ● Cost-based optimization
    ● Fault tolerance
    Distributed Training with TensorFlow

    View full-size slide

  28. Distributed Training
    Model Parallelism
    Sub-Graph ● Allows fine grained
    application of parallelism
    to slow graph
    components
    ● Larger more complex
    graph
    Full Graph ● Code is more similar to
    single process models
    ● Not necessarily as
    performant (large
    models)
    Data Parallelism
    Synchronous ● Prevents workers from
    “Falling behind”
    ● Workers progress at the
    speed of the slowest
    worker
    Asynchronous ● Workers advance as fast
    as they can
    ● Can result in runs that
    aren’t reproducible or
    difficult to debug behavior
    (large models)

    View full-size slide

  29. Fully managed, distributed training and prediction
    for custom TensorFlow graph
    Supports Regression and Classification initially
    Integrated with Cloud Dataflow and Cloud Datalab
    Limited Preview - cloud.google.com/ml
    Cloud Machine Learning (Cloud ML)

    View full-size slide

  30. Jeff Dean's keynote: YouTube video
    Define a custom TensorFlow graph
    Training at local: 8.3 hours w/ 1 node
    Training at cloud: 32 min w/ 20 nodes (15x faster)
    Prediction at cloud at 300 reqs / sec
    Cloud ML

    View full-size slide

  31. Tensor Processing Unit
    ASIC for TensorFlow
    Designed by Google
    10x better perf / watt
    8 bit quantization

    View full-size slide

  32. Confidential & Proprietary
    Google Cloud Platform 34
    Thank You
    https://www.tensorflow.org/
    https://cloud.google.com/ml/
    http://bit.ly/tensorflow-workshop
    Ian Lewis
    @IanMLewis

    View full-size slide