Distributed TensorFlow: Scaling Deep Learning Library

#tensorflow #scale #distributed

Transcript

1. Distributed TensorFlow Tien Dinh
6. TensorFlow: Expressing High-Level ML Computations Core in C++ • Very • low overhead Different • front ends for specifying/driving the computation Python • and C++ today, easy to add more
7. Computation is a dataflow graph Graph of Nodes • , called Operations or ops Edges are N • -dimensional arrays: Tensors
8. Computation is a dataflow graph WITH STATE
9. Computation is a dataflow graph Distributed
10. Computation is a dataflow graph Assign Devices to Ops • TensorFlow inserts Send/Recv Ops to transport tensors across devices • Recv ops pull data from Send ops
11. Computation is a dataflow graph Assign Devices to Ops TensorFlow inserts Send/Recv Ops to transport tensors across devices • Recv • ops pull data from Send ops
12. Distrubuted Training with TensorFlow
13. Distrubuted Training with TensorFlow
14. Model Parallelism = split model, share data
15. Distrubuted Training
16. Distrubuted Training with TensorFlow
17. Data Parallelism
18. Data Parallelism
19. Data Parallelism
20. Data Parallelism
21. Data Parallelism
22. Data Parallelism
23. Distributed training mechanisms Graph structure and low-level graph primitives (queues) allow us to play with synchronous vs. asynchronous update algorithms.
24. Thanks for your attention!