TensorFlow DevSummit 2019 Recap

Transcript

1. TensorFlow DevSummit 2019 Recap What is a Fundamental Change of

   TensorFlow v2? @shuhei_fujiwara 2019-03-20 TensorFlow User Group 1

2. Self-Introduction 藤原秀平 (FUJIWARA Shuhei) Twitter: @shuhei_fujiwara ▶ Merpay Inc., Machine

   Learning Team ▶ TensorFlow User Group Tokyo Organizer ▶ Google Developer Expert (Machine Learning) 2

3. Outline I’m talking about ▶ Major Change of TensorFlow v2

   ▶ AutoGraph ▶ Eager Execution ▶ Other Interesting Topics (if time permitted) 3

4. TensorFlow v2 Alpha is released! ▶ Eager execution (define-by-run) by

   default ▶ AutoGraph ▶ Convert Python statements to TensorFlow graph ▶ Cleanup APIs ▶ tf.keras would be main stream 4

5. Talk About AutoGraph ▶ YouTube ▶ https://www.youtube.com/watch?v=Up9CvRLIIIw ▶ Official document

   ▶ https://www.tensorflow.org/alpha/tutorials/eager/tf_function 5

6. Define and Run in TensorFlow v1 Define computation graph but

   not run for now constant 1 constant 2 matmul constant 3 add 6

7. Define and Run in TensorFlow v1 Execute a node by

   session.run() constant 1 constant 2 matmul constant 3 add Run 7

8. Why Graph? We need to calculate gradient easily b x

   Add W matmul ReLU ... L dL/db dAdd dL/dW dMatMul dReLU ... 1 8

9. Eager Execution (Define by Run) Listing 1: graph (define-and-run) 1

   >>> v1 = tf.constant(1) 2 >>> v2 = tf.constant(2) 3 >>> v3 = v1 + v2 4 >>> print(v3) 5 Tensor("add:0", shape=(), dtype= int32) 6 >>> sess = tf.Session() 7 >>> print(sess.run(v3)) 8 3 Listing 2: eager (define-by-run) 1 >>> v1 = tf.constant(1) 2 >>> v2 = tf.constant(2) 3 >>> v3 = v1 + v2 4 >>> print(v3) 5 tf.Tensor(3, shape=(), dtype=int32) 6 >>> print(v3.numpy()) 7 3 9

10. Compute Gradient in Eager Mode 1 v = tf.Variable(1.) 2

    3 with tf.GradientTape() as tape: 4 w = v * v 5 x = w * w 6 grad = tape.gradient(x, v) 7 8 print(grad) 9 # tf.Tensor(4.0, shape=(), dtype= float32) 1 v = tf.Variable(1.) 2 w = v * v 3 4 with tf.GradientTape() as tape: 5 x = w * w 6 grad = tape.gradient(x, v) 7 8 print(grad) 9 # None ▶ Computation graph is kept and gradient can be calculated in tf.GradientTape context 10

11. Use Optimizer in Custom Training 1 model = tf.keras.Sequential([...]) 2

    optim = tf.keras.optimizers.SGD(nesterov=True) 3 4 ... 5 6 with tf.GradientTape() as tape: 7 logits = model(x) 8 loss = ... 9 grads = tape.gradient(loss, model.trainable_variables) 10 optimizer.apply_gradients(zip(grads, model.trainable_variables)) ▶ https://www.tensorflow.org/alpha/tutorials/quickstart/advanced 11

12. Eager Execution Resolves All Problem? Pros of define-and-run ▶ Performance

    ▶ deployability (without Python runtime) ▶ Mobile, TensorFlow Serving, TensorFlow.js, ... Cons of define-and-run ▶ A little bit difficult ▶ Hard to debug ▶ Especially researcher want to try and error rapidly ▶ Hard to implement dynamic network 12

13. Why AutoGraph in TensorFlow v2? We want to enjoy pros

    of define-and-run and define-by-run... ▶ Use eager mode by default ▶ Convert (compile) Python function to TensorFlow graph if needed 13

14. tf.function ▶ @tf.function decorator automatically compile function to graph ▶

    if –> tf.cond ▶ while –> tf.while_loop 1 @tf.function 2 def abs(x): 3 if x >= 0: 4 return x 5 else: 6 return -x 14

15. Performance In terms of performance ▶ function with lots of

    small ops: @tf.function is effective ▶ ex. LSTM cell ▶ function with large ops: @tf.function is not effective ▶ ex. Convolutional network 15

16. Performance 1 @tf.function 2 def fn_graph(x): 3 for i in

    tf.range(1000): x += i 4 return x 5 6 def fn_eager(x): 7 for i in tf.range(1000): x += i 8 return x 9 10 time_graph = timeit.timeit(lambda: linear_graph(0), number=10) 11 time_eager = timeit.timeit(lambda: linear_eager(0), number=10) 12 13 # time_graph: 0.027 time_eager: 0.082 16

17. Eager Mode is Disabled in @tf.function 1 @tf.function 2 def

    fn_graph(): 3 tf.print(tf.executing_eagerly()) 4 return 5 6 def fn_eager(): 7 tf.print(tf.executing_eagerly()) 8 return 9 10 fn_graph() # False 11 fn_eager() # True 17

18. Note for tf.function ▶ Needless to say, tf.function can’t convert

    all Python code ▶ See Capabilities and Limitations ▶ tf.Variable in tf.function must be created only once 18

19. How to Use TensorFlow v2 ▶ Use eager mode when

    writing your code ▶ Use @tf.function when running your code ▶ Good bye session.run forever... ▶ Nothing to do as long as using tf.keras ▶ But you have to know about AutoGraph for custom training 19

20. Any Questions? I’m talking about ▶ TensorFlow Datasets ▶ TFX

    ▶ Reinforcement Learning ▶ etc... if time permitted. 20