TensorFlow DevSummit 2019 Recap What is a Fundamental Change of TensorFlow v2? @shuhei_fujiwara 2019-03-20 TensorFlow User Group 1

Self-Introduction 藤原秀平 (FUJIWARA Shuhei) Twitter: @shuhei_fujiwara ▶ Merpay Inc., Machine Learning Team ▶ TensorFlow User Group Tokyo Organizer ▶ Google Developer Expert (Machine Learning) 2

Outline I’m talking about ▶ Major Change of TensorFlow v2 ▶ AutoGraph ▶ Eager Execution ▶ Other Interesting Topics (if time permitted) 3

TensorFlow v2 Alpha is released! ▶ Eager execution (define-by-run) by default ▶ AutoGraph ▶ Convert Python statements to TensorFlow graph ▶ Cleanup APIs ▶ tf.keras would be main stream 4

Talk About AutoGraph ▶ YouTube ▶ https://www.youtube.com/watch?v=Up9CvRLIIIw ▶ Official document ▶ https://www.tensorflow.org/alpha/tutorials/eager/tf_function 5

Define and Run in TensorFlow v1 Define computation graph but not run for now constant 1 constant 2 matmul constant 3 add 6

Define and Run in TensorFlow v1 Execute a node by session.run() constant 1 constant 2 matmul constant 3 add Run 7

Why Graph? We need to calculate gradient easily b x Add W matmul ReLU ... L dL/db dAdd dL/dW dMatMul dReLU ... 1 8

Eager Execution (Define by Run) Listing 1: graph (define-and-run) 1 >>> v1 = tf.constant(1) 2 >>> v2 = tf.constant(2) 3 >>> v3 = v1 + v2 4 >>> print(v3) 5 Tensor("add:0", shape=(), dtype= int32) 6 >>> sess = tf.Session() 7 >>> print(sess.run(v3)) 8 3 Listing 2: eager (define-by-run) 1 >>> v1 = tf.constant(1) 2 >>> v2 = tf.constant(2) 3 >>> v3 = v1 + v2 4 >>> print(v3) 5 tf.Tensor(3, shape=(), dtype=int32) 6 >>> print(v3.numpy()) 7 3 9

Compute Gradient in Eager Mode 1 v = tf.Variable(1.) 2 3 with tf.GradientTape() as tape: 4 w = v * v 5 x = w * w 6 grad = tape.gradient(x, v) 7 8 print(grad) 9 # tf.Tensor(4.0, shape=(), dtype= float32) 1 v = tf.Variable(1.) 2 w = v * v 3 4 with tf.GradientTape() as tape: 5 x = w * w 6 grad = tape.gradient(x, v) 7 8 print(grad) 9 # None ▶ Computation graph is kept and gradient can be calculated in tf.GradientTape context 10

Use Optimizer in Custom Training 1 model = tf.keras.Sequential([...]) 2 optim = tf.keras.optimizers.SGD(nesterov=True) 3 4 ... 5 6 with tf.GradientTape() as tape: 7 logits = model(x) 8 loss = ... 9 grads = tape.gradient(loss, model.trainable_variables) 10 optimizer.apply_gradients(zip(grads, model.trainable_variables)) ▶ https://www.tensorflow.org/alpha/tutorials/quickstart/advanced 11

Eager Execution Resolves All Problem? Pros of define-and-run ▶ Performance ▶ deployability (without Python runtime) ▶ Mobile, TensorFlow Serving, TensorFlow.js, ... Cons of define-and-run ▶ A little bit difficult ▶ Hard to debug ▶ Especially researcher want to try and error rapidly ▶ Hard to implement dynamic network 12

Why AutoGraph in TensorFlow v2? We want to enjoy pros of define-and-run and define-by-run... ▶ Use eager mode by default ▶ Convert (compile) Python function to TensorFlow graph if needed 13

tf.function ▶ @tf.function decorator automatically compile function to graph ▶ if –> tf.cond ▶ while –> tf.while_loop 1 @tf.function 2 def abs(x): 3 if x >= 0: 4 return x 5 else: 6 return -x 14

Performance In terms of performance ▶ function with lots of small ops: @tf.function is effective ▶ ex. LSTM cell ▶ function with large ops: @tf.function is not effective ▶ ex. Convolutional network 15

Performance 1 @tf.function 2 def fn_graph(x): 3 for i in tf.range(1000): x += i 4 return x 5 6 def fn_eager(x): 7 for i in tf.range(1000): x += i 8 return x 9 10 time_graph = timeit.timeit(lambda: linear_graph(0), number=10) 11 time_eager = timeit.timeit(lambda: linear_eager(0), number=10) 12 13 # time_graph: 0.027 time_eager: 0.082 16

Eager Mode is Disabled in @tf.function 1 @tf.function 2 def fn_graph(): 3 tf.print(tf.executing_eagerly()) 4 return 5 6 def fn_eager(): 7 tf.print(tf.executing_eagerly()) 8 return 9 10 fn_graph() # False 11 fn_eager() # True 17

Note for tf.function ▶ Needless to say, tf.function can’t convert all Python code ▶ See Capabilities and Limitations ▶ tf.Variable in tf.function must be created only once 18

How to Use TensorFlow v2 ▶ Use eager mode when writing your code ▶ Use @tf.function when running your code ▶ Good bye session.run forever... ▶ Nothing to do as long as using tf.keras ▶ But you have to know about AutoGraph for custom training 19

Any Questions? I’m talking about ▶ TensorFlow Datasets ▶ TFX ▶ Reinforcement Learning ▶ etc... if time permitted. 20