PyData Meetup Amsterdam - Keras & Wavenet

Transcript

1. Rapid prototyping with Keras Implementing Deepmind’s WaveNet paper Bas Veeling 1
2. Rapid prototyping with Keras Implementing Deepmind’s WaveNet paper Bas Veeling 1
3. About me • AI at UvA, MSc thesis with Philips Research • Internship at Google Research, Scyfer. • ~2 years experience with Deep learning 2
4. WaveNet[1] Teaser • Can generate music/speech from examples. • Generative Auto-Regressive Deep Neural Network • Not fully text to speech: needs TTS features [1] Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." 3
5. WaveNet[1] Teaser • Can generate music/speech from examples. • Generative Auto-Regressive Deep Neural Network • Not fully text to speech: needs TTS features Speaker 1 Speaker 2 Music [1] Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." 3
6. Keras • Author: François Chollet ( @fchollet) & others • High-level neural network library • Python • Uses tensorflow or theano to run on GPU/CPU 4
7. 5 in = Input(8) out = Dense(9)(in) out = Dense(9)(out) out = Dense(9)(out) out = Dense(4)(out) model = Model(in, out) model.compile() model.fit(X,y)
8. About you • Know the basic principles of deep learning? • Worked with a machine learning framework? • Worked with Keras? • Worked with computation graph frameworks (tensorflow, theano, etc)? • Rough understanding of WaveNet? 6
9. Goals of this talk • Show the magic of Keras • Accessible without machine learning experience. • Tips and tricks on working efficiently. • Leave out on details (but feel free to ask!) • Non-goal: an objective comparison between frameworks. 7
10. Quick Deep Learning Intro • Image -> blackbox -> target ‘cat’ or ‘dog’ • Neural network consisting of layers of neurons that compute intermediate steps. • These layers have weights, which determine the outcome. • We ‘train’ these weights: show an input and adjust the weights towards target. 8
11. Computational graph libraries • Deep Learning Models are expensive • Run on GPU and other hardware • Solution: computational graph libary. • Keras uses Theano or Tensorflow, and provides common interface: import keras.backend as K • Example time: adding two numbers together. 9
12. WaveNet • x = raw audio (300ms or 4.800 samples) • ??????? • y = next unseen sample 10
13. Causal 1D Convolutions output = x[0]+x[1] Sum: output[t]=a*x[t ]+b*x[t+1] 1D Convolution: 11
14. Causal 1D Convolutions output = x[0]+x[1] Sum: output[t]=a*x[t-1]+b*x[t ] Causal 1D Convolution: 11
15. Causal 1D Convolutions output = x[0]+x[1] Sum: output[t]=a*x[t-1]+b*x[t ] Causal 1D Convolution: 11
16. Dilated Causal Convolutions 12
17. Dilated Causal Convolutions I.e. Divide and Conquer 12
18. Lots of other tricks! • Gated activation Unit[1] • Non-linear quantization • Categorical Distribution as output • Conditional Wavenet for TTS [1] Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." 13
19. Audio Generation 14
20. Model Architecture 15
21. Live Demo! Questions? 16
22. Training on Speech: Small network, 4khz, VCTK 1. Epoch 29 2. Epoch 65 3. Epoch 358 17
23. 1. Conditional WaveNet 2. Printing intermediate output w/ K.print_tensor() 3. PyCharm and working with remote GPU servers/Cluster 4. Having Keras in pycharm project with editable install. 5. Smart Experiment Management with IDSIA-Sacred 6. TDD with Keras 18 If we had more time:
24. Conclusion • Getting started with DL is easy! 19
25. Thanks! Contact me: basveeling@gmail.com Code: github.com/basveeling/wavenet Further reading: 
 github.com/tomlepaine/fast-wavenet
 keras.io
 deeplearningbook.org Figures: Google Deepmind (paper & blog) 20