Understanding Neural Network Architectures with Attention and Diffusion

Neural networks have revolutionized AI, enabling machines to learn from data and make intelligent decisions. In this talk, we'll explore two popular architectures: Attention models and Diffusion models.

First up, we'll discuss Attention models and how they've contributed to the success of large language models like ChatGPT. We'll explore how the Attention mechanism helps GPT focus on specific parts of a text sequence and how this mechanism has been applied to different tasks in natural language processing.

Next, we'll dive into Diffusion models, a class of generative models that have shown remarkable performance in image synthesis. We'll explain how they work and their potential applications in the creative industry.

By the end of the talk, you'll have a better understanding of these cutting-edge neural network architectures.

Transcript

1. ATTENTION AND DIFFUSION Understanding Neural Network Architectures Michał Karzyński EuroPython 2023
2. THE SPEAKER Michał Karzyński (@postrational) Software Architect Open Neural Network Exchange 
 Operators group
3. THE TALK: MODELS Transformers Diffusion Natural Language Images GPT BERT T5 Stable Diffusion Midjourney DALL-E Attention Convolution & Attention
4. CHAT GPT
5. DIFFUSION MODELS Stable Diffusion Midjourney DALL-E
6. THE TALK: OPERATIONS Linear a.k.a.: Dense, Fully-connected 
 Convolution Filter scan to produce feature maps Attention Key-value store lookup
7. MULTI-LAYER PERCEPTRON
8. linear linear input input Multi-layer Perceptron Multi-layer Perceptron
9. linear linear Multi-layer Perceptron Multi-layer Perceptron
10. OPERATION: CONVOLUTION
11. OPERATION: CONVOLUTION
12. CONVOLUTIONAL NETWORKS
13. convolution convolution max pool max pool linear linear VGG-16 VGG-16
14. VISUALIZATION OF CONVOLUTIONAL NETWORKS Zeiler and Fergus. "Visualizing and Understanding Convolutional Networks." arXiv:1311.2901v3 [cs.CV],
15. Zeiler and Fergus, 2013
16. ENCODER-DECODER ARCHITECTURE
17. convolution convolution max pool max pool deconvolution deconvolution Convolutional Autoencoder Convolutional Autoencoder unpooling unpooling
18. SKIP CONNECTIONS 
 AND RESIDUAL NETWORKS
19. convolution convolution max pool max pool linear linear ResNet-18 ResNet-18 average pool average pool add add + + + + + + + + + + + + + + + + + +
20. U-NET
21. convolution convolution max pool max pool deconvolution deconvolution Convolutional U-Net Convolutional U-Net unpooling unpooling concatenate concatenate c c c c c c c c c c
22. OPERATION: ATTENTION store = { 'key1': 'value1', 'key2': 'value2', 'key3': 'value3', } query = 'key1' value = store[query]
23. Vaswani, Ashish, et al. "Attention Is All You Need." arXiv:1706.03762v5 [cs.CL], 2017
24. TRANSFORMER ARCHITECTURE
25. linear linear inputs inputs Nx Nx append word to output append word to output K K V V V V Q Q multi-head attention multi-head attention Transformer Transformer positional encoding positional encoding embedding embedding add add + + + + + + + + + + + + + + + + + + + + + + + + Nx Nx
26. FORWARD AND BACKWARD DIFFUSION
27. FORWARD BACKWARD + + + + + + + + add genrated noise add genrated noise substract estimated noise substract estimated noise - - - -
28. LATENT DIFFUSION
29. Latent Space Latent Space + + + + + + + + "Logo for "Logo for EuroPython EuroPython in Prague" in Prague" + + add add + + + + + + + + K K V V V V Q Q + + + + K K V V V V Q Q + + + + BERT Encoder BERT Encoder ResBlock ResBlock ResBlock ResBlock Spatial Transformer Spatial Transformer Spatial Transformer Spatial Transformer linear linear multi-head attention multi-head attention Latent Diffusion Latent Diffusion timestep / positional encoding timestep / positional encoding embedding embedding deconvolution deconvolution K K V V V V Q Q + + + + convolution convolution up/down sample up/down sample c c c c c c c c concatenate concatenate c c
30. CLOSING REMARKS
31. Social media was the fi rst contact between A.I. and humanity, and humanity lost. YUVAL HARARI
32. THANK YOU