machine-learning-camp-2018

Slides from Machine Learning Camp in Finland 2018

Transcript

1. Piotr Wawryka Distributed Representation
2. Agenda • What it is distributed representation? • Words, Paragraphs and Documents • Images • Artistic Style Transfer • Videos - Youtube8M
3. Each concept represented by single unit. • easy to understand - intuitive • easy to code by hand • easy to associate with other representations • easy to learn Inefficient for componential structure. Local representation
4. Feature based representation Each concept represented by many units. Each unit represents many concept.
5. Local representation - new entity
6. Feature based representation - new entity
7. Feature based representation - relations
8. Distributed representation • real coordinate space • no clear interpretation of specific feature • representation length can be smaller
9. How to build distributed representation?
10. Text
11. Traditional approach: • one-hot encoding • every word is orthogonal one to another W woman ∙ W man = 0 Neural language modeling: • deep architectures • recurrent networks • simple models Words representation
12. Word2Vec • proposed by Tomas Mikolov • 2 log-linear models • distributed words representation
13. Sliding window center - the context - development, of, Hellenic, language
14. CBoW - Continuous Bag of Words
15. CBoW - details input layer • context - one-hot encoding projection layer • linear activation • shared weights - embedding lookup output layer • softmax
16. Skip-Gram
17. Skip-Gram - details input layer • center word - one-hot encoding projection layer • linear activation • embedding lookup output layer • softmax activation - shared weights
18. Word2Vec - optimizations Softmax - model conditional probability over vocabulary. Vocabulary can be very large! Optimizations: • Hierarchical Softmax • Noise Contrastive Estimation (NCE) • Negative Sampling
19. Word2Vec - expectations
20. Word2Vec - results
21. Word2Vec - results
22. Word2Vec - results
23. Word2Vec - basic building block • spam filtering • text classification • sentiment analysis • text search Training takes minutes or hours instead of days or weeks
24. Paragraph Vector • proposed by Quoc Le and Tomas Mikolov • models inspired by Word2Vec • distributed representation of sentences, paragraphs and documents
25. Paragraph Vector - Distributed Memory Source: Distributed Representations of Sentences and Documents, https://arxiv.org/pdf/1405.4053v2.pdf
26. Paragraph Vector - Distributed Bag of Words Source: Distributed Representations of Sentences and Documents, https://arxiv.org/pdf/1405.4053v2.pdf
27. Paragraph Vector - inference • fix projection layer word weights (W) • fix softmax weights • update paragraph representation (D) Source: Distributed Representations of Sentences and Documents, https://arxiv.org/pdf/1405.4053v2.pdf
28. Image
29. Image recognition Source: https://www.bdti.com/InsideDSP/2017/06/29/Microsoft
30. Convolutional Neural Network
31. Convolutions
32. Pooling
33. ReLU - Rectified Linear Unit • simple non-linear activation • fast to compute • fast to compute derivative • solves vanishing gradient problem • dead neurons
34. Leaky ReLU
35. Feature extraction
36. Distributed representation - example
37. Distributed representation - example
38. Modern architectures Source: http://knowyourmeme.com/memes/we-need-to-go-deeper
39. GoogLeNet - Inception Network Source: https://hacktilldawn.com/2016/09/25/inception-modules-explained-and-implemented/
40. Inception module Source: https://hacktilldawn.com/2016/09/25/inception-modules-explained-and-implemented/
41. Degradation problem Source: Deep Residual Learning for Image Recognition, https://arxiv.org/pdf/1512.03385v1.pdf
42. ResNet Source: Deep Residual Learning for Image Recognition, https://arxiv.org/pdf/1512.03385v1.pdf
43. Residual block Source: Deep Residual Learning for Image Recognition, https://arxiv.org/pdf/1512.03385v1.pdf learn F(x) and provide identity (x) with shortcut connection
44. plain network vs ResNet Source: Deep Residual Learning for Image Recognition, https://arxiv.org/pdf/1512.03385v1.pdf
45. Artistic Style Transfer
46. Artistic Style Transfer Source: A Neural Algorithm of Artistic Style, 2015, Gatys et al.
47. Textures Source: Texture Synthesis Using Convolutional Neural Networks, 2015, Gatys et al.
48. Textures Source: Texture Synthesis Using Convolutional Neural Networks, 2015, Gatys et al.
49. Style transfer Source: A Neural Algorithm of Artistic Style, 2015, Gatys et al.
50. Video
51. Youtube8M Source: https://research.google.com/youtube8m/ Large scale video dataset announced by Google in 2016: • 7+ million videos • 4700+ different labels • 450,000 hours of video How it was possible to distribute such large dataset?
52. Frames Source: http://www.mediacollege.com/video/frame-rate/ To understand video it is not required to have information about all frames. Sample frames from video. Distributed representation of each frame instead of raw image.
53. Dataset Frame level features • 1 frame per second • 1024 dimensional visual features (Inception Network) • 128 dimensional audio features (VGG inspired network) • quantized (8 bit ber unit) Dataset size: 1.71 Tb Video level features • aggregated frame level features using averaging and simple statistics
54. Summary
55. Thank you for your time!