Deep Graph Convolutional Recommenders

Transcript

1. Deep Graph Convolutional Recommenders Rianne van den Berg, 13 January 2017 Joint work with Thomas Kipf and Max Welling (University of Amsterdam) @ ADS Deep Dive into Machine Learning Slides based on a talk by Thomas Kipf … … … Input Hidden layer Hidden layer ReLU Output ReLU (Source: Thomas Kipf’s blog)
2. Deep Graph Convolutional Recommenders Rianne van den Berg The success story of deep learning 2 Speech data Natural language processing (NLP)
3. Deep Graph Convolutional Recommenders Rianne van den Berg Deep Learning on Euclidean data: CNN (Animation by Vincent Dumoulin) (Source: Wikipedia) 3 Convolutional neural networks (CNNs): Local filters Weight sharing
4. Deep Graph Convolutional Recommenders Rianne van den Berg Convolutional neural networks on grids 4 (Animation by Vincent Dumoulin) Single CNN layer with 3x3 filter: Update for a single pixel: • Transform neighbours individually • Add everything up • Apply nonlinearity h1 h2 h3 h8 h0 h4 h7 h6 h5 W8 W1 W0 W2 W3 W4 W5 W6 W7 W(l) i h(l) i X i W(l) i h(l) i h(l+1) 0 = X i W(l) i h(l) i ! 4
5. Deep Graph Convolutional Recommenders Rianne van den Berg or Graph-structured data 5 … Input ReLU Real-world examples: • Social networks • World-wide-web • Protein-interaction networks • Citation networks • Knowledge graphs • … 5
6. Deep Graph Convolutional Recommenders Rianne van den Berg A naive approach: fully connected NN 6 • Take adjacency matrix and feature matrix • Concatenate them • Feed them into deep (fully connected) neural net • Done? Problems: • Huge number of parameters • Needs to be re-trained if number of nodes changes • Does not generalize across graphs … … Input Hidden layer Hidden layer ReLU ? We need weight sharing! => CNNs on graphs or “Graph Convolutional Networks” (GCNs) 6
7. Deep Graph Convolutional Recommenders Rianne van den Berg Regular vs. Irregular Graph-structured data 7 vs. 7
8. Deep Graph Convolutional Recommenders Rianne van den Berg Regular vs. Irregular Graph-structured data 8 Key differences: • Different number of neighbors → weight sharing issue vs. W8 W1 W0 W2 W3 W4 W5 W6 W7 • No natural orientation/order of neighbors 8
9. Deep Graph Convolutional Recommenders Rianne van den Berg Regular vs. Irregular Graph-structured data 9 vs. W8 W1 W0 W2 W3 W4 W5 W6 W7 9 Key differences: • Different number of neighbors → weight sharing issue • No natural orientation/order of neighbors
10. Deep Graph Convolutional Recommenders Rianne van den Berg Spatial filters for irregular graphs 10 10 Propagation rule: W0 hi W1 h(l+1) i = 0 @h(l) i W(l) 0 + X j2Ni 1 cij h(l) j W(l) 1 1 A (related idea was first proposed in Scarselli et al. 2009) Kipf & Welling arXiv:1609.02907 (2016)
11. Deep Graph Convolutional Recommenders Rianne van den Berg Model architecture 11 11 … … … Input Hidden layer Hidden layer ReLU Output ReLU (Source: Thomas Kipf’s blog) Input: • Features for nodes (optional), • Adjacency matrix containing all links Embeddings: • Representations that combine features of neighborhood • Neighborhood size depends on number of layers Kipf & Welling arXiv:1609.02907 (2016)
12. Deep Graph Convolutional Recommenders Rianne van den Berg 12 12 Standard approach: Graph-based regularization (“smoothness constraints”) with Assumption: connected nodes likely to share same label [Zhu et al., 2003] Semi-supervised node classification on graphs Kipf & Welling arXiv:1609.02907 (2016) Setting: labeled and unlabeled nodes Task: Predict node label of unlabeled nodes
13. Deep Graph Convolutional Recommenders Rianne van den Berg Semi-supervised classification on graphs 13 13 Embedding-based approaches Two-step pipeline: 1)Get embedding for every node 2)Train classifier on node embedding Examples: • DeepWalk [Perozzi et al., 2014], • node2vec [Grover & Leskovec, 2016] Problem: Embeddings are not optimized for classification task! Solution: Train graph-based classifier end-to-end using GCN Evaluate loss on labeled nodes only: set of labeled node indices label matrix GCN output (after softmax) Kipf & Welling arXiv:1609.02907 (2016)
14. Deep Graph Convolutional Recommenders Rianne van den Berg Graph auto-encoders: Link prediction 14 14 Kipf & Welling, NIPS Bayesian Deep Learning Workshop, 2016 Input: , X A Aij=1 Xi Xj Zi Zj Encoder Embeddings: Z = f(A, X) Task: Create node embeddings that can be combined to reproduce edges.
15. Deep Graph Convolutional Recommenders Rianne van den Berg Graph auto-encoders: Link prediction 15 15 Kipf & Welling, NIPS Bayesian Deep Learning Workshop, 2016 Task: Create node embeddings that can be combined to reproduce edges. Decoder Zi Zj Embeddings: Z = f(A, X) Output: ˜ A = g(Z) Ãij=1
16. Deep Graph Convolutional Recommenders Rianne van den Berg List Price: $105.00 Price: $92.37 78 used & new from $64.57 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 6. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) by Trevor Hastie (April 12, 2011) Average Customer Review: (84) In Stock List Price: $89.95 Price: $73.02 124 used & new from $34.99 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 7. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning) by Richard S. Sutton (March 1, 1998) Average Customer Review: (22) In Stock List Price: $75.00 Price: $67.77 57 used & new from $43.64 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 8. Neural Network Design (2nd Edition) by Martin T Hagan (September 1, 2014) Average Customer Review: (29) In Stock List Price: $30.00 Price: $28.21 38 used & new from $24.21 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 9. Pattern Recognition and Machine Learning (Information Science and Statistics) by Christopher Bishop (October 1, 2007) Average Customer Review: (129) In Stock List Price: $94.95 Price: $68.03 111 used & new from $45.90 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 10. Probabilistic Graphical Models: Principles and Techniques (Adaptive Computation and Machine Learning series) by Daphne Koller (July 31, 2009) Average Customer Review: (37) In Stock List Price: $120.00 Recommender systems 16 16 Your Amazon.com > Recommended for You > Books Recommendations Books Arts & Photography Audible Audiobooks Biographies & Memoirs Books on CD Business & Money Calendars Children's Books Christian Books & Bibles Comics & Graphic Novels Computers & Technology Cookbooks, Food & Wine Crafts, Hobbies & Home Deals in Books Education & Teaching Engineering & Transportation Gay & Lesbian Health, Fitness & Dieting History Humor & Entertainment Law Libros en español Literature & Fiction Medical Books Mystery, Thriller & Suspense Parenting & Relationships Politics & Social Sciences Reference Religion & Spirituality Romance Science & Math Science Fiction & Fantasy Self-Help Sports & Outdoors Teens Test Preparation Textbooks Travel These recommendations are based on items you own and more. view: All | New Releases | Coming Soon 1. Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques for Building Intelligent Systems by Aurélien Géron (March 25, 2017) Available for Pre-order List Price: $49.99 Price: $28.56 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 2. Network Science by Albert-László Barabási (August 5, 2016) Average Customer Review: (9) In Stock List Price: $59.99 Price: $56.99 51 used & new from $38.82 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 3. Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies (MIT Press) by John D. Kelleher (July 24, 2015) Average Customer Review: (21) In Stock List Price: $80.00 Price: $74.00 63 used & new from $61.50 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 4. Python Machine Learning by Sebastian Raschka (September 23, 2015) Average Customer Review: (94) In Stock List Price: $44.99 Price: $40.49 77 used & new from $29.49 I own it Not interested Rate this item Recommended because you purchased Deep Learning ( Fix this ) 5. Machine Learning: A Probabilistic Perspective (Adaptive Computation and Machine Learning series) by Kevin P. Murphy (August 24, 2012) Average Customer Review: (65) In Stock Your Amazon.com Benefits & Offers For You Your Browsing History Recommended For You Improve Your Recommendations Your Profile Learn More Departments Account & Lists Hello, rianne Orders Try Prime Cart 0 Browsing History rianne's Amazon.com Try Prime Books
17. Deep Graph Convolutional Recommenders Rianne van den Berg Recommender systems 17 17 Source: http://www.cse.msu.edu/~forsati/ xu2 xu1 xu3 xu4 xu5 xu6 xu7 xi1 xi2 xi3 xi4 xi5 xi6 User features Item features (demographics, browse history, search history, …) (keywords, content categories, ...)
18. Deep Graph Convolutional Recommenders Rianne van den Berg Recommender systems: bipartite graphs 18 18 ? = missing link xu2 xu1 xu3 xu4 xu5 xu6 xu7 xi1 xi2 xi3 xi4 xi5 xi6 xu2 xu1 xu4 xu3 xu5 xi2 xi1 xi4 xi3 1 5 3 2 3 1 3 5 4 4
19. Deep Graph Convolutional Recommenders Rianne van den Berg Recommender systems: Link prediction 19 19 Aij=1 Xi Xj Special properties: • Bipartite structure • Multiple types of links 1 5 3 2 3 1 3 Ru5,i4 = 5 4 4 xu2 xu1 xu4 xu3 xu5 xi2 xi1 xi4 xi3
20. Deep Graph Convolutional Recommenders Rianne van den Berg Recommender systems: Link prediction 20 20 Task: Create user/item node embeddings that can be combined to reproduce known ratings. zu2 zu1 zu4 zu3 zu5 zi2 zi1 zi4 zi3 Input: Output: Embeddings: Auto-encoder 1 5 3 2 3 1 3 Ru5,i4 = 5 4 4 xu2 xu1 xu4 xu3 xu5 xi2 xi1 xi4 xi3 1 5 2 3 1 3 Řu5,i4 = 5 4 4 ˇ R = g(Zu, Zi) Zu, Zi = f(Xu, Xi, R) Xu, Xi, R
21. Deep Graph Convolutional Recommenders Rianne van den Berg References 21 Blog post Thomas Kipf: Graph Convolutional Networks http://tkipf.github.io/graph-convolutional-networks Code on Github: http://github.com/tkipf/gcn Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, 2016: https://arxiv.org/abs/1609.02907 Kipf & Welling, Variational Graph Auto-Encoders, NIPS BDL Workshop, 2016: https://arxiv.org/abs/1611.07308 Contact: {R.vandenBerg2, T.N.Kipf}@uva.nl Project funded by SAP 21 … … … Input Hidden layer Hidden layer ReLU Output ReLU