Sum Product Networks: A New Deep Architecture
! Hoifung Poon and Pedro Domingos (2011) ! St. Louis Machine Learning & Data Science March 20, 2013 ! ! Steven Borrelli [email protected] @stevendborrelli {twitter, github} 

No content

No content

http://deeplearningworkshopnips2010.files.wordpress.com/2010/09/nips10-workshop-tutorial-final.pdf Deep Learning 

Sum Product Network
Hoifung Poon and Pedro Domingos, 2011
! • Models with multiple layers of hidden variables allow for efficient inference in a much larger class of distributions. • However, deep architectures leads are difficult to learn and require approximate methods (MCMC, etc.) • Complex Networks have intractable Partition functions: 

Sum Product Network
Hoifung Poon and Pedro Domingos, 2011
! Mixture Feature Directed Acyclic Graph http://homes.cs.washington.edu/~pedrod/papers/nips12.pdf 

Sum Product Network
⊗ ⊕ 0.4 ⊗ ⊗ ⊗ 0.2 0.1 0.3 ⎯ X1 X2 X1 ⎯ X2 X1 X2 P(X) 1 1 0.4 1 0 0.2 0 1 0.1 0 0 0.3 P(X) = 0.4 ⋅ X1 ⋅ X2 ! + 0.2 ⋅ X1 ⋅ X2 ! + 0.1 ⋅ X1 ⋅ X2 ! + 0.3 ⋅ X1 ⋅ X2 ⎯ ⎯ ⎯ ⎯ 

Computing SPNs 

Mixture model vs. SPN N * 2N-1 2N-1 ~N 

Partition Function https://class.coursera.org/neuralnets-2012-001 

Image recognition: Google 

U of Toronto DRBM http://www.cs.toronto.edu/~hinton/science.pdf 

SPN Architecture http://homes.cs.washington.edu/~pedrod/papers/nips12.pdf 

Caltech 101 Results (MSE) 

Caltech 101 Image completion 64 SPN DBN Nearest Neighbor DBM PCA Original 

Potential Benefits over other Deep Architectures
! • Less complex engineering, exact inference due to tractability of partition function. • Order of magnitude faster in learning and inference • Better Mean Square Error, Precision/recall and qualitative results.