Gorgonia: Primitives for Building Neural Networks in Go

Lightning talk given at Golang Sydney September 2016. This are the slides that introduce Gorgonia which was published earlier that day.
Gorgonia: https://github.com/chewxy/gorgonia

https://www.youtube.com/watch?v=lLkhC6Ehre4

Transcript

1. Gorgonia go get -u github.com/chewxy/gorgonia
2. Gorgonia ¡  Library that provides the primitives to dynamically build neural networks and assorted machine learning algorithms ¡  Is like Theano and Tensorflow, but written in Go. Follow @chewxy on Twitter
3. Using Gorgonia ¡  Build a expression graph ¡  Perform symbolic differentiation ¡  Perform automatic differentiation Follow @chewxy on Twitter
4. Using Gorgonia ¡  Build an expression graph g  :=  NewGraph()   x  :=  NewMatrix(g,  Float64,  WithName("x"),  WithShape(5,5))   w  :=  NewVector(g,  Float64,  WithName("w"),  WithShape(5))   xw  :=  Must(Mul(x,w))   act  :=  Must(Sigmoid(xw))   Follow @chewxy on Twitter
5. Neural Network Refresher Courtesy Michael Nielsen Follow @chewxy on Twitter
6. Neural Network Refresher Courtesy Michael Nielsen x Follow @chewxy on Twitter
7. Neural Network Refresher Courtesy Michael Nielsen w Follow @chewxy on Twitter
8. Neural Network Refresher Courtesy Michael Nielsen wx Follow @chewxy on Twitter
9. Neural Network Refresher Courtesy Michael Nielsen act Follow @chewxy on Twitter
10. Expression Graph Follow @chewxy on Twitter
11. Using Gorgonia ¡  Symbolic differentiation cost  :=  Must(Sum(act))  //or  mean  if  you  do  batch  size  stuff   grads,  err  :=  Grad(cost,  w)   //  grads[0]  is     ∂cost ∂w Follow @chewxy on Twitter
12. Using Gorgonia ¡  Executing Expression Graph ¡  Two VMs available: ¡  TapeMachine   ¡  LispMachine   ¡  Created for different purposes Follow @chewxy on Twitter
13. Using Gorgonia ¡  TapeMachine ¡  Optimized for compile-once-execute-many-times expression graphs ¡  LispMachine ¡  Good for dynamically changing expression graphs (like LSTMs or GRUs) ¡  Mix and match running modes! Follow @chewxy on Twitter
14. Using Gorgonia ¡  Executing a Expression Graph prog,  locMap  :=  Compile(g)   m  :=  NewTapeMachine()   Let(x,  xVal)   Let(w,  wVal)  //  or  could  be  init  along  the  Node   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } Follow @chewxy on Twitter
15. Using Gorgonia prog,  locMap  :=  Compile(g)   m  :=  NewTapeMachine()   Let(x,  xVal)   Let(w,  wVal)   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } f  =  theano.function(inputs=[x],   outputs=grads)   f(xVal)   Follow @chewxy on Twitter
16. Using Gorgonia prog,  locMap  :=  Compile(g)   m  :=  NewTapeMachine()   Let(x,  xVal)   Let(w,  wVal)   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } f  =  theano.function(inputs=[x],   outputs=grads)   f(xVal)   More verbose, but more mental clarity with regards to the running mode of the expression* *pure opinion(probably biased) by the creator of Gorgonia Follow @chewxy on Twitter
17. Using Gorgonia ¡ The Program 0    loadArg  8208ce780  to  CPU0   1    loadArg  8208ce840  to  CPU1   2    Alloc  Vector  Float64  CPU2   3    A  ×  b  [CPU0  CPU1]  CPU2  [Matrix  Float64,  Vector  Float64]  true  false  true   4    Do  Batched  BLAS   5    sigmoid  [CPU2]  CPU2  [Vector  Float64]    false  true  false   6    Σ[0]  [CPU2]  CPU3  [Vector  Float64]    false  false  false   7    const  1  []  CPU4  []      false  false  false   8    SizeOf=5  [CPU2]  CPU5  [Vector  Float64]    false  false  false   9    Repeat[0][CPU4  CPU5]  CPU6  [Float64,  Float64]    false  false  false   10  -­‐  [CPU4  CPU2]  CPU7  [Float64,  Vector  Float64]  false  false  false   11  ⊙  [CPU2  CPU7]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false   12  ⊙  [CPU2  CPU6]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false   13  Alloc  Matrix  Float64  CPU8   14  a  ⊗  b  [CPU2  CPU1]  CPU8  [Vector  Float64,  Vector  Float64]  true  false  true   15  Alloc  Vector  Float64  CPU9   16  Aᵀ  ×  b  [CPU0  CPU2]  CPU9  [Matrix  Float64,  Vector  Float64]  true  false  true   Follow @chewxy on Twitter
18. Things I wrote with Gorgonia ¡  "Compose" music given inputs from music by John Williams, Murray Gold, and Bear McCreary ¡  Neural Tensor Machine ¡  Generative Models in general ¡  (also all the not-so-cool stuff like linear regression*) * The world's oldest machine learning algorithm, over 100years old and still going strong! Follow @chewxy on Twitter
19. Things Gorgonia Can't Do (Yet) ¡  Convolutional neural networks ¡  Im2Col not implemented yet ¡  Conv2D not implemented yet Follow @chewxy on Twitter
20. The Ask ¡  Help out Gorgonia's project (released today!) ¡  Test coverage is abysmal at 50%. 90% would be nice. ¡  Weird ASM bug wrt []float32   ¡  Add more fun Ops. ¡  Distributed Computing (tried to implement 3 times!!) ¡  Low-level Optimization (esp. re: malloc and gc) https://github.com/chewxy/gorgonia Follow @chewxy on Twitter