Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gorgonia: Primitives for Building Neural Networks in Go

Xuanyi
September 15, 2016

Gorgonia: Primitives for Building Neural Networks in Go

Lightning talk given at Golang Sydney September 2016. This are the slides that introduce Gorgonia which was published earlier that day.
Gorgonia: https://github.com/chewxy/gorgonia

https://www.youtube.com/watch?v=lLkhC6Ehre4

Xuanyi

September 15, 2016
Tweet

More Decks by Xuanyi

Other Decks in Programming

Transcript

  1. Gorgonia ¡  Library that provides the primitives to dynamically build

    neural networks and assorted machine learning algorithms ¡  Is like Theano and Tensorflow, but written in Go. Follow @chewxy on Twitter
  2. Using Gorgonia ¡  Build a expression graph ¡  Perform symbolic

    differentiation ¡  Perform automatic differentiation Follow @chewxy on Twitter
  3. Using Gorgonia ¡  Build an expression graph g  :=  NewGraph()

      x  :=  NewMatrix(g,  Float64,  WithName("x"),  WithShape(5,5))   w  :=  NewVector(g,  Float64,  WithName("w"),  WithShape(5))   xw  :=  Must(Mul(x,w))   act  :=  Must(Sigmoid(xw))   Follow @chewxy on Twitter
  4. Using Gorgonia ¡  Symbolic differentiation cost  :=  Must(Sum(act))  //or  mean

     if  you  do  batch  size  stuff   grads,  err  :=  Grad(cost,  w)   //  grads[0]  is     ∂cost ∂w Follow @chewxy on Twitter
  5. Using Gorgonia ¡  Executing Expression Graph ¡  Two VMs available:

    ¡  TapeMachine   ¡  LispMachine   ¡  Created for different purposes Follow @chewxy on Twitter
  6. Using Gorgonia ¡  TapeMachine ¡  Optimized for compile-once-execute-many-times expression graphs

    ¡  LispMachine ¡  Good for dynamically changing expression graphs (like LSTMs or GRUs) ¡  Mix and match running modes! Follow @chewxy on Twitter
  7. Using Gorgonia ¡  Executing a Expression Graph prog,  locMap  :=

     Compile(g)   m  :=  NewTapeMachine()   Let(x,  xVal)   Let(w,  wVal)  //  or  could  be  init  along  the  Node   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } Follow @chewxy on Twitter
  8. Using Gorgonia prog,  locMap  :=  Compile(g)   m  :=  NewTapeMachine()

      Let(x,  xVal)   Let(w,  wVal)   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } f  =  theano.function(inputs=[x],   outputs=grads)   f(xVal)   Follow @chewxy on Twitter
  9. Using Gorgonia prog,  locMap  :=  Compile(g)   m  :=  NewTapeMachine()

      Let(x,  xVal)   Let(w,  wVal)   if  err  :=  m.RunAll();  err  !=  nil  {    log.Fatal(err)   } f  =  theano.function(inputs=[x],   outputs=grads)   f(xVal)   More verbose, but more mental clarity with regards to the running mode of the expression* *pure opinion(probably biased) by the creator of Gorgonia Follow @chewxy on Twitter
  10. Using Gorgonia ¡ The Program 0    loadArg  8208ce780  to  CPU0

      1    loadArg  8208ce840  to  CPU1   2    Alloc  Vector  Float64  CPU2   3    A  ×  b  [CPU0  CPU1]  CPU2  [Matrix  Float64,  Vector  Float64]  true  false  true   4    Do  Batched  BLAS   5    sigmoid  [CPU2]  CPU2  [Vector  Float64]    false  true  false   6    Σ[0]  [CPU2]  CPU3  [Vector  Float64]    false  false  false   7    const  1  []  CPU4  []      false  false  false   8    SizeOf=5  [CPU2]  CPU5  [Vector  Float64]    false  false  false   9    Repeat[0][CPU4  CPU5]  CPU6  [Float64,  Float64]    false  false  false   10  -­‐  [CPU4  CPU2]  CPU7  [Float64,  Vector  Float64]  false  false  false   11  ⊙  [CPU2  CPU7]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false   12  ⊙  [CPU2  CPU6]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false   13  Alloc  Matrix  Float64  CPU8   14  a  ⊗  b  [CPU2  CPU1]  CPU8  [Vector  Float64,  Vector  Float64]  true  false  true   15  Alloc  Vector  Float64  CPU9   16  Aᵀ  ×  b  [CPU0  CPU2]  CPU9  [Matrix  Float64,  Vector  Float64]  true  false  true   Follow @chewxy on Twitter
  11. Things I wrote with Gorgonia ¡  "Compose" music given inputs

    from music by John Williams, Murray Gold, and Bear McCreary ¡  Neural Tensor Machine ¡  Generative Models in general ¡  (also all the not-so-cool stuff like linear regression*) * The world's oldest machine learning algorithm, over 100years old and still going strong! Follow @chewxy on Twitter
  12. Things Gorgonia Can't Do (Yet) ¡  Convolutional neural networks ¡ 

    Im2Col not implemented yet ¡  Conv2D not implemented yet Follow @chewxy on Twitter
  13. The Ask ¡  Help out Gorgonia's project (released today!) ¡ 

    Test coverage is abysmal at 50%. 90% would be nice. ¡  Weird ASM bug wrt []float32   ¡  Add more fun Ops. ¡  Distributed Computing (tried to implement 3 times!!) ¡  Low-level Optimization (esp. re: malloc and gc) https://github.com/chewxy/gorgonia Follow @chewxy on Twitter