Gorgonia: Primitives for Building Neural Networks in Go

Lightning talk given at Golang Sydney September 2016. This are the slides that introduce Gorgonia which was published earlier that day.
Gorgonia: https://github.com/chewxy/gorgonia

https://www.youtube.com/watch?v=lLkhC6Ehre4

Transcript

1. Gorgonia
   go get -u github.com/chewxy/gorgonia
   

   View Slide

2. Gorgonia
   ¡  Library that provides the primitives to dynamically
   build neural networks and assorted machine
   learning algorithms
   ¡  Is like Theano and Tensorflow, but written in Go.
   Follow @chewxy on Twitter
   

   View Slide

3. Using Gorgonia
   ¡  Build a expression graph
   ¡  Perform symbolic differentiation
   ¡  Perform automatic differentiation
   Follow @chewxy on Twitter
   

   View Slide

4. Using Gorgonia
   ¡  Build an expression graph
   g  :=  NewGraph()  
   x  :=  NewMatrix(g,  Float64,  WithName("x"),  WithShape(5,5))  
   w  :=  NewVector(g,  Float64,  WithName("w"),  WithShape(5))  
   xw  :=  Must(Mul(x,w))  
   act  :=  Must(Sigmoid(xw))  
   Follow @chewxy on Twitter
   

   View Slide

5. Neural Network Refresher
   Courtesy Michael Nielsen
   Follow @chewxy on Twitter
   

   View Slide

6. Neural Network Refresher
   Courtesy Michael Nielsen
   x
   Follow @chewxy on Twitter
   

   View Slide

7. Neural Network Refresher
   Courtesy Michael Nielsen
   w
   Follow @chewxy on Twitter
   

   View Slide

8. Neural Network Refresher
   Courtesy Michael Nielsen
   wx
   Follow @chewxy on Twitter
   

   View Slide

9. Neural Network Refresher
   Courtesy Michael Nielsen
   act
   Follow @chewxy on Twitter
   

   View Slide

10. Expression Graph
    Follow @chewxy on Twitter
    

    View Slide

11. Using Gorgonia
    ¡  Symbolic differentiation
    cost  :=  Must(Sum(act))  //or  mean  if  you  do  batch  size  stuff  
    grads,  err  :=  Grad(cost,  w)  
    //  grads[0]  is    
    ∂cost
    ∂w
    Follow @chewxy on Twitter
    

    View Slide

12. Using Gorgonia
    ¡  Executing Expression Graph
    ¡  Two VMs available:
    ¡  TapeMachine  
    ¡  LispMachine  
    ¡  Created for different purposes
    Follow @chewxy on Twitter
    

    View Slide

13. Using Gorgonia
    ¡  TapeMachine
    ¡  Optimized for compile-once-execute-many-times
    expression graphs
    ¡  LispMachine
    ¡  Good for dynamically changing expression graphs
    (like LSTMs or GRUs)
    ¡  Mix and match running modes!
    Follow @chewxy on Twitter
    

    View Slide

14. Using Gorgonia
    ¡  Executing a Expression Graph
    prog,  locMap  :=  Compile(g)  
    m  :=  NewTapeMachine()  
    Let(x,  xVal)  
    Let(w,  wVal)  //  or  could  be  init  along  the  Node  
    if  err  :=  m.RunAll();  err  !=  nil  {  
     log.Fatal(err)  
    }
    Follow @chewxy on Twitter
    

    View Slide

15. Using Gorgonia
    prog,  locMap  :=  Compile(g)  
    m  :=  NewTapeMachine()  
    Let(x,  xVal)  
    Let(w,  wVal)  
    if  err  :=  m.RunAll();  err  !=  nil  {  
     log.Fatal(err)  
    }
    f  =  theano.function(inputs=[x],  
    outputs=grads)  
    f(xVal)  
    Follow @chewxy on Twitter
    

    View Slide

16. Using Gorgonia
    prog,  locMap  :=  Compile(g)  
    m  :=  NewTapeMachine()  
    Let(x,  xVal)  
    Let(w,  wVal)  
    if  err  :=  m.RunAll();  err  !=  nil  {  
     log.Fatal(err)  
    }
    f  =  theano.function(inputs=[x],  
    outputs=grads)  
    f(xVal)  
    More verbose, but
    more mental clarity
    with regards to the
    running mode of the
    expression*
    *pure opinion(probably biased) by the creator of Gorgonia
    Follow @chewxy on Twitter
    

    View Slide

17. Using Gorgonia
    ¡ The Program
    0    loadArg  8208ce780  to  CPU0  
    1    loadArg  8208ce840  to  CPU1  
    2    Alloc  Vector  Float64  CPU2  
    3    A  ×  b  [CPU0  CPU1]  CPU2  [Matrix  Float64,  Vector  Float64]  true  false  true  
    4    Do  Batched  BLAS  
    5    sigmoid  [CPU2]  CPU2  [Vector  Float64]    false  true  false  
    6    Σ[0]  [CPU2]  CPU3  [Vector  Float64]    false  false  false  
    7    const  1  []  CPU4  []      false  false  false  
    8    SizeOf=5  [CPU2]  CPU5  [Vector  Float64]    false  false  false  
    9    Repeat[0][CPU4  CPU5]  CPU6  [Float64,  Float64]    false  false  false  
    10  -­‐  [CPU4  CPU2]  CPU7  [Float64,  Vector  Float64]  false  false  false  
    11  ⊙  [CPU2  CPU7]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false  
    12  ⊙  [CPU2  CPU6]  CPU2  [Vector  Float64,  Vector  Float64]  false  true  false  
    13  Alloc  Matrix  Float64  CPU8  
    14  a  ⊗  b  [CPU2  CPU1]  CPU8  [Vector  Float64,  Vector  Float64]  true  false  true  
    15  Alloc  Vector  Float64  CPU9  
    16  Aᵀ  ×  b  [CPU0  CPU2]  CPU9  [Matrix  Float64,  Vector  Float64]  true  false  true  
    Follow @chewxy on Twitter
    

    View Slide

18. Things I wrote with Gorgonia
    ¡  "Compose" music given inputs from music by John
    Williams, Murray Gold, and Bear McCreary
    ¡  Neural Tensor Machine
    ¡  Generative Models in general
    ¡  (also all the not-so-cool stuff like linear regression*)
    * The world's oldest machine learning algorithm, over 100years old and still going strong!
    Follow @chewxy on Twitter
    

    View Slide

19. Things Gorgonia Can't Do
    (Yet)
    ¡  Convolutional neural networks
    ¡  Im2Col not implemented yet
    ¡  Conv2D not implemented yet
    Follow @chewxy on Twitter
    

    View Slide

20. The Ask
    ¡  Help out Gorgonia's project (released today!)
    ¡  Test coverage is abysmal at 50%. 90% would be nice.
    ¡  Weird ASM bug wrt []float32  
    ¡  Add more fun Ops.
    ¡  Distributed Computing (tried to implement 3 times!!)
    ¡  Low-level Optimization (esp. re: malloc and gc)
    https://github.com/chewxy/gorgonia
    Follow @chewxy on Twitter
    

    View Slide