• Other bits and bobs: • Back propagate errors • Learning method (SGD, AdaGrad, RMSprop, L-‐BFGS…) • Errors (Cross Entropy, meansquare errors…) • Unique layers (ConvoluDon, RBM, etc …) • Mix and Match bits • All the above is just simple manipulaDon of matrix code
• map[coordinate]float64 • type sparseMat64 struct { d []float64 r []uint32 c []uint32 } • type sparseMat64 []*cell • type sparseMat64 [][MAXCOL]float64 type coordinate struct { r, c int } type cell struct { col int, val float64 }
• map[coordinate]float64 • type sparseMat64 struct { d []float64 r []uint32 c []uint32 } • type sparseMat64 []*cell • type sparseMat64 [][MAXCOL]float64 type coordinate struct { r, c int } type cell struct { col int, val float64 } Most flexible Most useful for Hadamard Fastest, good for small MAXCOL Fastest to build
chunkSize := len(mat) / threads costChan := make(chan *cost) for chunk := 0; chunk < threads; chunk++ { var start int = chunk * chunkSize var end int if start+chunkSize > len(mat) { end = len(mat) } else { end = start + chunkSize } go nn.costFn(mat[start:end], chunkSize, costChan) }
. // This loop took a helluva long time 226 4.93s 4.93s for _, a := range actives { 227 29.47s 29.47s layer2Grad[i][a] += delta * hiddenAct[a] 228 1.02mins 1.02mins hiddenActGrad[a] += delta * nn.weights2[i][a] 229 . . }
a scalar by a vector void avx_scale64(const double *s, const size_t n, double *x, double *z) { static const size_t elements = 4; // with double precision, the processor can only handle 4 at a time. Maybe switch to float32? const size_t end = n / elements; const __m256d vy = _mm256_broadcast_sd(s); // broadcast a scalar into vy // load x and z into AVX dedicated registers __m256d *vz = (__m256d *)z; __m256d *vx = (__m256d *)x; for (size_t i=0; i < end; ++i) { vz[i] = _mm256_mul_pd(vx[i], vy); } } */
int) []float64 { // pad it so that at most there will be 64 items extra -‐ worse comes to worst, there are a bunch of 0s to add or mul size := length * 64 ptr := C._mm_malloc((C.size_t)(size), 64) header := reflect.SliceHeader{ Data: uintptr(unsafe.Pointer(ptr)), Len: length, Cap: length, } goSlice := *(*[]float64)(unsafe.Pointer(&header)) return goSlice } // somewhere later defer C._mm_free(unsafe.Pointer(&vec[0]))
for i := range rows { // calculate delta omitted // This loop took a helluva long time for _, a := range actives { layer2Grad[i][a] += delta * hiddenAct[a] hiddenActGrad[a] += delta * nn.weights2[i][a] } }
TEXT ·∙scale64(SB), 7, $0 MOVSD s+0(FP), X0 MOVQ x_data+8(FP), SI // move the first element to the indexing register MOVQ x_len+24(FP), BP // move the length into BP // check that there is more than 4 values in the slice SUBQ $4, BP JL remainders // jump to remainder // manually broadcast s into X0's lower bits, which is used for SIMD MOVLHPS X0, X0 loop: // load the first two elements, and then multiply by s (which is in X0) MOVUPD (SI), X2 MULPD X0, X2 // load second two elements, then multiply by s MOVUPD 16(SI), X4 MULPD X0, X4 // move the values back to the original positions MOVUPD X2, (SI) MOVUPD X4, 16(SI) // now that we are done with the first 4 elements, update the pointers to the top of the next 4 elements ADDQ $32, SI SUBQ $4, BP // the len is now less 4 JGE loop remainders: ADDQ $4, BP // we subtracted earlier, remember??!!! JEQ done // if there are nothing left to process, we'll go to done remainingloop: // load element from x, and then multiply by s (which in X0) // TODO: check if there is a performance penalty on using SSE registers MOVSD (SI), X2 MULSD X0, X2 // save it back to the array MOVSD X2, (SI) // update the pointer ADDQ $8, SI DECQ BP JNE remainingloop done: RET
. testing: warning: no tests to run PASS BenchmarkGoScale64-‐8 1000000 1339 ns/op BenchmarkScale64-‐8 5000000 329 ns/op BenchmarkGoAdd64-‐8 1000000 1711 ns/op BenchmarkAdd64-‐8 1000000 1473 ns/op ok github.com/chewxy/testideas3 6.567s func GoAdd64(a, b []float64) { for i, v := range a{ a[i] = v + b[i] } } Not much difference
Be-er algorithms • PrecalculaDng acDvaDons and caching§ • Binary weights (no mulDplicaDon, only addiDon)* *very very very very very new. I'm sDll playing with it
network in Go is fairly simple. • Building a performant neural network in Go? Only embark with quesDonable sanity • Irony: deploying Theano is hard. But all my opDmizaDons make deploying this equally hard. • A default built in, well opDmized matrix type would be really really really sweet
chewxy
syossan27
jbvincey
xgouchet
yuukis
sasagar
ivargrimstad
splcywolf
itotsum
whatyouhide
ydah
gokana
kokushin
addyosmani
jonrohan
danielanewman
jeffersonlam
mongodb
thoeni
deanohume
mclloyd
kneath
wjessup
chrislema
notwaldorf
![@chewxy on Twi-er // func scale64(s float64, a []float64)](https://files.speakerdeck.com/presentations/ef112cdc8e95411897272b08170219a1/slide_38.jpg){kind=link}
