Go for Data Science

Transcript

1. Go for Data Science. 2018 Stockholm Go Conference. ROBIN BARTHOLDSON @ FORMULATE
2. Go is great for data science.
3. P( ) min getting a data science job in 2018
4. 14% 17% 20% 27% 30% 33% 39% 51% 64% 72% Tableau Hive Matlab Spark SAS Java Hadoop SQL R Python Most frequently occuring skills in data science job postings 2017 SOURCE: Glassdoor
5. Here is a very simple truth that every other possible argument boils down to: data science is not software engineering. “ “ SOURCE: Quora
10. WARNING Possibly partly unfair characterizations of other languages ahead* *Especially R. !
11. ✅ Type safety.
12. Data stuff Other stuff (the fun stuff with algos and so)
13. > x <- matrix(1:9, 3) > idx <- c(TRUE, FALSE, TRUE) > class(x[idx,]) [1] matrix
14. > x <- matrix(1:9, 3) > idx <- c(TRUE, FALSE, FALSE) > class(x[idx,]) [1] integer
15. > x <- matrix(1:9, 3) > idx <- c(TRUE, FALSE, FALSE) > class(x[idx,,drop=FALSE]) [1] matrix
16. ✅ Performance without obscurity.
17. for jj = 1:numCities B = zeros(numCities,numAircraft); for kk = 1:numAircraft B(:,kk) = A(ii-1,:)+(edgeCosts(:,jj,kk))’; end minNewPath = min(min(B)); A(ii,jj) = min(A(ii-1,jj),minNewPath); end Atmp = A(ii-1,:); A(ii,:) = min([A(ii-1,:);(min(min(Atmp(ones(numCities, 1),:,ones(numAircraft,1)) + edgeCostSwap,[],2),[],3))']); Example borrowed from Brendan Tracey. 10x
18. for (i in 1:n) { acc <- acc + x[i] } for i in range(0,n): acc += x[i] for i := range x { acc += x[i] } R Python 97 ms 18 ms 1 ms Summing 1 million float64s
19. ✅ Deployment made easy ;).
20. Virtual environments Compiling C++/ Fortran packages from scratch 2.7 vs 3.X 100MB+ interpreter $ go get -u github.com/amazing/pkg $ go build $ cat Dockerfile FROM scratch COPY bin . CMD [“./bin”]
21. ✅ Got a parallellizable algo? Np.
22. ✅ Idioms that make life easier. • Composition • Error handling • Cancellation
24. Person with the sexiest occupation in the 21st century: But can you really do X in Go and be productive?
25. Explore Model Deploy
26. Explore Model Deploy
27. Data frame Plots §  github.com/kniren/gota §  github.com/tobgu/qframe §  github.com/gonum/exp/dframe §  gonum.org/v1/gonum/plot §  github.com/wcharczuk/go-chart Demo:ed next Explore
28. Explore Model Deploy
29. Case study: from Python to Go Glmnet §  Variable selection and regularization §  Backend written in fixed width Fortran90 §  Interfaces with R, Python, MatLab 1 2 3 Baseline: Python code Rewrote the Fortran wrapper using CGO Translated the Fortran code into Go Autogenerating testdata and treating the reference implementations as golden data
30. SciPy/NumPy Gonum Sci-Kit learn §  pa-m/sklearn §  sjwhitworth/golearn §  [godoc.org] Tensorflow/ Pytorch/Caffe Gorgonia ≈ ≈ ≈ Model
31. BLAS LAPACK mat optimize integrate floats diff stat graph Model
32. Explore Model Deploy
33. “Scoring” engine Data pipelines Examples §  Apache Beam §  SciPipe §  Gleam §  dmitryikh/ leaves §  Tensorflow Go bindings Deploy §  This is where Go really shines –  “Just use the standard library” –  Apply the common idioms §  Model for (stateful) stream and batch processing §  Run on Flink, Spark, Dataflow etc — write in e.g. Go, Java, Python §  Read and predict classes from models created in major tree libs §  XGBoost §  LightGBM §  Scikit-learn
34. Person with the sexiest occupation in the 21st century: But can you really do X in Go and be productive? Yep.
35. • More models and algorithms • More mature data frames + plotting sugar • Portability/interoperability (e.g. Apache Arrow) (• Performance, e.g. SIMD and/or inline assembly)
36. Thanks!