DyND - Enabling Complex Analytics Across the Language Barrier, PyData London 2016

1. DYND: ENABLING COMPLEX ANALYTICS ACROSS THE LANGUAGE BARRIER IRWIN ZAID, CONTINUUM ANALYTICS
2. Part 1: The What and Why of DyND Part 2: DyND Snippets DYND: ENABLING COMPLEX ANALYTICS ACROSS THE LANGUAGE BARRIER
3. PYDATA LONDON 2016 What is DyND? DyND aims to be a modern library for array-oriented computing
4. PYDATA LONDON 2016 What is DyND? ‣ A library, not a language ‣ Helps do computation with arrays ‣ “Modern” means “uses software engineering practices from today, solving problems people have had for a while”
5. PYDATA LONDON 2016 What is DyND really? A type system and a callable object that operate together with an array container, engineered in C++ and bound to dynamic languages like Python
6. PYDATA LONDON 2016 What is DyND not? ‣ DyND is not NumPy 2.0
 - NumPy is really good at what it was designed to do: broadcasting 
 computation and stride-based manipulation
 - NumPy may not be “replaceable” — Fortran is still pretty widely used ‣ DyND is not a JIT for Python
 - See Numba for that ‣ DyND is not Boost.MultiArray
 - DyND is dynamic C++, templates are a hidden detail
7. PYDATA LONDON 2016 It’s 2016. Why write another array library? Because there are problems that are not being otherwise solved
8. PYDATA LONDON 2016 It’s 2016. Why write another array library? ‣ Example: Missing data
 - Values that may or may not be present
 - The masked arrays of numpy.ma are not sufficient, there is overhead related 
 to how the masked array is stored and NumPy is not always consistent 
 with how it treats mask arrays
 - Discussed at length in 2011, remains unsolved in 2016
9. PYDATA LONDON 2016 It’s 2016. Why write another array library? ‣ Example: Variable-length strings
 - NumPy can only efficiently represent strings of a predefined length
 - Variable-length strings have to be stored as Python objects
10. PYDATA LONDON 2016 It’s 2016. Why write another array library? ‣ Example: Custom types
 - NumPy dtypes are too primitive
 - How does one represent categorical data? Ragged dimensions? GPU data?
 - Cannot define user overloads on ufuncs, e.g. string concatenation
11. PYDATA LONDON 2016 It’s 2016. Why write another array library? ‣ Example: NumPy without the “Py”
 - Sometimes we don’t want to use Python
 - Why not have a representation of data that can go between several 
 languages? (R, Julia, Javascript, …)
12. PYDATA LONDON 2016 Four Traits of DyND ‣ Expressive ‣ Generic ‣ Extendable ‣ Pluggable
 
 

13. PYDATA LONDON 2016 Expressive ‣ DyND implements Datashape as its type system
 - A structured data description language, http://datashape.pydata.org
 type dimension * dtype
 
 var * int32
 3 * string
 4 * float64 Datashape: Struct type {x: int32, y: string, z: float64} Tabular data var * {x: int32, y: string, z: float64}
14. PYDATA LONDON 2016 Generic ‣ DyND’s type, callable, and array objects are reference- counted smart pointers that dynamically interpret data ‣ Types can be parameterized on other types
 - N * T, var[T], option[T] ‣ Callables can be transformed (in a functional sense) from inner operations to higher-order patterns 
 - Define the innermost operation, then build out the behavior you want with 
 predefined generic patterns
 - nd::functional::elwise([](int x, int y) { return x + y; });
15. PYDATA LONDON 2016 Extendable ‣ Types and callables are first-class objects that users should create directly
16. PYDATA LONDON 2016 Extendable ‣ Types and callables are first-class objects that users should create directly
17. PYDATA LONDON 2016 Pluggable ‣ DyND supports plug-in libraries
 - Define custom types and callables (or namespaces thereof) directly ‣ Use nd::set(“my_amazing_callable”, f) for a custom callable or nd::set(“my_amazing_namespace”, {{“my_amazing_callable”, f}, {“my_other_amazing_callable”, g}}) for a custom namespace ‣ Callables are dynamically propagated to Python, entirely removing the need for any user wrapper code
18. Part 1: The What and Why of DyND Part 2: DyND Snippets DYND: ENABLING COMPLEX ANALYTICS ACROSS THE LANGUAGE BARRIER
19. PYDATA LONDON 2016 Types ‣ Types are instances of simple classes
 - Write a class, get a type ‣ Types expose dynamic features to arrays
 - Either properties, like .real or .imag, or behavior, like .conj() ‣ Types can be kinds or patterns
 - Int, Scalar, Fixed, or Any; Fixed * T or (N * T, T) -> T
20. PYDATA LONDON 2016 Metadata and Data ‣ Array metadata can describe data other than strided
 - Offset (tuple or struct), indirect (pointer), ragged (variable-sized dimensions) ‣ Array data is poolable or allocatable in custom memory spaces
 - Variable-sized strings or dimensions; CUDA
21. PYDATA LONDON 2016 Fundamental Types
22. PYDATA LONDON 2016 Dimension Types
23. PYDATA LONDON 2016 Aggregate Types
24. PYDATA LONDON 2016 Option Type
25. PYDATA LONDON 2016 Symbolic Types
26. PYDATA LONDON 2016 Callable and Functionals ‣ Share functions alongside data
 - Callables are first-class objects that can be dynamically published ‣ Enable user-defined functions with generic patterns
 - Functionals like apply, elwise, reduction, multidispatch, outer, neighborhood, 
 and rolling transform one callable into another ‣ Built-in callable are overloadable
 - Users can define +, -, *, /, … for custom types
27. PYDATA LONDON 2016 Elementwise Functional
28. PYDATA LONDON 2016 Reduction Functional
29. PYDATA LONDON 2016 Multidispatch Functional
30. PYDATA LONDON 2016 Option Operations
31. PYDATA LONDON 2016 JSON Processing
32. PYDATA LONDON 2016 Thanks to… Mark Wiebe Ian Henriksen Stefan Krah Irwin Zaid
33. PYDATA LONDON 2016 Thanks to…
34. PYDATA LONDON 2016 Get DyND! conda install dynd-python -c dynd/channel/dev