DyND.pdf - Speaker Deck

DyND.pdf

by Irwin Zaid

Slide 1

Slide 1 text

DyND C++ dynamic array library, with Python exposure (http://github.com/libdynd) Irwin Zaid September 18, 2015

Slide 2

Slide 2 text

This Talk • Discussion at a high level – No implementation details • Code examples

Slide 3

Slide 3 text

Data?

Slide 4

Slide 4 text

Data? Missing Values Composite Structures Heterogeneous Storage The Usual Case Irregularly Shaped

Slide 5

Slide 5 text

Principles of DyND • Store data in its native layout • Enable computation with generic patterns • Be a cross-language intermediary (C++, Python, R?, Julia?, JavaScript?, …)

Slide 6

Slide 6 text

Types, Metadata, and Data • Types are instances of simple classes – Write a class, get a type • Types expose dynamic features to arrays – Either properties, like .real() or .imag(), or behavior, like .conj() • Types can be kinds, patterns, or constraints – Int, Scalar, Fixed, or Any; Fixed * T or (N * T, T) -> T; C[3 * 2 * float64] • Array metadata can describe data other than strided – Offset (tuple or struct), indirect (pointer), ragged (variable-sized dimensions) • Array data is poolable or allocatable in custom memory spaces – Variable-sized strings or dimensions; CUDA

Slide 7

Slide 7 text

Fundamental Types

Slide 8

Slide 8 text

Dimension Types

Slide 9

Slide 9 text

Composite Types

Slide 10

Slide 10 text

Option Type

Slide 11

Slide 11 text

Symbolic Types

Slide 12

Slide 12 text

Callables and Functionals • Share functions alongside data – Callables are first-class objects that can be dynamically published • Enable user-defined functions with generic patterns – Functionals like apply, elwise, reduction, multidispatch, outer, neighborhood, and rolling transform one callable into another • Built-in callables are overloadable – Users can define +, -, *, /, … for custom types • Write once, retarget later – Capture LLVM IR from C++ at compile-time

Slide 13

Slide 13 text

Elementwise Functional

Slide 14

Slide 14 text

Reduction Functional

Slide 15

Slide 15 text

Multidispatch Functional

Slide 16

Slide 16 text

JSON Processing

Slide 17

Slide 17 text

Option Operations

Slide 18

Slide 18 text

LLVM Support

Slide 19

Slide 19 text

Just-in-Time and Ahead-of-Time • JITing happens automatically with Numba if it is importable – Previous examples of user-defined functions can then drop the return type, it will be deduced at runtime • AOT planned, it will work by specifying the argument types of user-defined functions