The Tensor (n-dimensional array) is the principal software abstraction for designing and coupling the different components of modern machine learning systems. Unfortunately, writing code using popular tensor libraries is very challenging. First, the libraries expose the developers to a low-level, physical memory model, forcing them to translate between their semantic mental model and the physical model continuously. Second, the broadcast semantics between tensors of different dimensions are not robust and lead to surprising errors. Finally, in absence of a common language to specify tensor shapes explicitly, developers are compelled to write adhoc comments and woefully cryptic code.
In this talk,
We present a new language to specify tensor shapes and their transformations, by attaching "semantic" names to individual tensor dimensions.
The shape annotations are library-agnostic, improve code readability, and make it easy to write semantic shape assertions and transformations.
We present the tsalib library (http://github.com/ofnote/tsalib) which integrates named shape annotations into existing Python tensor programs seamlessly using optional type annotations. With examples (self-attention in Transformers, convolution-sequence in Resnets), we illustrate how shape annotations improve code readability dramatically and even reduce the code size.
We also discuss a tool, "tsanley" (http://github.com/ofnote/tsanley), for Python code, which (a) helps catch tricky shape errors at runtime and (b) improves third-party code understanding by annotating code with shape annotations.