First problem: Efficient Data Input “It’s Always About the Data” http://www.python.org/doc/essays/refcnt/ Reference Counting Essay May 1998 Guido van Rossum TableIO
1999 : Early SciPy emerges Discussions on the matrix-sig from 1997 to 1999 wanting a complete data analysis environment: Paul Barrett, Joe Harrington, Perry Greenfield, Paul Dubois, Konrad Hinsen, and others. Activity in 1998, led to increased interest in 1999.
! In response on 15 Jan, 1999, I posted to matrix-sig a list of routines I felt needed to be present and began wrapping / writing in earnest. On 6 April 1999, I announced I would be creating this uber-package which eventually became SciPy Gaussian quadrature 5 Jan 1999 cephes 1.0 30 Jan 1999 sigtools 0.40 23 Feb 1999 Numeric docs March 1999 cephes 1.1 9 Mar 1999 multipack 0.3 13 Apr 1999 Helper routines 14 Apr 1999 multipack 0.6 (leastsq, ode, fsolve, quad) 29 Apr 1999 sparse plan described 30 May 1999 multipack 0.7 14 Jun 1999 SparsePy 0.1 5 Nov 1999 cephes 1.2 (vectorize) 29 Dec 1999 Plotting??
Using Numeric circa 2000 Image from my PhD Thesis made using Python interface to DISLIN and using Numeric + hand- written C-extensions for inverting wave- equation.
Brief History Person Package Year Jim Fulton Matrix Object in Python 1994 Jim Hugunin Numeric 1995 Perry Greenfield, Rick White, Todd Miller Numarray 2001 Travis Oliphant NumPy 2005
Now an impressive community effort • Chuck Harris • Pauli Virtanen • Robert Kern • Warren Weckesser • Ralf Gommers • Mark Wiebe • Nathaniel Smith • Nathan Bell • Stefan van der Walt • Matthew Brett • Josef Perktold …
• This means other people. This will require sacrificing some of your ego to really listen!
• Someone will point out how you suck (listen to them, you probably do).
• Nurture empathy.
• Treat other people like they matter to you — the only successful way to do that is to actually care. This exposes you to get hurt — care about people anyway!
Zen of NumPy (ala import this) • strided is better than scattered • contiguous is better than strided • descriptive is better than imperative • array-oriented is better than object-oriented • broadcasting is a great idea • vectorized is better than an explicit loop • unless it’s too complicated --- then use Cython/Numba • think in higher dimensions
APL : the first array-oriented language • Appeared in 1964 • Originated by Ken Iverson • Direct descendants (J, K, Matlab) are still used heavily and people pay a lot of money for them • NumPy is a descendent APL J K Matlab Numeric NumPy Q
NumPy in Data Analytics • NumPy is a decent array object with some user- friendly features.
• NumPy “arrays of structures” can be used to handle arbitrary data. http://people.rit.edu/blbgse/pythonNotes/numpy.html First Name Last Name Score Dave Thomas 89.4 Tasha Hen 76.6 Cool Python 100 Stack Overflow 95.32 Py Py 75
Pandas is “Structure of Arrays” • Labels on the dimensions (indexes) • Easy manipulation of new columns • Missing Value handling • Time Series handling • General Split-Apply-Combine • Merge and Join • Integrated Plotting • Chained method calls the norm • Familiar to R users — more user-friendly features!
The most important part PEP 3118: Revising the buffer protocol Basically the “structure” of NumPy arrays as a protocol in Python itself to establish a memory-sharing standard between objects. It makes it possible for a heterogeneous world of powerful array-like objects outside of NumPy that communicate.
! Falls short in not defining a general data description language (DDL). http://python.org/dev/peps/pep-3118/
A Dual of Encapsulation? • The buffer protocol and data-types provide a different view on encapsulation (the dual of typical encapsulation approaches for those initiated to linear algebra)
! • Rather than attach methods to data and hide the data, you describe data completely and declaratively. You can then later apply arbitrary code to this data at run-time.
! • It makes it much easier to move code to data at all levels.
Blaze and Numba Blaze is motivated by generalizing PEP 3118 to all languages and data-sets — Python Glue 2.0 Numba is motivated by a desire for Python NumPy array-like code to reach the speeds of Fortran!
Interpreters Data Compute API Blaze Architecture • Flexible architecture to accommodate exploration • Use compilation of deferred expressions to optimize data interactions
Blaze Data • Single interface for data layers • Composition of different formats • Simple api to add custom data formats SQL CSV HDFS JSON Mem Custom HDF5 Data
Blaze Compute Compute DyND Pandas PyTables Spark • Computation abstraction over numerous data libraries • Simple multi-dispatched visitors to implement new backends • Allows plumbing between stacks to be seamless to user
Image Processing @jit('void(f8[:,:],f8[:,:],f8[:,:])') def filter(image, filt, output): M, N = image.shape m, n = filt.shape for i in range(m//2, M-m//2): for j in range(n//2, N-n//2): result = 0.0 for k in range(m): for l in range(n): result += image[i+k-m//2,j+l-n//2]*filt[k, l] output[i,j] = result ~1500x speed-up
Case-study -- j0 from scipy.special • scipy.special was one of the first libraries I wrote
• extended “umath” module by adding new “universal functions” to compute many scientific functions by wrapping C and Fortran libs.
• Bessel functions are solutions to a differential equation: x 2 d 2 y dx 2 + x dy dx + ( x 2 ↵ 2) y = 0 y = J↵ ( x ) Jn (x) = 1 ⇡ Z ⇡ 0 cos (n⌧ x sin (⌧)) d⌧
Result --- equivalent to compiled code In [6]: %timeit vj0(x) 10000 loops, best of 3: 75 us per loop ! In [7]: from scipy.special import j0 ! In [8]: %timeit j0(x) 10000 loops, best of 3: 75.3 us per loop But! Now code is in Python and can be experimented with more easily (and moved to the GPU / accelerator more easily)!
Numba can change the game! LLVM IR x86 C++ ARM PTX C Fortran Python Numba turns Python into a “compiled language” (but much more flexible). You don’t have to reach for C/C++
teoliphant
orgachem
cimadai
biwashi
niftycorp
nishiuma
yuj1osm
yoshiitaka
nttcom
poteboy
yukimatsuyoshi
comucal
h1kita
keathley
matthewcrist
sstephenson
rocio
frogandcode
marcduiker
philnash
jonrohan
62gerente
bkeepers
jlugia
addyosmani