simulation codes, data analysis packages, databases, visualization tools, and home-grown software-each of which presents the user with a different set of interfaces and file formats. As a result, a scientist may spend a considerable amount of time simply trying to get all of these components to work together in some manner...” - David Beazley Scientific Computing with Python (ACM vol. 216, 2000) 1990s: The Scripting Era
Perl scripts that called C++ numerical routines that would dump data files, and I would load them up into MatLab to plot them. After a while I got tired of the MatLab dependency… so I started loading them up in GnuPlot.” -John Hunter creator of Matplotlib SciPy 2012 Keynote 2000s: The SciPy Era
then Matlab and shell scripts & Fortran & C/C++ libraries. When I discovered Python, I really liked the language... But, it was very nascent and lacked a lot of libraries. I felt like I could add value to the world by connecting low-level libraries to high-level usage in Python.” - Travis Oliphant creator of NumPy & SciPy via email, 2015 2000s: The SciPy Era
and seeing all the books on languages I had. I literally had a stack with books on C, C++, Unix utilities (awk/sed/sh/etc), Perl, IDL manuals, the Mathematica book, Make printouts, etc. I realized I was probably spending more time switching between languages than getting anything done..” - Fernando Perez creator of IPython via email, 2015
requirements that were not well-addressed by any single tool at my disposal: - Data structures with labeled axes . . . - Integrated time series functionality . . . - Arithmetic operations and reductions . . . - Flexible handling of missing data - Merge and other relational operations . . . I wanted to be able to do all these things in one place, preferably in a language well-suited to general purpose software development” - Wes McKinney creator of Pandas (in Python for Data Analysis)
PyData Era Motto: “Python as Alternative to R” Motto: “Python as Alternative to MatLab” Motto: “Python as Alternative to Bash” * yes, this is all overly simplified . . .
on Python for scientific and data-intensive computing, It comes in two flavors: - Miniconda is a minimal install of the conda command-line tool - Anaconda is miniconda plus hundreds of common packages. I recommend Miniconda. http://conda.pydata.org/
Inc.) In order to continue the installation process, please review the license agreement. Please, press ENTER to continue >>> Installation Miniconda is a lightweight installation (~25MB) that gives you access to the conda package management tool. It creates a sandboxed Python installation, entirely disconnected from your system Python. http://conda.pydata.org/
Python 3.5.1 |Continuum Analytics, Inc.| (default ... Type "help", "copyright", "credits" or "license" ... >>> print("hello world") hello world Installation Both conda and python now point to the executables installed by miniconda. http://conda.pydata.org/
metadata ......... Solving package specifications: . Package plan for installation in environment /Users/jakevdp/anaconda/: The following NEW packages will be INSTALLED: appnope: 0.1.0-py36_0 bleach: 1.5.0-py36_0 cycler: 0.10.0-py36_0 decorator: 4.0.11-py36_0 Installation Installation of new packages can be done seamlessly with conda install http://conda.pydata.org/
metadata ......... Solving package specifications: . Package plan for installation in environment /Users/jakevdp/anaconda/envs/py2.7: The following NEW packages will be INSTALLED: mkl: 2017.0.3-0 numpy: 1.13.0-py27_0 openssl: 1.0.2l-0 pip: 9.0.1-py27_1 Installation New sandboxed environments can be created with specific versions of Python and its packages. Here we create an environment named py2.7 with Python 2.7 http://conda.pydata.org/
$ python --version Python 2.7.11 :: Continuum Analytics, Inc. Installation By “activating” the environment, we can now use this different Python version with a different set of packages. You can create as many of these environments as you’d like. http://conda.pydata.org/
installs python packages within any environment; conda installs any package within conda environments” For many more details on the distinctions, see my blog post, Conda: Myths and Misconceptions.1
from local directory: /Users/jakevdp [I 06:32:22.641 NotebookApp] 0 active kernels [I 06:32:22.641 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 06:32:22.642 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). http://jupyter.org/
from local directory: /Users/jakevdp [I 06:32:22.641 NotebookApp] 0 active kernels [I 06:32:22.641 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 06:32:22.642 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). http://jupyter.org/
from local directory: /Users/jakevdp [I 06:32:22.641 NotebookApp] 0 active kernels [I 06:32:22.641 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 06:32:22.642 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). http://jupyter.org/
from local directory: /Users/jakevdp [I 06:32:22.641 NotebookApp] 0 active kernels [I 06:32:22.641 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 06:32:22.642 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). http://jupyter.org/
from local directory: /Users/jakevdp [I 06:32:22.641 NotebookApp] 0 active kernels [I 06:32:22.641 NotebookApp] The IPython Notebook is running at: http://localhost:8888/ [I 06:32:22.642 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). http://jupyter.org/
for storing and manipulating numerical data arrays. import numpy as np x = np.arange(10) print(x) [0 1 2 3 4 5 6 7 8 9] Arithmetic and other operations are performed element-wise on these arrays: print(x * 2 + 1) [ 1 3 5 7 9 11 13 15 17 19] http://www.numpy.org/
algebra, Fast Fourier Transforms, etc. M = np.random.rand(5, 10) # 5x10 random matrix u, s, v = np.linalg.svd(M) print(s) [ 4.22083 1.091050 0.892570 0.55553 0.392541] x = np.random.randn(100) # 100 std normal values X = np.fft.fft(x) print(X[:4]) # first four entries [ -7.932434 +0.j -16.683935 -3.997685j 3.229016+16.658718j 2.366788-11.863747j] http://www.numpy.org/
in Python) is vectorization: x = np.random.rand(10000000) %%timeit y = np.empty(x.shape) for i in range(len(x)): y[i] = 2 * x[i] + 1 1 loop, best of 3: 6.4 s per loop If you write Python like C, you’ll have a bad time: http://www.numpy.org/
in Python) is vectorization: x = np.random.rand(10000000) %%timeit y = 2 * x + 1 10 loops, best of 3: 58.6 ms per loop Use vectorization for readability and speed ~ 100x speedup! http://www.numpy.org/
in Python) is vectorization: x = np.random.rand(10000000) %%timeit y = 2 * x + 1 10 loops, best of 3: 58.6 ms per loop Use vectorization for readability and speed https://www.youtube.com/watch?v=EEUXKG97YRw https://speakerdeck.com/jakevdp/losing-your-loops-fast-numerical-computing-with-numpy-pycon-2015 ~ 100x speedup! For a more complete intro to vectorization in NumPy, see Losing Your Loops: Fast Numerical Computation in Python (my talk at PyCon 2015)
a variety of formats. Start here to read virtually any data format! # contents of data.csv name, id peter, 321 paul, 605 mary, 444 name id 0 peter 321 1 paul 605 2 mary 444 df = pd.read_csv('data.csv') print(df) http://pandas.pydata.org
id val 0 A 1 1 B 2 2 A 3 3 B 4 df = pd.DataFrame({'id': ['A', 'B', 'A', 'B'], 'val': [1, 2, 3, 4]}) print(df) val id A 4 B 6 grouped = df.groupby('id').sum() print(grouped) http://pandas.pydata.org
interface to Vega-Lite: import pdvega # import makes vgplot attribute available data.vgplot.scatter('petalLength', 'petalWidth') http://jakevdp.github.io/pdvega;/
a huge and rapidly-developing space: See my PyCon 2017 talk, Python’s Visualization Landscape https://speakerdeck.com/jakevdp/pythons-visualization-landscape-pycon-2017 https://www.youtube.com/watch?v=FytuB8nFHPQ
e.g. scipy.sparse sparse matrix operations scipy.interpolate interpolation routines scipy.integrate numerical integration scipy.spatial spatial metrics & distances scipy.stats statistical functions scipy.optimize minimization & optimization scipy.linalg linear algebra scipy.special special mathematical functions scipy.fftpack Fourier & related transforms Most functionality comes from wrapping Netlib & related Fortran libraries, meaning it is blazing fast. http://www.scipy.org/
y) xfit = np.linspace(-1, 11, 1000) yfit = model.predict(xfit[:, np.newaxis]) plt.plot(x, y, '.k') plt.plot(xfit, yfit) Fit a random forest regression: Machine Learning with scikit-learn
= SVR() model.fit(x[:, np.newaxis], y) xfit = np.linspace(-1, 11, 1000) yfit = model.predict(xfit[:, np.newaxis]) plt.plot(x, y, '.k') plt.plot(xfit, yfit) Fit a support vector regression: Scikit-learn’s strength: provides a uniform API for the most common machine learning methods.
slow in Python: def fib(n): a, b = 0, 1 for i in range(n): a, b = b, a + b return a %timeit fib(10000) # ipython “timeit magic” 100 loops, best of 3: 2.73 ms per loop
b = 0, 1 for i in range(n): a, b = b, a + b return a %timeit fib(10000) # ipython “timeit magic” 100000 loops, best of 3: 6.06 µs per loop With a quick decorator, code can be ~1000x as fast! ~ 500x speedup!
compilation of the Python function to LLVM byte-code. import numba @numba.jit def fib(n): a, b = 0, 1 for i in range(n): a, b = b, a + b return a %timeit fib(10000) # ipython “timeit magic” 100000 loops, best of 3: 6.06 µs per loop With a quick decorator, code can be ~1000x as fast! ~ 500x speedup!
marginal speedups without even changing the code: %%cython def fib(n): a, b = 0, 1 for i in range(n): a, b = b, a + b return a 100 loops, best of 3: 2.42 ms per loop %timeit fib(10000) ~ 10% speedup!
for the compiler leads to much better performance: %%cython def fib(int n): cdef int a = 0, b = 1 for i in range(n): a, b = b, a + b return a 100000 loops, best of 3: 5.93 µs per loop %timeit fib(10000) ~ 500x speedup!
jakevdp
nealle
iharuoru
coe401_
algoartis
fadis
ike002jp
fast_doctor
1mash0
knakagawa
joergneumann
soutaro
gob
jeffersonlam
tammielis
3n
bkeepers
moore
chriscoyier
maggiecrowley
mongodb
nonsquared
lauravandoore
morganepeng
jonrohan
![Coding Environment: $ jupyter notebook [I 06:32:22.641 NotebookApp] Serving notebooks](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_39.jpg){kind=link}
![Coding Environment: $ jupyter notebook [I 06:32:22.641 NotebookApp] Serving notebooks](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_40.jpg){kind=link}
![Coding Environment: $ jupyter notebook [I 06:32:22.641 NotebookApp] Serving notebooks](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_41.jpg){kind=link}
![Coding Environment: $ jupyter notebook [I 06:32:22.641 NotebookApp] Serving notebooks](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_42.jpg){kind=link}
![Coding Environment: $ jupyter notebook [I 06:32:22.641 NotebookApp] Serving notebooks](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_43.jpg){kind=link}
![http://scikit-learn.org/ from sklearn.ensemble import RandomForestRegressor model = RandomForestRegressor() model.fit(x[:, np.newaxis],](https://files.speakerdeck.com/presentations/e5a862c2a11849d082bd9e41484bd7d1/slide_70.jpg){kind=link}
