 Scientific Python Eueung Mulyana http://eueung.github.io/python/sci Hint: Navigate with Arrow Keys | Attribution-ShareAlike CC BY-SA 1 / 31

Agenda 1. Jupyter / IPython 2. NumPy 3. SciPy 4. matplotlib 5. pandas 6. SymPy 7. scikit-learn 8. jakevdp: The State of the Stack 2 / 31

 Jupyter / IPython 3 / 31

IPython Powerful interactive shell Supports tab completion of just about everything Inline help system for modules, classes etc. with ? , source code with ? ? Browser based notebook (Jupyter) with support for (runnable) code, text, mathematical expressions using LATEX, inline plots etc. Could be used as a computational lab notes/worksheets Magic functions to access the shell, run R code etc. Parallel computing 4 / 31

Notes on Jupyter 1. The Jupyter Notebook works with over 40 languages 2. Jupyter Notebooks render on GitHub Jupyter Computational Narratives 1. Computers are optimized for producing, consuming and processing data. 2. Humans are optimized for producing, consuming and processing narratives/stories. 3. For code and data to be useful to humans, we need tools for creating and sharing narratives that involve code and data. The Jupyter Notebook is a tool for creating and sharing computational narratives. 5 / 31

Jupyter & Data Science The Jupyter Notebook is a tool that allows us to explore the fundamental questions of Data Science with a particular dataset with code and data in a manner that produces a computational narrative that can be shared, reproduced, modified, and extended. At the end of it all, those computational narratives encapsulate the goal or end point of Data Science. The character of the narrative (prediction, inference, data generation, insight, etc.) will vary from case to case. T h e p u r p o s e o f c o m p u t i n g i s i n s i g h t , n o t n u m b e r s . H a m m i n g , R i c h a r d ( 1 9 6 2 ) . N u m e r i c a l M e t h o d s f o r S c i e n t i s t s a n d 6 / 31

 NumPy 7 / 31

NumPy NumPy is the fundamental package for scientific computing with Python. It contains among other things: A powerful N-dimensional array object Sophisticated (broadcasting) functions Tools for integrating C/C++ and Fortran code Useful linear algebra, Fourier transform, and random number capabilities Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases. NumPy provides a powerful N-dimensions array object Methods on these arrays are fast because they relies on well-optimised librairies for linear algebra (BLAS, ATLAS, MKL) NumPy is tolerant to python’s lists NumPy inherits from years of computer based numerical analysis problem solving 8 / 31

i m p o r t n u m p y a s n p a = n p . a r r a y ( [ 1 , 2 , 3 ] ) # C r e a t e a r a n k 1 a r r a y p r i n t t y p e ( a ) # P r i n t s " < t y p e ' n u m p y . n d a r r a y ' > " p r i n t a . s h a p e # P r i n t s " ( 3 , ) " p r i n t a [ 0 ] , a [ 1 ] , a [ 2 ] # P r i n t s " 1 2 3 " a [ 0 ] = 5 # C h a n g e a n e l e m e n t o f t h e a r r a y p r i n t a # P r i n t s " [ 5 , 2 , 3 ] " b = n p . a r r a y ( [ [ 1 , 2 , 3 ] , [ 4 , 5 , 6 ] ] ) # C r e a t e a r a n k 2 a r r a y p r i n t b . s h a p e # P r i n t s " ( 2 , 3 ) " p r i n t b [ 0 , 0 ] , b [ 0 , 1 ] , b [ 1 , 0 ] # P r i n t s " 1 2 4 " # - - - - - a = n p . z e r o s ( ( 2 , 2 ) ) # C r e a t e a n a r r a y o f a l l z e r o s p r i n t a # P r i n t s " [ [ 0 . 0 . ] # [ 0 . 0 . ] ] " b = n p . o n e s ( ( 1 , 2 ) ) # C r e a t e a n a r r a y o f a l l o n e s p r i n t b # P r i n t s " [ [ 1 . 1 . ] ] " c = n p . f u l l ( ( 2 , 2 ) , 7 ) # C r e a t e a c o n s t a n t a r r a y p r i n t c # P r i n t s " [ [ 7 . 7 . ] # [ 7 . 7 . ] ] " d = n p . e y e ( 2 ) # C r e a t e a 2 x 2 i d e n t i t y m a t r i x p r i n t d # P r i n t s " [ [ 1 . 0 . ] # [ 0 . 1 . ] ] " e = n p . r a n d o m . r a n d o m ( ( 2 , 2 ) ) # C r e a t e a n a r r a y f i l l e d w i t h r a n d o m v a l u e s p r i n t e # M i g h t p r i n t " [ [ 0 . 9 1 9 4 0 1 6 7 0 . 0 8 1 4 3 9 4 1 ] # [ 0 . 6 8 7 4 4 1 3 4 0 . 8 7 2 3 6 6 8 7 ] ] " Numpy Numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object (MATLAB style), and tools for working with these arrays. Arrays A numpy array is a grid of values, all of the same type, and is indexed by a tuple of nonnegative integers. The number of dimensions is the rank of the array; the shape of an array is a tuple of integers giving the size of the array along each dimension. We can initialize numpy arrays from nested Python lists, and access elements using square brackets. Numpy also provides many functions to create arrays. 9 / 31

 SciPy 10 / 31

SciPy SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering. SciPy core packages: IPython, NumPy, SciPy Library, SimPy, matplotlib, pandas. SciPy Library SciPy is a collection of mathematical algorithms and convenience functions built on top of NumPy includes modules for: statistics, integration & ODE solvers, linear algebra, optimization, FFT, etc. We use the terms SciPy and SciPy Library interchangeably. Meaning depends on context. SciPy is a toolbox for researchers/scientists, it contains many hidden treasures for them. 11 / 31

SciPy & NumPy Numpy provides a high-performance multidimensional array and basic tools to compute with and manipulate these arrays. SciPy builds on this, and provides a large number of functions that operate on numpy arrays and are useful for different types of scientific and engineering applications. SciPy provides numerous numerical routines, that run efficiently on top of NumPy arrays for: optimization, signal processing, linear algebra and many more. It also provides some convenient data structures as compressed sparse matrix and spatial data structures. If you had already use some scikits (scikit-learn, scikit-image) you already used scipy extensively. A few thoughts on SciPy: Contains linear algebra routines that overlap with NumPy; SciPy’s linear algebra routines always run on the optimized system libraries (LAPACK, ATLAS, Intel Math Kernel Library, etc.) Sparse matrix support Extends NumPy’s statistical capabilities Under active development, new toys added constantly! 12 / 31

SciPy A big box of tools: Special functions (scipy.special) Integration (scipy.integrate) Optimization (scipy.optimize) Interpolation (scipy.interpolate) Fourier Transforms (scipy.fftpack) Signal Processing (scipy.signal) Statistics (scipy.stats) Linear Algebra (scipy.linalg) File IO (scipy.io) Sparse Eigenvalue Problems with ARPACK Compressed Sparse Graph Routines (scipy.sparse.csgraph) Spatial data structures and algorithms (scipy.spatial) Multi-dimensional image processing (scipy.ndimage) Weave (scipy.weave) f r o m s c i p y . s t a t s i m p o r t l i n r e g r e s s ( s l o p e , i n t e r c e p t , r , p , s e ) = l i n r e g r e s s ( x , n o i s y _ y ) # - - - f r o m s c i p y . s t a t s i m p o r t s p e a r m a n r , p e a r s o n r x _ c u b e d = x * * 3 x _ c u b e d + = n p . r a n d o m . n o r m a l ( 0 , 3 , 1 0 ) 13 / 31

 matplotlib 14 / 31

matplotlib The ultimate plotting library that renders 2D and 3D high-quality plots for python. pyplot implements Matlab-style plotting Object-oriented API for more advanced graphics The API mimics, in many ways the MATLAB one, easing the transition from MATLAB users to python Once again, no surprises, matplotlib is a very stable and mature project (expect one major release per year) Inline plots in the notebook: i p y t h o n n o t e b o o k - - p y l a b i n l i n e 15 / 31

i m p o r t n u m p y a s n p i m p o r t m a t p l o t l i b . p y p l o t a s p l t # C o m p u t e t h e x a n d y c o o r d i n a t e s f o r p o i n t s o n a s i n e c u r v e x = n p . a r a n g e ( 0 , 3 * n p . p i , 0 . 1 ) y = n p . s i n ( x ) # P l o t t h e p o i n t s u s i n g m a t p l o t l i b p l t . p l o t ( x , y ) p l t . s h o w ( ) # Y o u m u s t c a l l p l t . s h o w ( ) t o m a k e g r a p h i c s a p p e a r . i m p o r t n u m p y a s n p i m p o r t m a t p l o t l i b . p y p l o t a s p l t # C o m p u t e t h e x a n d y c o o r d i n a t e s f o r p o i n t s o n s i n e a n d c o s i n e c u r v e s x = n p . a r a n g e ( 0 , 3 * n p . p i , 0 . 1 ) y _ s i n = n p . s i n ( x ) y _ c o s = n p . c o s ( x ) # P l o t t h e p o i n t s u s i n g m a t p l o t l i b p l t . p l o t ( x , y _ s i n ) p l t . p l o t ( x , y _ c o s ) p l t . x l a b e l ( ' x a x i s l a b e l ' ) p l t . y l a b e l ( ' y a x i s l a b e l ' ) p l t . t i t l e ( ' S i n e a n d C o s i n e ' ) p l t . l e g e n d ( [ ' S i n e ' , ' C o s i n e ' ] ) p l t . s h o w ( ) matplotlib matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code. For simple plotting the pyplot interface provides a MATLAB-like interface, particularly when combined with IPython. For the power user, you have full control of line styles, font properties, axes properties, etc, via an object oriented interface or via a set of functions familiar to MATLAB users. With just a little bit of extra work we can easily plot a more complex chart e.g. multiple lines at once, and add a title, legend, and axis labels. 16 / 31

 Notes 17 / 31

TL;DR NumPy is the foundation SciPy is built upon NumPy, with some overlapping functionality matplotlib complements both NumPy, SciPy, matplotlib NumPy is the foundation of scientific and numerical computing with Python SciPy is a collection of mathematical and scientific tools matplotlib is a technical plotting package NumPy Arrays Implemented in C for efficiency Python indexing and slicing Elements are strongly typed Taking advantage of NumPy Think in parallel! Replace loops with vector operations matplotlib Primarily 2D plotting Basic 3D plots available with mplot3d (import mpl_toolkits.mplot3d) 18 / 31

Other Notes NumPy/SciPy/scikit-learn rely on many low-level Fortran/C library such as BLAS, ATLAS, the Intel MKL… most of these libraries are shipped by your favorite OS unoptimized (well, maybe not the case for Mac) you may want to re-compile these libraries or to use a packaged python distribution (anaconda, canopy) libraries for performance: numba, cython, ... 19 / 31

 pandas 20 / 31

pandas is an open source, BSD-licensed library providing high- performance, easy-to-use data structures and data analysis tools for the Python programming language. pandas "R for Python" Provides easy to use data structures & a ton of useful helper functions for data cleanup and transformations Fast! (backed by NumPy arrays) Integrates well with other libs e.g. scikit-learn 21 / 31

i m p o r t p a n d a s a s p d i m p o r t n u m p y a s n p i m p o r t m a t p l o t l i b . p y p l o t a s p l t s = p d . S e r i e s ( [ 1 , 3 , 5 , n p . n a n , 6 , 8 ] ) d a t e s = p d . d a t e _ r a n g e ( ' 2 0 1 3 0 1 0 1 ' , p e r i o d s = 6 ) d f = p d . D a t a F r a m e ( n p . r a n d o m . r a n d n ( 6 , 4 ) , i n d e x = d a t e s , c o l u m n s = l i s t ( d f 2 = p d . D a t a F r a m e ( { ' A ' : 1 . , ' B ' : p d . T i m e s t a m p ( ' 2 0 1 3 0 1 0 2 ' ) , ' C ' : p d . S e r i e s ( 1 , i n d e x = l i s t ( r a n g e ( 4 ) ) , d t y p e = ' D ' : n p . a r r a y ( [ 3 ] * 4 , d t y p e = ' i n t 3 2 ' ) , ' E ' : p d . C a t e g o r i c a l ( [ " t e s t " , " t r a i n " , " t e s t " ' F ' : ' f o o ' } ) pandas pandas provides the D a t a F r a m e class, which is very similar to a d a t a . f r a m e in R Built on top of NumPy arrays, and allows mixed column types Copes well with missing values (unlike NumPy) Intelligently matches on columns/indices (supports SQL- like joins etc.) Read and write .csv, .xls, HTML tables etc. Lots of useful data analysis tools built in 22 / 31

 SymPy 23 / 31

SymPy SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python and does not require any external libraries. i m p o r t s y m p y s y m p y . s q r t ( 8 ) # 2 * s q r t ( 2 ) f r o m s y m p y i m p o r t s y m b o l s x , y = s y m b o l s ( ' x y ' ) e x p r = x + 2 * y e x p r # x + 2 * y e x p r - x # 2 * y 24 / 31

 scikit-learn 25 / 31

scikit-learn Machine Learning algorithms implemented in Python on top of NumPy & SciPy Conveniently maintains the same interface to a wide range of algorithms Includes algorithms for: Classification, Regression, Clustering, Dimensionality reduction As well as lots of useful utilities (cross-validation, preprocessing etc.) f r o m s k l e a r n i m p o r t d a t a s e t s i r i s = d a t a s e t s . l o a d _ i r i s ( ) d i g i t s = d a t a s e t s . l o a d _ d i g i t s ( ) p r i n t ( d i g i t s . d a t a ) d i g i t s . t a r g e t d i g i t s . i m a g e s [ 0 ] f r o m s k l e a r n i m p o r t s v m c l f = s v m . S V C ( g a m m a = 0 . 0 0 1 , C = 1 0 0 . ) c l f . f i t ( d i g i t s . d a t a [ : - 1 ] , d i g i t s . t a r g e t [ : - 1 ] ) 26 / 31

 The State of the Stack 27 / 31

28 / 31

Many More Tools .. Performance Numba, Weave, Numexpr, Theano . . . Visualization Bokeh, Seaborn, Plotly, Chaco, mpld3, ggplot, MayaVi, vincent, toyplot, HoloViews . . . Data Structures & Computation Blaze, Dask, DistArray, XRay, Graphlab, SciDBpy, pySpark . . . Packaging & distribution: pip/wheels, conda, EPD, Canopy, Anaconda ... 29 / 31

References 1. Brian Granger: Project Jupyter as a Foundation for Open Data Science 2. Juan Luis Cano Rodriguez, IPython: How a notebook is changing science | Python as a real alternative to MATLAB, Mathematica and other commercial software 3. Olivier Hervieu: Introduction to scientific programming in python 4. CS231n: IPython Tutorial, http://cs231n.github.io/ipython-tutorial/ 5. J.R. Johansson: Introduction to scientific computing with Python 6. Introduction to solving biological problems with Python by pycam 7. Jake VanderPlas: The State of the Stack 30 / 31

 END Eueung Mulyana http://eueung.github.io/python/sci Hint: Navigate with Arrow Keys | Attribution-ShareAlike CC BY-SA 31 / 31