Cython - Making Python as Fast as C

Is your Python program too slow, or planning to integrate some C libraries?

Cython is a transcompiler to compile your Python source into C source. Then compile the C source into binary. It implies your program is able to take the benefits of the optimization from compiler. Cython is also a programming language which defined extra features from Python. The extra features make it is easier to access C, even C++ library from Python.

This talk will introduce you to Cython and cover how to set up, the extra features in Cython, examples, and many tips.

Transcript

1. Cython — 
 Making Python as Fast as C Mosky
2. Cython ➤ Cython is a source-to-source compiler (a.k.a. transcompiler). ➤ Cython is a superset of Python. ➤ Provides optional static type declarations. ➤ Makes writing C extensions for Python easier. ➤ Makes Python faster by pre-compiling and static type. ➤ Sometimes faster by orders of magnitude. ➤ The pandas uses Cython! 2
3. .so C compiler .c .py import Python .pyx Cython
4. Mosky ➤ Python Charmer at Pinkoi. ➤ Has spoken at: PyCons in 
 TW, MY, KR, JP , SG, HK,
 COSCUPs, and TEDx, etc. ➤ Countless hours 
 on teaching Python. ➤ Own the Python packages like ZIPCodeTW. ➤ http://mosky.tw/ 4
5. Outline 1. Setup 2. Foundation 3. Practicing 4. Tips 5. Uncovered Topics 6. Other Solutions 5
6. Setup
7. Install C Compiler ➤ Mac: ➤ $ xcode-select --install ➤ Ubuntu / Debian: ➤ $ sudo apt-get install build-essential ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 7
8. Install Cython ➤ Mac / Ubuntu / Debian: ➤ $ pip3 install cython ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 8
9. The setup.py from distutils.core import setup from Cython.Build import cythonize setup( name='cython-lab', ext_modules=cythonize( '*.pyx', language_level='3' ) ) 9
10. The hello_cython.pyx print('Hello Cython!') 10
11. Build ➤ Into package folder for development: ➤ $ python setup.py build_ext --inplace 11
12. Foundation
13. Define Static Types cdef int i, j, k cdef float f, g[42], *h 13
14. cdef struct Grail: int age float volume cdef union Food: char *spam float *eggs cdef enum CheeseType: cheddar, edam, camembert cdef enum CheeseState: hard = 1 soft = 2 runny = 3 14
15. ctypedef unsigned long ULong ctypedef int* IntPtr 15
16. cdef struct Point: int x int y # either `struct` or `ctypedef` is not need cdef Point p 16
17. cdef: struct Point: int x int y Point p 17
18. Define Functions def add(a, b): return a+b cdef add(a, b): return a+b 18
19. cdef object add(object a, object b): return a+b cdef int add(int a, int b): return a+b cpdef int add(int a, int b): return a+b 19
20. .pxd Exposes cdef Func # mylib.pxd cdef say_hello(char* name=?) # another.pyx from mylib cimport say_hello 20
21. Using C Lib from libc.math cimport sin # or cdef extern from "math.h": double sin(double x) 21
22. Function Visibility 22 Same 
 File Other .pyx .py Funcs in .h/.c Visible
 directly Visible
 via cdef extern Invisible cdef Visible
 via .pxd & cimport cpdef Visible
 via import def
23. Binding During 23 Same 
 File Other .pyx .py Funcs in .h/.c compile-time compile-time x cdef cpdef run-time def
24. Type Conversions 24 C From Py To Py short, int, long int int float, double int/float float char* str/bytes bytes array iterable list struct dict
25. Practicing
26. The Suggestion ➤ LIB_NAME.pyx ➤ Has functions. ➤ test_LIB_NAME.py ➤ Call the functions. 26
27. The Overflow ➤ Static types may also overflow in Cython silently. ➤ Try to make an overflow! ➤ In C: ➤ http://j.mp/test_overflow_in_c_c ➤ In Cython: ➤ http://j.mp/overflow_in_pyx_pyx 27
28. Using C Functions ➤ Try to use the functions in C. ➤ Playing with fork, the system call, may be fun. ➤ In C: ➤ http://j.mp/test_fork_c ➤ In Cython: ➤ http://j.mp/fork_in_pyx_pyx 28
29. The Functions ➤ Write three functions defined in 
 def, cdef, and cpdef. ➤ Try to call them in: ➤ The same file, ➤ Another .pyx file, ➤ And a .py file. ➤ Refer to the table, “Function Visibility”. ➤ The Cython code: ➤ http://j.mp/ lib_in_pyx_pyx ➤ http://j.mp/ use_lib_in_pyx_pyx ➤ http://j.mp/ test_lib_in_pyx_py 29
30. Tips
31. cython -a ➤ cython -a NAME.pyx ➤ cython -a -3 NAME.pyx ➤ open NAME.html 31
32. pyximport import pyximport pyximport.install() import my_pyx_lib # compile .pyx into .so 32
33. import pyximport pyximport.install(pyimport=True) import my_py_lib # compile .py into .so 33
34. Uncovered Topics
35. Uncovered Topics ➤ Differences between C and Cython expressions ➤ http://docs.cython.org/src/userguide/ language_basics.html#differences-between-c-and-cython- expressions ➤ Propagating Exceptions in cdef ➤ http://docs.cython.org/src/userguide/ language_basics.html#error-return-values 35
36. ➤ Extension Type — cdef class ➤ http://docs.cython.org/src/userguide/ extension_types.html ➤ Generic programming using Cython's Template ➤ http://docs.cython.org/src/userguide/fusedtypes.html ➤ Conditional Compilation ➤ http://docs.cython.org/src/userguide/ language_basics.html#conditional-compilation 36
37. ➤ Profiling ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html ➤ Parallelism (No GIL + OpenMP) ➤ http://docs.cython.org/src/userguide/parallelism.html ➤ Using C++ in Cython ➤ http://docs.cython.org/src/userguide/ wrapping_CPlusPlus.html 37
38. Other Solutions
40. Other Solutions ➤ Numba: translates Python to optimized machine code. ➤ @jit ➤ CFFI: calls C code in Python dynamically. ➤ Cython versus CFFI – fuzzy notepad ➤ pybind11: makes C++ code available in Python. ➤ Cython, pybind11, cffi – which tool should you choose? – Setfans Welt ➤ Dask: provides advanced parallelism for analytics. ➤ Comparison to Spark – Dask ➤ panda and Enhancing performance – pandas. 40
41. Cool Down ➤ Did you use NumPy and SciPy efficiently? ➤ Profile your program. ➤ Consider the portability — you are writing C programs! ➤ Algorithm still does matter. ➤ Pick the most suitable solution. 41
42. Recap ➤ static types by cdef and in function definition. ➤ cdef function & cpdef function. ➤ .pxd exposes cdef functions. ➤ cdef extern for C functions. 42