Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cython - Making Python as Fast as C

Cython - Making Python as Fast as C

Is your Python program too slow, or planning to integrate some C libraries?

Cython is a transcompiler to compile your Python source into C source. Then compile the C source into binary. It implies your program is able to take the benefits of the optimization from compiler. Cython is also a programming language which defined extra features from Python. The extra features make it is easier to access C, even C++ library from Python.

This talk will introduce you to Cython and cover how to set up, the extra features in Cython, examples, and many tips.

Mosky Liu

April 24, 2014
Tweet

More Decks by Mosky Liu

Other Decks in Programming

Transcript

  1. Cython — 
 Making Python as Fast as C Mosky

  2. Cython ➤ Cython is a source-to-source compiler (a.k.a. transcompiler). ➤

    Cython is a superset of Python. ➤ Provides optional static type declarations. ➤ Makes writing C extensions for Python easier. ➤ Makes Python faster by pre-compiling and static type. ➤ Sometimes faster by orders of magnitude. ➤ The pandas uses Cython! 2
  3. .so C compiler .c .py import Python .pyx Cython

  4. Mosky ➤ Python Charmer at Pinkoi. ➤ Has spoken at:

    PyCons in 
 TW, MY, KR, JP , SG, HK,
 COSCUPs, and TEDx, etc. ➤ Countless hours 
 on teaching Python. ➤ Own the Python packages like ZIPCodeTW. ➤ http://mosky.tw/ 4
  5. Outline 1. Setup 2. Foundation 3. Practicing 4. Tips 5.

    Uncovered Topics 6. Other Solutions 5
  6. Setup

  7. Install C Compiler ➤ Mac: ➤ $ xcode-select --install ➤

    Ubuntu / Debian: ➤ $ sudo apt-get install build-essential ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 7
  8. Install Cython ➤ Mac / Ubuntu / Debian: ➤ $

    pip3 install cython ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 8
  9. The setup.py from distutils.core import setup from Cython.Build import cythonize

    setup( name='cython-lab', ext_modules=cythonize( '*.pyx', language_level='3' ) ) 9
  10. The hello_cython.pyx print('Hello Cython!') 10

  11. Build ➤ Into package folder for development: ➤ $ python

    setup.py build_ext --inplace 11
  12. Foundation

  13. Define Static Types cdef int i, j, k cdef float

    f, g[42], *h 13
  14. cdef struct Grail: int age float volume cdef union Food:

    char *spam float *eggs cdef enum CheeseType: cheddar, edam, camembert cdef enum CheeseState: hard = 1 soft = 2 runny = 3 14
  15. ctypedef unsigned long ULong ctypedef int* IntPtr 15

  16. cdef struct Point: int x int y # either `struct`

    or `ctypedef` is not need cdef Point p 16
  17. cdef: struct Point: int x int y Point p 17

  18. Define Functions def add(a, b): return a+b cdef add(a, b):

    return a+b 18
  19. cdef object add(object a, object b): return a+b cdef int

    add(int a, int b): return a+b cpdef int add(int a, int b): return a+b 19
  20. .pxd Exposes cdef Func # mylib.pxd cdef say_hello(char* name=?) #

    another.pyx from mylib cimport say_hello 20
  21. Using C Lib from libc.math cimport sin # or cdef

    extern from "math.h": double sin(double x) 21
  22. Function Visibility 22 Same 
 File Other .pyx .py Funcs

    in .h/.c Visible
 directly Visible
 via cdef extern Invisible cdef Visible
 via .pxd & cimport cpdef Visible
 via import def
  23. Binding During 23 Same 
 File Other .pyx .py Funcs

    in .h/.c compile-time compile-time x cdef cpdef run-time def
  24. Type Conversions 24 C From Py To Py short, int,

    long int int float, double int/float float char* str/bytes bytes array iterable list struct dict
  25. Practicing

  26. The Suggestion ➤ LIB_NAME.pyx ➤ Has functions. ➤ test_LIB_NAME.py ➤

    Call the functions. 26
  27. The Overflow ➤ Static types may also overflow in Cython

    silently. ➤ Try to make an overflow! ➤ In C: ➤ http://j.mp/test_overflow_in_c_c ➤ In Cython: ➤ http://j.mp/overflow_in_pyx_pyx 27
  28. Using C Functions ➤ Try to use the functions in

    C. ➤ Playing with fork, the system call, may be fun. ➤ In C: ➤ http://j.mp/test_fork_c ➤ In Cython: ➤ http://j.mp/fork_in_pyx_pyx 28
  29. The Functions ➤ Write three functions defined in 
 def,

    cdef, and cpdef. ➤ Try to call them in: ➤ The same file, ➤ Another .pyx file, ➤ And a .py file. ➤ Refer to the table, “Function Visibility”. ➤ The Cython code: ➤ http://j.mp/ lib_in_pyx_pyx ➤ http://j.mp/ use_lib_in_pyx_pyx ➤ http://j.mp/ test_lib_in_pyx_py 29
  30. Tips

  31. cython -a ➤ cython -a NAME.pyx ➤ cython -a -3

    NAME.pyx ➤ open NAME.html 31
  32. pyximport import pyximport pyximport.install() import my_pyx_lib # compile .pyx into

    .so 32
  33. import pyximport pyximport.install(pyimport=True) import my_py_lib # compile .py into .so

    33
  34. Uncovered Topics

  35. Uncovered Topics ➤ Differences between C and Cython expressions ➤

    http://docs.cython.org/src/userguide/ language_basics.html#differences-between-c-and-cython- expressions ➤ Propagating Exceptions in cdef ➤ http://docs.cython.org/src/userguide/ language_basics.html#error-return-values 35
  36. ➤ Extension Type — cdef class ➤ http://docs.cython.org/src/userguide/ extension_types.html ➤

    Generic programming using Cython's Template ➤ http://docs.cython.org/src/userguide/fusedtypes.html ➤ Conditional Compilation ➤ http://docs.cython.org/src/userguide/ language_basics.html#conditional-compilation 36
  37. ➤ Profiling ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html ➤ Parallelism (No GIL + OpenMP)

    ➤ http://docs.cython.org/src/userguide/parallelism.html ➤ Using C++ in Cython ➤ http://docs.cython.org/src/userguide/ wrapping_CPlusPlus.html 37
  38. Other Solutions

  39. None
  40. Other Solutions ➤ Numba: translates Python to optimized machine code.

    ➤ @jit ➤ CFFI: calls C code in Python dynamically. ➤ Cython versus CFFI – fuzzy notepad ➤ pybind11: makes C++ code available in Python. ➤ Cython, pybind11, cffi – which tool should you choose? – Setfans Welt ➤ Dask: provides advanced parallelism for analytics. ➤ Comparison to Spark – Dask ➤ panda and Enhancing performance – pandas. 40
  41. Cool Down ➤ Did you use NumPy and SciPy efficiently?

    ➤ Profile your program. ➤ Consider the portability — you are writing C programs! ➤ Algorithm still does matter. ➤ Pick the most suitable solution. 41
  42. Recap ➤ static types by cdef and in function definition.

    ➤ cdef function & cpdef function. ➤ .pxd exposes cdef functions. ➤ cdef extern for C functions. 42