Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cython - Making Python as Fast as C

D16bc1f94b17ddc794c2dfb48ef59456?s=47 Mosky
April 24, 2014

Cython - Making Python as Fast as C

Is your Python program too slow, or planning to integrate some C libraries?

Cython is a transcompiler to compile your Python source into C source. Then compile the C source into binary. It implies your program is able to take the benefits of the optimization from compiler. Cython is also a programming language which defined extra features from Python. The extra features make it is easier to access C, even C++ library from Python.

This talk will introduce you to Cython and cover how to set up, the extra features in Cython, examples, and many tips.

D16bc1f94b17ddc794c2dfb48ef59456?s=128

Mosky

April 24, 2014
Tweet

Transcript

  1. Cython — 
 Making Python as Fast as C Mosky

  2. Cython ➤ Cython is a source-to-source compiler (a.k.a. transcompiler). ➤

    Cython is a superset of Python. ➤ Provides optional static type declarations. ➤ Makes writing C extensions for Python easier. ➤ Makes Python faster by pre-compiling and static type. ➤ Sometimes faster by orders of magnitude. ➤ The pandas uses Cython! 2
  3. .so C compiler .c .py import Python .pyx Cython

  4. Mosky ➤ Python Charmer at Pinkoi. ➤ Has spoken at:

    PyCons in 
 TW, MY, KR, JP , SG, HK,
 COSCUPs, and TEDx, etc. ➤ Countless hours 
 on teaching Python. ➤ Own the Python packages like ZIPCodeTW. ➤ http://mosky.tw/ 4
  5. Outline 1. Setup 2. Foundation 3. Practicing 4. Tips 5.

    Uncovered Topics 6. Other Solutions 5
  6. Setup

  7. Install C Compiler ➤ Mac: ➤ $ xcode-select --install ➤

    Ubuntu / Debian: ➤ $ sudo apt-get install build-essential ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 7
  8. Install Cython ➤ Mac / Ubuntu / Debian: ➤ $

    pip3 install cython ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 8
  9. The setup.py from distutils.core import setup from Cython.Build import cythonize

    setup( name='cython-lab', ext_modules=cythonize( '*.pyx', language_level='3' ) ) 9
  10. The hello_cython.pyx print('Hello Cython!') 10

  11. Build ➤ Into package folder for development: ➤ $ python

    setup.py build_ext --inplace 11
  12. Foundation

  13. Define Static Types cdef int i, j, k cdef float

    f, g[42], *h 13
  14. cdef struct Grail: int age float volume cdef union Food:

    char *spam float *eggs cdef enum CheeseType: cheddar, edam, camembert cdef enum CheeseState: hard = 1 soft = 2 runny = 3 14
  15. ctypedef unsigned long ULong ctypedef int* IntPtr 15

  16. cdef struct Point: int x int y # either `struct`

    or `ctypedef` is not need cdef Point p 16
  17. cdef: struct Point: int x int y Point p 17

  18. Define Functions def add(a, b): return a+b cdef add(a, b):

    return a+b 18
  19. cdef object add(object a, object b): return a+b cdef int

    add(int a, int b): return a+b cpdef int add(int a, int b): return a+b 19
  20. .pxd Exposes cdef Func # mylib.pxd cdef say_hello(char* name=?) #

    another.pyx from mylib cimport say_hello 20
  21. Using C Lib from libc.math cimport sin # or cdef

    extern from "math.h": double sin(double x) 21
  22. Function Visibility 22 Same 
 File Other .pyx .py Funcs

    in .h/.c Visible
 directly Visible
 via cdef extern Invisible cdef Visible
 via .pxd & cimport cpdef Visible
 via import def
  23. Binding During 23 Same 
 File Other .pyx .py Funcs

    in .h/.c compile-time compile-time x cdef cpdef run-time def
  24. Type Conversions 24 C From Py To Py short, int,

    long int int float, double int/float float char* str/bytes bytes array iterable list struct dict
  25. Practicing

  26. The Suggestion ➤ LIB_NAME.pyx ➤ Has functions. ➤ test_LIB_NAME.py ➤

    Call the functions. 26
  27. The Overflow ➤ Static types may also overflow in Cython

    silently. ➤ Try to make an overflow! ➤ In C: ➤ http://j.mp/test_overflow_in_c_c ➤ In Cython: ➤ http://j.mp/overflow_in_pyx_pyx 27
  28. Using C Functions ➤ Try to use the functions in

    C. ➤ Playing with fork, the system call, may be fun. ➤ In C: ➤ http://j.mp/test_fork_c ➤ In Cython: ➤ http://j.mp/fork_in_pyx_pyx 28
  29. The Functions ➤ Write three functions defined in 
 def,

    cdef, and cpdef. ➤ Try to call them in: ➤ The same file, ➤ Another .pyx file, ➤ And a .py file. ➤ Refer to the table, “Function Visibility”. ➤ The Cython code: ➤ http://j.mp/ lib_in_pyx_pyx ➤ http://j.mp/ use_lib_in_pyx_pyx ➤ http://j.mp/ test_lib_in_pyx_py 29
  30. Tips

  31. cython -a ➤ cython -a NAME.pyx ➤ cython -a -3

    NAME.pyx ➤ open NAME.html 31
  32. pyximport import pyximport pyximport.install() import my_pyx_lib # compile .pyx into

    .so 32
  33. import pyximport pyximport.install(pyimport=True) import my_py_lib # compile .py into .so

    33
  34. Uncovered Topics

  35. Uncovered Topics ➤ Differences between C and Cython expressions ➤

    http://docs.cython.org/src/userguide/ language_basics.html#differences-between-c-and-cython- expressions ➤ Propagating Exceptions in cdef ➤ http://docs.cython.org/src/userguide/ language_basics.html#error-return-values 35
  36. ➤ Extension Type — cdef class ➤ http://docs.cython.org/src/userguide/ extension_types.html ➤

    Generic programming using Cython's Template ➤ http://docs.cython.org/src/userguide/fusedtypes.html ➤ Conditional Compilation ➤ http://docs.cython.org/src/userguide/ language_basics.html#conditional-compilation 36
  37. ➤ Profiling ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html ➤ Parallelism (No GIL + OpenMP)

    ➤ http://docs.cython.org/src/userguide/parallelism.html ➤ Using C++ in Cython ➤ http://docs.cython.org/src/userguide/ wrapping_CPlusPlus.html 37
  38. Other Solutions

  39. None
  40. Other Solutions ➤ Numba: translates Python to optimized machine code.

    ➤ @jit ➤ CFFI: calls C code in Python dynamically. ➤ Cython versus CFFI – fuzzy notepad ➤ pybind11: makes C++ code available in Python. ➤ Cython, pybind11, cffi – which tool should you choose? – Setfans Welt ➤ Dask: provides advanced parallelism for analytics. ➤ Comparison to Spark – Dask ➤ panda and Enhancing performance – pandas. 40
  41. Cool Down ➤ Did you use NumPy and SciPy efficiently?

    ➤ Profile your program. ➤ Consider the portability — you are writing C programs! ➤ Algorithm still does matter. ➤ Pick the most suitable solution. 41
  42. Recap ➤ static types by cdef and in function definition.

    ➤ cdef function & cpdef function. ➤ .pxd exposes cdef functions. ➤ cdef extern for C functions. 42