Cython - Making Python as Fast as C

Is your Python program too slow, or planning to integrate some C libraries?

Cython is a transcompiler to compile your Python source into C source. Then compile the C source into binary. It implies your program is able to take the benefits of the optimization from compiler. Cython is also a programming language which defined extra features from Python. The extra features make it is easier to access C, even C++ library from Python.

This talk will introduce you to Cython and cover how to set up, the extra features in Cython, examples, and many tips.

Transcript

1. Cython — 

   Making Python as Fast as C
   Mosky
   

   View Slide

2. Cython
   ➤ Cython is a source-to-source compiler (a.k.a. transcompiler).
   ➤ Cython is a superset of Python.
   ➤ Provides optional static type declarations.
   ➤ Makes writing C extensions for Python easier.
   ➤ Makes Python faster by pre-compiling and static type.
   ➤ Sometimes faster by orders of magnitude.
   ➤ The pandas uses Cython!
   2
   

   View Slide

3. .so
   C compiler
   .c
   .py import
   Python
   .pyx
   Cython
   

   View Slide

4. Mosky
   ➤ Python Charmer at Pinkoi.
   ➤ Has spoken at: PyCons in 

   TW, MY, KR, JP
   , SG, HK,

   COSCUPs, and TEDx, etc.
   ➤ Countless hours 

   on teaching Python.
   ➤ Own the Python packages like
   ZIPCodeTW.
   ➤ http://mosky.tw/
   4
   

   View Slide

5. Outline
   1. Setup
   2. Foundation
   3. Practicing
   4. Tips
   5. Uncovered Topics
   6. Other Solutions
   5
   

   View Slide

6. Setup
   

   View Slide

7. Install C Compiler
   ➤ Mac:
   ➤ $ xcode-select --install
   ➤ Ubuntu / Debian:
   ➤ $ sudo apt-get install build-essential
   ➤ Other:
   ➤ http://docs.cython.org/src/quickstart/install.html
   7
   

   View Slide

8. Install Cython
   ➤ Mac / Ubuntu / Debian:
   ➤ $ pip3 install cython
   ➤ Other:
   ➤ http://docs.cython.org/src/quickstart/install.html
   8
   

   View Slide

9. The setup.py
   from distutils.core import setup
   from Cython.Build import cythonize
   setup(
   name='cython-lab',
   ext_modules=cythonize(
   '*.pyx',
   language_level='3'
   )
   )
   9
   

   View Slide

10. The hello_cython.pyx
    print('Hello Cython!')
    10
    

    View Slide

11. Build
    ➤ Into package folder for development:
    ➤ $ python setup.py build_ext --inplace
    11
    

    View Slide

12. Foundation
    

    View Slide

13. Define Static Types
    cdef int i, j, k
    cdef float f, g[42], *h
    13
    

    View Slide

14. cdef struct Grail:
    int age
    float volume
    cdef union Food:
    char *spam
    float *eggs
    cdef enum CheeseType:
    cheddar, edam,
    camembert
    cdef enum CheeseState:
    hard = 1
    soft = 2
    runny = 3
    14
    

    View Slide

15. ctypedef unsigned long ULong
    ctypedef int* IntPtr
    15
    

    View Slide

16. cdef struct Point:
    int x
    int y
    # either `struct` or `ctypedef` is not need
    cdef Point p
    16
    

    View Slide

17. cdef:
    struct Point:
    int x
    int y
    Point p
    17
    

    View Slide

18. Define Functions
    def add(a, b):
    return a+b
    cdef add(a, b):
    return a+b
    18
    

    View Slide

19. cdef object add(object a, object b):
    return a+b
    cdef int add(int a, int b):
    return a+b
    cpdef int add(int a, int b):
    return a+b
    19
    

    View Slide

20. .pxd Exposes cdef Func
    # mylib.pxd
    cdef say_hello(char* name=?)
    # another.pyx
    from mylib cimport say_hello
    20
    

    View Slide

21. Using C Lib
    from libc.math cimport sin
    # or
    cdef extern from "math.h":
    double sin(double x)
    21
    

    View Slide

22. Function Visibility
    22
    Same 

    File
    Other
    .pyx
    .py
    Funcs in .h/.c
    Visible

    directly
    Visible

    via cdef extern
    Invisible
    cdef
    Visible

    via .pxd &
    cimport
    cpdef
    Visible

    via import
    def
    

    View Slide

23. Binding During
    23
    Same 

    File
    Other
    .pyx
    .py
    Funcs in .h/.c
    compile-time
    compile-time x
    cdef
    cpdef
    run-time
    def
    

    View Slide

24. Type Conversions
    24
    C From Py To Py
    short, int, long int int
    float, double int/float float
    char* str/bytes bytes
    array iterable list
    struct dict
    

    View Slide

25. Practicing
    

    View Slide

26. The Suggestion
    ➤ LIB_NAME.pyx
    ➤ Has functions.
    ➤ test_LIB_NAME.py
    ➤ Call the functions.
    26
    

    View Slide

27. The Overflow
    ➤ Static types may also overflow in Cython silently.
    ➤ Try to make an overflow!
    ➤ In C:
    ➤ http://j.mp/test_overflow_in_c_c
    ➤ In Cython:
    ➤ http://j.mp/overflow_in_pyx_pyx
    27
    

    View Slide

28. Using C Functions
    ➤ Try to use the functions in C.
    ➤ Playing with fork, the system call, may be fun.
    ➤ In C:
    ➤ http://j.mp/test_fork_c
    ➤ In Cython:
    ➤ http://j.mp/fork_in_pyx_pyx
    28
    

    View Slide

29. The Functions
    ➤ Write three functions
    defined in 

    def, cdef, and cpdef.
    ➤ Try to call them in:
    ➤ The same file,
    ➤ Another .pyx file,
    ➤ And a .py file.
    ➤ Refer to the table,
    “Function Visibility”.
    ➤ The Cython code:
    ➤ http://j.mp/
    lib_in_pyx_pyx
    ➤ http://j.mp/
    use_lib_in_pyx_pyx
    ➤ http://j.mp/
    test_lib_in_pyx_py
    29
    

    View Slide

30. Tips
    

    View Slide

31. cython -a
    ➤ cython -a NAME.pyx
    ➤ cython -a -3 NAME.pyx
    ➤ open NAME.html
    31
    

    View Slide

32. pyximport
    import pyximport
    pyximport.install()
    import my_pyx_lib # compile .pyx into .so
    32
    

    View Slide

33. import pyximport
    pyximport.install(pyimport=True)
    import my_py_lib # compile .py into .so
    33
    

    View Slide

34. Uncovered Topics
    

    View Slide

35. Uncovered Topics
    ➤ Differences between C and Cython expressions
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#differences-between-c-and-cython-
    expressions
    ➤ Propagating Exceptions in cdef
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#error-return-values
    35
    

    View Slide

36. ➤ Extension Type — cdef class
    ➤ http://docs.cython.org/src/userguide/
    extension_types.html
    ➤ Generic programming using Cython's Template
    ➤ http://docs.cython.org/src/userguide/fusedtypes.html
    ➤ Conditional Compilation
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#conditional-compilation
    36
    

    View Slide

37. ➤ Profiling
    ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html
    ➤ Parallelism (No GIL + OpenMP)
    ➤ http://docs.cython.org/src/userguide/parallelism.html
    ➤ Using C++ in Cython
    ➤ http://docs.cython.org/src/userguide/
    wrapping_CPlusPlus.html
    37
    

    View Slide

38. Other Solutions
    

    View Slide

39. View Slide

40. Other Solutions
    ➤ Numba: translates Python to optimized machine code.
    ➤ @jit
    ➤ CFFI: calls C code in Python dynamically.
    ➤ Cython versus CFFI – fuzzy notepad
    ➤ pybind11: makes C++ code available in Python.
    ➤ Cython, pybind11, cffi – which tool should you choose? –
    Setfans Welt
    ➤ Dask: provides advanced parallelism for analytics.
    ➤ Comparison to Spark – Dask
    ➤ panda and Enhancing performance – pandas.
    40
    

    View Slide

41. Cool Down
    ➤ Did you use NumPy and SciPy efficiently?
    ➤ Profile your program.
    ➤ Consider the portability — you are writing C programs!
    ➤ Algorithm still does matter.
    ➤ Pick the most suitable solution.
    41
    

    View Slide

42. Recap
    ➤ static types by cdef and in function definition.
    ➤ cdef function & cpdef function.
    ➤ .pxd exposes cdef functions.
    ➤ cdef extern for C functions.
    42
    

    View Slide