$30 off During Our Annual Pro Sale. View Details »

Cython - Making Python as Fast as C

Cython - Making Python as Fast as C

Is your Python program too slow, or planning to integrate some C libraries?

Cython is a transcompiler to compile your Python source into C source. Then compile the C source into binary. It implies your program is able to take the benefits of the optimization from compiler. Cython is also a programming language which defined extra features from Python. The extra features make it is easier to access C, even C++ library from Python.

This talk will introduce you to Cython and cover how to set up, the extra features in Cython, examples, and many tips.

Mosky Liu

April 24, 2014
Tweet

More Decks by Mosky Liu

Other Decks in Programming

Transcript

  1. Cython — 

    Making Python as Fast as C
    Mosky

    View Slide

  2. Cython
    ➤ Cython is a source-to-source compiler (a.k.a. transcompiler).
    ➤ Cython is a superset of Python.
    ➤ Provides optional static type declarations.
    ➤ Makes writing C extensions for Python easier.
    ➤ Makes Python faster by pre-compiling and static type.
    ➤ Sometimes faster by orders of magnitude.
    ➤ The pandas uses Cython!
    2

    View Slide

  3. .so
    C compiler
    .c
    .py import
    Python
    .pyx
    Cython

    View Slide

  4. Mosky
    ➤ Python Charmer at Pinkoi.
    ➤ Has spoken at: PyCons in 

    TW, MY, KR, JP
    , SG, HK,

    COSCUPs, and TEDx, etc.
    ➤ Countless hours 

    on teaching Python.
    ➤ Own the Python packages like
    ZIPCodeTW.
    ➤ http://mosky.tw/
    4

    View Slide

  5. Outline
    1. Setup
    2. Foundation
    3. Practicing
    4. Tips
    5. Uncovered Topics
    6. Other Solutions
    5

    View Slide

  6. Setup

    View Slide

  7. Install C Compiler
    ➤ Mac:
    ➤ $ xcode-select --install
    ➤ Ubuntu / Debian:
    ➤ $ sudo apt-get install build-essential
    ➤ Other:
    ➤ http://docs.cython.org/src/quickstart/install.html
    7

    View Slide

  8. Install Cython
    ➤ Mac / Ubuntu / Debian:
    ➤ $ pip3 install cython
    ➤ Other:
    ➤ http://docs.cython.org/src/quickstart/install.html
    8

    View Slide

  9. The setup.py
    from distutils.core import setup
    from Cython.Build import cythonize
    setup(
    name='cython-lab',
    ext_modules=cythonize(
    '*.pyx',
    language_level='3'
    )
    )
    9

    View Slide

  10. The hello_cython.pyx
    print('Hello Cython!')
    10

    View Slide

  11. Build
    ➤ Into package folder for development:
    ➤ $ python setup.py build_ext --inplace
    11

    View Slide

  12. Foundation

    View Slide

  13. Define Static Types
    cdef int i, j, k
    cdef float f, g[42], *h
    13

    View Slide

  14. cdef struct Grail:
    int age
    float volume
    cdef union Food:
    char *spam
    float *eggs
    cdef enum CheeseType:
    cheddar, edam,
    camembert
    cdef enum CheeseState:
    hard = 1
    soft = 2
    runny = 3
    14

    View Slide

  15. ctypedef unsigned long ULong
    ctypedef int* IntPtr
    15

    View Slide

  16. cdef struct Point:
    int x
    int y
    # either `struct` or `ctypedef` is not need
    cdef Point p
    16

    View Slide

  17. cdef:
    struct Point:
    int x
    int y
    Point p
    17

    View Slide

  18. Define Functions
    def add(a, b):
    return a+b
    cdef add(a, b):
    return a+b
    18

    View Slide

  19. cdef object add(object a, object b):
    return a+b
    cdef int add(int a, int b):
    return a+b
    cpdef int add(int a, int b):
    return a+b
    19

    View Slide

  20. .pxd Exposes cdef Func
    # mylib.pxd
    cdef say_hello(char* name=?)
    # another.pyx
    from mylib cimport say_hello
    20

    View Slide

  21. Using C Lib
    from libc.math cimport sin
    # or
    cdef extern from "math.h":
    double sin(double x)
    21

    View Slide

  22. Function Visibility
    22
    Same 

    File
    Other
    .pyx
    .py
    Funcs in .h/.c
    Visible

    directly
    Visible

    via cdef extern
    Invisible
    cdef
    Visible

    via .pxd &
    cimport
    cpdef
    Visible

    via import
    def

    View Slide

  23. Binding During
    23
    Same 

    File
    Other
    .pyx
    .py
    Funcs in .h/.c
    compile-time
    compile-time x
    cdef
    cpdef
    run-time
    def

    View Slide

  24. Type Conversions
    24
    C From Py To Py
    short, int, long int int
    float, double int/float float
    char* str/bytes bytes
    array iterable list
    struct dict

    View Slide

  25. Practicing

    View Slide

  26. The Suggestion
    ➤ LIB_NAME.pyx
    ➤ Has functions.
    ➤ test_LIB_NAME.py
    ➤ Call the functions.
    26

    View Slide

  27. The Overflow
    ➤ Static types may also overflow in Cython silently.
    ➤ Try to make an overflow!
    ➤ In C:
    ➤ http://j.mp/test_overflow_in_c_c
    ➤ In Cython:
    ➤ http://j.mp/overflow_in_pyx_pyx
    27

    View Slide

  28. Using C Functions
    ➤ Try to use the functions in C.
    ➤ Playing with fork, the system call, may be fun.
    ➤ In C:
    ➤ http://j.mp/test_fork_c
    ➤ In Cython:
    ➤ http://j.mp/fork_in_pyx_pyx
    28

    View Slide

  29. The Functions
    ➤ Write three functions
    defined in 

    def, cdef, and cpdef.
    ➤ Try to call them in:
    ➤ The same file,
    ➤ Another .pyx file,
    ➤ And a .py file.
    ➤ Refer to the table,
    “Function Visibility”.
    ➤ The Cython code:
    ➤ http://j.mp/
    lib_in_pyx_pyx
    ➤ http://j.mp/
    use_lib_in_pyx_pyx
    ➤ http://j.mp/
    test_lib_in_pyx_py
    29

    View Slide

  30. Tips

    View Slide

  31. cython -a
    ➤ cython -a NAME.pyx
    ➤ cython -a -3 NAME.pyx
    ➤ open NAME.html
    31

    View Slide

  32. pyximport
    import pyximport
    pyximport.install()
    import my_pyx_lib # compile .pyx into .so
    32

    View Slide

  33. import pyximport
    pyximport.install(pyimport=True)
    import my_py_lib # compile .py into .so
    33

    View Slide

  34. Uncovered Topics

    View Slide

  35. Uncovered Topics
    ➤ Differences between C and Cython expressions
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#differences-between-c-and-cython-
    expressions
    ➤ Propagating Exceptions in cdef
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#error-return-values
    35

    View Slide

  36. ➤ Extension Type — cdef class
    ➤ http://docs.cython.org/src/userguide/
    extension_types.html
    ➤ Generic programming using Cython's Template
    ➤ http://docs.cython.org/src/userguide/fusedtypes.html
    ➤ Conditional Compilation
    ➤ http://docs.cython.org/src/userguide/
    language_basics.html#conditional-compilation
    36

    View Slide

  37. ➤ Profiling
    ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html
    ➤ Parallelism (No GIL + OpenMP)
    ➤ http://docs.cython.org/src/userguide/parallelism.html
    ➤ Using C++ in Cython
    ➤ http://docs.cython.org/src/userguide/
    wrapping_CPlusPlus.html
    37

    View Slide

  38. Other Solutions

    View Slide

  39. View Slide

  40. Other Solutions
    ➤ Numba: translates Python to optimized machine code.
    ➤ @jit
    ➤ CFFI: calls C code in Python dynamically.
    ➤ Cython versus CFFI – fuzzy notepad
    ➤ pybind11: makes C++ code available in Python.
    ➤ Cython, pybind11, cffi – which tool should you choose? –
    Setfans Welt
    ➤ Dask: provides advanced parallelism for analytics.
    ➤ Comparison to Spark – Dask
    ➤ panda and Enhancing performance – pandas.
    40

    View Slide

  41. Cool Down
    ➤ Did you use NumPy and SciPy efficiently?
    ➤ Profile your program.
    ➤ Consider the portability — you are writing C programs!
    ➤ Algorithm still does matter.
    ➤ Pick the most suitable solution.
    41

    View Slide

  42. Recap
    ➤ static types by cdef and in function definition.
    ➤ cdef function & cpdef function.
    ➤ .pxd exposes cdef functions.
    ➤ cdef extern for C functions.
    42

    View Slide