Cython

Cython presentation given in one of the Skyscanner lightning talks on 2013-03-07

Transcript

1. So, your Python has been slow recently? Saulius Lukauskas
2. https://secure.flickr.com/photos/restlessglobetrotter/3824486278/ Toy Example: min-coin change problem 1p 2p 5p 28p?
3. https://secure.flickr.com/photos/restlessglobetrotter/3824486278/ Toy Example: min-coin change problem 1p 2p 5p 28p? 5p 5p 5p 5p 1 + 2 + 5 = 8 coins
4. https://secure.flickr.com/photos/restlessglobetrotter/3824486278/ Toy Example: min-coin change problem 1p 2p 4p 28p? 5p
5. https://secure.flickr.com/photos/restlessglobetrotter/3824486278/ Toy Example: min-coin change problem 4p 28p? 5p 5p 5p 5p 4p 2+4 = 6 coins
6. Solution in pure Python
7. Solution in pure Python
8. In [1]: min_coin_change(28, [1,2,5]) Out[1]: 7 In [2]: min_coin_change(28, [1,2,4,5]) Out[2]: 6 Sanity check:
9. In [2]: %timeit min_coin_change(28, [1,2,5]) 10000 loops, best of 3: 163 µs per loop In [3]: %timeit min_coin_change(28, [1,2,4,5]) 1000 loops, best of 3: 262 µs per loop In [4]: %timeit min_coin_change(10000, [1,2,4,5]) 10 loops, best of 3: 73.9 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 803 ms per loop Runtime:
10. Removing the “P” from CPython https://secure.flickr.com/photos/togawanderings/8369738603/ Most importantly, Cython allows you to: Add static type declarations to Python Call C from back and forth natively
11. Optimisation #1: cp change.py change.pyx https://secure.flickr.com/photos/dmclear/5398254250/
12. Optimisation #1: cp change.py change.pyx https://secure.flickr.com/photos/dmclear/5398254250/ Add appropriate setup.py
13. https://secure.flickr.com/photos/dmclear/5398254250/ from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {'build_ext': build_ext}, ext_modules = [Extension("change", ["change.pyx"])] )
14. https://secure.flickr.com/photos/dmclear/5398254250/ In [2]: %timeit min_coin_change(28, [1,2,5]) 10000 loops, best of 3: 146 µs per loop In [3]: %timeit min_coin_change(28, [1,2,4,5]) 10000 loops, best of 3: 181 µs per loop In [4]: %timeit min_coin_change(10000, [1,2,4,5]) 10 loops, best of 3: 61.6 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 613 ms per loop
15. https://secure.flickr.com/photos/dmclear/5398254250/ In [2]: %timeit min_coin_change(28, [1,2,5]) 10000 loops, best of 3: 146 µs per loop In [3]: %timeit min_coin_change(28, [1,2,4,5]) 10000 loops, best of 3: 181 µs per loop In [4]: %timeit min_coin_change(10000, [1,2,4,5]) 10 loops, best of 3: 61.6 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 613 ms per loop In [2]: %timeit min_coin_change(28, [1,2,5]) 10000 loops, best of 3: 163 µs per loop In [3]: %timeit min_coin_change(28, [1,2,4,5]) 1000 loops, best of 3: 262 µs per loop In [4]: %timeit min_coin_change(10000, [1,2,4,5]) 10 loops, best of 3: 73.9 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 803 ms per loop
16. Optimisation #2: static typed variables https://secure.flickr.com/photos/jblndl/3934078162/
18. Import numpy’s c bindings to allow static typing arrays Make all integers C ints
19. In [4]: %timeit min_coin_change(28, [1,2,5]) 100000 loops, best of 3: 8.15 µs per loop In [5]: %timeit min_coin_change(28, [1,2,4,5]) 100000 loops, best of 3: 8.8 µs per loop In [6]: %timeit min_coin_change(10000, [1,2,4,5]) 1000 loops, best of 3: 1.74 ms per loop In [7]: %timeit min_coin_change(100000, [1,2,4,5]) 100 loops, best of 3: 17.6 ms per loop Optimised runtime:
20. In [4]: %timeit min_coin_change(28, [1,2,5]) 100000 loops, best of 3: 8.15 µs per loop In [5]: %timeit min_coin_change(28, [1,2,4,5]) 100000 loops, best of 3: 8.8 µs per loop In [6]: %timeit min_coin_change(10000, [1,2,4,5]) 1000 loops, best of 3: 1.74 ms per loop In [7]: %timeit min_coin_change(100000, [1,2,4,5]) 100 loops, best of 3: 17.6 ms per loop In [2]: %timeit min_coin_change(28, [1,2,5]) 10000 loops, best of 3: 163 µs per loop In [3]: %timeit min_coin_change(28, [1,2,4,5]) 1000 loops, best of 3: 262 µs per loop In [4]: %timeit min_coin_change(10000, [1,2,4,5]) 10 loops, best of 3: 73.9 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 803 ms per loop
21. In [7]: %timeit min_coin_change(100000, [1,2,4,5]) 100 loops, best of 3: 17.6 ms per loop In [5]: %timeit min_coin_change(100000, [1,2,4,5]) 1 loops, best of 3: 803 ms per loop 45.625 times faster
22. Native C Loop Fast lookup Native C variables
23. Further optimisations: Compiler flags, e.g. disabling bounds checking for array https://secure.flickr.com/photos/southerncomfort/86189992/ Rewriting part of the function in C
24. Other scenarios: One could use a native c library (e.g. lxml) directly from Cython bypassing the python layer https://secure.flickr.com/photos/southerncomfort/86189992/
25. More info: http://www.cython.org/ https://secure.flickr.com/photos/southerncomfort/86189992/ Saulius Lukauskas, 2013-03-07 "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" Donald Knuth