Ben Shaw: Python is slow, make it faster with C

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Ben Shaw:
Python is slow, make it faster with C
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Saturday, 13 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

Most people have heard that it's possible to integrate Python with C to give performance boosts to computationally heavy code, but because it seems daunting they've never given it a try. It's actually not as hard as you think. This talk covers some of the different methods of speeding up your code with C, and compares the results to those you can get from other methods, like using PyPy.

**Abstract**

Introduction

As developers, we like to work with Python because it's forgiving, quick to develop for and allows our code to be very dynamic. Unfortunately the trade-off for this magic is lower performance than compiled languages. Python can be sped up by offloading heavy algorithms to compiled C, using specially built C modules utilising the C Python API, or by integrating existing C libraries with using the python ctypes module. It is also possible to speed up Python using alternative interpreters, like PyPy, which uses a JIT compiler.

Pure Python Implementation

First we will take a look at a CPU bound algorithm written purely in Python, and see how it performs. The program reads data and prints results. The time it takes to run this will be considered the worst case scenario.

Pure C Implementation

The same program will be re-written in C, including the input and output logic, and we will compare the time it takes to run against the Python implementation. The results of the C implementation will be considered the best case scenario.

Custom Python Module with C Implementation

Python provides an API, C development headers and special C types, to allow the creation of a specially built bridges between Python and C code. In this example, the algorithm will be written in C, and bridged to Python with a custom Python/C module. Input and output takes place within Python, with C only performing the computation. With this method we can achieve near best-case speeds, at the cost of some additional (and sometimes complicated) C coding.

Bridging To C with ctypes

Introduced in Python 2.5, ctypes allows Python to integrate with pre-built C libraries without custom C code. This approach has the advantage over the custom Python/C Module of not needing to write a lot of boilerplate and bridging code in C. As with the Py/C implementation, C is used only to execute the algorithm, and Python takes care of input and output. Again, performance is close to best-case speeds, but the work to integrate with C is much less.

Alternative Python Implementations

PyPy uses a JIT compiler to offer impressive performance gains. The original Python code will be run through PyPy, and although the results might not be as quite as good as using compiled C, they come close, and the effort-to-gain ratio certainly makes it attractive option.

Conclusion

Each performance boosting option has its pros and cons, and when it's so easy to just use PyPy and get good results, why would you still use C? We'll look at some example use cases for each of the methods presented and why you would choose one over the others.

**YouTube**

https://www.youtube.com/watch?v=tGLSKXfx0m0

Transcript

1. PYTHON IS SLOW Make it faster with C Ben Shaw
2. –Everyone “It’s OK that Python isn’t fast, you can write your slow functions in C!”
3. TABLE OF CONTENTS C Module vs C Types
4. TABLE OF CONTENTS • C Module vs C Types A Simple Algorithm
5. TABLE OF CONTENTS • C Module vs C Types • A Simple Algorithm 5 Different Implementations
6. TABLE OF CONTENTS • C Module vs C Types • A Simple Algorithm • 5 Different Implementations Speed Comparisons
7. TABLE OF CONTENTS • C Module vs C Types • A Simple Algorithm • 5 Different Implementations • Speed Comparisons
8. C Python Module CTypes • The more traditional way • Write special wrapper code in C • More Pythonic • Integrate exceptions, help text etc • Newer (Only available since Python 2.5/2006 ) • Simpler to implement, only need to write Python code PYTHON C INTEGRATION
9. C Python CODE CONTINUUM CTypes C Module C Python PYTHON INTEGRATION C Module CTypes
10. A SIMPLE BUT COMPUTATIONALLY EXPENSIVE EXAMPLE
11. IS A NUMBER PRIME?
12. SUPER NAÏVE for  divisor  in  xrange(√number):*     if  number  %  divisor  ==  0:       return  False   return  True *With provisions for 1 and 2
13. ITERATE OVER A BUNCH OF NUMBERS and do math with each one.
14. LOL REAL WORLD EXAMPLE • Easy to understand • Easy to implement • Runs faster in C than in Python
15. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
16. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
17. PURE PYTHON def  is_prime_naive(number):          #  return  false  for  even  and  1,  true  for  2  
        test_max  =  prime_test_max(number)          for  divisor  in  xrange(3,  test_max  +  1,  2):                  if  number  %  divisor  ==  0:                          return  False          return  True
18. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          main(is_prime_naive)
19. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          main(is_prime_naive)
20. PURE PYTHON def  main(prime_func):          #  skip  some  parsey  stuff          if  argv[1]  ==  '-­‐b':                  benchmark_maximum  =  int(argv[2])                  benchmark(benchmark_maximum,  prime_func)          else:                  num_to_check  =  int(argv[1])                  if  prime_func(num_to_check):                          print  "{}  is  prime.".format(num_to_check)                  else:                          print  "{}  is  not  prime.".format(num_to_check)  
21. PURE PYTHON def  main(prime_func):          #  skip  some  parsey  stuff          if  argv[1]  ==  '-­‐b':                  benchmark_maximum  =  int(argv[2])                  benchmark(benchmark_maximum,  prime_func)          else:                  num_to_check  =  int(argv[1])                  if  prime_func(num_to_check):                          print  "{}  is  prime.".format(num_to_check)                  else:                          print  "{}  is  not  prime.".format(num_to_check)  
22. PURE PYTHON def  benchmark(benchmark_maximum,  prime_func):          for  i  in  xrange(benchmark_maximum):                  prime_func(i)
23. PURE PYTHON $  python  purepython.py  5   5  is  prime.   $  python  purepython.py  9   9  is  not  prime.   $  time  python  purepython.py  -­‐b  1000   …
24. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
25. PURE C bool  isPrimeNaive(const  unsigned  int  number)  {        /*     return  false  for  even  and  1,  true  for  2     */          unsigned  int  testMax  =  primeTestMax(number),  divisor;          for(divisor  =  3;  divisor  <=  testMax;  divisor  +=  2)                  if  (number  %  divisor  ==  0)                          return  false;          return  true;   }
26. PURE C bool  isPrimeNaive(const  unsigned  int  number)  {          /*     return  false  for  even  and  1,  true  for  2     */          unsigned  int  testMax  =  primeTestMax(number),  divisor;          for(divisor  =  3;  divisor  <=  testMax;  divisor  +=  2)                  if  (number  %  divisor  ==  0)                          return  false;          return  true;   } Get unsigned int typed ceil( sqrt( number ))
27. PURE C con_math.c libcon_math.so main.c pure c binary
28. PURE C $  ./purec  5   5  is  prime.   $  ./purec  9   9  is  not  prime.   $  time  ./purec  -­‐b  1000   …
29. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
30. PYTHON C MODULE • Write your functions in C (usually just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
31. PYTHON C MODULE • Write your functions in C (usually just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
32. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
33. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
34. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
35. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
36. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
37. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   } From con_math.c
38. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static  PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
39. PYTHON C MODULE • Write your functions in C (usually just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
40. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
41. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
42. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
43. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
44. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
45. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
46. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
47. PYTHON C MODULE • Write your functions in C (usually just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
48. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
49. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
50. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {          {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
51. PYTHON C MODULE setup.py from  distutils.core  import  setup,  Extension   setup(          ext_modules=[Extension("con_math",  ["con_math_py.c",                    "../lib/con_math.c"],  include_dirs=['../lib'])],   )  
52. PYTHON C MODULE from  con_math  import  is_prime_naive   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          main(is_prime_naive)
53. C MODULE CF. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive   becomes   from  con_math  import  is_prime_naive
54. PYTHON C MODULE $  python  python_c_extension.py  5   5  is  prime.   $  python  python_c_extension.py  9   9  is  not  prime.   $  time  python  python_c_extension.py  -­‐b  1000   …
55. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
56. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive)
57. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive)
58. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive) Reference to libcon_math.so
59. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import  main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive) From libcon_math.so
60. PYTHON CTYPES $  python  python_ctypes.py  5   5  is  prime.   $  python  python_ctypes.py  9   9  is  not  prime.   $  time  python  python_ctypes.py  -­‐b  1000   …
61. PRIME 5 WAYS • Pure Python (CPython) • Pure C • Python with C Module • Python with C Types • PyPy
62. PYPY • Download and uncompress • (Binaries available for a bunch of different OSs) • Use anywhere you would have called your python binary previously • No changes to your Python code • Your C libraries may not be supported
63. PURE PYTHON WITH PYPY $  ~/pypy/bin/pypy  purepython.py  5   5  is  prime.   $  ~/pypy/bin/pypy  purepython.py  9   9  is  not  prime.   $  time  ~/pypy/bin/pypy  purepython.py  -­‐b  1000   …
64. BENCHMARKS
65. BENCHMARKS • Use the -­‐b flag to benchmark, evaluate each number up to 1,000,000 for primality • Measure execution time using time • total time = user + sys • Run each implementation 10 times • $  for  i  in  {1..10};  do  time  ./purec  -­‐b  1000000;  done   • Take the mean of the results
66. EXECUTION TIME Execution Time (s) 0 1 2 3 4 5 CPython Pure C C Module C Types PyPy 1.45 0.82 0.57 0.42 4.6
67. SPEED UP Speed Up (x Times) 0 2 4 6 8 10 12 CPython Pure C C Module C Types PyPy 3.2x 5.7x 8.16x 11.14x 1x
68. CONCLUSIONS
69. PYPY • Can give you good gains without changes to your Python code • Provided you don’t need to use a library it doesn’t support • Is not a solution for integrating Python with C
70. CTYPES • No wrapper C code to write - provided your library already compiles to a shared object • Simple to use if you’re happy without Python exceptions and objects coming back from your library • Pretty darn fast
71. C MODULE • You gotta write a bunch of wrapper C • Good integration with Python types, exceptions and objects • Faster than C types, getting closer to Pure C
72. PURE C • What are you doing here?
73. PURE PYTHON • It’s slow • We love it • It’s the reason we’re here
74. RESOURCES • Code on GitHub
 https://github.com/beneboy/py-c-integration- example • Slides and writeup at http://bbit.co.nz/blog/4/