Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ben Shaw: Python is slow, make it faster with C

Ben Shaw: Python is slow, make it faster with C

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Ben Shaw:
Python is slow, make it faster with C
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Saturday, 13 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

Most people have heard that it's possible to integrate Python with C to give performance boosts to computationally heavy code, but because it seems daunting they've never given it a try. It's actually not as hard as you think. This talk covers some of the different methods of speeding up your code with C, and compares the results to those you can get from other methods, like using PyPy.

**Abstract**

Introduction

As developers, we like to work with Python because it's forgiving, quick to develop for and allows our code to be very dynamic. Unfortunately the trade-off for this magic is lower performance than compiled languages. Python can be sped up by offloading heavy algorithms to compiled C, using specially built C modules utilising the C Python API, or by integrating existing C libraries with using the python ctypes module. It is also possible to speed up Python using alternative interpreters, like PyPy, which uses a JIT compiler.

Pure Python Implementation

First we will take a look at a CPU bound algorithm written purely in Python, and see how it performs. The program reads data and prints results. The time it takes to run this will be considered the worst case scenario.

Pure C Implementation

The same program will be re-written in C, including the input and output logic, and we will compare the time it takes to run against the Python implementation. The results of the C implementation will be considered the best case scenario.

Custom Python Module with C Implementation

Python provides an API, C development headers and special C types, to allow the creation of a specially built bridges between Python and C code. In this example, the algorithm will be written in C, and bridged to Python with a custom Python/C module. Input and output takes place within Python, with C only performing the computation. With this method we can achieve near best-case speeds, at the cost of some additional (and sometimes complicated) C coding.

Bridging To C with ctypes

Introduced in Python 2.5, ctypes allows Python to integrate with pre-built C libraries without custom C code. This approach has the advantage over the custom Python/C Module of not needing to write a lot of boilerplate and bridging code in C. As with the Py/C implementation, C is used only to execute the algorithm, and Python takes care of input and output. Again, performance is close to best-case speeds, but the work to integrate with C is much less.

Alternative Python Implementations

PyPy uses a JIT compiler to offer impressive performance gains. The original Python code will be run through PyPy, and although the results might not be as quite as good as using compiled C, they come close, and the effort-to-gain ratio certainly makes it attractive option.

Conclusion

Each performance boosting option has its pros and cons, and when it's so easy to just use PyPy and get good results, why would you still use C? We'll look at some example use cases for each of the methods presented and why you would choose one over the others.

**YouTube**

https://www.youtube.com/watch?v=tGLSKXfx0m0

6b880a0b67fac54c42c77fe70d97334d?s=128

New Zealand Python User Group

September 13, 2014
Tweet

Transcript

  1. PYTHON IS SLOW Make it faster with C Ben Shaw

  2. –Everyone “It’s OK that Python isn’t fast, you can write

    your slow functions in C!”
  3. TABLE OF CONTENTS C Module vs C Types

  4. TABLE OF CONTENTS • C Module vs C Types A

    Simple Algorithm
  5. TABLE OF CONTENTS • C Module vs C Types •

    A Simple Algorithm 5 Different Implementations
  6. TABLE OF CONTENTS • C Module vs C Types •

    A Simple Algorithm • 5 Different Implementations Speed Comparisons
  7. TABLE OF CONTENTS • C Module vs C Types •

    A Simple Algorithm • 5 Different Implementations • Speed Comparisons
  8. C Python Module CTypes • The more traditional way •

    Write special wrapper code in C • More Pythonic • Integrate exceptions, help text etc • Newer (Only available since Python 2.5/2006 ) • Simpler to implement, only need to write Python code PYTHON C INTEGRATION
  9. C Python CODE CONTINUUM CTypes C Module C Python PYTHON

    INTEGRATION C Module CTypes
  10. A SIMPLE BUT COMPUTATIONALLY EXPENSIVE EXAMPLE

  11. IS A NUMBER PRIME?

  12. SUPER NAÏVE for  divisor  in  xrange(√number):*     if  number

     %  divisor  ==  0:       return  False   return  True *With provisions for 1 and 2
  13. ITERATE OVER A BUNCH OF NUMBERS and do math with

    each one.
  14. LOL REAL WORLD EXAMPLE • Easy to understand • Easy

    to implement • Runs faster in C than in Python
  15. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  16. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  17. PURE PYTHON def  is_prime_naive(number):          #  return

     false  for  even  and  1,  true  for  2  
        test_max  =  prime_test_max(number)          for  divisor  in  xrange(3,  test_max  +  1,  2):                  if  number  %  divisor  ==  0:                          return  False          return  True
  18. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          main(is_prime_naive)
  19. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          main(is_prime_naive)
  20. PURE PYTHON def  main(prime_func):          #  skip

     some  parsey  stuff          if  argv[1]  ==  '-­‐b':                  benchmark_maximum  =  int(argv[2])                  benchmark(benchmark_maximum,  prime_func)          else:                  num_to_check  =  int(argv[1])                  if  prime_func(num_to_check):                          print  "{}  is  prime.".format(num_to_check)                  else:                          print  "{}  is  not  prime.".format(num_to_check)  
  21. PURE PYTHON def  main(prime_func):          #  skip

     some  parsey  stuff          if  argv[1]  ==  '-­‐b':                  benchmark_maximum  =  int(argv[2])                  benchmark(benchmark_maximum,  prime_func)          else:                  num_to_check  =  int(argv[1])                  if  prime_func(num_to_check):                          print  "{}  is  prime.".format(num_to_check)                  else:                          print  "{}  is  not  prime.".format(num_to_check)  
  22. PURE PYTHON def  benchmark(benchmark_maximum,  prime_func):          for

     i  in  xrange(benchmark_maximum):                  prime_func(i)
  23. PURE PYTHON $  python  purepython.py  5   5  is  prime.

      $  python  purepython.py  9   9  is  not  prime.   $  time  python  purepython.py  -­‐b  1000   …
  24. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  25. PURE C bool  isPrimeNaive(const  unsigned  int  number)  {    

       /*     return  false  for  even  and  1,  true  for  2     */          unsigned  int  testMax  =  primeTestMax(number),  divisor;          for(divisor  =  3;  divisor  <=  testMax;  divisor  +=  2)                  if  (number  %  divisor  ==  0)                          return  false;          return  true;   }
  26. PURE C bool  isPrimeNaive(const  unsigned  int  number)  {    

         /*     return  false  for  even  and  1,  true  for  2     */          unsigned  int  testMax  =  primeTestMax(number),  divisor;          for(divisor  =  3;  divisor  <=  testMax;  divisor  +=  2)                  if  (number  %  divisor  ==  0)                          return  false;          return  true;   } Get unsigned int typed ceil( sqrt( number ))
  27. PURE C con_math.c libcon_math.so main.c pure c binary

  28. PURE C $  ./purec  5   5  is  prime.  

    $  ./purec  9   9  is  not  prime.   $  time  ./purec  -­‐b  1000   …
  29. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  30. PYTHON C MODULE • Write your functions in C (usually

    just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
  31. PYTHON C MODULE • Write your functions in C (usually

    just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
  32. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  33. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  34. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  35. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  36. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  37. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   } From con_math.c
  38. PYTHON C MODULE #include  <Python.h>   #include  <con_math.h>   static

     PyObject     *con_math_py_is_prime_naive(PyObject  *self,  PyObject  *args)    {          unsigned  int  numberToCheck;          if  (!PyArg_ParseTuple(args,  "I",  &numberToCheck))                  return  NULL;          bool  result  =  isPrimeNaive(numberToCheck);          return  Py_BuildValue("i",  result);   }
  39. PYTHON C MODULE • Write your functions in C (usually

    just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
  40. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  41. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  42. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  43. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  44. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  45. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  46. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  47. PYTHON C MODULE • Write your functions in C (usually

    just wrapper code) • Define the mapping between Python and C naming • Create init function to set up module
  48. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  49. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  50. PYTHON C MODULE static  PyMethodDef  ConMathMethods[]  =  {    

         {"is_prime_naive",    con_math_py_is_prime_naive,   METH_VARARGS,            "Test  if  a  number  is  prime."},          {NULL,  NULL,  0,  NULL}   };   PyMODINIT_FUNC  initcon_math(void)  {          (void)  Py_InitModule("con_math",  ConMathMethods);   }  
  51. PYTHON C MODULE setup.py from  distutils.core  import  setup,  Extension  

    setup(          ext_modules=[Extension("con_math",  ["con_math_py.c",                    "../lib/con_math.c"],  include_dirs=['../lib'])],   )  
  52. PYTHON C MODULE from  con_math  import  is_prime_naive   from  pythonlib.wrappers

     import  main   if  __name__  ==  '__main__':          main(is_prime_naive)
  53. C MODULE CF. PURE PYTHON from  pythonlib.con_math  import  is_prime_naive  

    becomes   from  con_math  import  is_prime_naive
  54. PYTHON C MODULE $  python  python_c_extension.py  5   5  is

     prime.   $  python  python_c_extension.py  9   9  is  not  prime.   $  time  python  python_c_extension.py  -­‐b  1000   …
  55. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  56. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive)
  57. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive)
  58. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive) Reference to libcon_math.so
  59. PYTHON CTYPES from  ctypes  import  CDLL   from  pythonlib.wrappers  import

     main   if  __name__  ==  '__main__':          lib  =  CDLL('../lib/libcon_math.so')          main(lib.isPrimeNaive) From libcon_math.so
  60. PYTHON CTYPES $  python  python_ctypes.py  5   5  is  prime.

      $  python  python_ctypes.py  9   9  is  not  prime.   $  time  python  python_ctypes.py  -­‐b  1000   …
  61. PRIME 5 WAYS • Pure Python (CPython) • Pure C

    • Python with C Module • Python with C Types • PyPy
  62. PYPY • Download and uncompress • (Binaries available for a

    bunch of different OSs) • Use anywhere you would have called your python binary previously • No changes to your Python code • Your C libraries may not be supported
  63. PURE PYTHON WITH PYPY $  ~/pypy/bin/pypy  purepython.py  5   5

     is  prime.   $  ~/pypy/bin/pypy  purepython.py  9   9  is  not  prime.   $  time  ~/pypy/bin/pypy  purepython.py  -­‐b  1000   …
  64. BENCHMARKS

  65. BENCHMARKS • Use the -­‐b flag to benchmark, evaluate each

    number up to 1,000,000 for primality • Measure execution time using time • total time = user + sys • Run each implementation 10 times • $  for  i  in  {1..10};  do  time  ./purec  -­‐b  1000000;  done   • Take the mean of the results
  66. EXECUTION TIME Execution Time (s) 0 1 2 3 4

    5 CPython Pure C C Module C Types PyPy 1.45 0.82 0.57 0.42 4.6
  67. SPEED UP Speed Up (x Times) 0 2 4 6

    8 10 12 CPython Pure C C Module C Types PyPy 3.2x 5.7x 8.16x 11.14x 1x
  68. CONCLUSIONS

  69. PYPY • Can give you good gains without changes to

    your Python code • Provided you don’t need to use a library it doesn’t support • Is not a solution for integrating Python with C
  70. CTYPES • No wrapper C code to write - provided

    your library already compiles to a shared object • Simple to use if you’re happy without Python exceptions and objects coming back from your library • Pretty darn fast
  71. C MODULE • You gotta write a bunch of wrapper

    C • Good integration with Python types, exceptions and objects • Faster than C types, getting closer to Pure C
  72. PURE C • What are you doing here?

  73. PURE PYTHON • It’s slow • We love it •

    It’s the reason we’re here
  74. RESOURCES • Code on GitHub
 https://github.com/beneboy/py-c-integration- example • Slides and

    writeup at http://bbit.co.nz/blog/4/