PyConZA 2012: "PyPy" by Armin Rigo

The PyPy project has recently gathered a lot of attention for its progress in speeding up the Python language -- it is the fastest, most compatible and most stable 'alternative' Python interpreter. No longer merely a research curiosity, PyPy is now suitable for production use.

The speed comes from a custom Just-in-Time compiler (JIT). It is the first Virtual Machine to have a JIT generated automatically from the interpreter of the language, for a complicated language like Python. It makes it correct and complete by construction. The JIT itself is a tracing JIT, roughly similar to TraceMonkey.

most Python benchmarks run much faster than with CPython or Psyco
the real-world PyPy compiler toolchain itself (200 KLocs) runs twice as fast
supports x86 (32 or 64 bit), ARM (v7), and soon POWER64
full compatibility with CPython (more than Jython/IronPython)
ctypes, CFFI and C++ support to call C/C++ libraries from Python (fast)
supports Stackless Python (in-progress)
integrates existing CPython C extensions (slowly)
In this talk we will see examples of what PyPy is best at (pure Python code that runs for a while), what compatibility issues you may run into (very few), how to use CPython C extension modules (you can more or less, but it's slow), as well as dig a bit below the surface and use some tools to view the x86 machine code that was produced by the JIT.

I will end the talk with an overview of Software Transactional Memory (STM) and how it promizes to give a PyPy without the Global Interpreter Lock (GIL), i.e. able to run a single process using multiple cores.

Transcript

1. PyPy Armin Rigo PyCon ZA 2012 October 4, 2012 Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 1 / 15
2. PyPy is... Another Python interpreter with a JIT compiler Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 1 / 15
3. PyPy was... Around since 2003 (advertised as) production ready since December 2010 I release 1.4 Funding I EU FP6 programme I Eurostars programme I donations I ... Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 2 / 15
4. PyPy 1.9: current status Faster I 1.7x than 1.5 (Summer 2011) I 2.2x than 1.4 (December 2010) I 5.5x than CPython Implements Python 2.7.3 Many more “PyPy-friendly” programs than before Packaging I Debian, Ubuntu, Fedora, Homebrew, Gentoo, ArchLinux, ... I Windows (32bit only), OS X C extension compatibility I runs (big part of) PyOpenSSL and lxml Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 3 / 15
5. PyPy organization Part of SFC -- Software Freedom Conservancy I Bradley successfully fighting U.S. bureaucracy I we are happy about it Funding model I py3k, numpy, STM I more than 100’000$ in donations I from individuals, large companies and the PSF Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 4 / 15
6. PyPy’s JIT compiler Removes abstraction Almost never gives up x86-32, x86-64, ARMv7, (POWER64) (Works with other languages) Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 5 / 15
7. Real world applications Positive feedback http://speed.pypy.org/ Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 6 / 15
8. py3k py3k branch in mercurial I developed in parallel I Python 3 written in Python 2 Focus on correctness Dropped some interpreter optimizations for now First 90% done, remaining 90% not done Majority of the funds by Google Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 7 / 15
9. NumPy progress going slowly multi dimensional arrays, broadcasting, fancy indexing all dtypes, except complex, strings and objects good results for performance Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 8 / 15
10. STM Software Transactional Memory “Remove the GIL” But also, new models (better than threads) Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 9 / 15
11. Calling C Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 10 / 15
12. Calling C landscape CPython C extensions SWIG, SIP, wrapper generators ctypes Cython CFFI (our new thing) Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 11 / 15
13. CFFI Example >>> from cffi import FFI >>> ffi = FFI() >>> ffi.cdef(""" ... int printf(const char *format, ...); ... """) >>> C = ffi.dlopen(None) >>> arg = ffi.new("char[]", "world") >>> C.printf("hi there, %s!\n", arg) hi there, world! Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 12 / 15
14. CFFI Many more examples Including macro calls and most subtleties of C http://cffi.readthedocs.org Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 13 / 15
15. STM Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 14 / 15
16. Conclusion Try out PyPy on real code http://pypy.org/ Thank you! Armin Rigo (PyCon ZA 2012) PyPy in Production October 4, 2012 15 / 15