Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyPy3, Numpy, and maybe a GIL-less future by Armin Rigo

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 05, 2017

PyPy3, Numpy, and maybe a GIL-less future by Armin Rigo

PyPy is a Python implementation, alternative to the standard CPython. PyPy comes with a JIT (Just-in-Time compiler). We'll see the recent developments:

* PyPy now supports either Python 2.7 or (in beta) Python 3.5.

* Numpy and the scientific stack are getting ever closer to fully working.

We will also mention a potential future direction: getting rid of the GIL (Global Interpreter Lock). The Python community has been discussing removing the GIL for a long time, with no success so far. Targeting PyPy instead of CPython would still be quite involved, but more doable.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 05, 2017
Tweet

Transcript

  1. PyPy meets Python 3 and Numpy (and other What’s New

    topics) Armin Rigo PyConZA 2017 October 2017
  2. What’s New In PyPy 1. Python 3.5 2. Numpy, Cython,

    Pandas, ... 3. pypy-stm? 4. RevDB: reverse debugging 5. Others...
  3. What is PyPy PyPy is another implementation of Python Mostly,

    drop-in replacement Supports Python 2.7, and almost Python 3.5 Comes with a JIT, good performance
  4. PyPy 3.5

  5. PyPy 3.5 Python 3.5 support released in gamma as stable

    (and mostly as fast) as PyPy 2.7, same JIT/GC/etc. what could be wrong is a few details everywhere please try it and report issues! Thanks to Mozilla for funding this work!
  6. PyPy 3.5 Async HTTP benchmarks:

  7. PyPy 3.5 status Roughly complete 3.5 support (plus f-strings!) Reasonably

    good performance Tested mostly on Linux so far First "final" to be released very soon Python 3.6 to follow
  8. Scientific stack

  9. Scientific stack Numpy or Numpypy? Numpy = the standard numpy

    library Numpypy = our own partial reimplementation
  10. Scientific stack Numpy or Numpypy? Numpy = the standard numpy

    library (use this) Numpypy = our own partial reimplementation (deprecated)
  11. Scientific stack Numpy works (99.9%) On both PyPy 2.7 and

    PyPy 3.5 The rest of the scientific stack mostly works too (Jupyter, Matplotlib, Pandas, etc.)
  12. Cython, cpyext Cython mostly works Actually, any CPython C extension

    module mostly works Thanks to cpyext, our CPython C API emulation layer
  13. Performance? Numpy/Pandas/etc. are all slow-ish at the Python-C boundary Less

    so than last year but still Complex algorithms written inside Numpy in C or Fortran have the same speed, of course lots of ndarray[index] => slow one call to numpy.linalg.eig() => fast speed hack: p = ffi.cast("double *", ffi.from_buffer(ndarray))
  14. Performance? We have plans to improve Funding help welcome For

    now: try it out on your own code and see
  15. Software Transactional Memory

  16. Software Transactional Memory pypy-stm: getting rid of the Global Interpreter

    Lock
  17. Software Transactional Memory ...unfortunately, this approach does not seem to

    work :-(
  18. Software Transactional Memory Unstable performance: "Conflicts" between threads are hard

    to find Fix one conflict, usually only to uncover the next one As long as there is one, performance is bad You may fix enough to get good performance... but: Continue developing the program, and you’ll often reintroduce conflicts (Also, hard to test for, reliably)
  19. PyPy-nogil? Another possibility: a GIL-free but non-STM PyPy Large amount

    of work, large maintenance burden Lots of hype, but we would need serious commercial funding and backing
  20. Reverse Debugger

  21. Reverse Debugger RevDB: The essential tool you need once a

    year
  22. Reverse Debugger Debugger with the ability to go forward and

    backward in time Watchpoints to know when a value changes, while going in either direction http://bitbucket.org/pypy/revdb
  23. Others

  24. JIT improvements Reductions in the warm-up time Consumes less memory,

    too
  25. VMProf A good high-performance profiler for Python code pip install

    vmprof Works on CPython and on PyPy The PyPy version shows the machine code generated by the JIT
  26. CFFI improvements CFFI: calling C from Python or the other

    way around Works identically on CPython and on PyPy Biggest improvement of last year is embedding Use CFFI to embed Python inside another program---much easier than with the CPython C API, and works identically on CPython or PyPy too
  27. Next year?

  28. Next year? Polish PyPy 3.5 / 3.6 Polish Numpy and

    the scientific stack Port RevDB to PyPy 3.5 Better memory profiling
  29. Question & answers Polish PyPy 3.5 / 3.6 Polish Numpy

    and the scientific stack Port RevDB to PyPy 3.5 Better memory profiling Thank you! http://pypy.org/