PyPy Status Update

Transcript

1. PyPy Status Update
   Antonio Cuni
   PyCon Otto
   April 07 2017
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 1 / 19
   

   View Slide

2. About me
   PyPy core dev
   pdb++, fancycompleter, capnpy, ...
   Consultant, trainer
   http://antocuni.eu
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 2 / 19
   

   View Slide

3. What is PyPy
   Alternative, fast Python implementation
   Performance: JIT compiler, advanced GC
   PyPy 5.7.1
   x86, ARM, s390x, PPC
   http://pypy.org
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 3 / 19
   

   View Slide

4. Speed
   7.4 times faster than CPython
   http://speed.pypy.org
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 4 / 19
   

   View Slide

5. Usual questions
   Is it stable/mature? Can I use it in production?
   Does package X work?
   What about Python 3?
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 5 / 19
   

   View Slide

6. PyPy in the real world (1)
   High frequency trading platform for sports bets
   low latency is a must
   PyPy used in production since 2012
   ~100 PyPy processes running 24/7
   up to 10x speedups
   after careful tuning and optimizing for PyPy
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 6 / 19
   

   View Slide

7. PyPy in the real world (2)
   Real-time online advertising auctions
   tight latency requirement (<100ms)
   high throughput (hundreds of thousands of requests per
   second)
   30% speedup
   We run PyPy basically everywhere
   Julian Berman
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 7 / 19
   

   View Slide

8. PyPy in the real world (3)
   IoT on the cloud
   5-10x faster
   We do not even run benchmarks on CPython
   because we just know that PyPy is way faster
   Tobias Oberstein
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 8 / 19
   

   View Slide

9. Usual questions
   Is it stable/mature? Can I use it in production?
   Does package X work?
   What about Python 3?
   antocuni (PyCon Otto) PyPy Status Update April 07 2017 9 / 19
   

   View Slide

10. cpyext
    Pure Python pacakges just work (TM)
    C extensions goes through cpyext
    Lots of progresses in the compatibility area
    Still slow
    Crossing the Python/C border takes ages
    lots of low hanging fruits to be taken :)
    Use CFFI if possible
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 10 / 19
    

    View Slide

11. demo
    live demo time :)
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 11 / 19
    

    View Slide

12. numpy
    It just works
    ~5 failing tests
    Better GC integration needed
    The GC does not know how much raw memory
    numpy objects are keeping alive
    It looks like a memory leak
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 12 / 19
    

    View Slide

13. Other libs
    cython
    it works
    pandas
    ~95% of passing tests
    most failures are because it cannot find Tkinter libs
    We saw some segfaults
    scipy
    ~99% of passing tests
    Numpy memory leaks?
    matplotlib
    Saving to disk works
    Jupyter inline works
    no GUI backends
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 13 / 19
    

    View Slide

14. Usual questions
    Is it stable/mature? Can I use it in production?
    Does package X work?
    What about Python 3?
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 14 / 19
    

    View Slide

15. PyPy 3.5
    PyPy 5.7.1 supports Python 3.5 (beta)
    Funded by a big grant by Mozilla (thanks!)
    https://morepypy.blogspot.com/2016/08/
    pypy-gets-funding-from-mozilla-for.html
    Still some failing tests
    Very good results on asyncio
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 15 / 19
    

    View Slide

16. PyPy 3.5 asyncio
    Between 2.87 and 8.08 times faster
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 16 / 19
    

    View Slide

17. Usual questions
    Is it stable/mature? Can I use it in production?
    Does package X work?
    What about Python 3?
    Bonus topic: vmprof
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 17 / 19
    

    View Slide

18. VMProf
    Statistical profiler
    Low overhead (~5%)
    For CPython and PyPy
    Distinguish between:
    Interpreter
    Native code
    JIT (PyPy only)
    GC (PyPy only)
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 18 / 19
    

    View Slide

19. VMProf example
    $ python -m vmprof --web foobar.py
    $ pypy -m vmprof --web foobar.py
    $ pypy -m vmprof --web --jitlog
    foobar.py
    antocuni (PyCon Otto) PyPy Status Update April 07 2017 19 / 19
    

    View Slide