HPy: a future-proof way of extending Python?

HPy (https://github.com/pyhandle/hpy) is a joint project which is being developed by PyPy, CPython and Cython developers. It aims to design a better C API for writing Python extensions which is more friendly to alternative implementations and which would allow CPython itself more freedom to experiment (e.g. by using a real GC instead of refcounting).

Transcript

1. HPy: a future-proof way of extending Python? Antonio Cuni Python Language Summit 2020 antocuni (Python Language Summit 2020) HPy 1 / 16
2. What is HPy? An attempt to "solve" some of the problems with the C-API Idea born at EP2019, discussion between: PyPy devs CPython devs Cython dev(s) https://github.com/pyhandle/hpy https://hpy.readthedocs.io/en/latest/ https://mail.python.org/archives/list/hpy-dev@python.org/ #hpy on freenode antocuni (Python Language Summit 2020) HPy 1 / 16
3. What is the problem? The C-API is too tied to CPython internals Many implementation details are exposed by/backed in the API CPython can’t evolve / change its details Alternative implementations have to "emulate" CPython Related work: Stable ABI, PEP 384 https://pythoncapi.readthedocs.io/ antocuni (Python Language Summit 2020) HPy 2 / 16
4. Exposed details Reference counting Objects as C pointers (PyObject *) Implicit assumption that Python-level is is the same as C-level == Structs not fully opaque ob_refcnt, ob_type, PyTypeObject, ... PEP 384 goes in the right direction Borrowed references PyList_GetItem ==> borrowed reference In PyPy, [1, 2, 3, 4] is represented as a C long[] No references to borrow! antocuni (Python Language Summit 2020) HPy 3 / 16
5. Refcounting vs GC Refcounting prevents using a "real" GC The GC is not (only) about collecting garbage! It should be called Memory Manager State-of-the-art GCs: super-fast allocation, fast deallocation no/minimal pauses multi-threading (not the PyPy GC) ... E.g., on gcbench.py, PyPy is ~25x faster than CPython! antocuni (Python Language Summit 2020) HPy 4 / 16
6. CPython Can’t evolve the VM Can’t experiment with many ideas Refcounting is a big problem Refcounting is the major blocker / problem for the Gilec- tomy. The next step with the Gilectomy is to switch CPython to tracing garbage collection, which is much more amenable to running across multiple threads. Larry Hastings antocuni (Python Language Summit 2020) HPy 5 / 16
7. Alternative implementations Standard solution: compatibility layer to emulate CPython PyPy: cpyext IronPython: IronClad Jython: Jython Native Interface ... Massive amout of precious developer hours wasted Poor results https://morepypy.blogspot.com/2018/09/ inside-cpyext-why-emulating-cpython-c.html antocuni (Python Language Summit 2020) HPy 6 / 16
8. HPy solution Fully opaque data structures by default GC-friendly: handles Like file descriptors of Windows’s HANDLE HPy_Dup ==> Py_INCREF HPy_Close() ==> Py_DECREF Each handle must be closed individually PyObject *a = PyLong_FromLong(42); PyObject *b = a; Py_INCREF(b); Py_DECREF(a); Py_DECREF(a); // Ok HPy a = HPyLong_FromLong(ctx, 42); HPy b = HPy_Dup(ctx, a); HPy_Close(a); HPy_Close(a); // WRONG! HPyContext passed everywhere (useful for subinterpreters, etc.) antocuni (Python Language Summit 2020) HPy 7 / 16
9. HPy strategy to conquer the world Zero overhead on CPython Using macros and static inline to map HPy to C-API Incremental adoption Port existing extensions one function at a time Faster on alternative implementations 3x faster than cpyext on PyPy 2x faster on GraalPython (could be optimized further) Better debugging experience "The handle created at foo.c:543 was never closed" (Optional) Universal ABI: one binary for multiple versions/implementations Cython backend antocuni (Python Language Summit 2020) HPy 8 / 16
10. HPy targets foo.c #include <hpy.h> CPython ABI hpy/cpython.h gcc foo.cpython-37m.so CPython antocuni (Python Language Summit 2020) HPy 9 / 16
11. HPy targets foo.c #include <hpy.h> CPython ABI hpy/cpython.h HPy Universal ABI hpy/universal.h gcc foo.cpython-37m.so foo.hpy-1.so CPython PyPy hpy.universal hpy/universal.cpython- 37m.so GraalPython antocuni (Python Language Summit 2020) HPy 10 / 16
12. HPy targets foo.c #include <hpy.h> CPython ABI hpy/cpython.h HPy Universal ABI hpy/universal.h BarPython ABI bar-python/hpy.h gcc foo.cpython-37m.so foo.hpy-1.so CPython PyPy hpy.universal hpy/universal.cpython- 37m.so GraalPython foo.bar-python.so BarPython antocuni (Python Language Summit 2020) HPy 11 / 16
13. CPython ABI // hpy/cpython.h typedef struct { PyObject *_o; } HPy; static inline HPy HPy_Dup(HPyContext ctx, HPy handle) { Py_XINCREF(handle._o); return handle; } static inline HPy HPyLong_FromLong(HPyContext ctx, long v) { return (HPy){PyLong_FromLong(v)}; } antocuni (Python Language Summit 2020) HPy 12 / 16
14. Universal ABI // hpy/universal.h /* a word-sized opaque field: can be an index, a pointer, whatever */ typedef struct { HPy_ssize_t _i; } HPy; struct _HPyContext_s { int ctx_version; ... HPy (*ctx_Dup)(HPyContext ctx, HPy h); HPy (*ctx_Long_FromLong)(HPyContext ctx, long value); ... }; typedef struct _HPyContext_s *HPyContext; static inline HPy HPy_Dup(HPyContext ctx, HPy h) { return ctx->ctx_Dup ( ctx, h ); } static inline HPy HPyLong_FromLong(HPyContext ctx, long value) { return ctx->ctx_Long_FromLong ( ctx, value ); } antocuni (Python Language Summit 2020) HPy 13 / 16
15. Implementation on PyPy # pseudocode class HandleManager: def __init__(self): # GC-managed! The items inside the list might move in memory self.handles_w = [] def new(self, w_obj): i = self._find_empty_index() self.handles_w[i] = w_obj return i ... def ctx_Long_FromLong(space, value): w_obj = space.newint(value) return handle_manager.new(w_obj) def make_context(): ctx = lltype.malloc(HPyContext) ctx.ctx_Long_FromLong = ctx_Long_FromLong ... return ctx antocuni (Python Language Summit 2020) HPy 14 / 16
16. Current status No type objects yet (WIP) Produce native CPython and HPy Universal extensions ultrajson-hpy CPython ABI: as fast as ultrajson Universal ABI on CPython: 10% slower Universal ABI on PyPy: 3x faster https://github.com/pyhandle/ultrajson-hpy The current approach works on the technical level The biggest challenge will be adoption Speed on PyPy might be the most important driving force in the short term antocuni (Python Language Summit 2020) HPy 15 / 16
17. Next steps Short term Custom types in C Validate the approach by porting PicoNumpy https://github.com/paugier/piconumpy Debug mode Medium term Cython backend Experiment with the real numpy Long term PEP Official PSF/CPython endorsement? Hypothetical sci-fi future :) CPython switches to HPy internally antocuni (Python Language Summit 2020) HPy 16 / 16