PyConZA 2015: "How I learnt to stop worrying and love Boost.Python" by Bruce Merry

The Zen of Python dictates that there should be one - and preferably only one - obvious way to do something. However, when it comes to interoperation with C and C++, there is a multitude of options: the Python C API, Boost.Python, ctypes, cffi, Cython. I will describe my quest to find the right interface for a high-performance networking library.

The talk is largely a case study in applying Boost.Python, and will look at some issues such as the Global Interpreter Lock, handling KeyboardInterrupt cleanly, and managing object lifetime. I will briefly mention some of the alternative tools to explain why I settled on Boost.Python. It is not a complete Boost.Python tutorial, but rather aims to give a sense of the flavour and show how it's used in a real application.

For obvious reasons, this talk will have a lot of C++ code in it, and some familiarity with C++ will be useful.

Transcript

1. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary How I Learnt To Stop Worrying And Love Boost.Python An adventure in high-performance networking Bruce Merry SKA South Africa PyCon ZA 2015 1 / 50
2. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 2 / 50
3. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 3 / 50
4. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary A MeerKAT Dish 4 / 50
5. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Lots Of Dishes 5 / 50
6. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 6 / 50
7. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary High-performance Networking Streaming Protocol for Exchange of Astronomical Data Generic, not astronomy-specific at all Designed for moving numpy arrays around Used extensively in KAT-7 and MeerKAT Lossy one-way UDP transmission, multicast-friendly New library handles 20 Gbps in a single stream 7 / 50
8. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 8 / 50
9. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Code 1 namespace petshop { 2 3 class Parrot { 4 int volts; 5 public: 6 void voom(); 7 void set_volts(int volts); 8 int get_volts() const; 9 }; 10 11 } 9 / 50
10. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 10 / 50
11. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Python C API 1 #include "Python.h" 2 #include "petshop.h" 3 #include <memory> 4 5 typedef struct { 6 PyObject_HEAD 7 petshop::Parrot wrapped; 8 } Parrot; 9 10 static PyTypeObject ParrotType = { 11 PyVarObject_HEAD_INIT(NULL, 0) 12 "capi_petshop.Parrot" 13 }; 14 15 static PyObject * 16 Parrot_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { 17 Parrot *self = (Parrot *) type->tp_alloc(type, 0); 18 if (self != NULL) 19 { 20 new (&self->wrapped) petshop::Parrot(); 21 } 22 return (PyObject *) self; 23 } 24 25 static void Parrot_dealloc(Parrot *self) { 26 self->wrapped.~Parrot(); 27 Py_TYPE(self)->tp_free((PyObject *) self); 28 } 29 30 static PyObject *Parrot_voom(Parrot *self) { 31 self->wrapped.voom(); 32 Py_RETURN_NONE; 33 } 34 35 static PyObject *Parrot_get_volts(Parrot *self, void *closure) { 36 return PyLong_FromLong(self->wrapped.get_volts()); 37 } 38 39 static int 40 Parrot_set_volts(Parrot *self, PyObject *value, void *closure) { 41 long volts = PyLong_AsLong(value); 42 if (PyErr_Occurred()) 43 return -1; 44 if (volts < INT_MIN || volts > INT_MAX) { 45 PyErr_SetString(PyExc_OverflowError, 46 "Python int too large to convert to C int"); 47 return -1; 48 } 49 self->wrapped.set_volts(volts); 50 return 0; 51 } 52 53 static PyMethodDef Parrot_methods[] = { 54 {"voom", (PyCFunction) Parrot_voom, METH_NOARGS, "voom"}, 55 {NULL} 56 }; 57 58 static PyGetSetDef Parrot_getseters[] = { 59 {"volts", (getter) Parrot_get_volts, 60 (setter) Parrot_set_volts, "volts", NULL}, 61 {NULL} 62 }; 63 64 static struct PyModuleDef capi_petshop_module = { 65 PyModuleDef_HEAD_INIT, 66 "capi_petshop", 67 NULL, 68 -1, 69 NULL, 70 }; 71 72 extern "C" { 73 74 PyMODINIT_FUNC PyInit_capi_petshop(void) { 75 PyObject *m; 76 ParrotType.tp_new = Parrot_new; 77 ParrotType.tp_basicsize = sizeof(Parrot); 78 ParrotType.tp_flags = Py_TPFLAGS_DEFAULT; 79 ParrotType.tp_dealloc = (destructor) Parrot_dealloc; 80 ParrotType.tp_methods = Parrot_methods; 81 ParrotType.tp_getset = Parrot_getseters; 82 if (PyType_Ready(&ParrotType) < 0) 83 return NULL; 84 m = PyModule_Create(&capi_petshop_module); 85 if (m == NULL) 86 return NULL; 87 Py_INCREF(&ParrotType); 88 PyModule_AddObject(m, "Parrot", (PyObject *) &ParrotType); 89 return m; 90 } 91 } 11 / 50
12. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Python C API 1 #include "Python.h" 2 #include "petshop.h" 3 #include <memory> 4 5 typedef struct { 6 PyObject_HEAD 7 petshop::Parrot wrapped; 8 } Parrot; 9 10 static PyTypeObject ParrotType = { 11 PyVarObject_HEAD_INIT(NULL, 0) 12 "capi_petshop.Parrot" 13 }; 14 15 static PyObject * 16 Parrot_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { 17 Parrot *self = (Parrot *) type->tp_alloc(type, 0); 18 if (self != NULL) 19 { 20 new (&self->wrapped) petshop::Parrot(); 21 } 22 return (PyObject *) self; 23 } 24 25 static void Parrot_dealloc(Parrot *self) { 26 self->wrapped.~Parrot(); 27 Py_TYPE(self)->tp_free((PyObject *) self); 28 } 29 30 static PyObject *Parrot_voom(Parrot *self) { 31 self->wrapped.voom(); 32 Py_RETURN_NONE; 33 } 34 35 static PyObject *Parrot_get_volts(Parrot *self, void *closure) { 36 return PyLong_FromLong(self->wrapped.get_volts()); 37 } 38 39 static int 40 Parrot_set_volts(Parrot *self, PyObject *value, void *closure) { 41 long volts = PyLong_AsLong(value); 42 if (PyErr_Occurred()) 43 return -1; 44 if (volts < INT_MIN || volts > INT_MAX) { 45 PyErr_SetString(PyExc_OverflowError, 46 "Python int too large to convert to C int"); 47 return -1; 48 } 49 self->wrapped.set_volts(volts); 50 return 0; 51 } 52 53 static PyMethodDef Parrot_methods[] = { 54 {"voom", (PyCFunction) Parrot_voom, METH_NOARGS, "voom"}, 55 {NULL} 56 }; 57 58 static PyGetSetDef Parrot_getseters[] = { 59 {"volts", (getter) Parrot_get_volts, 60 (setter) Parrot_set_volts, "volts", NULL}, 61 {NULL} 62 }; 63 64 static struct PyModuleDef capi_petshop_module = { 65 PyModuleDef_HEAD_INIT, 66 "capi_petshop", 67 NULL, 68 -1, 69 NULL, 70 }; 71 72 extern "C" { 73 74 PyMODINIT_FUNC PyInit_capi_petshop(void) { 75 PyObject *m; 76 ParrotType.tp_new = Parrot_new; 77 ParrotType.tp_basicsize = sizeof(Parrot); 78 ParrotType.tp_flags = Py_TPFLAGS_DEFAULT; 79 ParrotType.tp_dealloc = (destructor) Parrot_dealloc; 80 ParrotType.tp_methods = Parrot_methods; 81 ParrotType.tp_getset = Parrot_getseters; 82 if (PyType_Ready(&ParrotType) < 0) 83 return NULL; 84 m = PyModule_Create(&capi_petshop_module); 85 if (m == NULL) 86 return NULL; 87 Py_INCREF(&ParrotType); 88 PyModule_AddObject(m, "Parrot", (PyObject *) &ParrotType); 89 return m; 90 } 91 } Just Say No 11 / 50
13. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Cython Overview Cython .pyx .pxd .c GCC .so 12 / 50
14. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Cython Definition File c_petshop.pxd 1 cdef extern from "petshop.h" namespace "petshop": 2 cdef cppclass Parrot: 3 void voom() 4 void set_volts(int volts) 5 int get_volts() 13 / 50
15. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Cython Wrapper cython_petshop.pyx 1 cimport c_petshop 2 3 cdef class Parrot: 4 cdef c_petshop.Parrot _this 5 6 def voom(self): 7 self._this.voom() 8 9 property volts: 10 def __get__(self): 11 return self._this.get_volts() 12 13 def __set__(self, volts): 14 self._this.set_volts(volts) 14 / 50
16. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Cython Sometimes This Happens Error compiling Cython file: ------------------------------------------------------------ ... PyTypeObject *Py_TYPE(obj) bint PyMapping_Check(obj) object PyErr_Format(exc, const char *format, ...) @cname("__pyx_convert__from_py_spead::in::item") cdef item __pyx_convert__from_py_spead::in::item(obj) except *: ^ ------------------------------------------------------------ FromPyStructUtility:11:39: Expected an identifier or literal 15 / 50
17. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Boost.Python 16 / 50
18. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Boost.Python 1 #include <boost/python.hpp> 2 #include "petshop.h" 3 4 using namespace petshop; 5 using namespace boost::python; 6 7 BOOST_PYTHON_MODULE(bpy_petshop) 8 { 9 class_<Parrot>("Parrot") 10 .def("voom", &Parrot::voom) 11 .add_property("volts", 12 &Parrot::get_volts, &Parrot::set_volts); 13 } 16 / 50
19. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Sir-Not-Appearing-In-These-Slides SWIG Understands C++ and templates Compiles to Python C API Quick glance at docs look promising! cffi, ctypes C only weave 17 / 50
20. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Interface Summary Interface C++ PyPy No new lang Runtime C API % % " — Cython " % % — Boost.Python " % " Boost cffi % " " cffi ctypes % ( ") " — SWIG " % % ? 18 / 50
21. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 19 / 50
22. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Keyword And Default Arguments 1 .def("add_udp_reader", 2 &ring_stream_wrapper::add_udp_reader, 3 (arg("port"), 4 arg("max_size")=udp_reader::default_max_size, 5 arg("buffer_size")=udp_reader::default_buffer_size, 6 arg("bind_hostname")=std::string())) 20 / 50
23. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Map C++ Exceptions 1 static void translate_exception_stop_iteration( 2 const stop_iteration &e) 3 { 4 PyErr_SetString(PyExc_StopIteration, e.what()); 5 } 6 ... 7 8 register_exception_translator<stop_iteration>( 9 &translate_exception_stop_iteration); 21 / 50
24. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Operator Overloads 1 class_<flavour>("Flavour", ...) 2 .def(self == self) 3 .def(self != self); 22 / 50
25. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Calling Python 1 namespace py = boost::python; 2 void heap_wrapper::add_item(py::object item) 3 { 4 std::int64_t id = p::extract<std::int64_t>( 5 item.attr("id")); 6 py::object buffer = item.attr("to_buffer")(); 7 ... 8 } 23 / 50
26. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Wrappers For Python Builtins 1 class heap_wrapper : public heap 2 { 3 ... 4 py::list get_items() const 5 { 6 std::vector<item> base = heap::get_items(); 7 py::list out; 8 for (const item &it : base) 9 { 10 if (it.id != DESCRIPTOR_ID) 11 out.append(item_wrapper(it, self)); 12 } 13 return out; 14 } 15 ... 16 }; 24 / 50
27. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 25 / 50
28. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Refresher In CPython, objects have a reference count. Increment it to hold a reference. Decrement it to drop the reference. When it hits zero, object is deleted. Very easy to screw up with Python C API. 26 / 50
29. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary C++ To The Rescue boost::python::handle is a smart wrapper Copy constructor increments the refcount Destructor decrements the refcount 27 / 50
30. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary C++ To The Rescue boost::python::handle is a smart wrapper Copy constructor increments the refcount Destructor decrements the refcount Caution: not safe while release_gil active 27 / 50
31. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Putting Things On Top Of Other Things What about C++ objects referencing other C++ objects? 1 class flowers { 2 public: 3 string name; 4 explicit flowers(const string &name) : name(name) {} 5 }; 6 7 class vase { 8 flowers *contents = nullptr; 9 public: 10 void set_contents(flowers &value) { 11 contents = &value; 12 } 13 14 string str() const { 15 return (contents ? contents->name : "empty"); 16 } 17 }; 28 / 50
32. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Naïve Wrapping 1 BOOST_PYTHON_MODULE(custodian1) { 2 using namespace boost::python; 3 class_<flowers>("Flowers", init<string>()); 4 class_<vase>("Vase") 5 .def("set_contents", &vase::set_contents) 6 .def("__str__", &vase::str); 7 } 29 / 50
33. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Naïve Wrapping 1 BOOST_PYTHON_MODULE(custodian1) { 2 using namespace boost::python; 3 class_<flowers>("Flowers", init<string>()); 4 class_<vase>("Vase") 5 .def("set_contents", &vase::set_contents) 6 .def("__str__", &vase::str); 7 } >>> from custodian import * >>> v = Vase() >>> str(v) ’empty’ >>> v.set_contents(Flowers(’tulips’)) >>> str(v) Segmentation fault 29 / 50
34. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Custodians And Wards Let’s tell Boost.Python about the relationship: 1 BOOST_PYTHON_MODULE(custodian) { 2 using namespace boost::python; 3 class_<flowers>("Flowers", init<string>()); 4 class_<vase>("Vase") 5 .def("set_contents", &vase::set_contents, 6 with_custodian_and_ward_postcall<1, 2>()) 7 .def("__str__", &vase::str); 8 } 30 / 50
35. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Custodians And Wards What Happened? weakref Vase life support Flowers weak callback 31 / 50
36. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Help, My Python Is Leaking! What Happens Here? 1 v = Vase() 2 for line in f: 3 v.set_contents(Flowers(line)) 32 / 50
37. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary A More Direct Approach Let’s make vase explicitly hold a reference to flowers. 1 class vase { 2 flowers *contents = nullptr; 3 public: 4 handle<> flowers_handle; 5 6 void set_contents(flowers &value) { 7 contents = &value; 8 } 9 10 string str() const { 11 return (contents ? contents->name : "empty"); 12 } 13 }; 33 / 50
38. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Storing A Handle spead2 implements a call policy for this: 1 class_<vase>("Vase") 2 .def("set_contents", &vase::set_contents, 3 store_handle_postcall< 4 vase, &vase::flowers_handle, 1, 2>()) 5 .def("__str__", &vase::str); See spead2 source for store_handle_postcall implementation. 34 / 50
39. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Caution Caution: don’t create cycles involving C++ objects Python can’t garbage collect such cycles C API has a solution to this, but Boost.Python doesn’t expose it 35 / 50
40. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 36 / 50
41. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Quick Reminder Python 2 Python 3 Purpose str / bytes bytes Encoded byte sequence unicode str Unicode code points 37 / 50
42. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary What Would Boost.Python Do? Python 2 str ↔ std::string Python 3 str ↔ std::string, UTF-8 bytes → std::string 38 / 50
43. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary What Would Boost.Python Do? Python 2 str ↔ std::string Python 3 str ↔ std::string, UTF-8 bytes → std::string What if I want std::string → bytes? 38 / 50
44. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Custom Converters We’ll define a new C++ class for this. 1 class bytestring : public std::string { 2 { 3 public: 4 using std::string::string; 5 6 bytestring(const std::string &s) 7 : std::string(s) {} 8 bytestring(std::string &&s) 9 : std::string(std::move(s)) {} 10 }; This is just a class interchangeable with std::string. 39 / 50
45. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Custom Converters Define how to convert it to Python1: 1 class bytestring_to_python 2 { 3 public: 4 static PyObject *convert(const bytestring &s) 5 { 6 return PyBytes_FromStringAndSize( 7 s.data(), s.size()); 8 } 9 }; Register the converter 1 to_python_converter<bytestring, bytestring_to_python>(); 1only Python 3 version shown 40 / 50
46. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Outline 1 Motivation MeerKAT SPEAD 2 Talking to C++ An Example C++ Library The Interface Zoo 3 Boost.Python That’s Easy! Object Lifetime Unicode Global Interpreter Lock 41 / 50
47. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Why do we care? Python threads hold the GIL while executing. Waiting for a heap should not block other threads 42 / 50
48. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Why do we care? Python threads hold the GIL while executing. Waiting for a heap should not block other threads Non-Python threads calling Python code 42 / 50
49. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Releasing the GIL, C++-style 1 class release_gil { 2 PyThreadState *save; 3 public: 4 release_gil() { 5 save = PyEval_SaveThread(); 6 } 7 8 ~release_gil() { 9 PyEval_RestoreThread(save); 10 } 11 }; 43 / 50
50. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Acquiring the GIL, C++ style 1 class acquire_gil { 2 PyGILState_STATE gstate; 3 public: 4 acquire_gil() { 5 gstate = PyGILState_Ensure(); 6 } 7 8 ~acquire_gil() { 9 PyGILState_Release(gstate); 10 } 11 }; 44 / 50
51. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary KeyboardInterrupt Consider a simple synchronous receiver loop: 1 for heap in stream: 2 do_stuff_with(heap) What happens if I hit Ctrl-C? Ideally, KeyboardInterrupt exception. 45 / 50
52. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary KeyboardInterrupt Consider a simple synchronous receiver loop: 1 for heap in stream: 2 do_stuff_with(heap) What happens if I hit Ctrl-C? Ideally, KeyboardInterrupt exception. Early in spead2 development, nothing until next heap. 45 / 50
53. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary If I Can Just Interrupt You There... A POSIX solution: Signal handlers interrupt system calls (EINTR) PyErr_CheckInterrupts gives interrupt handles a chance to run. Return -1 if handler threw an exception 46 / 50
54. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary If I Can Just Interrupt You There... General pattern: 1 while (true) { 2 release_gil gil; 3 int result = blocking_syscall(); 4 if (result == -1) { 5 if (errno == EINTR) { 6 gil.acquire(); 7 if (PyErr_CheckSignals() == -1) 8 throw boost::python::error_already_set(); 9 } 10 else { 11 gil.acquire(); 12 PyErr_SetFromErrno(PyExc_OSError); 13 throw boost::python::error_already_set(); 14 } 15 } 16 else 17 break; 18 } 47 / 50
55. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Good Fantastic for integrating C++ classes 48 / 50
56. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Good Fantastic for integrating C++ classes Can drop down to Python C API when needed 48 / 50
57. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Good Fantastic for integrating C++ classes Can drop down to Python C API when needed No new language to learn 48 / 50
58. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Not So Good Introduces a system library dependency 2as far as I know 49 / 50
59. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Not So Good Introduces a system library dependency Lots of deep metaprogramming magic 2as far as I know 49 / 50
60. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Not So Good Introduces a system library dependency Lots of deep metaprogramming magic Doesn’t support all C API functionality 2as far as I know 49 / 50
61. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Summary The Not So Good Introduces a system library dependency Lots of deep metaprogramming magic Doesn’t support all C API functionality Doesn’t support PyPy2 2as far as I know 49 / 50
62. Boost.Python Bruce Merry Motivation MeerKAT SPEAD Talking to C++ An Example C++ Library The Interface Zoo Boost.Python That’s Easy! Object Lifetime Unicode GIL Summary Questions Slides and code samples: https: //github.com/bmerry/pyconza2015-bpy spead2: https://github.com/ska-sa/spead2 50 / 50