Integrating C++ with Python: How, When, and Why

How to integrate C++ and Python using Boost.Python, with an emphasis on calling Python code from C++ programs.

1. Integrating C++ with Python: How, When, and Why Austin Bingham Roxar Software Solutions
3. Named attributes
4. Serializable values
5. Metadata & introspection
6. Accessible
8. Named attributes Serializable values Metadata & introspection Accessible
9. ”Any sufficiently complicated C or Fortran program contains an ad hoc, informally- specified, bug-ridden, slow implementation of half of Common Lisp.” – Greenspun's 10th Rule
10. Concept
11. Python extension module C/C++ library
12. C-API SWIG ctypes boost.python
13. C/C++ application Python interpreter
14. C-API boost.python
15. Python module C/C++ application C/C++ wrapper library
16. Natural, idiomatic C++ APIs built on Python implementations. Goal
17. import logging import sys logging.basicConfig() log = logging.getLogger() log.addHandler( logging.StreamHandler()) log.addHandler( logging.StreamHandler( sys.stdout))
18. #include <logging.hpp> void initLogging() { basicConfig(); Logger log = getLogger(); // This handler will log to stderr log.addHandler(handlers::StreamHandler()); // This will log to stdout log.addHandler( handlers::StreamHandler( boost::python::import("sys") .attr("stdout"))); }
21. Why & Where
22. print('Hello, world.') vs. #include <iostream> int main(int, char**) { std::cout << "Hello, world." << std::endl; return 0; } Expressiveness and productivity
23. Edit, compile, run, debug
24. Access to Python modules • logging • uuid • re • difflib • shutil • template engines ...and so on C++
25. Existing C++ Code
27. But where does Python fit?
28. Everywhere Pareto Principle
29. Filesystem interaction Writing "servers" Logging Access the standard library
30. Reinventing the Wheel
31. Making the Case
32. Rapid prototyping and high productivity allow you to build a proof-of-concept in your ”spare time” Just do it!
33. Zero-cost • Software • Strong community support • Transparent, open- source, fixable Be cognizant of issue like training cost, finding developers, etc. Zero-cost
34. High Productivity L. Prechelt. An empirical comparison of seven programming languages. IEEE Computer, 2000. High Productivity
35. Understand Performance • Python when you can, C++ when you must • Proof-by-demonstration (again!) • Pareto Principle ...PLUS... Weave Understand Performance
36. Pretty Pictures/Shiny Objects Pretty Pictures/Shiny Objects
37. ...and so on... • Cross-Platform • Vendor-Neutral • ”Batteries Included” • ”5 Python Pluses for the Enterprise” [Hobbs] ...and so on...
38. Progression
39. C/C++ application C/C++ library
40. C/C++ application C/C++ wrapper library Python module
41. Python application Python module Extension main() C/C++ library
42. Python application Python module Python module
43. Techniques
44. mypackage Code structure lib_mypackage.so namespace mypackage { . . . }
45. Module initialization namespace mypackage { void initialize() { bp::object mod = bp::import("mypackage"); // register type-conversion // anything else: create // loggers, etc. } }
46. Type conversion How do we get objects across the Python- C++ boundary? The key is often to remember that PyObjects are wholly legitimate C-level entities. There is no magic.
47. Type conversion: A simple model C++ class bp::object obj_; PyObject
48. Type conversion: to C++ inspection Python to C++ if (PyObject_IsInstance(obj, class_obj)) return new T( object( handle<>( borrowed( obj))));
49. Type conversion: to Python templates C++ to Python template <typename T> struct to_python_object_ { static PyObject* convert(const T& t) { return boost::python::incref( t.obj().ptr()); } };
50. Type conversion: registration // Register from-python converter boost::python::converter::registry::push_back( &convertible, &construct, boost::python::type_id<MyType>()); // Register to-python converter boost::python::to_python_converter< MyType, to_python_object_<MyType> >(); // Covert from Python to C++ MyType x = boost::python::extract<MyType>(obj); // Convert from C++ to Python (implicit in bp::object constructor) python_class.attr("doit")(x);
51. Exceptions: with C-API PythonAPI_Foo(); if (PyErr_Occurred()) respondToPythonException(); PythonAPI_Bar(); if (PyErr_Occurred()) respondToPythonException();
52. Exceptions: error_already_set Python exception bp::error_already_set try { some_class.attr("method")(3); } catch (const bp::error_already_set&) { // Some exception has been thrown in Python throw SomeException(); }
53. Exception translation PyObject *t, *v, *tb; PyErr_Fetch(&t, &v, &tb); // Check if it's a ValueError if (PyErr_GivenExceptionMatches( value_error_class_obj, t)) { throw ValueError(); }
54. Exception translation // The easy way! try { some_python_function(); } catch (const bp::error_already_set&) { ackward::core::translatePythonException(); }
55. Iteration There's a very natural mapping between C++ and Python iterators • C++ iterator holds a Python iterator reference • Incrementing iterator calls next(iter)and stores values • StopIteration is caught in increment • The "end" iterator simply has a None Python iterator
56. Duck Typing vs. Static Typing mypackage::Database C++ Python mypackage.DB mypackage.DBProxy pkg2.FancyDB
57. Other stuff • Enumerations • Const-ness • Properties • Tuples • Kwargs • Subclassing • Lifetime management
58. Interesting Bits Python/C++ Integration
59. Debugging the full stack C++ application C++ wrapper library Python module C/C++ library debug?! • pdb.set_trace() • gdb with python extensions • VisualStudio python support (?) • SIGINT to break debugger • Common logging • Debugging individual layers
60. Debugging: Pythonizing the App main() extension C++ wrapper library Python module C/C++ library Python application pdb gdb attach
61. Performance • Python function call overhead can cause problems. • Be aware of unnecessary data copies. • Learn the buffer and memoryview APIs In general, the 80/20 principle works in your favor. Performance is not an issue for most code, and when it is you can bring many tools to bear.
62. • Lutz Prechelt. An empirical comparison of seven programming languages. IEEE Computer 33(10):23-29, October 2000. • Hobbs, Jeff. ”5 Python Pluses for the Enterprise” http://www.linuxinsider. com/rsstory/70337.html • misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/ • misspent.wordpress.com/2009/10/11/boost-python-and-handling-python- exceptions/ • misspent.wordpress.com/2012/03/24/debugging-cc-and-cpython-using-gdb-7s- new-python-extension-support/ • Ackward: code.google.com/p/ackward/ Links & References
63. THANKS! Austin Bingham @austin_bingham austin.bingham@emerson.com