Python from the Inside-Out (EuroPython 2010)

A talk from EuroPython 2010 on introducing Python into the guts of a large legacy C++ program.

1. Python from the Inside-Out EuroPython, July 19, 2010 Austin Bingham
2. Overview • Python at center of legacy C++ application • Finding the fit and making the pitch • Obstacles: technical, cultural, and political • Two APIs • Benefits and downsides • Lessons learned and the future
3. Motivation • Help increase Python use in ”unconventional” areas, but areas where it is actually well suited • Why? Wider adoption means a healthier language, more libraries, better community, etc. • Plus, it's a great tool from which developers can really benefit...spread the joy! • We feel our experience was a success story, so sharing it might help others in similar positions.
4. RMS • Flagship product • C/C++ • 15+ years old • ~2.7M LOC So
5. A Grumpy Old Man ”You damn kids stay off of my lawn!” – RMS, 2009
6. Update the Save Format Undocumented Inaccessible Fragile Modernize!
7. Features of the New System • Plugin storage modules • Named attributes • Serializable values • Metadata and introspection • Lazy loading • Accessible Hmmmm.....
8. Features of the New System • Plugin storage modules • Named attributes • Serializable values • Metadata and introspection • Lazy loading • Accessible
9. Features of the New System • Named attributes • Serializable values • Metadata and introspection • Accessible ”Any sufficiently complicated C or Fortran program contains an ad hoc, informally- specified, bug-ridden, slow implementation of half of Common Lisp.” – Greenspun's 10th Rule
10. The Road to Python
11. Objections
12. Isn't that for Scripting? • High-level • Scripting • Glue code Certainly not the kind of thing you want gumming up the guts of your lean, mean C++ program
13. No, It's a Library • Powerful (meta-)object system • Tons of useful functionality • Cross-platform • C / C++ API • World-class quality Precisely the kind of thing you want in the guts of your lean, mean C++ program... if you need what it offers.
14. Yet Another Language print”Hello world”
15. Speed Q: Isn't Python slow? A: Yes, but... • For many things this is irrelevant • You can drop into C/C++ Besides, if you really need the functionality of Python, do you think you'd do better by rewriting it?
16. Other Objections • Open source, viral licensing • Export laws (ssl, hashlib) • New build and install procedures
17. Making the Case
18. Just Do It! Rapid prototyping and high productivity allow you to build a proof-of-concept in your ”spare time”
19. Zero-cost • Software • Strong community support • Transparent, open- source, fixable Be cognizant of issue like training cost, finding developers, etc.
20. High Productivity L. Prechelt. An empirical comparison of seven programming languages. IEEE Computer, 2000.
21. Understand Performance • Python when you can, C++ when you must • Proof-by-demonstration (again!) • Pareto Principle ...PLUS... Pyrex Weave
22. Pretty Pictures/Shiny Objects
23. ...and so on... • Cross-Platform • Vendor-Neutral • ”Batteries Included” • ”5 Python Pluses for the Enterprise” [Hobbs]
24. Convincing the Decision Makers • ”Just do it”. Demonstrate benefits by example. • Zero-cost • Great community support • High productivity (see ”Just do it”) • Understand performance issues and how to address them (Python when you can, C++ when you must) • Cross-platform • Pretty pictures
25. Benefits
26. Rapid Development • Prototype replacement completed in ~1 month • Full system being shipped to customers completed in 1 year • Prototype ”grew” smoothly into production version; no ”It works, so rewrite it for real this time.”
27. High Productivity • Two developers • Python • 2075 (test) / 5103 (total) = ~40% test • C++ • 1730 (test) / 3993 (total) = ~43% test
28. High Productivity • Two developers • Python • 2075 (test) / 5103 (total) = ~40% test • C++ • 1730 (test) / 3993 (total) = ~43% test
29. Easy Unit Testing == High Quality • The ability to easily develop unittests along with the code meant that tests were actually developed • Writing unit tests becomes soothing and addictive • Key to ensuring that changes were good • Boosted credibility with the as-yet-unconvinced
30. Easy tool development • Simple tools and ”doodles” are easy to develop, and they can transition into more powerful tools as needed • Examples: • Graphviz toys • Object browsers • Text dumpers • Web interfaces
31. Platform for extensibility • Python makes a great platform for developing extensible system because it is extensible • Plugins are just modules loaded more explicitly • Duck-typing means that issues like binary compatibility are easier (though not completely erased, e.g. C extensions, language versions)
32. Low-cost Experimentation • It's very easy to try out new ideas/features/etc. with python • Related to high-productivity
33. High-quality Packages pyqt apsw
34. Benefits • Rapid prototyping; fast transition to working code • High productivity • Strong unit-test support → high quality code • Easy to develop tools • TODO: graphviz of stratus • Platform for extensibility • Easy to try out new ideas • Wide availability of high-quality third-party modules
35. Difficulties
36. Cross-Language Debugging • Debugging C++ and Python at same time • gdb macros improve things • Cross-language exceptions are tricky • Python exception semantics are complex in the C-API
37. Embedding • Somewhat more exotic than extending • Tools not as polished or available • Not as much information available
38. ”Intellisense” • Some people really want this • No great solution to the problem Prove me wrong!
39. Reliance on C-API • Needed for performance, embedding, extensions • What about the future? PyPy, Jython, IronPython...feels fragile
40. Windows Support • 2.6.1 build problems? • Registry modifications for a ”real” installation
41. ”Private” Python Install • Non-trivial to maintain
42. Difficulties • Cross-language debugger support • Embedding more esoteric than extending, less support, fewer tools • Intellisense • Much reliance on C API...jython, pypy, ironpython? • Windows support • Toting around our own private python installation is non-trivial
43. You Can Do It, Too!
44. The Important Points • Look for Python ”reimplementation” • Learn extension and embedding technologies • Learn everything that Python makes available • Be realistic about Python capabilities • Just Do It!
45. You can has Python • Look for ”reimplementation” of python functionality. This is a hallmark of the need for python...don't reinvent. • Learn C API and extension technology to understand how it can fit into your system • Understand everything python can do, both standard and non-standard modules. • Just do it!
46. Links, misc. • Hobbs, Jeff. ”5 Python Pluses for the Enterprise” http: //www.linuxinsider.com/rsstory/70337.html