Reintroducing SWIG

Reintroducing SWIG, February 5, 2002, 10th International Python Conference 1
[email protected] Reintroducing SWIG (www.swig.org) David M. Beazley Department of Computer Science University of Chicago [email protected] February 5, 2002 PY-MA

[email protected] Introduction SWIG • "Simplified" Wrapper and Interface Generator • A freely available tool for automatically creating C/C++ extension modules. • Currently supports Python, Perl, Tcl, Ruby, Java, PHP , Guile, Mzscheme. • The predecessor to many similar wrapper generation tools. A brief history • 1995. SWIG originally developed at Los Alamos National Laboratory • 1996. Development moves to the University of Utah. • 1996. First public release (first announced on comp.lang.tcl) • 1996. Presented at the 4th International Python Conference, Livermore, CA. • 1997. SWIG-1.1 released. • 1998. Development moves to the University of Chicago • 1999. The lost year • 2000. The horror, the horror. • 2000. SWIG-1.3 project launched. • 2001. SWIG-1.3.6 - 1.3.10 (the rewrite) • 2002. SWIG-1.3.11 released (February 1,2002)

[email protected] Brief Overview Classic SWIG • A brief overview of SWIG-1.1. Crash and burn • Problems with the original SWIG implementation • Failed rewrite attempts • Near death experience. The new SWIG • SWIG-1.3 project. • A near-total rewrite of the internals. • Many interesting new features. • Current and future work. Goals • Introduce SWIG to new users • Describe new features to long-time users.

[email protected] Extension Building An important Python application area • Python used as control language for C,C++, and Fortran applications. • Python access to popular programming libraries. Can write extension modules by hand • Just write some wrappers like this PyObject *wrap_gcd(PyObject *self, PyObject *args) { int x,y,r; if (!PyArg_ParseTuple(args,"ii",&x,&y)) return NULL; r = gcd(x,y); return Py_BuildValue("i",r); } Problem: • Very tedious and error prone. • Especially if you have a huge application. • No one really wants to spend time doing this.

[email protected] Automated Tools Extension building tools/libraries • Modulator • bgen • Boost Python • Wrappy • Weave • SIP • f2py • pyfort • CXX • SWIG • (Apologies to anyone I missed). Benefits • Automate/simplify the process of interfacing Python with C, C++, Fortran, etc. • Make it possible for programmers to concentrate on more important problems. Tool classification • IDL compilers, parsers, stub generators, etc. • Programming libraries.

[email protected] SWIG SWIG is a specialized C/C++ compiler • Parses annotated C++ header files • Produces Python wrappers from standard C++ declarations. • Additional directives used to guide wrapper generation process. Example: // example.i : simple SWIG module file %module example %{ #include "someheader.h" %} // Functions to wrap int gcd(int x, int y); Running SWIG $ swig -python example.i $ cc -c -I/usr/local/include/python2.1 example_wrap.c $ cc -shared $(OBJS) example_wrap.o -o examplemodule.so $ python >>> import example >>> example.gcd(42,12) 6

[email protected] SWIG Example A more complex example: %module example %{ #include "someheader.h" %} class Account { int balance; public: Account(int initial); ~Account(); void deposit(int amt); int getbalance(); }; Use in Python (ommitting some details) >>> import example >>> a = example.Account(100) >>> a.deposit(50) >>> a.getbalance() 150 >>>

[email protected] SWIG Overview C/C++ features supported in SWIG-1.1 • Functions • Global variables • Constants • Simple classes and structures • Member data, member functions, static members. • Inheritance and multiple inheritance. • Typedef. SWIG attempts to map these features directly to Python • Functions accessed as Python functions • Constants mapped to Python variables. • Classes accessed as Python classes • Primitive C datatypes mapped to equivalent types in Python (int, float, etc.). Goal: • Make Python interface look as much like C API as possible. • Make it extremely easy to build modules from existing source code.

[email protected] How it works Combination of C/C++ wrappers and Python code • Low-level C/C++ accessors turned into Python extension module. • High-level Python wrappers used to provide a more natural interface. • "Shadow classes". Python class wrappers around C++ classes. • SWIG does not use extension types or extension classes. • Easier to do things in Python (less code, better portability, easier to customize). module.i class Foo { public: void bar(); }; Foo *new_Foo(); void delete_Foo(Foo*f); void Foo_bar(Foo *f); class Foo: def __init__(self): self.this = new_Foo() def bar(self): Foo_bar(self.this) ... C accessors (extension module) Python wrappers (shadow classes) SWIG

[email protected] Additional Features SWIG provides about two dozen special directives • %name, %new, %insert, %readonly, etc. • Inserted into the SWIG interface file. • Used to guide the interface generation process. • Used for customization. Full coverage of every SWIG feature impossible here • The topic of a half-day tutorial perhaps. Highlights • C/C++ pointer handling • Structure/class extension • Exception handling • Typemaps

[email protected] Pointer Handling C/C++ pointers fully supported. • Represented as value-type strings. • Fully type-checked. Example: Image *create_image(int width, int height); void plot(Image *x, int x, int y, int color); ColorMap *new_colormap(); ... In Python >>> im = create_image(400,400) >>> cmap = new_colormap() >>> im ’108efea8_p_Image’ >>> cmap ’108f8a00_p_ColorMap’ >>> plot(im, 20, 30, 1) >>> plot(cmap, 20, 30, 1) Traceback (most recent call last): TypeError: Type error. Expected _p_Image >>>

[email protected] Pointer Handling (cont.) All non-primitive objects handled as pointers • No marshalling • Pass-by-reference for all C/C++ objects Pointers are opaque • No dereferencing from Python. Definitions are optional • SWIG does not require structure/class definitions to operate. • Pointer-only model avoids the need to know the representation. C++ and type-checking • Inheritance/multiple inheritance built into type checking mechanism • Pointer values are properly cast • Class hierarchies recognized (pointers to derived classes ok). • Typedef also supported. Pointer model essential • Can create wrappers to almost any C/C++ library and data structure. • Just pass pointers around in Python. • Just like C except for lack of dereferencing.

[email protected] Structure/Class Extension Can extend structures/classes with new methods: struct Vector { double x, y, z; }; %addmethods Vector { char *__str__() { static buf[1024]; sprintf(buf,"Vector(%g,%g,%g)", self->x,self->y,self->z); return buf; } }; In Python >>> v = Vector() >>> v.x = 3; v.y = 4; v.z = 5 >>> print v Vector(3,4,5) Uses • Making C programs appear object-oriented in Python • Adding special Python methods to C++ classes.

[email protected] Exception Handling Convert a C++ exception to a Python exception %except(python) { try { $function } catch (indexerror) { PyErr_SetString(PyExc_RangeError,"Bad index"); return NULL; } }; ... Object *getobject(int objnum); Idea: • Function calls get extra error-handling code inserted. • Can catch C++ errors (or C errors). • Raise Python exception and return. Allows more seamless integration with C code • Especially if a library defines its own error handling mechanism.

[email protected] Typemaps An advanced customization mechanism • Used to modify the default type-conversion. • Used if you wanted to marshal data or handle datatypes differently. A very very simple example: %typemap(python,in) int positive { $target = PyInt_AsLong($source); if ($target <= 0) { PyErr_SetString(PyExc_ValueError,"Expected > 0"); return NULL; } } %typemap(python,out) int { $target = Py_BuildValue("i",$source); } ... int fact(int positive); int blah(int x, int positive); Also a great way to shoot yourself in the foot! • Requires a detailed knowledge of the Python API.

[email protected] SWIG-1.1 Summary SWIG-1.1 supported a wide variety of features • Used successfully in many projects. • Further details on www.swig.org. Extension building • Define interface files as mix of SWIG directives and ANSI C/C++ • Add customization features to refine the interface. SWIG-1.1 was also a implementation nightmare • A hodge-podge of features. • No clear design focus. • A variety of minor annoyances (missing features, problematic behavior). • Weak C++ support. • Really hard to maintain. SWIG-1.1 death • Development on 1.1 mostly stopped in 1998.

[email protected] Hard Problems Broken C++ type-system • SWIG-1.1 did not implement the full C++ type system. • And the partial implementation was broken. • More than a simple "parsing" issue. • Couldn’t even represent types properly internally • So parsing was not enough to fix this problem. Examples: const char *const a; int *&b; int (*pf)(int); int (*x[10]); long long; typedef double Mat[4][4]; void foo(Mat x); Also • Inconsistent type-handling behavior (especially with typemaps) • Odd type-related workarounds.

[email protected] Problems (cont.) Customization features too limited • Most added as optional features. • Didn’t work as some users wanted. Example: typemaps for multiple arguments int count(char *buffer, int len, char *pattern); int transpose(double *mat, int rows, int cols); • Can I make multiple arguments work with a single Python object? • Strings, Numeric arrays, etc. C++ support • Weak at best • Internal implementation horrible (actually, beyond horrible). • No overloading, no operators, no templates, no namespaces, etc. • No idea how to fix with broken parser and type system. Module system too complicated • Module API restricted extensibility. • Made it difficult to enhance SWIG with new features.

[email protected] Crash and Burn The dark years (1997-2000) • Problems with the implementation were obvious during 1.1 development. • At least 4-5 different attempts at a SWIG rewrite (I’ve lost count). • A complete top-down redesign of the system. • Plus an attempt at a "proper" C++ implementation. Head explosion • Let’s create the ultimate extensible wrapper generator that does everything. • Strong second system effect. • Couldn’t get the different pieces of the system to work together nicely. Near death • Nearly abandoned the project altogether in 2000. • Rewrite prospects seemed utterly hopeless. • ARGH!!!!!!

[email protected] SWIG-1.3 Effort Background • SWIG1.1p5 was the last stable release (1998). • Had implemented a special C preprocessor in one of the rewrites. • A few other developers were interested in making contributions. • Mostly minor enhancements for Perl5, Java, Guile. SWIG-1.3.1 (Feb. 2000) • Silently abandoned the glorious rewrite effort. • SWIG1.1p5 + preprocessor + long list of bugs and minor enhancements. • Killed a large number of SWIG features • Tcl 7.x, Perl4.x, documentation system, Objective-C, dozens of SWIG directives. SWIG-1.3.x (ongoing) • Total destruction and bottom-up rewrite of SWIG internals (in ANSI C). • All low-level data structures (strings, lists, hashes). • Reimplementation of the C++ type system. • Reimplementation of parser, almost all utility functions, etc. • Piecemeal migration of all language modules during development. • Idea: Just keep the system working (somehow).

[email protected] SWIG-1.3 Effort (cont.) Major contributors • William Fulton • Matthias Koeppe • Lyle Johnson • Loic Dachary • Thien-Thi Nguyen • Richard Palmer • Masaki Fukushima • Luigi Ballabio • Harco de Hilster Overview • Public CVS access • 11 major releases (1.3.11 current) • ~400 feature changes and enhancements since 1.3.1. Bottom line • SWIG-1.3 is not a minor enhancement of SWIG-1.1. • It’s not even backwards compatible (but it mostly looks similar).

[email protected] SWIG-1.3 Overview C/C++ parsing • Fully integrated C preprocessor with macros. • Full C++ type-system (const, pointers to functions, arrays, references, etc.). • Greatly improved parsing capabilities. • A lot of "invisible" changes. Internals • Complete construction of parse trees before code generation. • Allows for multi-pass compilation/semantic analysis. • Fully extensible data structures. • Internally based on ideas from XML. DOM-like API used for tree traversal. • Pattern matching for types and declaration signatures. Code generation • New language modules : Ruby, Mzscheme, Java, PHP • Completely new module API. • Code generation completely driven by typemap rules/patterns. • A lot of cleanup and refinement.

[email protected] Rethinking Wrapper Generation What does a wrapper generator do exactly? • It converts declarations. • It marshals data (type conversion) • Special directives to customize both of these tasks. Question • Is there any way to reduce SWIG to a couple of simple ideas? • Is there any similarity between all of the features added to SWIG-1.1? Answer: Maybe • Typemap rules (type conversion) • "Features" (declaration annotation and customization).

[email protected] Typemaps Revisited Typemap rules define all type conversion in SWIG-1.3 • They were only optional in SWIG-1.1 (defaults were hard-wired in C++) • Essentially a set of patterns for specifying data conversions. %typemap(in) int "$1 = PyInt_AsLong($input);"; %typemap(in) double "$1 = PyFloat_AsDouble($input);"; %typemap(out) int "$result = PyInt_FromLong($1);"; ... int blah(int, double); Language modules • Now each target defines type conversion with a large set of typemaps. • Contained in a special configuration file. • All conversions can be fully reconfigured by the user. Type system • Typemap matching fully integrated with the C++ type system. • Qualified/non-qualified matching, typedef, etc.

[email protected] Multi-Argument Typemaps Type matching is more powerful • Can span groups of consecutive arguments. • Expansion of a single Python object into multiple C arguments. • A long requested feature (but hard to implement). Example: %typemap(in) (char *buffer, int len) { $1 = PyString_AsString($input); $2 = PyString_Size($input); } int count(char *buffer, int len, char *pattern); ... In Python >>> example.count("Some string", "st") 1 >>>

[email protected] Multi-Argument Typemaps (cont) A very useful feature • Access to buffers, arrays, shape parameters, and other information %typemap(in) (double *mat, int rows, int columns) { $1 = get_some_pointer($input); $2 = get_rows($input); $3 = get_columns($input); ... } • More interesting interaction with Numeric Python, other extensions. Also extends to other SWIG directives • %apply directive (typemap copy). %apply (double *mat, int rows, int columns) { (double *rot, int rotr, int rotc), (double *trans, int transr, int transc) };

[email protected] Declaration Annotation Many old SWIG directives are declaration modifiers • %name, %readonly, %new, etc. • Example: %name(output) void print(FILE *f); What if this could be generalized somehow? • Kind of like typemaps? • Pattern matching rules for declarations? • Decoupling of modifiers from declarations. %rename(output) print; ... %include "someheader.h" ... /* someheader.h */ ... void print(FILE *f); ...

[email protected] The %feature Directive Generalized declaration annotation • Used to attach arbitrary properties to specific declarations Example: %feature("blah") print "spam"; ... void print(FILE *f); A peek inside the parser... • Parse tree node gets new attribute attached to it. • Ok, so what?? node: cdecl name = "print" type = "void" feature:blah = "spam" ...

[email protected] The %feature Directive Declaration specifications • Fully integrated with the C++ type system (including inheritance). Example: exception handling for a method %feature("except") Object::getitem { try { $action } catch (badindex) { PyErr_SetString(PyExc_RangeError,"bad index"); return NULL; } } class Object { ... Blah *getitem(int index); ... } class Derived: public Object { ... Blah *getitem(int index); ... }

[email protected] The %feature Directive Type parameterization • Declaration specifiers can be fully qualified with type signatures • Allows declarations to be exactly pinpointed. • Allows SWIG to work with overloaded methods (can pick the one you want). Example: %feature("ignore") Object::foo(Object *) const; ... class Object { public: void foo(Object *); void foo(Object *) const; ... }; class Derived: public Object { public: void foo(Object *); void foo(Object *) const; ... };

[email protected] New Capabilities %feature used to implement other SWIG directives • Also allows customizations to be passed to language modules. Example: %rename and %ignore • Used to more easily resolve overloaded functions %rename(foo_i) foo(int); %rename(foo_d) foo(double); %ignore foo(float); ... %include "someheader.h" • Applies across entire inheritance hierarchies • Can apply a consistent renaming with only a few extra specifications. • Very predictable---not at the mercy of an arbitrary overload resolution scheme. • Principle of least surprise.

[email protected] Overloading Example %rename(searchn) Object::search(char *, int); class Object { public: ... int search(char *str); int search(char *str, int len); // searchn ... } class Derived : public Object { public: ... int search(char *str); int search(char *str, int len); // searchn ... };

[email protected] Operator Overloading Can bind operators to Python operators %rename(__add__) Complex::operator+; %rename(__sub__) Complex::operator-; %rename(__neg__) Complex::operator-(); %rename(__mul__) Complex::operator*; ... Can capture all other operators • Maybe not as elegant, but still possible. %rename(add_cd) operator+(const Complex &, double); Note: • SWIG-1.3 automatically maps certain C++ operators to Python. • May not be necessary to do anything. • Certain C++ operators more problematic (no direct mapping to Python).

[email protected] New Exception Handling %exception • Allows parameterization • More efficient code generation (can pinpoint declarations). Example: %exception *::getitem(int index) { try { $action } catch (badindex) { PyErr_SetString(PyExc_RangeError,"..."); return NULL; } } %exception *::clone { try { $action} catch (memoryerror) { PyErr_SetString(PyExc_MemoryError,"..."); return NULL; } }; ... • Note: *:: matches any class.

[email protected] Other %feature customizations Python-specific enhancements %module example %rename(add_cd) operator+(const Complex &, double); %feature("shadow") Complex::operator+(const Complex &) %{ def __add__(self,other): if isinstance(other,Complex): return examplec.Complex___add__(self,other) else: return examplec.add_cd(self,float(other)) %} ... class Complex { Complex operator+(const Complex &); friend Complex operator+(const Complex &, double); }; In Python >>> a = Complex(3,4) >>> b = Complex(10,2) >>> c = a + b # Invokes Complex::operator+ >>> d = c + 3.5 # Invokes operator+ (add_cd)

[email protected] Interlude Most SWIG directives rewritten/replaced • Typemaps • Features • Implementation becoming a lot more consistent Slight change in how SWIG can be used • Preamble of declaration modifiers and typemaps • Inclusion of raw headers • Requires even less code modification than before %module example %{ #include "header.h" %} // Typemaps and declaration modifiers ... // Parse the raw header %include "header.h"

[email protected] Macro Handling SWIG-1.3 features a full preprocessor • Supports traditional macro expansion • Also provides support for special "SWIG" macros • Can define new SWIG directives as macros as well. Examples: #define %exception %feature("except") %define MemoryException { try { $action } catch(memoryerror) { PyErr_SetString(PyExc_MemoryError,"out of memory"); return NULL; } } %enddef %exception clone MemoryException; %exception copy MemoryException; %exception alloc MemoryException; ...

[email protected] Template Wrapping C++ templates • Can’t wrap a raw template. • You can wrap an instantiation of a template. Experimental SWIG feature : %template template<typename T> T max(T a, T b) { a >= b ? a : b } ... %template(maxint) max<int>; %template(maxdouble) max<double>; • Just have to give a unique name to the instantiation Python >>> maxint(3,10) 10 >>> maxdouble(3.5,14.2) 14.2 Also works with classes • Some limitations (no member templates, no partial specialization, etc.) • SWIG does not attempt to support all forms of template abuse.

[email protected] Minor Enhancements A lot of minor SWIG enhancements • Parsing • Type handling • Subtle semantic fixes. • Better code generation • Better support for cross-module linking. • const handled correctly. • Pointers to functions. • Pointers to C++ members. And I’m probably forgetting many more...

[email protected] Current Status SWIG is alive and well • Better than ever • 73000 downloads from SourceForge since 1999. • 720 subscribers on the mailing list ([email protected]) Development is ongoing • SWIG-1.3 project is incomplete. • We’re still working on cleanup and migration. • Some old SWIG features not yet revamped. • Still needs more documentation and cleanup. • Some experimental features only partially implemented. Long term • Convergence of documentation with new features. • Will release SWIG-1.4 (or 2.0) as the next stable release. • No estimated date. Maybe 2003. • These things take time.

[email protected] Future Work Python • Need support for Python 2.2 and newer releases. • Think we can support new objects with a little work. • Declaration annotation (%feature) may allow very interesting customizations AOP? • Certain aspects of SWIG seem similar to Aspect Oriented Programming. • Maybe there is some kind of connection • A new direction? (We don’t really know). Refinement of SWIG module system • Rewrite has focused on bottom-up reimplementation. • Language modules are last stage of the rewrite. Restoration of some lost features • Objective-C support. • Documentation system.

[email protected] Limitations C++ • Namespaces • Nested classes • Weren’t supported in SWIG-1.1. • Will support eventually. • Better integration with standard C++ library. Code generation • Would like to generate less code • SWIG-1.3 is much better than SWIG-1.1. • Still could do better (I think).

[email protected] Summary SWIG-1.3 is a major enhancement • Better C/C++ parsing • Full C++ type system. • Simplification of internals. • Useful set of primitives (typemaps, features, etc.). • Very powerful customization options. • Better code generation. I’m excited • SWIG-1.3.11 is more capable than I ever imagined. • And it’s only getting better. Bottom line: • If you looked at SWIG before, you might want to look at it again.

[email protected] More Information Web page • www.swig.org Volunteers wanted • SWIG is an all volunteer effort • Not a funded research project nor a commercial operation. • We work on it because it’s fun and useful. • Anyone who wants to work on it can work on it. Discussion

Reintroducing SWIG

Reintroducing SWIG

More Decks by David Beazley

Other Decks in Programming

Featured

Transcript