Reintroducing SWIG

Reintroducing SWIG, February 5, 2002, 10th International Python Conference 1 [email protected]
Reintroducing SWIG
(www.swig.org)
David M. Beazley
Department of Computer Science
University of Chicago
[email protected]
February 5, 2002
PY-MA

View Slide

Introduction
SWIG
• "Simplified" Wrapper and Interface Generator
• A freely available tool for automatically creating C/C++ extension modules.
• Currently supports Python, Perl, Tcl, Ruby, Java, PHP
, Guile, Mzscheme.
• The predecessor to many similar wrapper generation tools.
A brief history
• 1995. SWIG originally developed at Los Alamos National Laboratory
• 1996. Development moves to the University of Utah.
• 1996. First public release (first announced on comp.lang.tcl)
• 1996. Presented at the 4th International Python Conference, Livermore, CA.
• 1997. SWIG-1.1 released.
• 1998. Development moves to the University of Chicago
• 1999. The lost year
• 2000. The horror, the horror.
• 2000. SWIG-1.3 project launched.
• 2001. SWIG-1.3.6 - 1.3.10 (the rewrite)
• 2002. SWIG-1.3.11 released (February 1,2002)

View Slide

Brief Overview
Classic SWIG
• A brief overview of SWIG-1.1.
Crash and burn
• Problems with the original SWIG implementation
• Failed rewrite attempts
• Near death experience.
The new SWIG
• SWIG-1.3 project.
• A near-total rewrite of the internals.
• Many interesting new features.
• Current and future work.
Goals
• Introduce SWIG to new users
• Describe new features to long-time users.

View Slide

Extension Building
An important Python application area
• Python used as control language for C,C++, and Fortran applications.
• Python access to popular programming libraries.
Can write extension modules by hand
• Just write some wrappers like this
PyObject *wrap_gcd(PyObject *self, PyObject *args) {
int x,y,r;
if (!PyArg_ParseTuple(args,"ii",&x,&y)) return NULL;
r = gcd(x,y);
return Py_BuildValue("i",r);
}
Problem:
• Very tedious and error prone.
• Especially if you have a huge application.
• No one really wants to spend time doing this.

View Slide

Automated Tools
Extension building tools/libraries
• Modulator
• bgen
• Boost Python
• Wrappy
• Weave
• SIP
• f2py
• pyfort
• CXX
• SWIG
• (Apologies to anyone I missed).
Benefits
• Automate/simplify the process of interfacing Python with C, C++, Fortran, etc.
• Make it possible for programmers to concentrate on more important problems.
Tool classification
• IDL compilers, parsers, stub generators, etc.
• Programming libraries.

View Slide

SWIG
SWIG is a specialized C/C++ compiler
• Parses annotated C++ header files
• Produces Python wrappers from standard C++ declarations.
• Additional directives used to guide wrapper generation process.
Example:
// example.i : simple SWIG module file
%module example
%{
#include "someheader.h"
%}
// Functions to wrap
int gcd(int x, int y);
Running SWIG
$ swig -python example.i
$ cc -c -I/usr/local/include/python2.1 example_wrap.c
$ cc -shared $(OBJS) example_wrap.o -o examplemodule.so
$ python
>>> import example
>>> example.gcd(42,12)
6

View Slide

SWIG Example
A more complex example:
%module example
%{
#include "someheader.h"
%}
class Account {
int balance;
public:
Account(int initial);
~Account();
void deposit(int amt);
int getbalance();
};
Use in Python (ommitting some details)
>>> import example
>>> a = example.Account(100)
>>> a.deposit(50)
>>> a.getbalance()
150
>>>

View Slide

SWIG Overview
C/C++ features supported in SWIG-1.1
• Functions
• Global variables
• Constants
• Simple classes and structures
• Member data, member functions, static members.
• Inheritance and multiple inheritance.
• Typedef.
SWIG attempts to map these features directly to Python
• Functions accessed as Python functions
• Constants mapped to Python variables.
• Classes accessed as Python classes
• Primitive C datatypes mapped to equivalent types in Python (int, float, etc.).
Goal:
• Make Python interface look as much like C API as possible.
• Make it extremely easy to build modules from existing source code.

View Slide

How it works
Combination of C/C++ wrappers and Python code
• Low-level C/C++ accessors turned into Python extension module.
• High-level Python wrappers used to provide a more natural interface.
• "Shadow classes". Python class wrappers around C++ classes.
• SWIG does not use extension types or extension classes.
• Easier to do things in Python (less code, better portability, easier to customize).
module.i
class Foo {
public:
void bar();
};
Foo *new_Foo();
void delete_Foo(Foo*f);
void Foo_bar(Foo *f);
class Foo:
def __init__(self):
self.this = new_Foo()
def bar(self):
Foo_bar(self.this)
...
C accessors (extension module)
Python wrappers (shadow classes)
SWIG

View Slide

Additional Features
SWIG provides about two dozen special directives
• %name, %new, %insert, %readonly, etc.
• Inserted into the SWIG interface file.
• Used to guide the interface generation process.
• Used for customization.
Full coverage of every SWIG feature impossible here
• The topic of a half-day tutorial perhaps.
Highlights
• C/C++ pointer handling
• Structure/class extension
• Exception handling
• Typemaps

View Slide

Pointer Handling
C/C++ pointers fully supported.
• Represented as value-type strings.
• Fully type-checked.
Example:
Image *create_image(int width, int height);
void plot(Image *x, int x, int y, int color);
ColorMap *new_colormap();
...
In Python
>>> im = create_image(400,400)
>>> cmap = new_colormap()
>>> im
’108efea8_p_Image’
>>> cmap
’108f8a00_p_ColorMap’
>>> plot(im, 20, 30, 1)
>>> plot(cmap, 20, 30, 1)
Traceback (most recent call last):
TypeError: Type error. Expected _p_Image
>>>

View Slide

Pointer Handling (cont.)
All non-primitive objects handled as pointers
• No marshalling
• Pass-by-reference for all C/C++ objects
Pointers are opaque
• No dereferencing from Python.
Definitions are optional
• SWIG does not require structure/class definitions to operate.
• Pointer-only model avoids the need to know the representation.
C++ and type-checking
• Inheritance/multiple inheritance built into type checking mechanism
• Pointer values are properly cast
• Class hierarchies recognized (pointers to derived classes ok).
• Typedef also supported.
Pointer model essential
• Can create wrappers to almost any C/C++ library and data structure.
• Just pass pointers around in Python.
• Just like C except for lack of dereferencing.

View Slide

Structure/Class Extension
Can extend structures/classes with new methods:
struct Vector {
double x, y, z;
};
%addmethods Vector {
char *__str__() {
static buf[1024];
sprintf(buf,"Vector(%g,%g,%g)",
self->x,self->y,self->z);
return buf;
}
};
In Python
>>> v = Vector()
>>> v.x = 3; v.y = 4; v.z = 5
>>> print v
Vector(3,4,5)
Uses
• Making C programs appear object-oriented in Python
• Adding special Python methods to C++ classes.

View Slide

Exception Handling
Convert a C++ exception to a Python exception
%except(python) {
try {
$function
} catch (indexerror) {
PyErr_SetString(PyExc_RangeError,"Bad index");
return NULL;
}
};
...
Object *getobject(int objnum);
Idea:
• Function calls get extra error-handling code inserted.
• Can catch C++ errors (or C errors).
• Raise Python exception and return.
Allows more seamless integration with C code
• Especially if a library defines its own error handling mechanism.

View Slide

Typemaps
An advanced customization mechanism
• Used to modify the default type-conversion.
• Used if you wanted to marshal data or handle datatypes differently.
A very very simple example:
%typemap(python,in) int positive {
$target = PyInt_AsLong($source);
if ($target <= 0) {
PyErr_SetString(PyExc_ValueError,"Expected > 0");
return NULL;
}
}
%typemap(python,out) int {
$target = Py_BuildValue("i",$source);
}
...
int fact(int positive);
int blah(int x, int positive);
Also a great way to shoot yourself in the foot!
• Requires a detailed knowledge of the Python API.

View Slide

SWIG-1.1 Summary
SWIG-1.1 supported a wide variety of features
• Used successfully in many projects.
• Further details on www.swig.org.
Extension building
• Define interface files as mix of SWIG directives and ANSI C/C++
• Add customization features to refine the interface.
SWIG-1.1 was also a implementation nightmare
• A hodge-podge of features.
• No clear design focus.
• A variety of minor annoyances (missing features, problematic behavior).
• Weak C++ support.
• Really hard to maintain.
SWIG-1.1 death
• Development on 1.1 mostly stopped in 1998.

View Slide

Hard Problems
Broken C++ type-system
• SWIG-1.1 did not implement the full C++ type system.
• And the partial implementation was broken.
• More than a simple "parsing" issue.
• Couldn’t even represent types properly internally
• So parsing was not enough to fix this problem.
Examples:
const char *const a;
int *&b;
int (*pf)(int);
int (*x[10]);
long long;
typedef double Mat[4][4];
void foo(Mat x);
Also
• Inconsistent type-handling behavior (especially with typemaps)
• Odd type-related workarounds.

View Slide

Problems (cont.)
Customization features too limited
• Most added as optional features.
• Didn’t work as some users wanted.
Example: typemaps for multiple arguments
int count(char *buffer, int len, char *pattern);
int transpose(double *mat, int rows, int cols);
• Can I make multiple arguments work with a single Python object?
• Strings, Numeric arrays, etc.
C++ support
• Weak at best
• Internal implementation horrible (actually, beyond horrible).
• No overloading, no operators, no templates, no namespaces, etc.
• No idea how to fix with broken parser and type system.
Module system too complicated
• Module API restricted extensibility.
• Made it difficult to enhance SWIG with new features.

View Slide

Crash and Burn
The dark years (1997-2000)
• Problems with the implementation were obvious during 1.1 development.
• At least 4-5 different attempts at a SWIG rewrite (I’ve lost count).
• A complete top-down redesign of the system.
• Plus an attempt at a "proper" C++ implementation.
Head explosion
• Let’s create the ultimate extensible wrapper generator that does everything.
• Strong second system effect.
• Couldn’t get the different pieces of the system to work together nicely.
Near death
• Nearly abandoned the project altogether in 2000.
• Rewrite prospects seemed utterly hopeless.
• ARGH!!!!!!

View Slide

SWIG-1.3 Effort
Background
• SWIG1.1p5 was the last stable release (1998).
• Had implemented a special C preprocessor in one of the rewrites.
• A few other developers were interested in making contributions.
• Mostly minor enhancements for Perl5, Java, Guile.
SWIG-1.3.1 (Feb. 2000)
• Silently abandoned the glorious rewrite effort.
• SWIG1.1p5 + preprocessor + long list of bugs and minor enhancements.
• Killed a large number of SWIG features
• Tcl 7.x, Perl4.x, documentation system, Objective-C, dozens of SWIG directives.
SWIG-1.3.x (ongoing)
• Total destruction and bottom-up rewrite of SWIG internals (in ANSI C).
• All low-level data structures (strings, lists, hashes).
• Reimplementation of the C++ type system.
• Reimplementation of parser, almost all utility functions, etc.
• Piecemeal migration of all language modules during development.
• Idea: Just keep the system working (somehow).

View Slide

SWIG-1.3 Effort (cont.)
Major contributors
• William Fulton
• Matthias Koeppe
• Lyle Johnson
• Loic Dachary
• Thien-Thi Nguyen
• Richard Palmer
• Masaki Fukushima
• Luigi Ballabio
• Harco de Hilster
Overview
• Public CVS access
• 11 major releases (1.3.11 current)
• ~400 feature changes and enhancements since 1.3.1.
Bottom line
• SWIG-1.3 is not a minor enhancement of SWIG-1.1.
• It’s not even backwards compatible (but it mostly looks similar).

View Slide

SWIG-1.3 Overview
C/C++ parsing
• Fully integrated C preprocessor with macros.
• Full C++ type-system (const, pointers to functions, arrays, references, etc.).
• Greatly improved parsing capabilities.
• A lot of "invisible" changes.
Internals
• Complete construction of parse trees before code generation.
• Allows for multi-pass compilation/semantic analysis.
• Fully extensible data structures.
• Internally based on ideas from XML. DOM-like API used for tree traversal.
• Pattern matching for types and declaration signatures.
Code generation
• New language modules : Ruby, Mzscheme, Java, PHP
• Completely new module API.
• Code generation completely driven by typemap rules/patterns.
• A lot of cleanup and refinement.

View Slide

Rethinking Wrapper Generation
What does a wrapper generator do exactly?
• It converts declarations.
• It marshals data (type conversion)
• Special directives to customize both of these tasks.
Question
• Is there any way to reduce SWIG to a couple of simple ideas?
• Is there any similarity between all of the features added to SWIG-1.1?
Answer: Maybe
• Typemap rules (type conversion)
• "Features" (declaration annotation and customization).

View Slide

Typemaps Revisited
Typemap rules define all type conversion in SWIG-1.3
• They were only optional in SWIG-1.1 (defaults were hard-wired in C++)
• Essentially a set of patterns for specifying data conversions.
%typemap(in) int "$1 = PyInt_AsLong($input);";
%typemap(in) double "$1 = PyFloat_AsDouble($input);";
%typemap(out) int "$result = PyInt_FromLong($1);";
...
int blah(int, double);
Language modules
• Now each target defines type conversion with a large set of typemaps.
• Contained in a special configuration file.
• All conversions can be fully reconfigured by the user.
Type system
• Typemap matching fully integrated with the C++ type system.
• Qualified/non-qualified matching, typedef, etc.

View Slide

Multi-Argument Typemaps
Type matching is more powerful
• Can span groups of consecutive arguments.
• Expansion of a single Python object into multiple C arguments.
• A long requested feature (but hard to implement).
Example:
%typemap(in) (char *buffer, int len) {
$1 = PyString_AsString($input);
$2 = PyString_Size($input);
}
int count(char *buffer, int len, char *pattern);
...
In Python
>>> example.count("Some string", "st")
1
>>>

View Slide

Multi-Argument Typemaps (cont)
A very useful feature
• Access to buffers, arrays, shape parameters, and other information
%typemap(in) (double *mat, int rows, int columns) {
$1 = get_some_pointer($input);
$2 = get_rows($input);
$3 = get_columns($input);
...
}
• More interesting interaction with Numeric Python, other extensions.
Also extends to other SWIG directives
• %apply directive (typemap copy).
%apply (double *mat, int rows, int columns) {
(double *rot, int rotr, int rotc),
(double *trans, int transr, int transc)
};

View Slide

Declaration Annotation
Many old SWIG directives are declaration modifiers
• %name, %readonly, %new, etc.
• Example:
%name(output) void print(FILE *f);
What if this could be generalized somehow?
• Kind of like typemaps?
• Pattern matching rules for declarations?
• Decoupling of modifiers from declarations.
%rename(output) print;
...
%include "someheader.h"
...
/* someheader.h */
...
void print(FILE *f);
...

View Slide

The %feature Directive
Generalized declaration annotation
• Used to attach arbitrary properties to specific declarations
Example:
%feature("blah") print "spam";
...
void print(FILE *f);
A peek inside the parser...
• Parse tree node gets new attribute attached to it.
• Ok, so what??
node: cdecl
name = "print"
type = "void"
feature:blah = "spam"
...

View Slide

Declaration specifications
• Fully integrated with the C++ type system (including inheritance).
Example: exception handling for a method
%feature("except") Object::getitem {
try { $action } catch (badindex) {
PyErr_SetString(PyExc_RangeError,"bad index");
return NULL;
}
}
class Object {
...
Blah *getitem(int index);
...
}
class Derived: public Object {
...
Blah *getitem(int index);
...
}

View Slide

Type parameterization
• Declaration specifiers can be fully qualified with type signatures
• Allows declarations to be exactly pinpointed.
• Allows SWIG to work with overloaded methods (can pick the one you want).
Example:
%feature("ignore") Object::foo(Object *) const;
...
class Object {
public:
void foo(Object *);
void foo(Object *) const;
...
};
class Derived: public Object {
public:
void foo(Object *);
void foo(Object *) const;
...
};

View Slide

New Capabilities
%feature used to implement other SWIG directives
• Also allows customizations to be passed to language modules.
Example: %rename and %ignore
• Used to more easily resolve overloaded functions
%rename(foo_i) foo(int);
%rename(foo_d) foo(double);
%ignore foo(float);
...
%include "someheader.h"
• Applies across entire inheritance hierarchies
• Can apply a consistent renaming with only a few extra specifications.
• Very predictable---not at the mercy of an arbitrary overload resolution scheme.
• Principle of least surprise.

View Slide

Overloading Example
%rename(searchn) Object::search(char *, int);
class Object {
public:
...
int search(char *str);
int search(char *str, int len); // searchn
...
}
class Derived : public Object {
public:
...
int search(char *str);
int search(char *str, int len); // searchn
...
};

View Slide

Operator Overloading
Can bind operators to Python operators
%rename(__add__) Complex::operator+;
%rename(__sub__) Complex::operator-;
%rename(__neg__) Complex::operator-();
%rename(__mul__) Complex::operator*;
...
Can capture all other operators
• Maybe not as elegant, but still possible.
%rename(add_cd) operator+(const Complex &, double);
Note:
• SWIG-1.3 automatically maps certain C++ operators to Python.
• May not be necessary to do anything.
• Certain C++ operators more problematic (no direct mapping to Python).

View Slide

New Exception Handling
%exception
• Allows parameterization
• More efficient code generation (can pinpoint declarations).
Example:
%exception *::getitem(int index) {
try { $action } catch (badindex) {
PyErr_SetString(PyExc_RangeError,"...");
return NULL;
}
}
%exception *::clone {
try { $action} catch (memoryerror) {
PyErr_SetString(PyExc_MemoryError,"...");
return NULL;
}
};
...
• Note: *:: matches any class.

View Slide

Other %feature customizations
Python-specific enhancements
%module example
%rename(add_cd) operator+(const Complex &, double);
%feature("shadow") Complex::operator+(const Complex &) %{
def __add__(self,other):
if isinstance(other,Complex):
return examplec.Complex___add__(self,other)
else:
return examplec.add_cd(self,float(other))
%}
...
class Complex {
Complex operator+(const Complex &);
friend Complex operator+(const Complex &, double);
};
In Python
>>> a = Complex(3,4)
>>> b = Complex(10,2)
>>> c = a + b # Invokes Complex::operator+
>>> d = c + 3.5 # Invokes operator+ (add_cd)

View Slide

Interlude
Most SWIG directives rewritten/replaced
• Typemaps
• Features
• Implementation becoming a lot more consistent
Slight change in how SWIG can be used
• Preamble of declaration modifiers and typemaps
• Inclusion of raw headers
• Requires even less code modification than before
%module example
%{
#include "header.h"
%}
// Typemaps and declaration modifiers
...
// Parse the raw header
%include "header.h"

View Slide

Macro Handling
SWIG-1.3 features a full preprocessor
• Supports traditional macro expansion
• Also provides support for special "SWIG" macros
• Can define new SWIG directives as macros as well.
Examples:
#define %exception %feature("except")
%define MemoryException {
try { $action } catch(memoryerror) {
PyErr_SetString(PyExc_MemoryError,"out of memory");
return NULL;
}
}
%enddef
%exception clone MemoryException;
%exception copy MemoryException;
%exception alloc MemoryException;
...

View Slide

Template Wrapping
C++ templates
• Can’t wrap a raw template.
• You can wrap an instantiation of a template.
Experimental SWIG feature : %template
template T max(T a, T b) { a >= b ? a : b }
...
%template(maxint) max;
%template(maxdouble) max;
• Just have to give a unique name to the instantiation
Python
>>> maxint(3,10)
10
>>> maxdouble(3.5,14.2)
14.2
Also works with classes
• Some limitations (no member templates, no partial specialization, etc.)
• SWIG does not attempt to support all forms of template abuse.

View Slide

Minor Enhancements
A lot of minor SWIG enhancements
• Parsing
• Type handling
• Subtle semantic fixes.
• Better code generation
• Better support for cross-module linking.
• const handled correctly.
• Pointers to functions.
• Pointers to C++ members.
And I’m probably forgetting many more...

View Slide

Current Status
SWIG is alive and well
• Better than ever
• 73000 downloads from SourceForge since 1999.
• 720 subscribers on the mailing list ([email protected])
Development is ongoing
• SWIG-1.3 project is incomplete.
• We’re still working on cleanup and migration.
• Some old SWIG features not yet revamped.
• Still needs more documentation and cleanup.
• Some experimental features only partially implemented.
Long term
• Convergence of documentation with new features.
• Will release SWIG-1.4 (or 2.0) as the next stable release.
• No estimated date. Maybe 2003.
• These things take time.

View Slide

Future Work
Python
• Need support for Python 2.2 and newer releases.
• Think we can support new objects with a little work.
• Declaration annotation (%feature) may allow very interesting customizations
AOP?
• Certain aspects of SWIG seem similar to Aspect Oriented Programming.
• Maybe there is some kind of connection
• A new direction? (We don’t really know).
Refinement of SWIG module system
• Rewrite has focused on bottom-up reimplementation.
• Language modules are last stage of the rewrite.
Restoration of some lost features
• Objective-C support.
• Documentation system.

View Slide

Limitations
C++
• Namespaces
• Nested classes
• Weren’t supported in SWIG-1.1.
• Will support eventually.
• Better integration with standard C++ library.
Code generation
• Would like to generate less code
• SWIG-1.3 is much better than SWIG-1.1.
• Still could do better (I think).

View Slide

Summary
SWIG-1.3 is a major enhancement
• Better C/C++ parsing
• Full C++ type system.
• Simplification of internals.
• Useful set of primitives (typemaps, features, etc.).
• Very powerful customization options.
• Better code generation.
I’m excited
• SWIG-1.3.11 is more capable than I ever imagined.
• And it’s only getting better.
Bottom line:
• If you looked at SWIG before, you might want to look at it again.

View Slide

More Information
Web page
• www.swig.org
Volunteers wanted
• SWIG is an all volunteer effort
• Not a funded research project nor a commercial operation.
• We work on it because it’s fun and useful.
• Anyone who wants to work on it can work on it.
Discussion

View Slide

Reintroducing SWIG

Reintroducing SWIG

David Beazley

More Decks by David Beazley

Other Decks in Programming

Featured

Transcript