Scientific Computing and Scripting Language Extension Programming

Presentation. At ONERA, Paris. 2002.

Transcript

1. Scripting Language Extensions, April 18, 2002 1 beazley@cs.uchicago.edu Scientific Computing and Scripting Language Extension Programming David M. Beazley Department of Computer Science The University of Chicago beazley@cs.uchicago.edu April 18, 2002 PY-MA
2. Scripting Language Extensions, April 18, 2002 2 beazley@cs.uchicago.edu Scientific Software The ongoing software problems • It’s too hard to write. • It’s too hard to use. • It’s too hard to extend. • It’s too hard to maintain. • It’s too slow. • A sad story. Issue : how do you make it better? • Object orientation? • Software components? • New programming languages? • Problem solving environments? • Genetic engineering?
3. Scripting Language Extensions, April 18, 2002 3 beazley@cs.uchicago.edu Scripted Scientific Software Many programs are being transformed... Questions: • Why are computational scientists doing this? • How are they doing it? • Are there any limitations? C/C++ Application C/C++ Application Scripting Language Non-interactive batch processing (Python, Perl, Tcl, etc.) (extension)
4. Scripting Language Extensions, April 18, 2002 4 beazley@cs.uchicago.edu Motivation My background • Theoretical physics division at Los Alamos (1991-1997) • Worked on materials science software (molecular dynamics) • Parallel computing (CM-5, Cray T3D, clusters, etc.). I was frustrated • Could run very large simulations. • And we could run demos. • And the code ran fast and was scalable. • But we weren’t getting any useful scientific results. Questions • Why? • Can we do anything about it?
5. Scripting Language Extensions, April 18, 2002 5 beazley@cs.uchicago.edu A Fracture Experiment An experiment: crack propagation in a notched plate • Plate of some material. • Notch. • Apply forces. • Observe crack formation at notch tip. stress stress
6. Scripting Language Extensions, April 18, 2002 6 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Obvious parameters • What is the size of the plate? • How big is the notch? • What forces are applied? width thickness length ndepth nwidth F(t) F(t)
7. Scripting Language Extensions, April 18, 2002 7 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Boundary conditions? • Free surfaces? free surface
8. Scripting Language Extensions, April 18, 2002 8 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Boundary conditions? • Free surfaces? • Periodic boundary (infinite plate)?
9. Scripting Language Extensions, April 18, 2002 9 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Boundary conditions? • Free surfaces? • Periodic boundary (infinite plate)? • Damping?
10. Scripting Language Extensions, April 18, 2002 10 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Material • What is the plate made of? • A single element? An alloy? • What is the temperature of the plate? temp Cu
11. Scripting Language Extensions, April 18, 2002 11 beazley@cs.uchicago.edu A Fracture Experiment (cont.) What is the internal structure? • Crystal lattice? • Imperfections?
12. Scripting Language Extensions, April 18, 2002 12 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Force model? • Pair-wise interaction? • Density effects? Angular effects? Quantum effects? Potential energy r
13. Scripting Language Extensions, April 18, 2002 13 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Domain decomposition • Parallel processing? • Load balancing? Numerics • Numerical integration. • Discretization (timestep, etc.) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Processors
14. Scripting Language Extensions, April 18, 2002 14 beazley@cs.uchicago.edu A Fracture Experiment (cont.) Low level system issues • Message passing. • Threads. • I/O. • File formats. • Databases? • Networking?
15. Scripting Language Extensions, April 18, 2002 15 beazley@cs.uchicago.edu Observations Problem specification is hard • Complicated initial conditions. • Many parameters. • Complicated system configuration (parallel computing). • Difficult to debug. Data analysis is harder • Consider the dynamics of a fracture experiment. Software development • Code is generally not written by huge teams of programmers. • Few formal software requirements (often dictated by physics). • Piecemeal development. • Very long development cycles (continual code development). • Not practical (or desirable) to rewrite existing code from scratch. This is only part of the problem
16. Scripting Language Extensions, April 18, 2002 16 beazley@cs.uchicago.edu Scientific Software Traditional programming model • Monolithic Fortran, C, C++ application. • Non-interactive batch processing. • No user interface. Many codes controlled with scanf() % crack Nx : 1000 Ny : 400 Nz : 40 notch_width : 10 notch_depth : 20 Material : 1 Force model : 1 Integrator : 2 ... Nsteps : 10000 Running...
17. Scripting Language Extensions, April 18, 2002 17 beazley@cs.uchicago.edu Scientific Interpreters May have a home-grown interpreter % crack crack > nx 1000 crack > ny 400 crack > notch_depth 20 ... crack > run 10000 Running... Features • Partial implementation of an interpreted language. • Can set variables, issue commands, etc. • No control flow, no user defined procedures. • Usually a bad interpreted language. Problems: • Not very flexible. • Difficult to maintain and extend.
18. Scripting Language Extensions, April 18, 2002 18 beazley@cs.uchicago.edu Scripting Languages Scripting languages • Interpreted programming environment. • Usually interactive. • Extensible with C/C++/Fortran code. Use in scientific software • Can replace existing user interfaces. • A logical extension of existing user interfaces. Benefits • Programs controlled with a real programming language. • Interactive debugging, data exploration. • Rapid prototyping (via. scripts). • Advanced simulation control. • Graphical user interface (Tk) • Systems integration. Goal • An better environment for scientific experimentation.
19. Scripting Language Extensions, April 18, 2002 19 beazley@cs.uchicago.edu Background Interpreters in scientific computing • Used widely in scientific software. • Used in common scientific packages (MATLAB, Mathematica, IDL, etc.). • Many home-grown domain specific languages. What’s different? • Development of next-generation scripting languages. • Perl (1987), Tcl (1990), Python (1991), Ruby (1995). • Strong support for extension programming, embedding. • Freely available. • Documented. Large user community. • Development of extension building tools. Result • Ideally suited for use in scientific programs. • Interpreters can be retrofitted onto existing projects. • They are an essential part of new projects. • It’s not necessary to write your own interpreter.
20. Scripting Language Extensions, April 18, 2002 20 beazley@cs.uchicago.edu Extension Programming Wrappers • main() replaced by wrappers (data marshalling, error handling, etc.) • Similar to stub-code with RPC, CORBA, COM, etc. • Goal: expose application internals to interpreter (functions, classes, variables,...) >>> gcd(12,16) 4 Scripting Interpreter Scientific Application (C/C++/Fortran) Wrapper Layer Scientific Application (C/C++/Fortran) Wrapper Layer Scientific Application (C/C++/Fortran) main() Original Application
21. Scripting Language Extensions, April 18, 2002 21 beazley@cs.uchicago.edu Extension Code Example Python wrapper #include "Python.h" extern int gcd(int,int); PyObject *wrap_gcd(PyObject *self, PyObject *args) { int x,y; int result; if (!PyArg_ParseTuple(args,"ii", &x, &y)) { return NULL; } result = gcd(x,y); return PyInt_FromLong(result); } • Data conversion from C <---> Python. • You would write a wrapper for each part of your program. • Ex: 300 C functions ==> 300 wrapper functions
22. Scripting Language Extensions, April 18, 2002 22 beazley@cs.uchicago.edu The Extension Building Problem No one wants to write extension code • Highly repetitive. • Prone to error. • Difficult for complicated programs. Other problems • Scientific programs characterized by rapid change. • Functions change, variables change, objects change. • Piecemeal development. • Would require continual maintenance of the wrappers. • Complicates development. • Makes scripting languages difficult to use.
23. Scripting Language Extensions, April 18, 2002 23 beazley@cs.uchicago.edu Extension Tools Significant work in extension building tools Goals • Simplify extension programming • Automate extension programming. Common approaches • Programming libraries. • Specialized compilers. • Mixed language procedure inlining. • SWIG • CABLE • Inline • Boost Python • Wrappy • Grad • f2py • pyfort • G-wrap • Tolua • CXX • Pyrex • Weave • Many others
24. Scripting Language Extensions, April 18, 2002 24 beazley@cs.uchicago.edu The SWIG Project SWIG • A C/C++ compiler for generating wrappers to existing code. • Freely available and in development since 1995. • Currently targets Python, Perl, Tcl, Ruby, Java, PHP , Guile, and Mzscheme. Source translation • C++ header files are parsed to generate wrappers Goals • Make it extremely easy for users (scientists) to build wrappers. • Allow scripting interface to automatically track changes in underlying source. • Make the binding process as invisible as possible. .h .h .h swig Wrapper Code C/C++ Perl, Python,Tcl, Ruby, ...
25. Scripting Language Extensions, April 18, 2002 25 beazley@cs.uchicago.edu SWIG Overview Key components: • Header file parsing. • Special SWIG directives. Supported C++ features • Functions, variables, constants. • Classes • Inheritance and multiple inheritance. • Pointers, references, arrays, member pointers. • Overloading (with renaming) • Operators. • Namespaces. • Templates. • Preprocessor. Not supported • Nested classes, member templates, partial specialization Will show some examples • Not a complete coverage of SWIG.
26. Scripting Language Extensions, April 18, 2002 26 beazley@cs.uchicago.edu Input files C/C++ declarations mixed with special SWIG directives. // example.i : Sample SWIG input file %module example %{ #include "example.h" %} // Resolve a name clash with Python %rename(cprint) print; // C/C++ declarations extern int gcd(int x, int y); extern int fact(int n); ... %include "example.h" // Parse a header file ...
27. Scripting Language Extensions, April 18, 2002 27 beazley@cs.uchicago.edu Creating a Module Compilation and linking % swig -python example.i % cc -c -I/usr/local/include/python2.1 example_wrap.c % cc -shared example_wrap.o $(OBJS) -o examplemodule.so Use % python Python 2.1 (#3, Aug 20 2001, 15:41:42) [GCC 2.95.2 19991024 (release)] on sunos5 >>> import example >>> example.gcd(12,16) 4 >>> Comments: • Modules built as shared libraries/DLLs • Dynamic loading used to import into interpreter. • Contents of the module similar to C.
28. Scripting Language Extensions, April 18, 2002 28 beazley@cs.uchicago.edu A More Complicated Example • Structures/classes mapped into wrapper objects. • Provides natural access from an interpreter. class Complex { double real, imag; public: Complex(double r = 0, double i = 0); Complex(const Complex &c); Complex &operator=(const Complex &c); Complex operator+(const Complex &); Complex operator-(const Complex &); Complex operator*(const Complex &); Complex operator-(); double re(); double im(); ... }; >>> a = Complex(3,4) >>> b = Complex(5,6) >>> c = a + b >>> c.re() 8.0 >>> c.im() 10.0 >>> C++ Python
29. Scripting Language Extensions, April 18, 2002 29 beazley@cs.uchicago.edu Structure Extension Converting C structures to classes • Can make C programs look OO (or extend C++ classes) typedef struct { double x,y,z; } Vector; ... %addmethods Vector { Vector(double x, double y, double z) { Vector *r = (Vector *) malloc(sizeof(Vector)); r->x = x; r->y = y; r->z = z; return r; } double magnitude() { return sqrt(self->x*self->x+self->y*self->y + self->z*self->z); } ... };
30. Scripting Language Extensions, April 18, 2002 30 beazley@cs.uchicago.edu Overloaded Methods Must disambiguate %rename(barstr) Foo::bar(char *, int); ... class Foo { public: virtual void bar(int); virtual void bar(char *,int); // barstr ... }; class Spam : public Foo { public: virtual void bar(int); virtual void bar(char *,int); // barstr ... }; Comment: • Overloading tricky to implement in dynamically typed languages.
31. Scripting Language Extensions, April 18, 2002 31 beazley@cs.uchicago.edu Template Wrapping %template directive template<class T> class vector { public: vector(); ~vector(); T get(int index); int size(); ... }; // Instantiate templates %template(intvector) vector<int>; %template(doublevector) vector<double>; In Python >>> v = intvector() ... >>> x = v.get(2) >>> print v.size() 10 >>>
32. Scripting Language Extensions, April 18, 2002 32 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; C++ (input)
33. Scripting Language Extensions, April 18, 2002 33 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Complex * new_Complex(double r, double i) { return new Complex(r,i); } Complex * Complex_operator___add__( Complex *self,Complex *other) { Complex *r; r=new Complex(self->operator+(*other)); return r; } double Complex_re(Complex *self) { return self->re(); } C++ (input) Procedure Wrappers
34. Scripting Language Extensions, April 18, 2002 34 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Complex * new_Complex(double r, double i) { return new Complex(r,i); } Complex * Complex_operator___add__( Complex *self,Complex *other) { Complex *r; r=new Complex(self->operator+(*other)); return r; } double Complex_re(Complex *self) { return self->re(); } Extension Module (DLL) C++ (input) Procedure Wrappers
35. Scripting Language Extensions, April 18, 2002 35 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Complex * new_Complex(double r, double i) { return new Complex(r,i); } Complex * Complex_operator___add__( Complex *self,Complex *other) { Complex *r; r=new Complex(self->operator+(*other)); return r; } double Complex_re(Complex *self) { return self->re(); } Extension Module (DLL) class Complex: def __init__(self,r,i): self.this = new_Complex(r,i) def __add__(self,other): return Complex_operator___add__ self.this,other) def re(self): return Complex_re(self.this) ... C++ (input) Procedure Wrappers Python class
36. Scripting Language Extensions, April 18, 2002 36 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Complex * new_Complex(double r, double i) { return new Complex(r,i); } Complex * Complex_operator___add__( Complex *self,Complex *other) { Complex *r; r=new Complex(self->operator+(*other)); return r; } double Complex_re(Complex *self) { return self->re(); } Extension Module (DLL) class Complex: def __init__(self,r,i): self.this = new_Complex(r,i) def __add__(self,other): return Complex_operator___add__ self.this,other) def re(self): return Complex_re(self.this) ... >>> a = Complex(2,3) >>> b = Complex(4,5) >>> c = a + b >>> c.re() 6 >>> C++ (input) Procedure Wrappers Python class Python script
37. Scripting Language Extensions, April 18, 2002 37 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Complex * new_Complex(double r, double i) { return new Complex(r,i); } Complex * Complex_operator___add__( Complex *self,Complex *other) { Complex *r; r=new Complex(self->operator+(*other)); return r; } double Complex_re(Complex *self) { return self->re(); } Extension Module (DLL) class Complex: def __init__(self,r,i): self.this = new_Complex(r,i) def __add__(self,other): return Complex_operator___add__ self.this,other) def re(self): return Complex_re(self.this) ... >>> a = Complex(2,3) >>> b = Complex(4,5) >>> c = a + b >>> c.re() 6 >>> C++ (input) Procedure Wrappers Python class Python script
38. Scripting Language Extensions, April 18, 2002 38 beazley@cs.uchicago.edu How it Works class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Extension Module (DLL) >>> a = Complex(2,3) >>> b = Complex(4,5) >>> c = a + b >>> c.re() 6 >>> C++ (input) Procedure Wrappers Python class Python script SWIG generated SWIG generated
39. Scripting Language Extensions, April 18, 2002 39 beazley@cs.uchicago.edu How it Works • User only works with input file (C++) and scripts • Details of wrappers hidden. • Not modified by user, only used to compile DLL. class Complex { public: Complex(double r = 0, double i = 0); Complex operator+(const Complex &); double re(); ... }; Extension Module (DLL) >>> a = Complex(2,3) >>> b = Complex(4,5) >>> c = a + b >>> c.re() 6 >>> C++ (input) Python script swig
40. Scripting Language Extensions, April 18, 2002 40 beazley@cs.uchicago.edu Using SWIG Summary • Existing C/C++ header files used to build wrappers. • Process guided by some special SWIG directives • Most details hidden from user. Result • Internals of application are available in interpreter. • Functions, variables, data structures, etc. .h .h .h .h Scientific Application (C/C++/Fortran) Scientific Application (C/C++/Fortran) .i Scientific Application (C/C++/Fortran) Wrapper Layer swig DLL
41. Scripting Language Extensions, April 18, 2002 41 beazley@cs.uchicago.edu Challenges C/C++ is a bad interface definition language • Type system complexity: typedef int (*PFIA[20])(int, double *x); double foo(PFIA *const x); • Ambiguity in data conversion (pointers, arrays, output values, etc.) double bar(double *x, double *y, double *r); • Structures, classes, unions. • Templates, namespaces, overloading, operators, etc. SWIG solution • Declaration annotation. • Pattern based type conversion. • Will provide a brief tour of internals.
42. Scripting Language Extensions, April 18, 2002 42 beazley@cs.uchicago.edu Declaration Annotation The underlying customization mechanism %module example %rename(cprint) print; %ignore Complex::operator=; %include "example.h" // example.h void print(char *s); class Complex { public: void print(); ... Complex& operator=(const Complex &); ... }; Declaration modifiers (special directives) Pattern matching (unmodified C/C++)
43. Scripting Language Extensions, April 18, 2002 43 beazley@cs.uchicago.edu Declaration Annotation Advanced features • Fully integrated with the C++ type system. • Annotations can be parameterized with type signatures. Example: %ignore Object::bar(string *s) const; ... class Object { ... void bar(string *s); void bar(string *s) const; // Ignored ... } ; class Foo : public Object { ... void bar(string *s); void bar(string *s) const; // Ignored ... };
44. Scripting Language Extensions, April 18, 2002 44 beazley@cs.uchicago.edu Type Conversion Problem: marshalling • Must convert data between scripting and C representation. Example: In Python >>> gcd(12,16) 4 >>> count("Hello",5,"e") 1 int gcd(int x, int y); int count(char *buf, int len, char c); Integers String Single character
45. Scripting Language Extensions, April 18, 2002 45 beazley@cs.uchicago.edu Pattern-Based Type Conversion Typemap patterns %typemap(in) int { $1 = PyInt_AsLong($input); } %typemap(out) int { $result = PyInt_FromLong($1); } %typemap(in) char * { $1 = PyString_AsString($input); } ... %include "example.h" int gcd(int x, int y); ... int count(char *buf, int len, char c); ... C datatype conversion code. (depends on target language) C header Note: user rarely writes this.
46. Scripting Language Extensions, April 18, 2002 46 beazley@cs.uchicago.edu Typemaps Named typemaps: %typemap(in) double nonnegative { $1 = PyFloat_AsDouble($input); if ($1 < 0) { PyErr_SetString(PyExc_ValueError,"domain error!"); return NULL; } } double sqrt(double nonnegative); Sequences %typemap(in) (char *buf, int len) { $1 = PyString_AsString($input); $2 = PyString_Size($input); } int count(char *buf, int len, char c); >>> count("Hello","e") 1
47. Scripting Language Extensions, April 18, 2002 47 beazley@cs.uchicago.edu Typemaps and Datatypes Pattern matching integrated with C++ typesystem %typemap(in) int { ... } typedef int Integer; ... Integer gcd(Integer x, Integer y); namespace std { class string; %typemap(in) string * { ... }; } namespace S = std; using std::string; ... void foo(string *a, S::string *b); Comments: • All type conversion in SWIG is pattern based. • Type conversion by naming convention (very different than IDL). • Mostly hidden from users. • Allows advanced customization.
48. Scripting Language Extensions, April 18, 2002 48 beazley@cs.uchicago.edu Advanced Typemap Example Conversion of Numeric Python array to C %typemap(in) (double *mat, int nx, int ny) { PyArrayObject *array; if (!PyArray_Check($input)) { PyErr_SetString(PyExc_TypeError,"Expected an array"); return NULL; } array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (!array) { PyErr_SetString(PyExc_ValueError, "array must be two-dimensional and of type float"); return NULL; } $1 = (double *) array->data; /* Assign grid */ $2 = array->dimensions[0]; /* Assign nx */ $3 = array->dimensions[1]; /* Assign ny */ } ... double determinant(double *mat, int nx, int ny); Key point • SWIG can be customized to handle new datatypes. • Customized data marshalling.
49. Scripting Language Extensions, April 18, 2002 49 beazley@cs.uchicago.edu SWIG versus ... There are many extension building tools • You should try several • Each has advantages/disadvantages SWIG strengths • Supports both ANSI C and C++. • Supports C++ type system and most features. • Many customization features. • Multiple languages. • Large user base, stable, and documented. SWIG limitations • Callbacks. • Overloading (automatic) • Does not provide same control as hand-crafted wrappers. • Fortran (must work with C interface).
50. Scripting Language Extensions, April 18, 2002 50 beazley@cs.uchicago.edu Results We used this approach at Los Alamos • SPaSM molecular dynamics code. • Built an interpreted interface (Python) • Added data analysis, visualization modules. • Exposed internal data structures to interpreter. • Restructured code as collection of extension modules. • Scripting interfaces all automatically generated. Results • More simulations per month than previous 3 years of the project combined. • Directly led to publications in Science, Physical Review Letters, etc. • Difficult problems became easy. • Impossible problems became feasible. • Program was a lot more fun to use. Automation critical • Focus on physics, not on scripting interface.
51. Scripting Language Extensions, April 18, 2002 51 beazley@cs.uchicago.edu Example Interpreter Data analysis
52. Scripting Language Extensions, April 18, 2002 52 beazley@cs.uchicago.edu Open Issues Scripting is not a silver bullet • It improves the way in which programs are controlled • It simplifies many development issues. • But it introduces new types of problems. Issues: • Debugging • Software reliability. • Software components. • Project management issues.
53. Scripting Language Extensions, April 18, 2002 53 beazley@cs.uchicago.edu Debugging A scripting language error: % python spam.py Traceback (most recent call last): File "spam.py", line 15, in ? blah() File "spam.py", line 12, in blah bar() File "spam.py", line 9, in bar foo() File "spam.py", line 6, in foo spam(3) File "spam.py", line 3, in spam doh(n) NameError: There is no variable named 'doh'
54. Scripting Language Extensions, April 18, 2002 54 beazley@cs.uchicago.edu Debugging An extension error % python spam.py Segmentation fault (core dumped) % Very descriptive • No script-level traceback. • No details on what (or who) to blame. • No information for software maintainers. Could run the C/C++ debugger • But not on the extension module. • Instead, you run it on the interpreter.
55. Scripting Language Extensions, April 18, 2002 55 beazley@cs.uchicago.edu GDB Traceback (gdb) where #0 0xff1d9bf0 in __sigprocmask () from /usr/lib/libthread.so.1 #1 0xff1ce628 in _resetsig () from /usr/lib/libthread.so.1 #2 0xff1cdd18 in _sigon () from /usr/lib/libthread.so.1 #3 0xff1d0e8c in _thrp_kill () from /usr/lib/libthread.so.1 #4 0xfee49b10 in raise () from /usr/lib/libc.so.1 #5 0xfee3512c in abort () from /usr/lib/libc.so.1 #6 0xfee353d0 in _assert () from /usr/lib/libc.so.1 #7 0xfeee13ec in abort_crash () from /u0/beazley/Projects/WAD/WAD/Test/./ debugmodule.so #8 0xfeee28ec in _wrap_abort_crash () from /u0/beazley/Projects/WAD/WAD/Test/./debugmodule.so #9 0x281c8 in call_builtin (func=0x1cc4f0, arg=0x1f9424, kw=0x0) at ceval.c:2650 #10 0x28094 in PyEval_CallObjectWithKeywords (func=0x1cc4f0, arg=0x1f9424, kw=0x0) at ceval.c:2618 #11 0x26764 in eval_code2 (co=0x1d37e0, globals=0x0, locals=0x1d37cf, args=0x1cc4f0, argcount=1762552, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at ceval.c:1951 #12 0x263a0 in eval_code2 (co=0x1d3858, globals=0x0, locals=0x1cc4f0, args=0x19b1a4, argcount=1883008, kws=0x1d7318, kwcount=0, defs=0x0, defcount=0, owner=0x0) at ceval.c:1850 #13 0x263a0 in eval_code2 (co=0x1d3e50, globals=0x0, locals=0x19b1a4, args=0x1a7374, argcount=1883128, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at ceval.c:1850 #14 0x285e0 in call_function (func=0x1a73a4, arg=0x18f114, kw=0x0) at ceval.c:2772 #15 0x28080 in PyEval_CallObjectWithKeywords (func=0x1a73a4, arg=0x18f114, kw=0x0) at ceval.c:2616 #16 0x680b0 in builtin_apply (self=0x0, args=0x0) at bltinmodule.c:88 #17 0x281c8 in call_builtin (func=0x1910c8, arg=0x1f9b54, kw=0x0) at ceval.c:2650 #18 0x28094 in PyEval_CallObjectWithKeywords (func=0x1910c8, arg=0x1f9b54, kw=0x0) at ceval.c:2618 #19 0x26764 in eval_code2 (co=0x1f3948, globals=0x0, locals=0x1f38f0, args=0x1910c8, argcount=1733540, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x2436e4) at ceval.c:1951 #20 0x285e0 in call_function (func=0x24374c, arg=0x1a606c, kw=0x0) at ceval.c:2772 #21 0x28080 in PyEval_CallObjectWithKeywords (func=0x261414, arg=0x18f114, kw=0x0) at ceval.c:2616 #22 0x98064 in PythonCmd (clientData=0x1cc8e0, interp=0x20e658, argc=0, argv=0xffbee060) at ./_tkinter.c:1274 #23 0xff122064 in TclInvokeStringCommand (clientData=0x278538, interp=0x20e658, objc=1, objv=0x24ec84) at ./../generic/tclBasic.c:1752 #24 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0cd0) at ./../generic/tclExecute.c:845 #25 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370c8, flags=0) at ./../generic/tclBasic.c:2723 #26 0xff258220 in TkInvokeButton (butPtr=0x279188) at ./../generic/tkButton.c:1457 #27 0xff257698 in ButtonWidgetObjCmd (clientData=0x279188, interp=0x20e658, objc=2, objv=0x295e00) at ./../generic/tkButton.c:835 #28 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, command=0xff182128 "", length=0, flags=262144) at ./../generic/tclParse.c:932 #29 0xff15e2b8 in Tcl_EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, flags=262144) at ./../generic/tclParse.c:1019 #30 0xff122928 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370e0, flags=262144) at ./../generic/tclBasic.c:2565 #31 0xff165544 in Tcl_UplevelObjCmd (dummy=0x1, interp=0x20e658, objc=1, objv=0x24ec80) at ./../generic/tclProc.c:614 #32 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0b70) at ./../generic/tclExecute.c:845 #33 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x274d50, flags=0) at ./../generic/tclBasic.c:2723 #34 0xff165afc in TclObjInterpProc (clientData=0x1, interp=0x20e658, objc=0, objv=0xffbeebd8) at ./../generic/tclProc.c:1001 #35 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0xffbeebd8, command=0xffbef024 "\n tkButtonUp .1907556\n", length=25, flags=0) at ./../generic/tclParse.c:932 #36 0xff15e7d0 in Tcl_EvalEx (interp=0x20e658, script=0xffbef024 "\n tkButtonUp .1907556\n", numBytes=25, flags=-4264800) at ./../generic/tclParse.c:1393 #37 0xff15e9c0 in Tcl_Eval (interp=0x20e658, string=0xffbef024 "\n tkButtonUp .1907556\n") at ./../generic/tclParse.c:1512 #38 0xff1243d0 in Tcl_GlobalEval (interp=0x20e658, command=0xffbef024 "\n tkButtonUp .1907556\n") at ./../generic/tclBasic.c:4139 #39 0xff221a40 in Tk_BindEvent (bindingTable=0xffbef024, eventPtr=0x29ffa0, tkwin=0x2790a8, numObjects=2045728, objectPtr=0xffbef170) at ./../generic/ tkBind.c:1784 #40 0xff226450 in TkBindEventProc (winPtr=0x2790a8, eventPtr=0x29ffa0) at ./../generic/tkCmds.c:244 #41 0xff22c218 in Tk_HandleEvent (eventPtr=0x29ffa0) at ./../generic/tkEvent.c:737 #42 0xff22c61c in WindowEventProc (evPtr=0x29ff98, flags=-1) at ./../generic/ tkEvent.c:1072 #43 0xff15bb54 in Tcl_ServiceEvent (flags=-1) at ./../generic/tclNotify.c:607 #44 0xff15beec in Tcl_DoOneEvent (flags=-1) at ./../generic/tclNotify.c:846 #45 0x99314 in EventHook () at ./_tkinter.c:2020 #46 0xbaf30 in rl_read_key () at input.c:374 #47 0xac920 in readline_internal_char () at readline.c:454 #48 0xaca64 in readline_internal_charloop () at readline.c:507 #49 0xaca94 in readline_internal () at readline.c:521 #50 0xac704 in readline (prompt=0x1cbd9c ">>> ") at readline.c:349 #51 0x8249c in call_readline (prompt=0x1cbd9c ">>> ") at ./readline.c:462 #52 0x21ae0 in PyOS_Readline (prompt=0x1cbd9c ">>> ") at myreadline.c:118 #53 0x205a0 in tok_nextc (tok=0x27abd0) at tokenizer.c:192 #54 0x20fb4 in PyTokenizer_Get (tok=0x27abd0, p_start=0xffbef8c4, p_end=0xffbef8c0) at tokenizer.c:516 #55 0x20274 in parsetok (tok=0x27abd0, g=0x17026c, start=256, err_ret=0xffbef9b0) at parsetok.c:128 #56 0x20158 in PyParser_ParseFile (fp=0x18ebe8, filename=0xbf628 "<stdin>", g=0x17026c, start=256, ps1=0x1cbd9c ">>> ", ps2=0x25a7e4 "... ", err_ret=0xffbef9b0) at parsetok.c:75 #57 0x3a9c0 in PyRun_InteractiveOne (fp=0x18ebe8, filename=0xbf628 "<stdin>") at pythonrun.c:514 #58 0x3a8bc in PyRun_InteractiveLoop (fp=0x18ebe8, filename=0xbf628 "<stdin>") at pythonrun.c:478 #59 0x3a7ac in PyRun_AnyFileEx (fp=0x18ebe8, filename=0xbf628 "<stdin>", closeit=0) at pythonrun.c:453 #60 0x3a76c in PyRun_AnyFile (fp=0x18ebe8, filename=0xbf628 "<stdin>") at pythonrun.c:444 #61 0x1ff20 in Py_Main (argc=3, argv=0xffbefc74) at main.c:297 #62 0x1f90c in main (argc=3, argv=0xffbefc74) at python.c:10 (gdb)
56. Scripting Language Extensions, April 18, 2002 56 beazley@cs.uchicago.edu Mixed Language Debugging Very little work in this area • Apparently some prior work with Common Lisp, Java • Uncharted territory for scripting extension programming. The WAD project • Wrapped Application Debugger [ Beazley, USENIX 2001 ] • Experimental work in cross-language exception handling: • Idea: convert fatal extension errors into scripting language exceptions. • Allow "fatal" errors to propagate back into scripting environment.
57. Scripting Language Extensions, April 18, 2002 57 beazley@cs.uchicago.edu WAD Example % python >>> import debug >>> debug.seg_crash() Segmentation fault (core dumped) % % python >>> import debug >>> import libwadpy WAD Enabled >>> debug.seg_crash() Traceback (most recent call last): File "<stdin>", line 1, in ? SegFault: [ C stack trace ] #2 0x000281c0 in call_builtin(func=0x1cbaf0,arg=0x18f114,kw=0x0) in 'ceval.c', line 2650 #1 0xfeee26b8 in _wrap_seg_crash(self=0x0,args=0x18f114) in 'pydebug.c', line 510 #0 0xfeee1258 in seg_crash(0x1,0xfeef2d48,0x19a9f8,0x0,0x7365675f,0x5f5f6469) in 'debug.c', line 18 /u0/beazley/Projects/WAD/WAD/Test/debug.c, line 18 int seg_crash() { int *a = 0; => *a = 3; return 1; } >>> [ Interpreter is still alive here ]
58. Scripting Language Extensions, April 18, 2002 58 beazley@cs.uchicago.edu Software Reliability Legacy code issues • Scripting environment significantly increases flexibility. • Sloppy programming ==> nasty problems later. • Batch programs are notorious for lack of error checking. Failure modes • Uninitialized data. • Partial initialization. • Domain errors. • Synchronization errors. • Non-reentrant functions. • Error handling. Problem • Very difficult to detect and diagose these problems (may be silent). • None of the existing extension building tools address these problems. Solutions • Extension building + contracts?
59. Scripting Language Extensions, April 18, 2002 59 beazley@cs.uchicago.edu Software Components Components • Scripting language as a framework for loosely coupled software components • Data analysis, visualization, database, networking, debugging, etc. Issues • Components may interact with each other. • However, may have no formal component specification. • Very adhoc and piecemeal. • Need to be very careful to think about interactions. • Scripting can work with other component frameworks. C/C++ Application Scripting Language C/C++ Application Scripting Language Data Analysis Visualization
60. Scripting Language Extensions, April 18, 2002 60 beazley@cs.uchicago.edu Project Management Scripting changes the development environment. • Shared libraries. • Dynamic loading. • Extension compilers. • Makefiles. • Scripting language configuration. • Debugging. • Version control. Comments • Many users seem to run into trouble here.
61. Scripting Language Extensions, April 18, 2002 61 beazley@cs.uchicago.edu Summary Scripting languages • Becoming an essential part of scientific programming efforts. • Part of a bigger picture involving components, frameworks, PSEs. • Used widely elsewhere. Extension building • An active sub-culture within the scripting language community. • Has grown primarily from scientific/industrial projects. • Active work in compilers, languages, systems, etc. • Largely unknown and unrecognized in mainstream CS community. Opportunities for CS • Compilers. Compiler design, extensible compilers. • Programming languages. Mixed language environments. • Software engineering. Components, configuration management. • Systems. Run-time environments, debugging.
62. Scripting Language Extensions, April 18, 2002 62 beazley@cs.uchicago.edu Discussion Acknowledgments • LANL: Peter Lomdahl, Brad Holian, Tim Germann, Shujia Zhou • SWIG: William Fulton, Lyle Johnson, Matthias Koeppe, Richard Palmer, Luigi Ballabio, Loic Dachary, Masaki Fukushima, Jason Stewart, Harco de Hilster, David Fletcher. Links: • SWIG. http://www.swig.org