Bridging Python and C & Introduction to ctypes

Some solutions to work with C code/library in Python, and introduction to ctypes, a pure-Python solution to make use of compiled C shared objects. Also some tips and tricks when working with ctypes (and C in general).

Notes:

Slide 6
========

So we’re not going to talk about SWIG, SIP, Shiboken, etc. Just Python and C.

Slide 9
========

First: You don’t have the code (yet).
Second: You have the code, but you need to compile it.
Third: You have a compiled binary, and you want to use it in Python.

Slide 12
========

Suitable if: You need to write some C anyway, and is knowledgable enough with CPython internals.

Slide 13
========

Suitable if: You’re dealing with very little or non existing C code. Writing Cython is a lot easier than C, so you get speed in the binding with much hassle.

Slide 17
========

You can see how similar ctypes code is to straight C.

Slide 21
========

Also there are some types from stdint.h and others, e.g. c_int32 (mapped to int32_t). The naming rules are pretty straightforward.

Slide 23
========

Another example. (Python equivalent in the next slide.)

Notice three topics:

void * and non-NULL-terminated char * printing.
int * “out” parameter, pass by pointer, and address-of (&) operator.
int declaration.

Slide 24
========

Rough Python equivalent of the previous C code. Look at how each set of syntaxes is translated.

Note: C type variables are always initialised in ctypes, even if you don’t pass a value! c_int() is equivalent to c_int(0).

Slide 30
========

This is “Hello World” with ctypes. The point here is, function annotations are just hints, and are optional.

They are recommended, but you should use them as a tool, not an obligation.

Slide 31 & 32
========

You might get into trouble if you follow the C declaration blindly. Remember, C is weakly-typed.

Slide 35
========

Remember, unions and structures are all Python classes, and can have normal Python attributes! Use them!

Slide 41
========

This last one helps me very often. I find it extremely helpful to write some simple C programs first, then translate them into Python with ctypes.

Transcript

1. Bridging Python and C
   Also: Introduction to ctypes
   1
   

   View Slide

2. Me
   • Call me TP
   • Follow @uranusjr
   • https://uranusjr.com
   2
   

   View Slide

3. View Slide

4. 4
   http://macdown.uranusjr.com
   

   View Slide

5. 5
   www. .com
   

   View Slide

6. Notice: C++ is not C
   6
   

   View Slide

7. 7
   Python C
   

   View Slide

8. 8
   Python C
   Binding
   

   View Slide

9. Why?
   • I want to write C code!!
   • I have some C code
   • I want to access a C library
   9
   

   View Slide

10. 10
    Python .so
    Binding
    

    View Slide

11. To compile,
    or not to compile.
    11
    

    View Slide

12. Python C API
    12
    Python .so
    Binding
    C source
    compile compile
    C source
    

    View Slide

13. Cython
    13
    Python .so
    Binding
    C source
    Cython
    compile
    compile
    

    View Slide

14. No Compiling
    14
    Python .so
    Binding
    

    View Slide

15. https://en.wikipedia.org/wiki/Foreign_function_interface
    A foreign function interface (FFI)
    is a mechanism by which a program
    written in one programming
    language can call routines or make
    use of services written in another.
    15
    

    View Slide

16. FFI
    • libffi: Use C functions in other languages
    • Bundled in CPython
    • ctypes: Expose libffi to Python
    • Available in many implementations
    16
    

    View Slide

17. Show me the code!
    http://bit.ly/ejdb-example
    17
    

    View Slide

18. Basics
    18
    

    View Slide

19. Find the library
    19
    import  ctypes  
    from  ctypes.util  import  find_library  
    path  =  find_library('ejdb')  
    _  =  ctypes.cdll.LoadLibrary(path)
    

    View Slide

20. 20
    ejdbversion  =  _.ejdbversion  
    ejdbversion.argtypes  =  []  
    ejdbversion.restype  =  ctypes.c_char_p
    const  char  *ejdbversion(); C
    Python
    

    View Slide

21. 21
    ctypes C Python
    c_bool _Bool bool
    c_char char 1-char binary
    c_wchar wchar_t 1-char text
    c_int int int
    c_uint unsigned int int
    c_double double float
    c_char_p char * binary or None
    c_wchar_p wchar_t * text or None
    c_void_p void * int
    POINTER(type) type * (Not used directly)
    

    View Slide

22. 22
    ejdbversion  =  _.ejdbversion  
    ejdbversion.argtypes  =  []  
    ejdbversion.restype  =  c_char_p  
    version  =  ejdbversion()  
    print(version)  
    #  b'1.2.6'  
    #  Notice  the  auto  type  conversion.  
    #  BTW  this  is  Python  3.
    

    View Slide

23. 23
    void  *bson_data2(bson  *bs,  int  *len);  
    int  size  =  0;  
    void  *data  =  bson_data2(bs,  &size);  
    if  (data)  
           printf("%.*s\n",  size,  (char  *)data);
    

    View Slide

24. 24
    bson_data2.argtypes  =  [BSONP,  POINTER(c_int)]  
    bson_data2.restype  =  c_void_p  
    size  =  c_int(0)  
    data  =  bson_data2(bs,  byref(size))  
    if  data:        #  0  for  NULL.  
           print(string_at(data,  size.value))
    

    View Slide

25. 25
    typedef  struct  {  
           char  *cname;  
           int  cnamesz;  
           TCTDB  *tdb;  
           EJDB  *jb;  
           void  *mmtx;  
    }  EJCOLL;
    class  EJCOLL(Structure):  
           _fields_  =  [  
                   ('cname',  c_char_p),  
                   ('cnamesz',  c_int),  
                   ('tdb',  TCTDBREF),  
                   ('jb',  EJDBREF),  
                   ('mmtx',  c_void_p),  
           ]
    C
    Python
    

    View Slide

26. 26
    typedef  union  {  
           char  bytes[12];  
           int  ints[3];  
    }  bson_oid_t;
    class  BSONOID(Union):  
         _fields_  =  [  
                   ('bytes',  c_char  *  12),  
                   ('ints',  c_int  *  3),  
           ]
    C
    Python
    

    View Slide

27. 27
    enum  bson_binary_subtype_t  {  
           BSON_BIN_BINARY  =  0,  
           BSON_BIN_FUNC  =  1,  
           BSON_BIN_BINARY_OLD  =  2,  
           BSON_BIN_UUID  =  3,  
           BSON_BIN_MD5  =  5,  
           BSON_BIN_USER  =  128  
    };
    BSON_BIN_BINARY  =  0  
    BSON_BIN_FUNC  =  1  
    BSON_BIN_BINARY_OLD  =  2  
    BSON_BIN_UUID  =  3  
    BSON_BIN_MD5  =  5  
    BSON_BIN_USER  =  128
    C
    Python
    

    View Slide

28. 28
    ejdbgetcoll.argtypes  =  [EJDBREF,  c_char_p]  
    ejdbgetcoll.restype  =  EJCOLLREF  
    coll_r  =  ejdbgetcoll(jb,  b'people')  
    coll  =  coll_r.contents        #  Dereferencing.  
    print(coll.cname)                  #  Field  access.
    

    View Slide

29. More Tricks
    29
    

    View Slide

30. 30
    import  ctypes  
    import  ctypes.util  
    #  POSIX.  
    path  =  ctypes.util.find_library('libc')  
    libc  =  ctypes.cdll.LoadLibrary(path)  
    #  Windows.  
    #  libc  =  ctypes.cdll.LoadLibrary('msvcrt')  
    _  =  lib.puts(b'Hello  World!')
    

    View Slide

31. 31
    #  const  char  *bson_data2(  
    #          const  bson  *bs,  int  *size);  
    bson_data2  =  _.bson_data2  
    bson_data2.argtypes  =  [BSONREF,  POINTER(c_int)]  
    bson_data2.restype  =  c_char_p  
    bs  =  ...  
    size  =  c_int()  
    print(bson_data2(bs,  byref(size)))  
    #  b'\x0e'              <-­‐  Oops
    

    View Slide

32. 32
    #  const  char  *bson_data2(  
    #          const  bson  *bs,  int  *size);  
    bson_data2  =  _.bson_data2  
    bson_data2.argtypes  =  [BSONREF,  POINTER(c_int)]  
    bson_data2.restype  =  c_void_p  
    bs  =  ...  
    size  =  c_int()  
    data_ptr  =  bson_data2(bs,  byref(size))  
    print(string_at(data_ptr,  size.value))  
    #  b'\x0e\x00\x00\x00\x10A\x00\x01\x00\x00\x00\x00'
    

    View Slide

33. When in Rome
    • Array and pointer manipulation
    • pointer and addressof
    • Type-casting is your weapon
    • cast
    • Opaque pointers
    • c_void_p
    33
    

    View Slide

34. 34
    data  =  b'...'  
    #  NO!  
    bson_append_binary(  
           bs,  b'photo',  BSON_BIN_BINARY,  
           data,  len(data))  
    #  YES.  
    buf  =  create_string_buffer(data,  len(data))  
    bson_append_binary(  
           bs,  b'photo',  BSON_BIN_BINARY,  
           buf,  len(data))
    

    View Slide

35. 35
    class  BSONOID(Union):  
           _fields_  =  [...]  
           def  __str__(self):  
                   #  ...  
           @classmethod  
           def  from_string(cls,  s):  
                   oid  =  cls()  
                   bson_oid_from_string(byref(oid),  s)  
                   return  oid
    

    View Slide

36. 36
    int  main(int,  const  char  *[])  
    {  
           EJDB  *jb  =  ejdbnew();  
           //  Do  things...  
           ejdbdel(jb);  
           return  0;  
    }
    def  main():  
           jb  =  EJDB()  
           #  Garbage  collection  FTW!
    C
    Python
    

    View Slide

37. 37
    class  EJDB:  
           def  __init__(self):  
                   self._wrapped  =  ejdbnew()  
           #  When  do  we  call  `ejdbdel()`?  
    

    View Slide

38. 38
    class  EJDB:  
           def  __init__(self):  
                   self._wrapped  =  ejdbnew()  
           def  __del__(self):  
                   ejdbdel(self._wrapped)
    NO!
    

    View Slide

39. bit.ly/python-ctypes-raii
    39
    

    View Slide

40. Warnings
    • Using ctypes ≈ writing C
    • It will crash if things go wrong
    • I mean Segmentation Fault
    40
    

    View Slide

41. Advices
    • Test your modules
    • Protect things with wrappers
    • Python 3 faulthandler
    • When in doubt, write some C
    41
    

    View Slide

42. Anyway,
    should I use it?
    42
    

    View Slide

43. Yes
    • Beyond CPython
    • No need of a C compiler
    • Don’t need the C source
    43
    

    View Slide

44. Maybe Not
    • Need to compile
    dependencies anyway
    • C is fucking annoying
    • Don’t have a C API?
    44
    

    View Slide

45. Choose the Right Tool
    45
    

    View Slide

46. To be continued…?
    46
    

    View Slide