Bridging Python and C & Introduction to ctypes

Slide 1

Slide 1 text

Bridging Python and C Also: Introduction to ctypes 1

Slide 2

Slide 2 text

Me • Call me TP • Follow @uranusjr • https://uranusjr.com 2

Slide 3

Slide 3 text

Slide 4

Slide 4 text

4 http://macdown.uranusjr.com

Slide 5

Slide 5 text

5 www. .com

Slide 6

Slide 6 text

Notice: C++ is not C 6

Slide 7

Slide 7 text

7 Python C

Slide 8

Slide 8 text

8 Python C Binding

Slide 9

Slide 9 text

Why? • I want to write C code!! • I have some C code • I want to access a C library 9

Slide 10

Slide 10 text

10 Python .so Binding

Slide 11

Slide 11 text

To compile, or not to compile. 11

Slide 12

Slide 12 text

Python C API 12 Python .so Binding C source compile compile C source

Slide 13

Slide 13 text

Cython 13 Python .so Binding C source Cython compile compile

Slide 14

Slide 14 text

No Compiling 14 Python .so Binding

Slide 15

Slide 15 text

https://en.wikipedia.org/wiki/Foreign_function_interface A foreign function interface (FFI) is a mechanism by which a program written in one programming language can call routines or make use of services written in another. 15

Slide 16

Slide 16 text

FFI • libfﬁ: Use C functions in other languages • Bundled in CPython • ctypes: Expose libfﬁ to Python • Available in many implementations 16

Slide 17

Slide 17 text

Show me the code! http://bit.ly/ejdb-example 17

Slide 18

Slide 18 text

Basics 18

Slide 19

Slide 19 text

Find the library 19 import ctypes from ctypes.util import find_library path = find_library('ejdb') _ = ctypes.cdll.LoadLibrary(path)

Slide 20

Slide 20 text

20 ejdbversion = _.ejdbversion ejdbversion.argtypes = [] ejdbversion.restype = ctypes.c_char_p const char *ejdbversion(); C Python

Slide 21

Slide 21 text

21 ctypes C Python c_bool _Bool bool c_char char 1-char binary c_wchar wchar_t 1-char text c_int int int c_uint unsigned int int c_double double ﬂoat c_char_p char * binary or None c_wchar_p wchar_t * text or None c_void_p void * int POINTER(type) type * (Not used directly)

Slide 22

Slide 22 text

22 ejdbversion = _.ejdbversion ejdbversion.argtypes = [] ejdbversion.restype = c_char_p version = ejdbversion() print(version) # b'1.2.6' # Notice the auto type conversion. # BTW this is Python 3.

Slide 23

Slide 23 text

23 void *bson_data2(bson *bs, int *len); int size = 0; void *data = bson_data2(bs, &size); if (data) printf("%.*s\n", size, (char *)data);

Slide 24

Slide 24 text

24 bson_data2.argtypes = [BSONP, POINTER(c_int)] bson_data2.restype = c_void_p size = c_int(0) data = bson_data2(bs, byref(size)) if data: # 0 for NULL. print(string_at(data, size.value))

Slide 25

Slide 25 text

25 typedef struct { char *cname; int cnamesz; TCTDB *tdb; EJDB *jb; void *mmtx; } EJCOLL; class EJCOLL(Structure): _fields_ = [ ('cname', c_char_p), ('cnamesz', c_int), ('tdb', TCTDBREF), ('jb', EJDBREF), ('mmtx', c_void_p), ] C Python

Slide 26

Slide 26 text

26 typedef union { char bytes[12]; int ints[3]; } bson_oid_t; class BSONOID(Union): _fields_ = [ ('bytes', c_char * 12), ('ints', c_int * 3), ] C Python

Slide 27

Slide 27 text

27 enum bson_binary_subtype_t { BSON_BIN_BINARY = 0, BSON_BIN_FUNC = 1, BSON_BIN_BINARY_OLD = 2, BSON_BIN_UUID = 3, BSON_BIN_MD5 = 5, BSON_BIN_USER = 128 }; BSON_BIN_BINARY = 0 BSON_BIN_FUNC = 1 BSON_BIN_BINARY_OLD = 2 BSON_BIN_UUID = 3 BSON_BIN_MD5 = 5 BSON_BIN_USER = 128 C Python

Slide 28

Slide 28 text

28 ejdbgetcoll.argtypes = [EJDBREF, c_char_p] ejdbgetcoll.restype = EJCOLLREF coll_r = ejdbgetcoll(jb, b'people') coll = coll_r.contents # Dereferencing. print(coll.cname) # Field access.

Slide 29

Slide 29 text

More Tricks 29

Slide 30

Slide 30 text

30 import ctypes import ctypes.util # POSIX. path = ctypes.util.find_library('libc') libc = ctypes.cdll.LoadLibrary(path) # Windows. # libc = ctypes.cdll.LoadLibrary('msvcrt') _ = lib.puts(b'Hello World!')

Slide 31

Slide 31 text

31 # const char *bson_data2( # const bson *bs, int *size); bson_data2 = _.bson_data2 bson_data2.argtypes = [BSONREF, POINTER(c_int)] bson_data2.restype = c_char_p bs = ... size = c_int() print(bson_data2(bs, byref(size))) # b'\x0e' <-‐ Oops

Slide 32

Slide 32 text

32 # const char *bson_data2( # const bson *bs, int *size); bson_data2 = _.bson_data2 bson_data2.argtypes = [BSONREF, POINTER(c_int)] bson_data2.restype = c_void_p bs = ... size = c_int() data_ptr = bson_data2(bs, byref(size)) print(string_at(data_ptr, size.value)) # b'\x0e\x00\x00\x00\x10A\x00\x01\x00\x00\x00\x00'

Slide 33

Slide 33 text

When in Rome • Array and pointer manipulation • pointer and addressof • Type-casting is your weapon • cast • Opaque pointers • c_void_p 33

Slide 34

Slide 34 text

34 data = b'...' # NO! bson_append_binary( bs, b'photo', BSON_BIN_BINARY, data, len(data)) # YES. buf = create_string_buffer(data, len(data)) bson_append_binary( bs, b'photo', BSON_BIN_BINARY, buf, len(data))

Slide 35

Slide 35 text

35 class BSONOID(Union): _fields_ = [...] def __str__(self): # ... @classmethod def from_string(cls, s): oid = cls() bson_oid_from_string(byref(oid), s) return oid

Slide 36

Slide 36 text

36 int main(int, const char *[]) { EJDB *jb = ejdbnew(); // Do things... ejdbdel(jb); return 0; } def main(): jb = EJDB() # Garbage collection FTW! C Python

Slide 37

Slide 37 text

37 class EJDB: def __init__(self): self._wrapped = ejdbnew() # When do we call `ejdbdel()`?

Slide 38

Slide 38 text

38 class EJDB: def __init__(self): self._wrapped = ejdbnew() def __del__(self): ejdbdel(self._wrapped) NO!

Slide 39

Slide 39 text

bit.ly/python-ctypes-raii 39

Slide 40

Slide 40 text

Warnings • Using ctypes ≈ writing C • It will crash if things go wrong • I mean Segmentation Fault 40

Slide 41

Slide 41 text

Advices • Test your modules • Protect things with wrappers • Python 3 faulthandler • When in doubt, write some C 41

Slide 42

Slide 42 text

Anyway, should I use it? 42

Slide 43

Slide 43 text

Yes • Beyond CPython • No need of a C compiler • Don’t need the C source 43

Slide 44

Slide 44 text

Maybe Not • Need to compile dependencies anyway • C is fucking annoying • Don’t have a C API? 44