Extending Python with Compiled Languages

Transcript

1. Extending Python with Compiled Languages Emil Sharifullin @litleleprikon 1
2. Why? 2
3. Show me the Code http:!//bit.ly/ExtendingPython 3
4. Extending with C 4
5. C/C++ • Python native libraries • Basically a C shared libraries • Python working perfectly with C/C!++ shared libraries • C functions are needed to be wrapped with python-compatible functions using python C API 5
6. Code typedef struct { int length; int *values; } Array; Array *SieveOfEratosthenes(int n) 6 C/CExtExample.c
7. Add Python Support • Include Python.h • Wrap functions with python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_<name> that returns PyModuleDef 7
8. Add Python Support #define PY_SSIZE_T_CLEAN #include <Python/Python.h> #if PY_MAJOR_VERSION >= 3 … … … #endif C/CExtExample.c 8
9. Add Python Support static PyObject *CEratosphenes_SieveOfEratosphenes( PyObject *self, PyObject *args) C/CExtExample.c 9
10. Add Python Support if (!PyArg_ParseTuple(args, "I", &n)) { return NULL; } C/CExtExample.c 10
11. Add Python Support PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, …); PyAPI_FUNC(int) PyArg_ParseTupleAndKeywords( PyObject *, PyObject *, const char *, char **, ...); https://docs.python.org/3/c-api/arg.html 11
12. Add Python Support PyObject *result = PyList_New(0); … PyList_Append(result, PyLong_FromLong(p)); C/CExtExample.c 12
13. Add Python Support C/CExtExample.c static PyMethodDef Methods[] = { {“sieve_of_eratosphenes", CExtExample_SieveOfEratosphenes, METH_VARARGS, "Return list of prime numbers until given number."}, {NULL, NULL, 0, NULL} "/* Sentinel "*/ }; 13
14. Add Python Support C/CExtExample.c static PyModuleDef CExtExample = { PyModuleDef_HEAD_INIT, "CExtExample", "Example module written in C", -1, Methods}; 14
15. Add Python Support C/CExtExample.c PyObject *PyInit_CExtExample(void) { return PyModule_Create(&CExtExample); } 15
16. Setup   esharifu  $ ~/projects  python3 %--version Python 3.7.5   esharifu  $ ~/projects  clang -v Apple clang version 11.0.0 (clang-1100.0.20.17) Target: x86_64-apple-darwin18.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/ usr/bin 16
17. Building from setuptools import setup, Extension extension = Extension('CExtExample', sources=['CExtExample.c']) setup(name='CExtExample', version='1.0', description='This is Example module written in C', ext_modules=[extension]) C/setup.py 17
18. Building $ ~/p/E/C  nm CExtExample.cpython-37m-darwin.so 0000000000001050 d _CExtExample 0000000000000dd0 t _CExtExample_SieveOfEratosphenes 00000000000010c0 d _Methods 0000000000000db0 T _PyInit_CExtExample U _PyList_Append U _PyList_New U _PyLong_FromLong U _PyModule_Create2 U %__PyArg_ParseTuple_SizeT U _free U _malloc U dyld_stub_binder 18
19. Memory Management • Do not use malloc/free on python objects • Use Py<StructName>_New instead • Python have it’s own allocator and own private heap • You can use malloc/free on c structures 19
20. Pros & Cons • Freedom of GIL • Manual memory management • High speed 20
21. Extending with Go 21
22. Extending with Go 22
23. Go • Compiling Go code to shared libraries • Using CGo • CGo !!= Go • Use Go shared library in C extension 23
24. Code func SieveOfEratosphenes(n int) []int { … } Go/GoExtExample.go 24
25. Add Python Support • Import C • Include Python.h • Wrap go functions to CGo-compatible wrappers • Include Python.h • Wrap functions with python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_<name> that returns PyModuleDef 25
26. Add Python Support pt. 1 // #cgo pkg-config: python3 // #include <Python.h> import "C" Go/GoExtExample.go 26
27. Add Python Support pt. 1 // SieveOfEratosphenes calculates prime numbers //export SieveOfEratosphenes func func SieveOfEratosphenes(n int, result *[]int) { … } Go/GoExtExample.go 27
28. Setup   esharifu  $ ~/projects  python3 %--version Python 3.7.5   esharifu  $ ~/projects  go version go version go1.13.1 darwin/amd64 28
29. Building pt. 1 go build -v -buildmode=c-shared -o GoExtExampleGo.so \ GoExtExample.go Go/Makefile 29
30. Add Python Support pt. 2 #define PY_SSIZE_T_CLEAN #include <Python.h> #include "GoExtExampleGo.h" #if PY_MAJOR_VERSION >= 3 Go/GoExtExample.c 30
31. Add Python Support pt. 2 Go/GoExtExample.c static PyObject *GoExtExample_SieveOfEratosphenes(PyObject *self, PyObject *args) { int n; if (!PyArg_ParseTuple(args, "I", &n)) { return NULL; } GoSlice primes = {}; SieveOfEratosphenes((GoInt)n, &primes); PyObject *result = PyList_New(0); for (int i = 0; i < primes.len; i"++) { GoInt current = ((GoInt *)(primes.data))[i]; PyList_Append(result, PyLong_FromLong(current)); } return result; } 31
32. Building pt. 2 gcc GoExtExample.c -dynamiclib GoExtExampleGo.so -o \ GoExtExample.so \ $(python3-config —cflags) \ $(python3-config --ldflags) 32
33. Pros & Cons • Unable to use Go tooling • Unable return go structs from go funcs and multiple values • Possible memory leaks • Blocking goroutines during CGo calls • Unable to cross-compile • A bunch of side effects from CGo 33
34. Memory Management • https:!//karthikkaranth.me/blog/calling-c- code-from-go/ • Python and Go both have private heaps • GC of both languages operates only inside it’s private heap and knows only about objects it allocated 34
35. Memory Management 35
36. Benchmarks import pytest import GoExtExample @pytest.mark.parametrize("input", [1000, 100000, 1000000]) def test_go(benchmark, input): benchmark(GoExtExample.sieve_of_eratosphenes, input) Go/test_benchmark.py 36
37. Benchmarks ms 0 10 20 30 40 Sieve Limit 1000 100000 1000000 C Go 37
38. Benchmarks Ms 0 3000 6000 9000 12000 Sieve Limit 1000 100000 1000000 C Go Python 38
39. Why? • Speed(Not Guaranteed) • Avoiding GIL • Step-by-step migration from Python • Using language-specific libraries 39
40. Why Not? • Easy way to shot to your leg • Harder Memory Management 40
41. Thank You! •https:!//docs.python.org/3/extending/ extending.html •https:!//docs.python.org/3.7/c-api/index.html •https:!//hackernoon.com/extending-python-3- in-go-78f3a69552ac 41 Emil Sharifullin iam@litleleprikon.me @litleleprikon