Extending Python with Compiled Languages Emil Sharifullin @litleleprikon 1 

Why? 2 

Show me the Code http:!//bit.ly/ExtendingPython 3 

Extending with C 4 

C/C++ • Python native libraries • Basically a C shared libraries • Python working perfectly with C/C!++ shared libraries • C functions are needed to be wrapped with python-compatible functions using python C API 5 

Code typedef struct { int length; int *values; } Array; Array *SieveOfEratosthenes(int n) 6 C/CExtExample.c 

Add Python Support • Include Python.h • Wrap functions with python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_ that returns PyModuleDef 7 

Add Python Support #define PY_SSIZE_T_CLEAN #include #if PY_MAJOR_VERSION >= 3 … … … #endif C/CExtExample.c 8 

Add Python Support static PyObject *CEratosphenes_SieveOfEratosphenes( PyObject *self, PyObject *args) C/CExtExample.c 9 

Add Python Support if (!PyArg_ParseTuple(args, "I", &n)) { return NULL; } C/CExtExample.c 10 

Add Python Support PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, …); PyAPI_FUNC(int) PyArg_ParseTupleAndKeywords( PyObject *, PyObject *, const char *, char **, ...); https://docs.python.org/3/c-api/arg.html 11 

Add Python Support PyObject *result = PyList_New(0); … PyList_Append(result, PyLong_FromLong(p)); C/CExtExample.c 12 

Add Python Support C/CExtExample.c static PyMethodDef Methods[] = { {“sieve_of_eratosphenes", CExtExample_SieveOfEratosphenes, METH_VARARGS, "Return list of prime numbers until given number."}, {NULL, NULL, 0, NULL} "/* Sentinel "*/ }; 13 

Add Python Support C/CExtExample.c static PyModuleDef CExtExample = { PyModuleDef_HEAD_INIT, "CExtExample", "Example module written in C", -1, Methods}; 14 

Add Python Support C/CExtExample.c PyObject *PyInit_CExtExample(void) { return PyModule_Create(&CExtExample); } 15 

Setup   esharifu  $ ~/projects  python3 %--version Python 3.7.5   esharifu  $ ~/projects  clang -v Apple clang version 11.0.0 (clang-1100.0.20.17) Target: x86_64-apple-darwin18.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/ usr/bin 16 

Building from setuptools import setup, Extension extension = Extension('CExtExample', sources=['CExtExample.c']) setup(name='CExtExample', version='1.0', description='This is Example module written in C', ext_modules=[extension]) C/setup.py 17 

Building $ ~/p/E/C  nm CExtExample.cpython-37m-darwin.so 0000000000001050 d _CExtExample 0000000000000dd0 t _CExtExample_SieveOfEratosphenes 00000000000010c0 d _Methods 0000000000000db0 T _PyInit_CExtExample U _PyList_Append U _PyList_New U _PyLong_FromLong U _PyModule_Create2 U %__PyArg_ParseTuple_SizeT U _free U _malloc U dyld_stub_binder 18 

Memory Management • Do not use malloc/free on python objects • Use Py_New instead • Python have it’s own allocator and own private heap • You can use malloc/free on c structures 19 

Pros & Cons • Freedom of GIL • Manual memory management • High speed 20 

Extending with Go 21 

Extending with Go 22 

Go • Compiling Go code to shared libraries • Using CGo • CGo !!= Go • Use Go shared library in C extension 23 

Code func SieveOfEratosphenes(n int) []int { … } Go/GoExtExample.go 24 

Add Python Support • Import C • Include Python.h • Wrap go functions to CGo-compatible wrappers • Include Python.h • Wrap functions with python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_ that returns PyModuleDef 25 

Add Python Support pt. 1 // #cgo pkg-config: python3 // #include import "C" Go/GoExtExample.go 26 

Add Python Support pt. 1 // SieveOfEratosphenes calculates prime numbers //export SieveOfEratosphenes func func SieveOfEratosphenes(n int, result *[]int) { … } Go/GoExtExample.go 27 

Setup   esharifu  $ ~/projects  python3 %--version Python 3.7.5   esharifu  $ ~/projects  go version go version go1.13.1 darwin/amd64 28 

Building pt. 1 go build -v -buildmode=c-shared -o GoExtExampleGo.so \ GoExtExample.go Go/Makefile 29 

Add Python Support pt. 2 #define PY_SSIZE_T_CLEAN #include #include "GoExtExampleGo.h" #if PY_MAJOR_VERSION >= 3 Go/GoExtExample.c 30 

Add Python Support pt. 2 Go/GoExtExample.c static PyObject *GoExtExample_SieveOfEratosphenes(PyObject *self, PyObject *args) { int n; if (!PyArg_ParseTuple(args, "I", &n)) { return NULL; } GoSlice primes = {}; SieveOfEratosphenes((GoInt)n, &primes); PyObject *result = PyList_New(0); for (int i = 0; i < primes.len; i"++) { GoInt current = ((GoInt *)(primes.data))[i]; PyList_Append(result, PyLong_FromLong(current)); } return result; } 31 

Building pt. 2 gcc GoExtExample.c -dynamiclib GoExtExampleGo.so -o \ GoExtExample.so \ $(python3-config —cflags) \ $(python3-config --ldflags) 32 

Pros & Cons • Unable to use Go tooling • Unable return go structs from go funcs and multiple values • Possible memory leaks • Blocking goroutines during CGo calls • Unable to cross-compile • A bunch of side effects from CGo 33 

Memory Management • https:!//karthikkaranth.me/blog/calling-c- code-from-go/ • Python and Go both have private heaps • GC of both languages operates only inside it’s private heap and knows only about objects it allocated 34 

Memory Management 35 

Benchmarks import pytest import GoExtExample @pytest.mark.parametrize("input", [1000, 100000, 1000000]) def test_go(benchmark, input): benchmark(GoExtExample.sieve_of_eratosphenes, input) Go/test_benchmark.py 36 

Benchmarks ms 0 10 20 30 40 Sieve Limit 1000 100000 1000000 C Go 37 

Benchmarks Ms 0 3000 6000 9000 12000 Sieve Limit 1000 100000 1000000 C Go Python 38 

Why? • Speed(Not Guaranteed) • Avoiding GIL • Step-by-step migration from Python • Using language-specific libraries 39 

Why Not? • Easy way to shot to your leg • Harder Memory Management 40 

Thank You! •https:!//docs.python.org/3/extending/ extending.html •https:!//docs.python.org/3.7/c-api/index.html •https:!//hackernoon.com/extending-python-3- in-go-78f3a69552ac 41 Emil Sharifullin [email protected] @litleleprikon