Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending Python with Compiled Languages

Extending Python with Compiled Languages

5b8d20aa7d63c5d391b1c881e1764460?s=128

Iskander (Alex) Sharipov

November 30, 2019
Tweet

Transcript

  1. Extending Python with Compiled Languages Emil Sharifullin @litleleprikon 1

  2. Why? 2

  3. Show me the Code http:!//bit.ly/ExtendingPython 3

  4. Extending with C 4

  5. C/C++ • Python native libraries • Basically a C shared

    libraries • Python working perfectly with C/C!++ shared libraries • C functions are needed to be wrapped with python-compatible functions using python C API 5
  6. Code typedef struct { int length; int *values; } Array;

    Array *SieveOfEratosthenes(int n) 6 C/CExtExample.c
  7. Add Python Support • Include Python.h • Wrap functions with

    python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_<name> that returns PyModuleDef 7
  8. Add Python Support #define PY_SSIZE_T_CLEAN #include <Python/Python.h> #if PY_MAJOR_VERSION >=

    3 … … … #endif C/CExtExample.c 8
  9. Add Python Support static PyObject *CEratosphenes_SieveOfEratosphenes( PyObject *self, PyObject *args)

    C/CExtExample.c 9
  10. Add Python Support if (!PyArg_ParseTuple(args, "I", &n)) { return NULL;

    } C/CExtExample.c 10
  11. Add Python Support PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, …);

    PyAPI_FUNC(int) PyArg_ParseTupleAndKeywords( PyObject *, PyObject *, const char *, char **, ...); https://docs.python.org/3/c-api/arg.html 11
  12. Add Python Support PyObject *result = PyList_New(0); … PyList_Append(result, PyLong_FromLong(p));

    C/CExtExample.c 12
  13. Add Python Support C/CExtExample.c static PyMethodDef Methods[] = { {“sieve_of_eratosphenes",

    CExtExample_SieveOfEratosphenes, METH_VARARGS, "Return list of prime numbers until given number."}, {NULL, NULL, 0, NULL} "/* Sentinel "*/ }; 13
  14. Add Python Support C/CExtExample.c static PyModuleDef CExtExample = { PyModuleDef_HEAD_INIT,

    "CExtExample", "Example module written in C", -1, Methods}; 14
  15. Add Python Support C/CExtExample.c PyObject *PyInit_CExtExample(void) { return PyModule_Create(&CExtExample); }

    15
  16. Setup   esharifu  $ ~/projects  python3 %--version

    Python 3.7.5   esharifu  $ ~/projects  clang -v Apple clang version 11.0.0 (clang-1100.0.20.17) Target: x86_64-apple-darwin18.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/ usr/bin 16
  17. Building from setuptools import setup, Extension extension = Extension('CExtExample', sources=['CExtExample.c'])

    setup(name='CExtExample', version='1.0', description='This is Example module written in C', ext_modules=[extension]) C/setup.py 17
  18. Building $ ~/p/E/C  nm CExtExample.cpython-37m-darwin.so 0000000000001050 d _CExtExample 0000000000000dd0

    t _CExtExample_SieveOfEratosphenes 00000000000010c0 d _Methods 0000000000000db0 T _PyInit_CExtExample U _PyList_Append U _PyList_New U _PyLong_FromLong U _PyModule_Create2 U %__PyArg_ParseTuple_SizeT U _free U _malloc U dyld_stub_binder 18
  19. Memory Management • Do not use malloc/free on python objects

    • Use Py<StructName>_New instead • Python have it’s own allocator and own private heap • You can use malloc/free on c structures 19
  20. Pros & Cons • Freedom of GIL • Manual memory

    management • High speed 20
  21. Extending with Go 21

  22. Extending with Go 22

  23. Go • Compiling Go code to shared libraries • Using

    CGo • CGo !!= Go • Use Go shared library in C extension 23
  24. Code func SieveOfEratosphenes(n int) []int { … } Go/GoExtExample.go 24

  25. Add Python Support • Import C • Include Python.h •

    Wrap go functions to CGo-compatible wrappers • Include Python.h • Wrap functions with python-compatible wrappers • Define array of methods • Define PyModuleDef • Create function PyInit_<name> that returns PyModuleDef 25
  26. Add Python Support pt. 1 // #cgo pkg-config: python3 //

    #include <Python.h> import "C" Go/GoExtExample.go 26
  27. Add Python Support pt. 1 // SieveOfEratosphenes calculates prime numbers

    //export SieveOfEratosphenes func func SieveOfEratosphenes(n int, result *[]int) { … } Go/GoExtExample.go 27
  28. Setup   esharifu  $ ~/projects  python3 %--version

    Python 3.7.5   esharifu  $ ~/projects  go version go version go1.13.1 darwin/amd64 28
  29. Building pt. 1 go build -v -buildmode=c-shared -o GoExtExampleGo.so \

    GoExtExample.go Go/Makefile 29
  30. Add Python Support pt. 2 #define PY_SSIZE_T_CLEAN #include <Python.h> #include

    "GoExtExampleGo.h" #if PY_MAJOR_VERSION >= 3 Go/GoExtExample.c 30
  31. Add Python Support pt. 2 Go/GoExtExample.c static PyObject *GoExtExample_SieveOfEratosphenes(PyObject *self,

    PyObject *args) { int n; if (!PyArg_ParseTuple(args, "I", &n)) { return NULL; } GoSlice primes = {}; SieveOfEratosphenes((GoInt)n, &primes); PyObject *result = PyList_New(0); for (int i = 0; i < primes.len; i"++) { GoInt current = ((GoInt *)(primes.data))[i]; PyList_Append(result, PyLong_FromLong(current)); } return result; } 31
  32. Building pt. 2 gcc GoExtExample.c -dynamiclib GoExtExampleGo.so -o \ GoExtExample.so

    \ $(python3-config —cflags) \ $(python3-config --ldflags) 32
  33. Pros & Cons • Unable to use Go tooling •

    Unable return go structs from go funcs and multiple values • Possible memory leaks • Blocking goroutines during CGo calls • Unable to cross-compile • A bunch of side effects from CGo 33
  34. Memory Management • https:!//karthikkaranth.me/blog/calling-c- code-from-go/ • Python and Go both

    have private heaps • GC of both languages operates only inside it’s private heap and knows only about objects it allocated 34
  35. Memory Management 35

  36. Benchmarks import pytest import GoExtExample @pytest.mark.parametrize("input", [1000, 100000, 1000000]) def

    test_go(benchmark, input): benchmark(GoExtExample.sieve_of_eratosphenes, input) Go/test_benchmark.py 36
  37. Benchmarks ms 0 10 20 30 40 Sieve Limit 1000

    100000 1000000 C Go 37
  38. Benchmarks Ms 0 3000 6000 9000 12000 Sieve Limit 1000

    100000 1000000 C Go Python 38
  39. Why? • Speed(Not Guaranteed) • Avoiding GIL • Step-by-step migration

    from Python • Using language-specific libraries 39
  40. Why Not? • Easy way to shot to your leg

    • Harder Memory Management 40
  41. Thank You! •https:!//docs.python.org/3/extending/ extending.html •https:!//docs.python.org/3.7/c-api/index.html •https:!//hackernoon.com/extending-python-3- in-go-78f3a69552ac 41 Emil Sharifullin

    iam@litleleprikon.me @litleleprikon