Writing a Python C extension in 2018

Transcript

1. Writing a C Python extension in 2018 Paris.py 2018
2. Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at Apple (Red Team) PyMiniRacer author jb@sqreen.io @jbaviat Who Am I?
3. Someday… we needed to use V8 from Python. What we ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!
5. PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer
6. The problem • V8 is C++ • How do you run C++ in Python? …We need some kind of binding between these 2 worlds.
7. Many popular packages rely on binary cryptography numpy pymongo psycopg simplejson lxml sqlalchemy…
8. People do it!
 Let’s do it too.
9. What are our goals? • minimize maintenance • make setup easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…
10. built-in pythonic Python version independant open to other languages high throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔
11. ctypes Built into Python Binary is Python independant: • can be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++
12. $ python >>> path = "./hello.so" >>> import ctypes >>> lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary
 object
13. Overview V8 (C++ interface) C interface to V8 Python interface 3rd party binaries import ctypes class PyMiniRacer(object): … #include <v8.h> int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linking ctypes C/C++ code Python library
14. How to put this together? $ cat setup.py from distutils.core import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so
17. Ship it
18. Ever had trouble installing packages?
20. (at Sqreen, we are dog friendly) Come get your dog sticker!
21. Python packaging
 history sdist (source distribution) eggs wheels —> manylinux wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤
22. manylinux wheels Python standard: PEP513 / PEP571 Compatible on most (real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on CentOS 5
23. CentOS 5?? PEP571 → bumped requirement to CentOS 6!
24. Wheels vs compilation iso builds (crash can be reproduced) many packages need maintenance one build per user only one package harder to install…
25. Best of 2 worlds We can have: ctypes: use a Python independant binary wheels: ship compiled binaries By combining these… ctypes + wheels = ship only one binary
26. Thanks! Questions? We’re
 hiring! https://sqreen.io/jobs