Writing a Python C extension in 2018

Transcript

1. Writing a C Python extension in 2018 Paris.py 2018

2. Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at

   Apple (Red Team) PyMiniRacer author jb@sqreen.io @jbaviat Who Am I?

3. Someday… we needed to use V8 from Python. What we

   ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!
4. None

5. PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer

6. The problem • V8 is C++ • How do you

   run C++ in Python? …We need some kind of binding between these 2 worlds.

7. Many popular packages rely on binary cryptography numpy pymongo psycopg

   simplejson lxml sqlalchemy…

8. People do it!
 Let’s do it too.

9. What are our goals? • minimize maintenance • make setup

   easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…

10. built-in pythonic Python version independant open to other languages high

    throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔

11. ctypes Built into Python Binary is Python independant: • can

    be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++

12. $ python >>> path = "./hello.so" >>> import ctypes >>>

    lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary
 object

13. Overview V8 (C++ interface) C interface to V8 Python interface

    3rd party binaries import ctypes class PyMiniRacer(object): … #include <v8.h> int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linking ctypes C/C++ code Python library

14. How to put this together? $ cat setup.py from distutils.core

    import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so
15. None
16. None

17. Ship it

18. Ever had trouble installing packages?

19. None

20. (at Sqreen, we are dog friendly) Come get your dog

    sticker!

21. Python packaging
 history sdist (source distribution) eggs wheels —> manylinux

    wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤

22. manylinux wheels Python standard: PEP513 / PEP571 Compatible on most

    (real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on CentOS 5

23. CentOS 5?? PEP571 → bumped requirement to CentOS 6!

24. Wheels vs compilation iso builds (crash can be reproduced) many

    packages need maintenance one build per user only one package harder to install…

25. Best of 2 worlds We can have: ctypes: use a

    Python independant binary wheels: ship compiled binaries By combining these… ctypes + wheels = ship only one binary

26. Thanks! Questions? We’re
 hiring! https://sqreen.io/jobs