Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing a Python C extension in 2018

Writing a Python C extension in 2018


Jean-Baptiste Aviat

May 16, 2018


  1. Writing a C Python extension in 2018 Paris.py 2018

  2. Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at

    Apple (Red Team) PyMiniRacer author jb@sqreen.io @jbaviat Who Am I?
  3. Someday… we needed to use V8 from Python. What we

    ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!
  4. None
  5. PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer

  6. The problem • V8 is C++ • How do you

    run C++ in Python? …We need some kind of binding between these 2 worlds.
  7. Many popular packages rely on binary cryptography numpy pymongo psycopg

    simplejson lxml sqlalchemy…
  8. People do it!
 Let’s do it too.

  9. What are our goals? • minimize maintenance • make setup

    easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…
  10. built-in pythonic Python version independant open to other languages high

    throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔
  11. ctypes Built into Python Binary is Python independant: • can

    be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++
  12. $ python >>> path = "./hello.so" >>> import ctypes >>>

    lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary
  13. Overview V8 (C++ interface) C interface to V8 Python interface

    3rd party binaries import ctypes class PyMiniRacer(object): … #include <v8.h> int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linking ctypes C/C++ code Python library
  14. How to put this together? $ cat setup.py from distutils.core

    import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so
  15. None
  16. None
  17. Ship it

  18. Ever had trouble installing packages?

  19. None
  20. (at Sqreen, we are dog friendly) Come get your dog

  21. Python packaging
 history sdist (source distribution) eggs wheels —> manylinux

    wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤
  22. manylinux wheels Python standard: PEP513 / PEP571 Compatible on most

    (real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on CentOS 5
  23. CentOS 5?? PEP571 → bumped requirement to CentOS 6!

  24. Wheels vs compilation iso builds (crash can be reproduced) many

    packages need maintenance one build per user only one package harder to install…
  25. Best of 2 worlds We can have: ctypes: use a

    Python independant binary wheels: ship compiled binaries By combining these… ctypes + wheels = ship only one binary
  26. Thanks! Questions? We’re
 hiring! https://sqreen.io/jobs