Tweet
Tweet

Slide 1

Slide 1 text

Writing a C Python extension in 2018 Paris.py 2018

Slide 2

Slide 2 text

Jean-Baptiste Aviat CTO & Co-founder of sqreen.io Former Hacker at Apple (Red Team) PyMiniRacer author jb@sqreen.io @jbaviat Who Am I?

Slide 3

Slide 3 text

Someday… we needed to use V8 from Python. What we ship: • is public • is widely used • need to have frictionless install $ pip install sqreen —> it just works™!

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

PyMiniRacer: cool JS binding github.com/sqreen/PyMiniRacer

Slide 6

Slide 6 text

The problem • V8 is C++ • How do you run C++ in Python? …We need some kind of binding between these 2 worlds.

Slide 7

Slide 7 text

Many popular packages rely on binary cryptography numpy pymongo psycopg simplejson lxml sqlalchemy…

Slide 8

Slide 8 text

People do it!
 Let’s do it too.

Slide 9

Slide 9 text

What are our goals? • minimize maintenance • make setup easy • make testing easy • have great performance • have low memory fingerprint • dev time is a constraint We want to: And (obviously)…

Slide 10

Slide 10 text

built-in pythonic Python version independant open to other languages high throughput capable CPython ✔ ✔ ✔ ctypes ✔ ✔ ✔ cffi ✔ ✔ ✔ ✔ Cython ✔ ✔ SWIG ✔ ✔

Slide 11

Slide 11 text

ctypes Built into Python Binary is Python independant: • can be used on any Python version • can be used in other languages! No tight integration to Python • not high throughput capable • less Pythonic Complex syntax (C types wrapped in Python…) Not for C++

Slide 12

Slide 12 text

$ python >>> path = "./hello.so" >>> import ctypes >>> lib = ctypes.cdll.LoadLibrary(path) >>> lib.hello_world() Hello world! C file Python interface binary
 object

Slide 13

Slide 13 text

Overview V8 (C++ interface) C interface to V8 Python interface 3rd party binaries import ctypes class PyMiniRacer(object): … #include int miniracer_init(); … V8 library (libv8.a) V8 headers (v8.h) linking ctypes C/C++ code Python library

Slide 14

Slide 14 text

How to put this together? $ cat setup.py from distutils.core import setup, Extension extension = Extension('hello', ['hello.c']) setup(name=‘hello', version='1.0', ext_modules=[extension]) $ python setup.py build running build running build_ext building 'hello' extension clang […] -c hello.c -o hello.o creating build/lib.macosx-10.6-intel-2.7 clang -bundle […] hello.o -o hello.so

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Ship it

Slide 18

Slide 18 text

Ever had trouble installing packages?

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

(at Sqreen, we are dog friendly) Come get your dog sticker!

Slide 21

Slide 21 text

Python packaging
 history sdist (source distribution) eggs wheels —> manylinux wheels (built distribution) 2004 2012 2016 Python 2.4 Python 3.3 Python 3.6 ❤

Slide 22

Slide 22 text

manylinux wheels Python standard: PEP513 / PEP571 Compatible on most (real world) Linux Only in pip >= 8.1 Need build for all Python versions Binaries need to be built on CentOS 5

Slide 23

Slide 23 text

CentOS 5?? PEP571 → bumped requirement to CentOS 6!

Slide 24

Slide 24 text

Wheels vs compilation iso builds (crash can be reproduced) many packages need maintenance one build per user only one package harder to install…

Slide 25

Slide 25 text

Best of 2 worlds We can have: ctypes: use a Python independant binary wheels: ship compiled binaries By combining these… ctypes + wheels = ship only one binary

Slide 26

Slide 26 text

Thanks! Questions? We’re
 hiring! https://sqreen.io/jobs