Binary Python - Speaker Deck

Slide 1

Slide 1 text

BINARY PYTHON i n t r o d u c i n g n a t i v e c o d e Armin @mitsuhiko Ronacher

Slide 2

Slide 2 text

Hi I'm Armin Hailing from ice cold Vienna Austria (where stores are closed on sundays)

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Werkzeug, Jinja, Flask, Sentry, ... I <3 and build Open Source

Slide 7

Slide 7 text

Python OUR Heart beats for

Slide 8

Slide 8 text

But we also have other things we need to interface with …

Slide 9

Slide 9 text

C / C++ / RUST

Slide 10

Slide 10 text

Why do we have native code?

Slide 11

Slide 11 text

Speed

Slide 12

Slide 12 text

Functionality

Slide 13

Slide 13 text

Necessity

Slide 14

Slide 14 text

Importing Native Modules ❁

Slide 15

Slide 15 text

Import System ★ package/mylib.so ★ package/mylib.pyd ★ package/mylib.dylib

Slide 16

Slide 16 text

Local Development ★ lib/lib.c -> package/_lib.so ★ python setup.py build ★ pip install --editable . -v

Slide 17

Slide 17 text

Build for Distribution ★ lib/lib.c -> build/…/_lib.so ★ python setup.py bdist_wheel

Slide 18

Slide 18 text

WHY HANDROLL ❁

Slide 19

Slide 19 text

Many Systems Many Developers Run “everywhere”

Slide 20

Slide 20 text

that rules out most already existing solutions. SAD

Slide 21

Slide 21 text

Distributing ❁

Slide 22

Slide 22 text

Python Wheels ★ .py files are portable ★ .pyc files are generated on install ★ wheel is largely universal

Slide 23

Slide 23 text

sentry-8.12.0-py27-none-any.whl Flask-0.12-py2.py3-none-any.whl Package Name Version Python Tag ABI Tag Platform Tag

Slide 24

Slide 24 text

Binaries ★ Platform Specific ★ libc specific :( ★ might link against system libraries ★ typically cannot compile on  installation time

Slide 25

Slide 25 text

Binary Wheels ★ “easy” on OS X ★ trivial on Windows ★ limited support on Linux (manylinux1)

Slide 26

Slide 26 text

symsynd-1.3.0-cp27-none-manylinux1_x86_64.whl Package Name Version Python Tag ABI Tag Platform Tag

Slide 27

Slide 27 text

Pillow-4.0.0-cp36-cp36m-manylinux1_x86_64.whl Package Name Version Python Tag ABI Tag Platform Tag

Slide 28

Slide 28 text

All The Tags ❁

Slide 29

Slide 29 text

Python Tag ★ any python version ★ any 2.x / 3.x version ★ a specific version ★ a specific Python implementation  (cpython, pypy, …)

Slide 30

Slide 30 text

ABI Tag ★ the Python Interpreter ABI version  (eg: UC2 vs UC4)

Slide 31

Slide 31 text

Platform Tag ★ identifies the platform ★ eg: 32bit Intel OS X, x86_64 ★ platform can be complex.  (eg: manylinux1_x86_64)

Slide 32

Slide 32 text

WTF is manylinux1? ❁

Slide 33

Slide 33 text

linux binary compatibility is fraking terrible

Slide 34

Slide 34 text

manylinux1 ★ compile on super old CentOS version ★ do not link against fancy libraries ★ only use old C++ compilers if at all ★ static link all the things you can

Slide 35

Slide 35 text

where to get ancient CentOS?

Slide 36

Slide 36 text

Fewer Dimensions ❁

Slide 37

Slide 37 text

Pillow-4.0.0-cp36-cp36m-manylinux1_x86_64.whl Python 2 builds: Python 3 builds: Versions: 2.7 ABI: cpm + cpmu Platforms: OS X + 2 Linux Total: 1 ×2 × 3 = 6 Versions: 3.3 + 3.4 + 3.5 + 3.6 + 3.7 ABI: cpm Platforms: OS X + 2 Linux Total: 5 ×1 × 3 = 15

Slide 38

Slide 38 text

21 BUILDS!!!

Slide 39

Slide 39 text

that's a lot of wheels. SAD

Slide 40

Slide 40 text

Can we kill tags? ★ Python version tag: write Python  2.x and 3.x source compatible code ★ ABI Tag: do not link against libpython ★ Platform Tag: we can't do anything  about this one :(

Slide 41

Slide 41 text

path to success: • do not link to libpython • use cffi • 2.x/3.x compatible sources • fuck around with setuptools

Slide 42

Slide 42 text

SETUPTOOLS ❁

Slide 43

Slide 43 text

import os from distutils.command.build_py import build_py from distutils.command.build_ext import build_ext PACKAGE = 'mypackage' class CustomBuildPy(build_py): def run(self): build_py.run(self) build_mylib(os.path.join(self.build_lib, *PACKAGE.split('.'))) class CustomBuildExt(build_ext): def run(self): build_ext.run(self) if self.inplace: build_py = self.get_finalized_command('build_py') build_mylib(build_py.get_package_dir(PACKAGE))

Slide 44

Slide 44 text

from wheel.bdist_wheel import bdist_wheel class CustomBdistWheel(bdist_wheel): def get_tag(self): rv = bdist_wheel.get_tag(self) return ('py2.py3', 'none') + rv[2:]

Slide 45

Slide 45 text

from setuptools import setup setup( ... cffi_modules=['build.py:my_ffi'], install_requires=['cffi>=1.0.0'], setup_requires=['cffi>=1.0.0'], cmdclass={ 'build_ext': CustomBuildExt, 'build_py': CustomBuildPy, 'bdist_wheel': CustomBdistWheel, } )

Slide 46

Slide 46 text

Build My Lib ❁

Slide 47

Slide 47 text

import os import sys import shutil import subprocess EXT = sys.platform == 'darwin' and '.dylib' or '.so' def build_mylib(base_path): lib_path = os.path.join(base_path, '_nativelib.so') here = os.path.abspath(os.path.dirname(__file__)) cmdline = ['make', 'build-ext'] rv = subprocess.Popen(cmdline, cwd=here).wait() if rv != 0: sys.exit(rv) src_path = os.path.join(here, 'target', 'release', 'libnativelib' + EXT) if os.path.isfile(src_path): shutil.copy2(src_path, lib_path) build output path build command

Slide 48

Slide 48 text

develop: pip install --editable . -v build-ext: cargo build --release

Slide 49

Slide 49 text

CFFI (build.py) ❁

Slide 50

Slide 50 text

import sys import subprocess from cffi import FFI def _to_source(x): if sys.version_info >= (3, 0) and isinstance(x, bytes): x = x.decode('utf-8') return x my_ffi = FFI() my_ffi.cdef(_to_source(subprocess.Popen([ 'cc', '-E', '-DPYTHON_HEADER', 'mynativelib/mynativelib.h'], stdout=subprocess.PIPE).communicate()[0])) my_ffi.set_source('mypackage._nativelib', None) header only good for typedefs

Slide 51

Slide 51 text

my_ffi = FFI() my_ffi.cdef(_to_source(subprocess.Popen([ 'cc', '-E', '-DPYTHON_HEADER', 'mynativelib/mynativelib.h'], stdout=subprocess.PIPE).communicate()[0])) with open('mynativelib/mynativelib.cpp', 'rb') as source: my_ffi.set_source( 'mypackage/_nativelib', _to_source(source.read()), include_dirs=['mynativelib'], extra_compile_args=['-std=c++11'], source_extension='.cpp' ) with source compilation

Slide 52

Slide 52 text

GITIGNORE ❁

Slide 53

Slide 53 text

mypackage/_nativelib.py mypackage/*.so mypackage/*.dylib build dist *.pyc *.egg-info

Slide 54

Slide 54 text

Wrapping with CFFI ❁

Slide 55

Slide 55 text

from ._nativelib import ffi as _ffi _lib = _ffi.dlopen(os.path.join( os.path.dirname(__file__), '_nativelib.so')) _lib.mylib_global_init_if_needed() class MyObject(object): def __init__(self): self._ptr = _lib.my_object_new() def __del__(self): if self._ptr: _lib.my_object_free(self._ptr) self._ptr = None

Slide 56

Slide 56 text

from ._nativelib import ffi as _ffi, lib as _lib _lib.mylib_global_init_if_needed() class MyObject(object): def __init__(self): self._ptr = _lib.my_object_new() def __del__(self): if self._ptr: _lib.my_object_free(self._ptr) self._ptr = None

Slide 57

Slide 57 text

now for building. SO SAD

Slide 58

Slide 58 text

BASICS ❁

Slide 59

Slide 59 text

$ pip install wheel $ python setup.py bdist_wheel

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Useful Images ❁

Slide 62

Slide 62 text

For Python in General ★ quay.io/pypa/manylinux1_i686 ★ quay.io/pypa/manylinux1_x86_64

Slide 63

Slide 63 text

Things of note ★ It's an ancient CentOS (for instance  it has no SNI Support) ★ 32bit builds on on 64bit Docker  typically. Use the linux32 command ★ Dockerfile allows you to "cache" steps

Slide 64

Slide 64 text

How we do it ★ travis all the things ★ upload artifacts to github releases ★ download from there an upload to  pypi with twine

Slide 65

Slide 65 text

what about macOS?

Slide 66

Slide 66 text

build on travis / locally ★ travis better because you can build on  old macOS for higher portability ★ you can find old SDKs on github! ★ Use MACOS_DEPLOYMENT_TARGET

Slide 67

Slide 67 text

WHEEL_OPTIONS= if [ `uname` == "Darwin" ]; then WHEEL_OPTIONS="--plat-name=macosx-10.10-intel" fi python setup.py bdist_wheel $WHEEL_OPTIONS

Slide 68

Slide 68 text

Patterns ❁

Slide 69

Slide 69 text

Library Design

Slide 70

Slide 70 text

#ifndef MYLIB_H_INCLUDED #define MYLIB_H_INCLUDED #ifdef __cplusplus extern "C" { #endif typedef void mylib_type_t; mylib_type_t *mylib_type_new(void); void mylib_type_free(mylib_type_t *self); #ifdef __cplusplus } #endif #endif

Slide 71

Slide 71 text

#include "mylib.h" class Type { Type(); ~Type(); }; mylib_type_t *mylib_type_new() { Type *rv = new Type(); (mylib_type_t *)rv; } void mylib_type_free(mylib_type_t *self) { if (self) { Type *t = (Type *)self; delete t; } }

Slide 72

Slide 72 text

Error Handling

Slide 73

Slide 73 text

typedef struct mylib_error_t { int code; char *msg; }; void mylib_error_free(mylib_error_t *err) { if (err) { free(err->msg); free(err); } }

Slide 74

Slide 74 text

int mylib_do_stuff(int a, int b, mylib_error_t **err_out) { if (a + b > 255) { mylib_error_t *err = malloc(mylib_error_t); err->msg = strdup("Adding those chars overflows"); err->code = MYLIB_CHAR_OVERFLOW; *err_out = err; return -1; } return a + b; }

Slide 75

Slide 75 text

special_errors = {} def invoke_with_exc(func, *args): err = _ffi.new('mylib_error_t **') try: rv = func(*(args + (err,))) if not err[0]: return rv cls = special_errors.get(err[0].code, RuntimeError) raise cls(_ffi.string(err[0].msg).decode('utf-8', 'replace')) finally: if err[0]: _lib.mylib_error_free(err[0])

Slide 76

Slide 76 text

try: rv = invoke_with_exc(_lib.mylib_do_stuff, arg1, arg2) except DefaultError as e: print 'An error happened: %s' % e else: print 'The result is %r' % rv

Slide 77

Slide 77 text

Conclusions ❁

Slide 78

Slide 78 text

how painful is it?

Slide 79

Slide 79 text

it's pretty bad. SAD

Slide 80

Slide 80 text

but when it works it keeps working. LOVE IT

Slide 81

Slide 81 text

what do we use it for?

Slide 82

Slide 82 text

Native Symbolication C/C++

Slide 83

Slide 83 text

Javascript Source Maps Rust

Slide 84

Slide 84 text

QA &

Slide 85

Slide 85 text

No content

Slide 86

Slide 86 text

def rustcall(func, *args): err = _ffi.new('lsm_error_t *') rv = func(*(args + (err,))) if not err[0].failed: return rv try: cls = special_errors.get(err[0].code, SourceMapError) exc = cls(_ffi.string(err[0].message).decode('utf-8', 'replace')) finally: _lib.lsm_buffer_free(err[0].message) raise exc

Slide 87

Slide 87 text

use std::mem; use std::panic; fn silent_panic_handler(_pi: &panic::PanicInfo) { /* don't do anything here */ } #[no_mangle] pub unsafe extern "C" fn mylib_init() { panic::set_hook(Box::new(silent_panic_handler)); }

Slide 88

Slide 88 text

unsafe fn set_err(err: Error, err_out: *mut CError) { if err_out.is_null() { return; } let s = format!("{}\x00", err); (*err_out).message = Box::into_raw(s.into_boxed_str()) as *mut u8; (*err_out).code = err.get_error_code(); (*err_out).failed = 1; }

Slide 89

Slide 89 text

unsafe fn landingpad Result + panic::UnwindSafe, T>( f: F, err_out: *mut CError) -> T { if let Ok(rv) = panic::catch_unwind(f) { rv.map_err(|err| set_err(err, err_out)).unwrap_or(mem::zeroed()) } else { set_err(ErrorKind::InternalError.into(), err_out); mem::zeroed() } }

Slide 90

Slide 90 text

macro_rules! export ( ($n:ident($($an:ident: $aty:ty),*) -> Result<$rv:ty> $body:block) => ( #[no_mangle] pub unsafe extern "C" fn $n($($an: $aty,)* err: *mut CError) -> $rv { landingpad(|| $body, err) } ); );

Slide 91

Slide 91 text

export!(lsm_view_dump_memdb( view: *mut View, len_out: *mut c_uint, with_source_contents: c_int, with_names: c_int) -> Result<*mut u8> { let memdb = (*view).dump_memdb(DumpOptions { with_source_contents: with_source_contents != 0, with_names: with_names != 0, })?; *len_out = memdb.len() as c_uint; Ok(Box::into_raw(memdb.into_boxed_slice()) as *mut u8) });