Tweet
Tweet

Slide 1

Slide 1 text

MY PYTHON IS RUSTING Armin @mitsuhiko Ronacher — A PYTHON AND RUST LOVE STORY —

Slide 2

Slide 2 text

)J *N"SNJO BOE*EP0QFO4PVSDF MPUTPG1ZUIPOBOE4BB4 'MBTL 4FOUSZ j

Slide 3

Slide 3 text

… and here is where you can find me twitter.com/@mitsuhiko github.com/mitsuhiko lucumr.pocoo.org/

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

“so I heard you are doing Rust now …”

Slide 7

Slide 7 text

that's true but …

Slide 8

Slide 8 text

we love python

Slide 9

Slide 9 text

Strong Ecosystem

Slide 10

Slide 10 text

Fast Iteration

Slide 11

Slide 11 text

Stable Environment

Slide 12

Slide 12 text

Powerful
 Metaprogramming

Slide 13

Slide 13 text

Fast Interpreter Introspection

Slide 14

Slide 14 text

and rust?

Slide 15

Slide 15 text

Speed = -

Slide 16

Slide 16 text

Functionality

Slide 17

Slide 17 text

Reliability

Slide 18

Slide 18 text

what we use it for

Slide 19

Slide 19 text

Mach-O / Dwarf Parsing

Slide 20

Slide 20 text

Javascript Source Maps

Slide 21

Slide 21 text

Proguard Mappings

Slide 22

Slide 22 text

Command Line Tools

Slide 23

Slide 23 text

one to the other

Slide 24

Slide 24 text

virtualenv & pip & distutils & setuptools rustup & cargo

Slide 25

Slide 25 text

rustup the rust toolchain manager

Slide 26

Slide 26 text

cargo the rust package manager

Slide 27

Slide 27 text

pydoc & sphinx rustdoc

Slide 28

Slide 28 text

rustdoc the rust documentation builder

Slide 29

Slide 29 text

a rust primer

Slide 30

Slide 30 text

code ahead

Slide 31

Slide 31 text

fn main() { println!("Hello World!"); }

Slide 32

Slide 32 text

use std::io::{stdin, BufRead, BufReader}; use std::collections::HashMap; fn main() { let mut counts = HashMap::new(); for line_rv in BufReader::new(stdin()).lines() { let line = line_rv.unwrap(); *counts.entry(line).or_insert(0) += 1; } let mut items: Vec<_> = counts.into_iter().collect(); items.sort_by_key(|&(_, count)| -count); for (item, count) in items.into_iter().take(10) { println!("{}: {}", item, count); } }

Slide 33

Slide 33 text

use std::io::{stdin, BufRead, BufReader}; use std::collections::HashMap; fn main() { let mut counts = HashMap::new(); for line_rv in BufReader::new(stdin()).lines() { let line = line_rv.unwrap(); *counts.entry(line).or_insert(0) += 1; } let mut items: Vec<_> = counts.into_iter().collect(); items.sort_by_key(|&(_, count)| -count); for (item, count) in counts { println!("{}: {}", item, count); } }

Slide 34

Slide 34 text

error[E0382]: use of moved value: `counts` --> test.rs:16:26 | 13 | let mut items: Vec<_> = counts.into_iter().collect(); | ------ value moved here ... 16 | for (item, count) in counts { | ^^^^^^ value used here after move | = note: move occurs because `counts` has type `std::collections::HashMap`, which does not implement the `Copy` trait

Slide 35

Slide 35 text

use std::fmt; struct User { id: i64, name: String, } impl fmt::Display for User { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "", self.id, self.name) } } fn main() { println!("{}", User { id: 42, name: "Peter".to_string() }); }

Slide 36

Slide 36 text

the zen of python

Slide 37

Slide 37 text

Beautiful is better than ugly. #[derive(Serialize, Deserialize, Debug)] pub struct Deploy { #[serde(rename="environment")] pub env: String, pub name: Option, pub url: Option, } impl Deploy { pub fn list(&self, api: &Api, id: u64) -> ApiResult> { api.get(&format!("/deploys/{}/", id))?.convert() } }

Slide 38

Slide 38 text

Explicit is better than implicit. fn parse_rev_range(rng: &str) -> (Option<&str>, &str) { if rng == "" { return (None, "HEAD".into()); } let mut iter = rng.rsplitn(2, ".."); let rev = iter.next().unwrap_or("HEAD"); (iter.next(), rev) }

Slide 39

Slide 39 text

Simple is better than complex. use std::{fs, env, io}; let here = env::current_dir()?; for dent_rv in fs::read_dir(here)? { let dent = dent_rv?; let md = dent.metadata()?; println!("{: <60}{: <12}{}", dent.path().display(), md.len(), if md.is_file() { "file" } else { "dir" }); }

Slide 40

Slide 40 text

Complex is better than complicated. use redis::{Client, PipelineCommands, pipe}; let client = Client::open("redis://127.0.0.1/")?; let con = client.get_connection()?; let (k1, k2) : (i32, i32) = pipe() .atomic() .set("key_1", 42).ignore() .set("key_2", 43).ignore() .get("key_1") .get("key_2").query(&con)?;

Slide 41

Slide 41 text

Errors should never pass silently. use std::fs; fn main() { fs::File::open("/tmp/test.txt"); } $ rustc test.rs warning: unused result which must be used, #[warn(unused_must_use)] on by default --> test.rs:4:5 | 4 | fs::File::open("/tmp/test.txt"); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Slide 42

Slide 42 text

side by side

Slide 43

Slide 43 text

protocols vs traits __del__ __add__ __str__ __repr__ __getitem__ Drop::drop Add::add Display::fmt Debug::fmt Index::index

Slide 44

Slide 44 text

error causing throw … panic!(…) return Err(…);

Slide 45

Slide 45 text

error conversion try: x = foo() except SomeError as e: raise NewError(e) let x = foo()?

Slide 46

Slide 46 text

so you want to try it

Slide 47

Slide 47 text

[ Intermission ]

Slide 48

Slide 48 text

monolithic SOA /

Slide 49

Slide 49 text

this is absurd

Slide 50

Slide 50 text

modular code same process +

Slide 51

Slide 51 text

marriage

Slide 52

Slide 52 text

cffi you Python and Rust you declared your compatibility and now I

Slide 53

Slide 53 text

Rust Library > Rust CABI + C header > CFFI > Python cargo | cffi | wheel | setuptools

Slide 54

Slide 54 text

GO AWAY LIBPYTHON repeat after me:

Slide 55

Slide 55 text

do start threads and thread pools

Slide 56

Slide 56 text

don't pass complex
 data structures around

Slide 57

Slide 57 text

do wrap some rust objects in python

Slide 58

Slide 58 text

don't move complex logic
 from Python to Rust

Slide 59

Slide 59 text

do use docker for builds

Slide 60

Slide 60 text

setuptools

Slide 61

Slide 61 text

please halp

Slide 62

Slide 62 text

building

Slide 63

Slide 63 text

Pillow-4.0.0-cp36-cp36m-manylinux1_x86_64.whl Python 2 builds: Python 3 builds: Versions: 2.7 ABI: cpm + cpmu Platforms: OS X + 2 Linux Total: 1 ×2 × 3 = 6 Versions: 3.3 + 3.4 + 3.5 + 3.6 + 3.7 ABI: cpm Platforms: OS X + 2 Linux Total: 5 ×1 × 3 = 15

Slide 64

Slide 64 text

21 BUILDS!!!

Slide 65

Slide 65 text

path to success: • do not link to libpython • use cffi • 2.x/3.x compatible sources • fuck around with setuptools

Slide 66

Slide 66 text

symsynd-1.3.0-py2.py3-none-manylinux1_x86_64.whl Package Name Version Python Tag ABI Tag Platform Tag

Slide 67

Slide 67 text

3 builds!

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

useful images quay.io/pypa/manylinux1_i686 quay.io/pypa/manylinux1_x86_64

Slide 70

Slide 70 text

★ It's an ancient CentOS (for instance
 it has no SNI Support) ★ 32bit builds on on 64bit Docker
 typically. Use the linux32 command ★ Dockerfile allows you to "cache" steps

Slide 71

Slide 71 text

the bridge

Slide 72

Slide 72 text

use std::mem; use std::panic; fn silent_panic_handler(_pi: &panic::PanicInfo) { /* don't do anything here */ } #[no_mangle] pub unsafe extern "C" fn mylib_init() { panic::set_hook(Box::new(silent_panic_handler)); }

Slide 73

Slide 73 text

unsafe fn set_err(err: Error, err_out: *mut CError) { if err_out.is_null() { return; } let s = format!("{}\x00", err); (*err_out).message = Box::into_raw(s.into_boxed_str()) as *mut u8; (*err_out).code = err.get_error_code(); (*err_out).failed = 1; }

Slide 74

Slide 74 text

unsafe fn landingpad Result + panic::UnwindSafe, T>( f: F, err_out: *mut CError) -> T { if let Ok(rv) = panic::catch_unwind(f) { rv.map_err(|err| set_err(err, err_out)).unwrap_or(mem::zeroed()) } else { set_err(ErrorKind::InternalError.into(), err_out); mem::zeroed() } }

Slide 75

Slide 75 text

macro_rules! export ( ($n:ident($($an:ident: $aty:ty),*) -> Result<$rv:ty> $body:block) => ( #[no_mangle] pub unsafe extern "C" fn $n($($an: $aty,)* err: *mut CError) -> $rv { landingpad(|| $body, err) } ); );

Slide 76

Slide 76 text

export!(lsm_view_dump_memdb( view: *mut View, len_out: *mut c_uint, with_source_contents: c_int, with_names: c_int) -> Result<*mut u8> { let memdb = (*view).dump_memdb(DumpOptions { with_source_contents: with_source_contents != 0, with_names: with_names != 0, })?; *len_out = memdb.len() as c_uint; Ok(Box::into_raw(memdb.into_boxed_slice()) as *mut u8) });

Slide 77

Slide 77 text

typedef void lsm_view_t; typedef struct lsm_error_s { char *message; int failed; int code; } lsm_error_t; char *lsm_view_dump_memdb(const lsm_view_t *view, unsigned int *len_out, int with_source_contents, int with_names, lsm_error_t *err);

Slide 78

Slide 78 text

def rustcall(func, *args): err = _ffi.new('lsm_error_t *') rv = func(*(args + (err,))) if not err[0].failed: return rv try: cls = special_errors.get(err[0].code, SourceMapError) exc = cls(_ffi.string(err[0].message).decode('utf-8', 'replace')) finally: _lib.lsm_buffer_free(err[0].message) raise exc

Slide 79

Slide 79 text

what is missing

Slide 80

Slide 80 text

better bdist_wheel

Slide 81

Slide 81 text

Reusable Docker Setups

Slide 82

Slide 82 text

RustFFI + Shims

Slide 83

Slide 83 text

results

Slide 84

Slide 84 text

(python vs rust source map handling)

Slide 85

Slide 85 text

and when shit goes wrong

Slide 86

Slide 86 text

(bug in the binding caused memory leak)

Slide 87

Slide 87 text

it's great, but we need better tooling

Slide 88

Slide 88 text

QA &