RustとPython - Speaker Deck

Slide 1

Slide 1 text

Rust Rust とと Python Python PyCon PyCon J JP P 2018 2018 2018.09.17 Hideo Hattori 2018.09.17 Hideo Hattori 1

Slide 2

Slide 2 text

whoami whoami Software Engineer @ Tech Bureau, Corp. using Python from version 1.5 ❤ @hhatto 2

Slide 3

Slide 3 text

My Python Project My Python Project etc... genzshcomp pgmagick autopep8 aiogrn pyramid_flamegraph 3

Slide 4

Slide 4 text

AGENDA AGENDA What is Rust? Why use Rust with Python? Example Pros & Cons 4

Slide 5

Slide 5 text

What is Rust? What is Rust? 5

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Hello World Hello World fn main() { let name = "Rust"; println!("Hello {}!", name); } 7

Slide 8

Slide 8 text

version 1.29.0（2018.09.11） Memory Safety Blazingly Fast Rust Rust 8

Slide 9

Slide 9 text

Installation Installation $ curl https://sh.rustup.rs -sSf | sh 9

Slide 10

Slide 10 text

rustup rustup Installer like venv , pyenv , ... stable and nightly rustup install nightly 10

Slide 11

Slide 11 text

cargo cargo & & package manager & package repository like setup.py , pip & PyPI.org crates.io crates.io 11

Slide 12

Slide 12 text

cargo cargo - Start Project - Start Project $ cargo new myproject $ tree myproject myproject ├── Cargo.toml └── src └── main.rs 12

Slide 13

Slide 13 text

cargo cargo - Build - Build $ cargo build $ cargo run Hello Rust! 13

Slide 14

Slide 14 text

Memory Safety Memory Safety Ownership Borrowing 14

Slide 15

Slide 15 text

Ownership and Borrowing Ownership and Borrowing fn main() { let a = vec![1, 2, 3]; let b = a; println!("a={:?}", a); println!("b={:?}", b); } 15

Slide 16

Slide 16 text

$ rustc ownership.rs error[E0382]: use of moved value: `a` --> ownership.rs:5:24 | 3 | let b = a; | - value moved here 4 | 5 | println!("a={:?}", a); | ^ value used here after move | = 16

Slide 17

Slide 17 text

fn main() { let a = vec![1, 2, 3]; let b = &a; println!("a={:?}", a); println!("b={:?}", b); } 17

Slide 18

Slide 18 text

Concurrency with Concurrency with std::thread std::thread use std::thread; fn main() { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(|| { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 18

Slide 19

Slide 19 text

error[E0373]: closure may outlive the current function, but it borrows ì`, which is owned --> concurrency.rs:7:36 | 7 | threads.push(thread::spawn(|| { | ^^ may outlive borrowed value ì` 8 | println!("this is thread number {}", i); | - ì` is borrowed here help: to force the closure to take ownership of ì` (and any other referenced variables), u | 7 | threads.push(thread::spawn(move || { | ^^^^^^^ error: aborting due to previous error For more information about this error, try `rustc --explain E0373`. 19

Slide 20

Slide 20 text

use std::thread; fn main() { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(move || { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 20

Slide 21

Slide 21 text

Blazingly Fast Blazingly Fast minimal runtime zero-cost abstractions std::thread (rayon, crossbeam, ...) The computer language Benchmarks Game - Rust vs C++ g++ 21

Slide 22

Slide 22 text

Other feature Other feature Traits Pattern Match Macros ... 22

Slide 23

Slide 23 text

Uses Uses Command-Line Tools Web Application Extension Modules ... 23

Slide 24

Slide 24 text

Command-Line Tools Command-Line Tools - like grep tool - fuzzy finder in Rust - Replacement for ls written in Rust - blazingly fast CLOC program ripgrep skim exa tokei 24

Slide 25

Slide 25 text

Web Application Web Application https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=plaintext 25

Slide 26

Slide 26 text

Extension Modules Extension Modules Redis, Nginx, Varnish, etc... Programming Language. Python, Ruby, etc... 26

Slide 27

Slide 27 text

Python Python 27

Slide 28

Slide 28 text

Why use Rust with Why use Rust with Python? Python? 28

Slide 29

Slide 29 text

Motivation Motivation more fast ... C Extension more safety write Extension modules use Rust crates in Python world 29

Slide 30

Slide 30 text

How to implement How to implement ctypes + shared library rust-cpython PyO3 30

Slide 31

Slide 31 text

PyO3 PyO3 （ぱいおーすりー） 31

Slide 32

Slide 32 text

Requrements Requrements Rust（Nightly） setuptools-rust 32

Slide 33

Slide 33 text

Start Python Extension Project Start Python Extension Project $ rustup default nightly $ cargo new --lib example $ cd example 33

Slide 34

Slide 34 text

$ tree example example ├── Cargo.toml ├── setup.py └── src └── main.rs 34

Slide 35

Slide 35 text

Cargo.toml Cargo.toml [package] name = "example" version = "0.1.0" authors = ["Hideo Hattori"] [lib] name = "example" crate-type = ["cdylib"] [dependencies] [dependencies.pyo3] version = "0.4" features = ["extension-module"] 35

Slide 36

Slide 36 text

setup.py setup.py from setuptools import setup from setuptools_rust import Binding, RustExtension setup(name='example', version='0.1', rust_extensions=[ RustExtension('example', 'Cargo.toml', binding=Binding.PyO3)], zip_safe=False) 36

Slide 37

Slide 37 text

src/lib.rs src/lib.rs #![feature(specialization)] extern crate pyo3; use pyo3::prelude::*; use pyo3::py::modinit as pymodinit; #[pymodinit(example)] fn init_example(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "hello")] fn hello(name: &str) -> PyResult<()> { println!("Hello {}", name); Ok(()) } Ok(()) } 37

Slide 38

Slide 38 text

Build Build $ pip install . 38

Slide 39

Slide 39 text

Let s use!! Let s use!! $ python >>> import example >>> example.hello() Traceback (most recent call last): File "", line 1, in TypeError: Required argument ('name') (pos 1) not found >>> example.hello('Rust') Hello Rust 39

Slide 40

Slide 40 text

Packaging Packaging use pyo3-pack 40

Slide 41

Slide 41 text

Example of real world Example of real world 41

Slide 42

Slide 42 text

fast-woothee-python fast-woothee-python Multi-Language User-Agent Strings Parser （Python) （Rust）（Rust+Python) Project Woothee woothee-python woothee-rust fast-woothee-python 42

Slide 43

Slide 43 text

Benchmark Benchmark ## Ranking real uap 0.0813 (100.0) ******************** fast-woothee 1.0989 ( 7.4) * woothee 3.2219 ( 2.5) * uap(non-cache) 64.3114 ( 0.1) ## Matrix real [01] [02] [03] [04] [01] uap 0.0813 100.0 1352.1 3964.4 79132.1 [02] fast-woothee 1.0989 7.4 100.0 293.2 5852.5 [03] woothee 3.2219 2.5 34.1 100.0 1996.1 [04] uap(non-cache) 64.3114 0.1 1.7 5.0 100.0 https://gist.github.com/hhatto/c951a981e8a3ee4d1bbcf96cb93d5f5e 43

Slide 44

Slide 44 text

wrapper of Python s Class written in Rust fcsv.Reader fcsv.Writer fcsv fcsv Rust s csv crate 44

Slide 45

Slide 45 text

Implement Python Class Implement Python Class use pyo3::prelude::*; #[pyclass] struct Writer { _wtr: csv::Writer>, token: PyToken, } #[pymodinit(_fcsv)] fn init_mod(_py: Python, m: &PyModule) -> PyResult<()> { m.add_class::()?; Ok(()) } 45

Slide 46

Slide 46 text

Implement Python Class Implement Python Class #[pymethods] impl Writer { #[new] fn __new__(obj: &PyRawObject, path: String, kwargs: Option<&PyDict>) -> PyResult<()> { : obj.init(|t| Writer { _wtr: wtr, token: t, }) } fn writerow(&mut self, py: Python, arg: PyObject) -> PyResult<()> { : Ok(()) } } 46

Slide 47

Slide 47 text

Implement Python Iterator Implement Python Iterator #[pyproto] impl PyIterProtocol for Reader { fn __iter__(&mut self) -> PyResult { Ok(self.into()) } fn __next__(&mut self) -> PyResult> { let mut record = csv::StringRecord::new(); match self._rdr.read_record(&mut record) { Ok(true) => { let py = self.py(); Ok(Some(record.iter().collect::>().to_object(py))) } _ => Err(exc::StopIteration::new("stop")), } } } 47

Slide 48

Slide 48 text

use this use this # example.py import fcsv for row in fcsv.reader('hello.csv'): print(row) 48

Slide 49

Slide 49 text

Benchmark of Benchmark of Writer Writer ## Ranking real fcsv.writerows 0.0519 (100.0) ******************** fcsv.writerow 0.0677 ( 76.7) *************** unicodecsv.writerows 1.3566 ( 3.8) * std.writerows 1.4500 ( 3.6) * ## Matrix real [01] [02] [03] [04] [01] fcsv.writerows 0.0519 100.0 130.4 2614.4 2794.3 [02] fcsv.writerow 0.0677 76.7 100.0 2005.3 2143.4 [03] unicodecsv.writerows 1.3566 3.8 5.0 100.0 106.9 [04] std.writerows 1.4500 3.6 4.7 93.6 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 49

Slide 50

Slide 50 text

Benchmark of Benchmark of Reader Reader ## Ranking real fcsv.reader 0.5444 (100.0) ******************** unicodecsv.reader 0.6474 ( 84.1) ***************** std.reader 0.7892 ( 69.0) ************** ## Matrix real [01] [02] [03] [01] fcsv.reader 0.5444 100.0 118.9 145.0 [02] unicodecsv.reader 0.6474 84.1 100.0 121.9 [03] std.reader 0.7892 69.0 82.0 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 50

Slide 51

Slide 51 text

Python s os.path module re-written in Rust already rewrite in Rust about 50% fpath fpath 51

Slide 52

Slide 52 text

memchr, memrchr memchr, memrchr use fast character search memchr crate 52

Slide 53

Slide 53 text

fn _basename(path_str: &str, is_bytes: bool) -> PyObject { let i = match memchr::memrchr(MAIN_SEPARATOR as u8, path_str.as_bytes()) { Some(v) => v + 1, None => 0, }; : } 53

Slide 54

Slide 54 text

Benchmark Benchmark methodname % real[p,r] user[p,r] sys[p,r] n abspath 45.53% 10.15s, 5.53s 6.86s, 2.81s 3.25s, 2.69s 100000 basename 53.52% 0.71s, 0.33s 0.70s, 0.33s 0.00s, 0.00s 100000 dirname 57.43% 1.02s, 0.43s 1.01s, 0.43s 0.00s, 0.00s 100000 isabs 56.55% 0.59s, 0.25s 0.59s, 0.25s 0.00s, 0.00s 100000 islink 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 exists 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 lexists 0.70% 3.77s, 3.74s 0.01s, 0.01s 0.01s, 0.00s 50 54

Slide 55

Slide 55 text

Benchmark (2) Benchmark (2) methodname % real[p,r] user[p,r] sys[p,r] n split 53.85% 1.17s, 0.54s 1.17s, 0.54s 0.00s, 0.00s 100000 splitext 62.02% 1.22s, 0.46s 1.21s, 0.46s 0.00s, 0.00s 100000 relpath 52.97% 0.02s, 0.01s 0.01s, 0.01s 0.01s, 0.00s 50 normpath 57.27% 2.02s, 0.86s 2.01s, 0.86s 0.00s, 0.00s 100000 realpath 1.08% 13.39s, 13.25s 0.05s, 0.02s 0.02s, 0.03s 50 join 23.01% 0.24s, 0.19s 0.24s, 0.18s 0.00s, 0.00s 100000 expanduser 67.45% 1.50s, 0.49s 1.49s, 0.48s 0.00s, 0.00s 100000 expandvars 61.37% 1.21s, 0.47s 1.19s, 0.47s 0.00s, 0.00s 100000 55

Slide 56

Slide 56 text

Pros & Cons Pros & Cons 56

Slide 57

Slide 57 text

Pros Pros Good Performance better than C （in my opinion... ） use Rust packages(creates) 57