RustとPython

PyCon JP 2018 9/17 talk session
Python Extension Module written in Rust.

Transcript

1. Rust Rust と と Python Python PyCon PyCon J JP P 2018 2018 2018.09.17 Hideo Hattori 2018.09.17 Hideo Hattori 1
2. whoami whoami Software Engineer @ Tech Bureau, Corp. using Python from version 1.5 ❤ @hhatto 2
3. My Python Project My Python Project etc... genzshcomp pgmagick autopep8 aiogrn pyramid_flamegraph 3
4. AGENDA AGENDA What is Rust? Why use Rust with Python? Example Pros & Cons 4
5. What is Rust? What is Rust? 5
7. Hello World Hello World fn main() { let name = "Rust"; println!("Hello {}!", name); } 7
8. version 1.29.0（2018.09.11） Memory Safety Blazingly Fast Rust Rust 8
9. Installation Installation $ curl https://sh.rustup.rs -sSf | sh 9
10. rustup rustup Installer like venv , pyenv , ... stable and nightly rustup install nightly 10
11. cargo cargo & & package manager & package repository like setup.py , pip & PyPI.org crates.io crates.io 11
12. cargo cargo - Start Project - Start Project $ cargo new myproject $ tree myproject myproject ├── Cargo.toml └── src └── main.rs 12
13. cargo cargo - Build - Build $ cargo build $ cargo run Hello Rust! 13
14. Memory Safety Memory Safety Ownership Borrowing 14
15. Ownership and Borrowing Ownership and Borrowing fn main() { let a = vec![1, 2, 3]; let b = a; println!("a={:?}", a); println!("b={:?}", b); } 15
16. $ rustc ownership.rs error[E0382]: use of moved value: `a` --> ownership.rs:5:24 | 3 | let b = a; | - value moved here 4 | 5 | println!("a={:?}", a); | ^ value used here after move | = 16
17. fn main() { let a = vec![1, 2, 3]; let b = &a; println!("a={:?}", a); println!("b={:?}", b); } 17
18. Concurrency with Concurrency with std::thread std::thread use std::thread; fn main() { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(|| { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 18
19. error[E0373]: closure may outlive the current function, but it borrows `i`, which is owned --> concurrency.rs:7:36 | 7 | threads.push(thread::spawn(|| { | ^^ may outlive borrowed value `i` 8 | println!("this is thread number {}", i); | - `i` is borrowed here help: to force the closure to take ownership of `i` (and any other referenced variables), u | 7 | threads.push(thread::spawn(move || { | ^^^^^^^ error: aborting due to previous error For more information about this error, try `rustc --explain E0373`. 19
20. use std::thread; fn main() { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(move || { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 20
21. Blazingly Fast Blazingly Fast minimal runtime zero-cost abstractions std::thread (rayon, crossbeam, ...) The computer language Benchmarks Game - Rust vs C++ g++ 21
22. Other feature Other feature Traits Pattern Match Macros ... 22
23. Uses Uses Command-Line Tools Web Application Extension Modules ... 23
24. Command-Line Tools Command-Line Tools - like grep tool - fuzzy finder in Rust - Replacement for ls written in Rust - blazingly fast CLOC program ripgrep skim exa tokei 24
25. Web Application Web Application https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=plaintext 25
26. Extension Modules Extension Modules Redis, Nginx, Varnish, etc... Programming Language. Python, Ruby, etc... 26
27. Python Python 27
28. Why use Rust with Why use Rust with Python? Python? 28
29. Motivation Motivation more fast ... C Extension more safety write Extension modules use Rust crates in Python world 29
30. How to implement How to implement ctypes + shared library rust-cpython PyO3 30
31. PyO3 PyO3 （ぱいおーすりー） 31
32. Requrements Requrements Rust（Nightly） setuptools-rust 32
33. Start Python Extension Project Start Python Extension Project $ rustup default nightly $ cargo new --lib example $ cd example 33
34. $ tree example example ├── Cargo.toml ├── setup.py └── src └── main.rs 34
35. Cargo.toml Cargo.toml [package] name = "example" version = "0.1.0" authors = ["Hideo Hattori"] [lib] name = "example" crate-type = ["cdylib"] [dependencies] [dependencies.pyo3] version = "0.4" features = ["extension-module"] 35
36. setup.py setup.py from setuptools import setup from setuptools_rust import Binding, RustExtension setup(name='example', version='0.1', rust_extensions=[ RustExtension('example', 'Cargo.toml', binding=Binding.PyO3)], zip_safe=False) 36
37. src/lib.rs src/lib.rs #![feature(specialization)] extern crate pyo3; use pyo3::prelude::*; use pyo3::py::modinit as pymodinit; #[pymodinit(example)] fn init_example(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "hello")] fn hello(name: &str) -> PyResult<()> { println!("Hello {}", name); Ok(()) } Ok(()) } 37
38. Build Build $ pip install . 38
39. Let s use!! Let s use!! $ python >>> import example >>> example.hello() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Required argument ('name') (pos 1) not found >>> example.hello('Rust') Hello Rust 39
40. Packaging Packaging use pyo3-pack 40
41. Example of real world Example of real world 41
42. fast-woothee-python fast-woothee-python Multi-Language User-Agent Strings Parser （Python) （Rust） （Rust+Python) Project Woothee woothee-python woothee-rust fast-woothee-python 42
43. Benchmark Benchmark ## Ranking real uap 0.0813 (100.0) ******************** fast-woothee 1.0989 ( 7.4) * woothee 3.2219 ( 2.5) * uap(non-cache) 64.3114 ( 0.1) ## Matrix real [01] [02] [03] [04] [01] uap 0.0813 100.0 1352.1 3964.4 79132.1 [02] fast-woothee 1.0989 7.4 100.0 293.2 5852.5 [03] woothee 3.2219 2.5 34.1 100.0 1996.1 [04] uap(non-cache) 64.3114 0.1 1.7 5.0 100.0 https://gist.github.com/hhatto/c951a981e8a3ee4d1bbcf96cb93d5f5e 43
44. wrapper of Python s Class written in Rust fcsv.Reader fcsv.Writer fcsv fcsv Rust s csv crate 44
45. Implement Python Class Implement Python Class use pyo3::prelude::*; #[pyclass] struct Writer { _wtr: csv::Writer<BufWriter<fs::File>>, token: PyToken, } #[pymodinit(_fcsv)] fn init_mod(_py: Python, m: &PyModule) -> PyResult<()> { m.add_class::<Writer>()?; Ok(()) } 45
46. Implement Python Class Implement Python Class #[pymethods] impl Writer { #[new] fn __new__(obj: &PyRawObject, path: String, kwargs: Option<&PyDict>) -> PyResult<()> { : obj.init(|t| Writer { _wtr: wtr, token: t, }) } fn writerow(&mut self, py: Python, arg: PyObject) -> PyResult<()> { : Ok(()) } } 46
47. Implement Python Iterator Implement Python Iterator #[pyproto] impl PyIterProtocol for Reader { fn __iter__(&mut self) -> PyResult<pyobject> { Ok(self.into()) } fn __next__(&mut self) -> PyResult<option<pyobject>> { let mut record = csv::StringRecord::new(); match self._rdr.read_record(&mut record) { Ok(true) => { let py = self.py(); Ok(Some(record.iter().collect::<vec<&str>>().to_object(py))) } _ => Err(exc::StopIteration::new("stop")), } } } 47
48. use this use this # example.py import fcsv for row in fcsv.reader('hello.csv'): print(row) 48
49. Benchmark of Benchmark of Writer Writer ## Ranking real fcsv.writerows 0.0519 (100.0) ******************** fcsv.writerow 0.0677 ( 76.7) *************** unicodecsv.writerows 1.3566 ( 3.8) * std.writerows 1.4500 ( 3.6) * ## Matrix real [01] [02] [03] [04] [01] fcsv.writerows 0.0519 100.0 130.4 2614.4 2794.3 [02] fcsv.writerow 0.0677 76.7 100.0 2005.3 2143.4 [03] unicodecsv.writerows 1.3566 3.8 5.0 100.0 106.9 [04] std.writerows 1.4500 3.6 4.7 93.6 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 49
50. Benchmark of Benchmark of Reader Reader ## Ranking real fcsv.reader 0.5444 (100.0) ******************** unicodecsv.reader 0.6474 ( 84.1) ***************** std.reader 0.7892 ( 69.0) ************** ## Matrix real [01] [02] [03] [01] fcsv.reader 0.5444 100.0 118.9 145.0 [02] unicodecsv.reader 0.6474 84.1 100.0 121.9 [03] std.reader 0.7892 69.0 82.0 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 50
51. Python s os.path module re-written in Rust already rewrite in Rust about 50% fpath fpath 51
52. memchr, memrchr memchr, memrchr use fast character search memchr crate 52
53. fn _basename(path_str: &str, is_bytes: bool) -> PyObject { let i = match memchr::memrchr(MAIN_SEPARATOR as u8, path_str.as_bytes()) { Some(v) => v + 1, None => 0, }; : } 53
54. Benchmark Benchmark methodname % real[p,r] user[p,r] sys[p,r] n abspath 45.53% 10.15s, 5.53s 6.86s, 2.81s 3.25s, 2.69s 100000 basename 53.52% 0.71s, 0.33s 0.70s, 0.33s 0.00s, 0.00s 100000 dirname 57.43% 1.02s, 0.43s 1.01s, 0.43s 0.00s, 0.00s 100000 isabs 56.55% 0.59s, 0.25s 0.59s, 0.25s 0.00s, 0.00s 100000 islink 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 exists 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 lexists 0.70% 3.77s, 3.74s 0.01s, 0.01s 0.01s, 0.00s 50 54
55. Benchmark (2) Benchmark (2) methodname % real[p,r] user[p,r] sys[p,r] n split 53.85% 1.17s, 0.54s 1.17s, 0.54s 0.00s, 0.00s 100000 splitext 62.02% 1.22s, 0.46s 1.21s, 0.46s 0.00s, 0.00s 100000 relpath 52.97% 0.02s, 0.01s 0.01s, 0.01s 0.01s, 0.00s 50 normpath 57.27% 2.02s, 0.86s 2.01s, 0.86s 0.00s, 0.00s 100000 realpath 1.08% 13.39s, 13.25s 0.05s, 0.02s 0.02s, 0.03s 50 join 23.01% 0.24s, 0.19s 0.24s, 0.18s 0.00s, 0.00s 100000 expanduser 67.45% 1.50s, 0.49s 1.49s, 0.48s 0.00s, 0.00s 100000 expandvars 61.37% 1.21s, 0.47s 1.19s, 0.47s 0.00s, 0.00s 100000 55
56. Pros & Cons Pros & Cons 56
57. Pros Pros Good Performance better than C （in my opinion... ） use Rust packages(creates) 57
58. Cons Cons unstable Rust s Learning curve 58
59. Conclusion Conclusion 59
60. Thank you!! Thank you!! 60